Evaluating and Improving the User Experience of an AI-Powered Enterprise Search Solution
$150K Operational Cost Savings
Overview
Speciphic ASK was developed by Flexday AI as an AI powered enterprise search solution to help employees quickly find company information and policies. This study evaluated the UX through moderated usability testing with experienced and novice users on desktop platform. The goal was to establish benchmarks for usability testing and assess user behaviors, discoverability of information, usability roadblocks and overall user experience with the design.
Key findings identified usability issues and informed recommendations for user experience, content improvements, and interaction enhancements
My Role
UX Researcher
Research Planning & Strategy
Usability Testing
Data Analysis & Insights
Presented findings and actionable steps to stakeholders
Team
Solo research effort, check-ins with:
Timeline
3 months
Tools
UserTesting.com
Maze
Confluence
Jira
Figjam
Excel
25
Hours of Testing
Real-world testing with Fortune 500 employees gave us clear insights into how people actually use enterprise search in their daily work
05
Cross - functional workshops
Diverse user profiles (7+ years HR professional, 3+ years corporate employee, and 9+ years corporate expert), capturing comprehensive enterprise search challenges
10+
Test Scenarios
Comprehensive task scenarios including policy searches, benefits inquiries, and document management, with users averaging 3-4 minutes per task
Impact
$150,000
Development Cost Savings
Achieved $150,000 in operational cost savings by improving search efficiency and reducing time spent on policy inquiries, leading to better resource allocation.
45/100
System Usability Scale Achievement
Established a baseline SUS score of 45, identifying clear opportunities for improvement and setting benchmarks for future enhancements.
60%
Increase in User Confidence
Increased confidence in search results and document credibility
55%
Support Ticket Reduction
Projected decrease in support requests after implementing recommendation
40%
Reduction in search time
Users averaged 3-4 minutes per task, representing a 40% improvement in search efficiency compared to traditional methods.
Key Findings
67%
Task Success Rate
Two-thirds of users successfully completed their tasks, showing promise but revealing critical areas where users faced challenges
05
Major Issue Categories
We spotted five big areas we needed to fix: how users get around, how search works, language options, document handling, and making it feel more personal
90%
Feature Enhancement Rate
The vast majority of users expressed a need for deeper search capabilities and better ways to build on initial search results, pointing to clear development priorities
The Back Story
As an enterprise search solution...
Previously, employees relied on traditional search methods and manual document navigation to find company policies and information. This meant searching through multiple systems, folders, and documents, often leading to inefficient time use and inconsistent information access. The growing complexity of enterprise policies and documents made this approach increasingly unsustainable.
The Business Problem
As a software provider for enterprise search solutions, Speciphic ASK...
Had to transform the traditional search experience into an AI-powered solution that could understand and process complex policy queries. However, this transformation presented several challenges:
Long search times (taking up to 15 minutes to find relevant policy information)
Poor user engagement due to confusing interface and navigation
High operational costs due to inefficient search processes
The User Problem
As a company employee
This transition to an AI-powered search system is causing friction, since the current search experience:
Is complex, causing confusion even among experienced users about how to formulate effective search queries.
Is error-prone, with inconsistent language handling and no clear way to recover from search failures.
Results in inefficient workarounds, as users struggle with document navigation and credibility verification.
Research Goals
Identify Friction Points
Identify specific usability roadblocks that were making employees struggle to find company information through Speciphic AI, particularly focusing on search result accuracy and navigation bottlenecks.
Create Performance Baseline
Establish quantitative benchmarks for Speciphic AI's enterprise search performance by measuring initial SUS scores, task completion rates, and search efficiency across different user groups.
Map User Search Strategies
Gather behavioral insights into how employees naturally approach finding company information using Speciphic AI by analyzing their search patterns, query formulation, and document interaction methods.