By Kerry Bodine
Forrester Research
This article originally published in Self-Service World magazine, Jul/Aug 2006.
The success or failure of a self-service device depends on whether it enables customers to accomplish their goals easily. Evaluating the effectiveness of kiosks requires a portfolio of tools, each providing unique insights and requiring different resources and skills.Focus groups validate concepts, not user experience. A focus group can uncover reactions to new brand positioning or a potential new service, so the best time to conduct these sessions is after a company has clearly defined its idea and before it has gone into production. But beware: Focus groups don't test usability. Because participants don't attempt to complete actual tasks, their focus is on how the kiosk looks, not how it works.
Surveys show which users failed to reach their goals, not why. Surveys are best at gathering valuable quantitative information about who's coming to the kiosk and why, whether they succeed at their goals and their general level of satisfaction. But while surveys can identify that the kiosk has a problem, they don't effectively capture what that problem is. That is because most users can't identify the specific reasons why they were or were not able to accomplish their goals.
Analytics reveal behavior, not intent. Software from vendors like Nanonation and Netkey map kiosk traffic, showing major trends like where users drop off or spend time and the individual paths they take through the system. But analytics only show what users did — not why. A customer who views several screens and then leaves may be have been interrupted by screaming children or an impatient spouse. Or she may have left in frustration because she couldn't find the details she needed to make a decision.
Expert reviews find known usability flaws, not every possible problem. Expert reviews — like Forrester's Kiosk Hardware, Software and Environment reviews — are completed by trained professionals who attempt to accomplish specific user goals while checking for compliance with research-based criteria. Expert reviews can quickly identify a large number of usability issues, making them a cost-effective evaluation method. But because the criteria are based on past research, they only find problems that are known to create barriers for users.
Usability tests provide insight about real users, but can be costly. Project teams need usability tests to find problems that are unique to their kiosk application. With this method, testers recruit real users and have each try to accomplish a set of realistic goals, observing and identifying areas where users express confusion or go down the wrong path. But usability tests can be resource-intensive in terms of the time and money required to recruit users, conduct tests and analyze results.
Developing a holistic view of whether customers can accomplish their business-critical goals requires using evaluation tools in conjunction with one another. For example, to best gain insight into what drives kiosk transaction rates, start with a survey to determine a baseline of how many customers want to complete a particular transaction. Then, look to analytics data to understand whether transaction volumes are as expected and if not, where users drop off along the path. Then, use an expert review to examine the scenarios and look for design flaws. Once well understood problems are eliminated, conduct usability tests to reveal any remaining issues that block maximum transaction rates.
Kerry Bodine is a senior analyst on Forrester's customer experience team.
