Qualitative Usability Testing

Qualitative usability testing takes a closer look at user interactions with websites and software applications using skilled behavior observation and probing questions. This is an excellent approach to gather first-hand data from real users.

Overall, qualitative usability testing is best-suited for:

• Detecting obvious, critical issues, without concerns about the magnitude of the problems.
• Doing rapid design iterations to test changes and different concepts.
• Gaining a deeper understanding of user behaviors.

Why You Should Do It

Website and app design involves both technology and art. When it comes to designing them, both clients and designers go for the “gut” feeling and personal likes and dislikes. Unfortunately, often, the ultimate target audience is poorly understood or simply ignored.

Qualitative usability testing allows for a deeper understanding of the gaps between what users do and say, which often is not the same. Using small samples, this type of testing can show us where large usability issues exist and why. With a little bit of investment upfront in these tests, you can save time and money in the long run by avoiding the higher costs of redoing and redeveloping an ineffective website or application.

Back to Top

The Approach

Finding the optimal design is often an iterative process where changes are made as feedback is gathered from the target audience at different stages of a website or app conception.

We conduct research using qualitative methods at any stage of the development, from the idea/wireframe phase, to design prototypes to full developed websites or apps, providing unbiased feedback of what makes them effective in supporting customer acquisition and retention goals.

When Should You Conduct Usability Testing?

• Before the development of design concepts: Usability testing of current website or application in comparison with competitor websites and apps can provide invaluable insights during the requirement gathering phase.
• Early concept development: Usability testing with low-resolution prototypes (e.g. paper prototypes) in combination with information architecture testing techniques (e.g. card sorting, tree testing) in an iterative process can help detect problems with content organization and user errors.
• Prototype development: Before launching the “final” version of the website or app, you will benefit from testing navigation-enabled (clickable) prototypes to detect usability problems related to learnability, efficiency of use, memorability and more user errors.
• Final version development: Once the final version of the design has been chosen, it is time to do a final check on usability issues to make sure the implementation of the prior insights was effective, and no major problems remain. If at this point, no test has been done before committing to a final design, we strongly recommend doing the testing before sending it to the developers. It will cost you more to fix the problems after investing in development than what you would pay for the research at this point in the process.

There are also small-scale un-moderated usability testing modalities that can be more appropriate depending on the target audience location, timeline, and budget. When possible, we recommend combining qualitative usability testing with large-scale quantitative unmoderated tests to validate certain hypotheses.

Usability Test Modalities

Qualitative usability testing can be moderated, in-person and remotely, or un-moderated, which are often done remotely. They are based on the “think-aloud” methodology, in which we observe users’ behavior and listen to their feedback as they perform the tasks. These studies can be conducted with eye-tracking technology.

The choice of modality depends on the study goals, types of users, the technology available, budget, and timeline.

Moderated, in-person qualitative user testing is most appropriate when:

• Users are less computer-savvy.
• Non-computer based tasks are included.
• Personal tasks with sensitive data are included.
• There is an interest in observing users’ faces and body language.
• There is a need to protect product ideas, designs, and company information from being subject to screenshots or photos.
• User behaviors need to seem more immediate and “real” to the internal stakeholders.
• There are complex tasks that are difficult to administer remotely.
• It needs to be made more personal, sometimes more comfortable for all.
• There is a poor internet connection.
• When testing is conducted with kids, which requires consent.

Remote, moderated or unmoderated qualitative user testing is more appropriate when:

• There is a limited budget.
• There is a short timeline.
• It is important to understand the users’ own computer setup, especially if it is highly customized.
• It is the best way to reach users who are currently using the website or application.
• Testing with users in many different locations, including other countries.
• It is more convenient for participants.
• Observers are in multiple locations.

Back to Top

Target Audience Recruitment

Relevant Insights can recruit qualified 2C and B2B participants through our sample provider partners for both qualitative and quantitative research. Relevant Insights can also recruit participants from your customer database and will implement procedures to protect any identifiable personal information.

As an alternative, you may also recruit directly from your own customer database if data privacy restrictions don’t allow for customer data sharing. In these cases, we will provide the necessary data collection tools or work with your data collection tools (i.e., survey tools) if the data is required to stay in the account.

After discussions with you, we determine the sample parameters and assist in developing screeners. We also manage participant incentives.

Back to Top

Implementation

In direct collaboration with you, Relevant Insights will:

1. Develop tasks, scenarios, before-task, and follow-up questions to support the study objectives.
2. Recommend sample size, target audience, and sample sources. The sample of qualitative usability tests may include from 5 to 20 participants.
3. Set up and host usability tests with appropriate tools for both moderated and unmoderated usability testing. Relevant Insights uses a variety of tools to conduct usability testing on desktop and mobile versions of websites and apps with and out eye-tracking studies.
4. Rent facilities or online platforms as needed.
5. Moderate in-person or remote user test sessions.

Back to Top

Analytical Plan

Metrics that can be included in qualitative usability tests:

• Success rates for specific tasks.
• Time on tasks.
• Error rate.
• Perceived level of difficulty of the tasks.
• Perceived level of satisfaction with aspects of the tasks.
• Design elements and features that work well.
• Points of frustration.
• Level of difficulty to recover from a problem.
• Problems found in critical tasks.
• Users’ path.
• Page goal’s clarity.
• Whether a feature is used or works as intended.
• Learnability.
• Efficiency of use.
• Memorability.

Back to Top

Reporting

Relevant insights can provide different reporting options, which can be chosen based on budget and time constraints:

1. Video and audio recordings of interviews.
2. Interview transcripts.
3. Short written summary report with key findings.
4. Full detailed report with analysis, charts, quotes in a graphically appealing format.

Back to Top

Typical Project Duration

Four to six weeks.

Factors that affect project duration include:

• The incidence rate of the target sample: The lower the incidence rate, the longer we need to stay in the field to gather the required data.
• Client team responsiveness: A delayed response to requests for feedback at different steps of the project, will stall a project and affect the delivery date.
• The type of reporting selected.

Back to Top

Investment

The cost will vary depending on:

• Chosen test modality (moderated, in-person, remote, un-moderated).
• Sample specifications and sample size.
• Reporting requirements.
• The number of participants.
• Whether an eye-tracking study is included.

Back to Top