Quantitative Usability Testing

Quantitative usability testing allows you to understand the user’s actual experience. In contrast with usability testing conducted in a lab or focus group facility, quantitative usability testing often takes place in the same context users normally would interact with a website or application.

Why You Should Do It

Quantitative usability testing has the advantage of including larger samples to help validate hypotheses regarding usability issues users may find while interacting with a website or application in their natural environments.

This type of usability testing is well-suited for:

• Benchmarking studies in your industry.
• Detecting serious usability issues with your website/app and comparing them to the competition.
• Identifying navigational issues on your website/app.
• Understanding the impact of user’s technical setup (e.g. monitor size, browser, screen resolution, system performance

Back to Top

The Approach

Quantitative usability testing is mostly conducted remotely without the direct intervention of a moderator; thus, they are unmoderated and look very much like a survey from the user’s perspective.

These tests include task-based questions related to specific usage scenarios and other question types (e.g., rating, single-choice, multiple-choice, open-ended, etc.).

They can be conducted on desktop and mobile devices. The type of data collected depends on the device type, data collection application used and budget. There are many available. However, the collected metrics vary from app to app.

How Do We Collect Data?

Quantitative usability testing requires specialized data collection tools.

Current applications that can be used for quantitative usability testing collect one or more of these data types:

• Video recording of tasks completed by users, including users’ comments about the task (think-aloud approach). This may include recordings of screen activities, users’ faces, and audio, and sometimes screen touches (mobile testing).
• Clickstream, capturing the navigation path users take when trying to accomplish the task. This requires saving the users’ clicks.
• Heatmaps, showing the density of clicks in different areas where users click during the task. This requires saving the users’ clicks.
• Data from survey questions related to the tasks.

The choice of data collection application will depend on the study objectives, timeline, and budget.

When Is It Appropriate to Use Quantitative Usability Testing?

Quantitative usability testing can be conducted at different stages of the design process:

• Before the development of design concepts: Usability testing of the current website or application in comparison with competitor websites and apps can provide invaluable insights during the requirement gathering phase.
• Prototype development: Before launching the “final” version of the website or app, you will benefit from testing navigation-enabled (clickable) prototypes to detect usability problems related to learnability, the efficiency of use, memorability, and user errors.
• Final version development: Once the final version of the design has been chosen, it is time to do a final check on usability issues to ensure that the implementation of the prior insights was effective, and no major problems remain. At this point, if no test has been done before committing to a final design, we strongly recommend doing the testing before sending it to the developers. Compared to the research expenses at this point in the process, you will spend more money to fix the problems after investing in development.

Before deciding when to do it, you should consider some of the limitations of quantitative usability testing.

It is not appropriate when:

• You want to identify major usability issues as part of a rapid design iteration cycle. If you need a quick turnaround, you should do small-scale qualitative usability testing remotely or in person.
• You need a deeper understanding of the users and their behaviors. Although we can ask open-ended questions in quantitative usability testing, you can’t probe further if needed. Moreover, this is not ideally suited to ask complex questions, and you can’t observe users’ behaviors.

We often recommend combining quantitative usability testing with qualitative usability testing for a comprehensive understanding of usability issues.

Back to Top

Target Audience Recruitment

Relevant Insights can recruit qualified B2C and B2B participants through our sample provider partners. Relevant Insights can also recruit participants from your customer database and will implement procedures to protect any identifiable personal information.

As an alternative, you may also recruit directly from your own customer database if data privacy policies restrict customer data sharing. In these cases, we will provide the necessary data collection tools.

After discussions with you, we determine the sample parameters and assist in developing screeners. We also manage participant incentives.

Back to Top

Implementation

In direct collaboration with you, Relevant Insights will:

1. Develop tasks, scenarios, before-task, and follow-up questions to support the study objectives.
2. Recommend data collection tools for the research objectives.
3. Recommend sample size, target audience, and sample sources. The sample of qualitative usability tests may include from 100 to 200 participants.
4. Set up and host the usability test with appropriate tools for unmoderated usability. testing. Relevant Insights uses a variety of tools to conduct usability testing on desktop and mobile versions of websites and apps.

Back to Top

Analytical Plan

Common metrics monitored in quantitative usability testing include:

• Success rates for specific tasks.
• Time on tasks.
• Error rate.
• Perceived level of difficulty of the tasks.
• Perceived level of satisfaction with aspects of the tasks.
• Clickstream through different navigation paths.
• Heatmaps.
• Comparison between websites/apps, when appropriate.

Back to Top

Reporting

Relevant Insights can provide different levels of reporting based on budget and time constraints including:

1. A full, detailed report with analysis, quotes, and graphics in an appealing format.
2. Video of user sessions (depending on data collection tool).
3. Transcripts (when video with audio are collected).
4. Heatmaps (depending on data collection tool).
5. Clickstream patterns.

Back to Top

Typical Project Duration

Five to eight weeks.

Factors that affect project duration include:

• The incidence rate of the target sample: The lower the incidence rate, the longer we need to stay in the field to gather the required data.
• Client team responsiveness: A delayed response to requests for feedback at different steps of the project will stall a project and affect the delivery date.
• The number of participants that need to be recruited.

Back to Top

Investment

The cost will vary depending on:

• Data collection tool selected.
• Sample specifications and sample size.
• Reporting requirements.
• Number of participants.
• Whether competitive analysis is requested.

Back to Top