Moderated or Unmoderated Usability Tests?

Moderated or Unmoderated Usability Tests? That’s the question.

Usability tests are one of the UX research tools we can use to understand the user experience and improve it. Traditionally, we have done these tests in a lab setting often with a moderator, but new technologies are offering options to go outside the lab.

In moderated usability tests, which can be in-person or done remotely, a moderator follows a task-based discussion guide and asks probing questions after each task.

In contrast, in unmoderated usability tests, there is no moderator. Participants in these tests complete tasks on their own and answer follow-up pre-programmed questions.

Decision Criteria

The decision to conduct moderated or unmoderated usability tests requires consideration of several factors.

Among those factors are:

• Test objectives
• Type of data needed
• Usability tasks
• Test logistics

Test Objectives

Moderated and unmoderated usability tests are good at different things. Moderated tests are better for detecting critical issues, quickly and with a deep understanding of the “why” behind user behavior.

In the case of unmoderated usability tests, we use them to understand the magnitude of those issues (how many users really find them), measure actual user behavior, and compare design concepts, prototypes or competitor alternatives.

Type of Data Needed

Your test objectives will also guide the type of data needed. Moderated usability tests provide in-depth qualitative data due to the presence of a moderator and a small sample.

Unmoderated usability tests can provide both quantitative and qualitative data. However, the depth of the qualitative data is limited, as we are not able to ask probing questions. All you may get are responses to open-ended questions of varying quality.

Quantitative behavioral data is the strong suit of unmoderated usability tests. We can deploy them to large samples, allowing for generalization to a target population

Usability Tasks

Moderated and unmoderated usability tests also differ in the context in which we present tasks, the number of tasks and follow-up questions, and session length. In addition, the moderator influence and the social desirability effect are factors affecting the interviewing setting.

The main difference between both types of usability tests is the focus on interactive behavior in real-world conditions. Unmoderated usability tests allow capturing natural user interactions with the interface in the context where they normally occur (e.g. background noises, interruptions, device settings, etc.)

Test Logistics

There are advantages and disadvantages of moderated and unmoderated tests regarding their setup. On the one hand, we can conduct moderated tests quickly without technology barriers on the user side. However, they can pose geographic barriers when done in person. They also tend to be more expensive.

By contrast, unmoderated usability tests can reach many different participants across geographic regions and time zones at a low cost. The challenge is finding qualified participants with access to technology (e.g. internet access, computers, smartphones, etc.) in geographic areas of interest.

In Conclusion

Each usability testing approach has a purpose and complements each other. If timing and budget permit, you should use both. If you can only afford one or the other, think hard about the objectives and the type of decision you will make.

There are many tools available to do quantitative usability testing online faster and cheaper. Although these can also provide qualitative data, moderated usability testing can give you a deeper layer of insights into user behavior on specific issues.

In short, for detecting major problems or understanding at a deeper level of users’ behavior and expectations,  we recommend moderated usability tests with a qualitative approach.

Finally, if you need to know how big of a problem you have in your hands, which design concept to choose, or how you compare to your competitors, and unmoderated usability test with a quantitative approach would do the trick.

(An earlier version of this article was originally published on January 19, 2012. The article was last updated and revised on June 3, 2019.)