Multiple-choice questions (check all that apply) are very common in surveys. However, there are a couple of problems with this type of question.
This type of question often makes it easy for respondents to engage in satisficing behavior. This occurs when respondents select answer options without giving them too much thought. They go for the most effortless mental activity trying to satisfy the question’s’ minimum requirements, rather than working on finding the optimal answers that best represent their opinion.
Meaning of Non-Choices
We really don’t know what it means when an item from the list is not chosen. This could happen (Sudman and Bradburn, 1982) because:
- The option didn’t apply to the respondent
- The respondent is neutral or undecided
- The respondent overlooked the item
Alternative To The Multiple-Choice Format
A suggested solution to this problem is to ask multiple-choice questions as a series of forced yes/no answers for each of the question items.
This format requires that respondents report a judgment on each of the items. Research has shown that forced yes/no questions encourage deeper processing time and discourage satisficing response strategies. This is measured by the time spent on answering forced yes/no vs. check-all questions, and the number of items marked affirmatively in each question format.
Research by Smyth et al. (2003), comparing results from both types of formats in online surveys has found that:
- Respondents who answered forced yes/no questions spent significantly more time responding than did respondents to the check-all formatted questions.
- The forced yes/no format yielded more options marked affirmatively than the check-all format.
We can argue that the longer time spent answering the forced yes/no questions is a mechanical function of the fact that respondents are forced to give an answer for each item and spend extra time marking “no,” which is not required in the check-all question.
However, the positive correlation between time spent on answering the question and the number of options selected has also been shown to be an indicator of deeper processing and more thoughtful answers for the check-all formatted questions as well.
Respondents who spend more time answering check-all questions mark significantly more answers than those who answer check-all questions in less time.
Another research result supporting the hypothesis of deep processing is that there are no significant differences in the number of options marked affirmatively between respondents that take a longer time answering yes/no questions and check-all questions.
Not A Foolproof Solution
The yes/no format for multiple-choice questions is not 100% foolproof, as some respondents may still show satisficing behavior by marking yes or no for all options or marking them randomly.
In this case we need to put quality checks in place during programming that take into account the time spent on the question and any straightlining patterns.
An issue that we also need to manage is the fact that sometimes respondents can make a decision or think an option doesn’t apply to them. In this case, it would be wrong to force them to give a yes or no answer.
The best remedies against this problem are respondent screening and survey skips that would avoid showing options that don’t apply. In cases in which there is still room for this problem, I recommend adding a third “Don’t Know/Not Applicable” option.
Implications For Mixed Data Collection Modes
Phone surveys often include the forced yes/no format for multiple-choice questions since it is impractical to read all the options to respondents and expect them to remember them all to answer the question.
Unfortunately, in mixed data collection modes, (phone/online, phone/paper), the yes/no and check-all question formats are sometimes treated as equivalent. In other words, there is an assumption that respondents answer them in the same way. Research suggests that this would be a mistake.
Experiments carried out by Smyth et al. (2008) with phone and online surveys using both question formats have shown that the forced yes/no format yields consistently more options marked affirmatively than check-all formatted questions in online self-administrated and phone-administrated surveys.
This supports the idea that results from both question formats are not comparable and shouldn’t be treated interchangeably.
Based on the research, it seems you are better off using forced yes/no format for multiple-choice questions in order to elicit deeper processing and minimize satisficing behaviors.
However, consider also the visual format you will be using. If you use grids to consolidate several items in a question, the user experience may worsen for the respondent. For more on this check the article: Multi-Response or Dichotomous Grid Questions?
Finally, do not mix the yes/no and check-all formats across data collection modes, as results are not comparable.