A Better Format for Multiple-Choice Questions in Online Surveys

 |  Posted: by

Multiple Choice Questions

 Multiple-choice questions (check all that apply) are one of the most common question formats found in online surveys. However, there are a couple of problems with this type of question:

  • It often makes it easy for respondents to engage in satisficing behavior, which occurs when respondents select the answer options without giving them too much thought. They go for the most effortless mental activity trying to satisfy the question requirement, rather than work on finding the optimal answers that best represent their opinion.
  • We really don’t know what it means when an item from the list is not chosen. This could happen (Sudman and Bradburn, 1982) because:
  1. The option didn’t apply to the respondent
  2. The respondent is neutral or undecided
  3. The respondent overlooked the item

 WHAT CAN WE DO ABOUT IT?

A solution to this problem is to ask multiple-choice questions as a series of forced yes/no answers for each of the question items. This format requires that respondents report a judgment on each of the items. Research has shown that forced yes/no questions encourage deeper processing time and discourage satisficing response strategies as measured by the time spent on answering forced yes/no vs. check-all questions and the number of items marked affirmatively in each question format. Research by Smyth et al. (2003), comparing results from both types of formats in online surveys has found that:

  • Respondents who answered forced yes/no questions spent significantly more time responding than did respondents to the check-all formatted questions.
  • The forced yes/no format yielded more options marked affirmatively than the check-all format.

We can argue that the longer time spent answering the forced yes/no questions is a mechanical function of the fact that respondents are forced to give an answer for each item and spend extra time marking “no,” which is not required in the check-all question.

However, the positive correlation between time spent on answering the question and the number of options selected has also been shown to be an indicator of deeper processing and more thoughtful answers for the check-all formatted questions as well. Respondents who spend more time answering check-all questions mark significantly more answers than those who answer check-all questions in less time.

Another research result supporting the hypothesis of deep processing is that no significant differences have been found in the number of options marked affirmatively between respondents that take longer time answering yes/no questions and check-all questions.

The yes/no format for multiple-choice question are not a 100% foolproof, as some respondent may still show satisficing behavior by marking yes or no for all options or marking them randomly. In this case we need to put quality checks in place during programming that take into account the time spent on the question and any straightlining patterns.

An issue that we also need to manage is the fact that sometimes respondents are undecided or think an option doesn’t apply to them. In this case it would be wrong to force them to give a yes or no answer. The best remedies against this problem are respondent screening and survey skips that would avoid showing options that don’t apply. In cases in which there is still room for this problem, I recommend adding a third “Don’t Know/Not Applicable” option.

IMPLICATIONS FOR MIXED DATA COLLECTION MODES

 The forced yes/no format for multiple-choice questions is commonly used in phone surveys since it is impractical to read all the options to respondents and expect them to remember them all to answer the question. Often, when mixed data collection modes are used, (phone/online, phone/paper), the yes/no and check-all question formats are treated as equivalent, assuming they are answered the same way. Research suggests that this would be a mistake.

Experiments carried out by Smyth et al. (2008) with phone and online survey using both question formats have shown that the forced yes/no format yields consistently more options marked affirmatively than check-all formatted questions in online self-administrated and phone-administrated surveys. This supports the idea that results from both question formats are not comparable and shouldn’t be treated interchangeably.

KEY TAKEAWAYS

  • You are better off using forced yes/no format for multiple-choice questions in order to elicit deeper processing and minimize satisficing behaviors.
  • Do not mix the yes/no and check-all formats across data collection modes, as results are not comparable.

Comments Comments

Anti A Posted: March 31, 2011

Hi Michaela, interesting article. Would you also recommend yes/no format for a longer list, say a brand list of 30 items?

And just a comment, I think with online panels ‘speeders’ are regularly kicked out of the valid completes. At least they are in the panel I use …

Michaela Mora Posted: March 31, 2011

Hi Anti,
Good to hear from you! Thanks for your comment. Yes, I would recommend it, since in such long lists it is easy to overlook items. Having said that, a list of 30 items is pretty long and exhausting for respondents. If possible make it shorter or at least divide it in blocks.
Quality controls to eliminate speeders and straightliners are a must these days.

Research Rants Posted: April 4, 2011

No offense intended, but if you’re giving respondents a list of thirty items, you may want to ask yourself what sort of respondent ISN’T going to speed through it, and whether or not someone with that much free time is even vaguely representative of an average consumer. I recognize that in certain categories, there’s just no easy way around this, but it’s really rough on respondents.

Michaela Mora Posted: April 4, 2011

Good point Research Rants! I agree. We must strive for fewer items in multiple choice questions.

Stephanie Posted: January 13, 2012

Can you recommend a website to administer forced choice questionnaires?

Only logged in users can leave comments.