Multi-Response or Dichotomous Grid Questions?

Survey researchers have been debating and researching whether to use dichotomous grid questions (dual-column grid format) or multi-response questions for a while. Both are common survey question types together with rating scales.

Acquiescence Bias

The driver behind this effort is the search for ways to minimize respondent acquiescence bias. This is the tendency to positively agree with all questions or statements in a survey.

Multi-response questions, in which the respondents are asked to select all the options that apply, are a common question format used in surveys. They tend to elicit fewer selections in comparison with dichotomous grid questions.

In dichotomous grids, respondents are more likely to select positive answers (e.g., Yes, Agree, etc.), when forced to choose between a positive or negative answer (e.g., Yes/No, Agree/Disagree) (Smyth, Dillman, Christian & Stern, 2006; Thomas & Klein, 2006).

In 2015, Callegaro, Murakami, Tempman & Henderson suggested that the acquiescence bias explained the larger number of positive responses in the dichotomous grid format.

One way to minimize the effect of acquiescence bias in Yes/No answers, in any format, is to, explicitly explain what the Yes and No mean, namely being specific by connecting the answers to the question’s root. This is called being construct-specific in survey methodology parlance.

If you ask:

       Q. Have you taken a road trip in the last 12 months (month, year – month, year)? The specific answers would be:

            A1. Yes, I have taken a road trip in the last 12 months (month, year – month, year)

           A2. No, I have not taken a road trip in the last 12 months (month, year – month, year)

Salience Hypothesis

However, research conducted by GFK and the University of Nebraska – Lincoln (Thomas, Barlas, Buttermore & Smyth, 2017), presented in a poster (“Acquiescence Bias in Yes-No Grids? The Survey Says… No.”) at the 2017 AAPOR conference, couldn’t confirm that the acquiescence bias was the key driver for differences between the dichotomous grid and multi-response formats. They hypothesized that the differences were more likely due to how the items stand out or resonate with respondents (salience hypothesis).

Initial research by this group of researchers, presented at the 2016 AAPOR conference, in which dichotomous “Yes/No”, and “Describe/Do Not Describe” grids were compared to multi-response questions, found support for the salience hypothesis. However, it was objected that these grids were also prone to acquiescence bias, explicitly or implicitly.

Grids with Labeled Columns

In the most recent research, the researchers expanded the experiment to include construct-specific dichotomous grid questions with column labels. These were based on the specific question asked (e.g., Like – Do Not Like), in which there is no explicit or implicit agreement.

This time, the research showed that the average number of positive responses was similar across scales. They were also higher than those for multi-response questions.

Interestingly, more items with lower salience also received positive answers in the dichotomous grids, compared to the multi-response. This is most likely due to forcing respondents to consider each item before selecting an answer. As a result, this brings their attention to items that may be overlooked in a list of select-all-that-apply options.

To validate their results, the researchers asked a follow-up question about the frequency of engagement in behaviors presented in the multi-response and dichotomous grids. They, then, examined how well responses from the different question formats predicted the frequency behavior.

They expected low correlations between the answers to the grid questions and the frequency behavior if the responses to the dichotomous grids were a result of acquiescence bias. However, the analysis found comparable correlations between the frequency behavior question and all question formats tested, suggesting support for the salience hypothesis.

Based on these results, the researchers recommend using dichotomous grids, as they elicit greater item consideration, particularly in screeners.

The Problem With Grids

These results are encouraging and provide a promising practical application to improve respondent quality. Unfortunately, the use of grids has other problems.

Grid/Matrix questions:

• Pose a higher cognitive burden and take longer to answer.
• Can be tiring and tedious.
• Are susceptible to satisficing behavior (selecting answers that meet minimum requirements) which leads to straight-lining (selecting the same answers for all items).
• Take more space on any survey mode (online screen, mobile screen, paper survey).
• Need quality controls to filter out bad responses.

In short, dichotomous grids seem to be better to encourage more thoughtful answers about items that may be ignored in a long multi-response list. Nonetheless, we should use them with caution and add controls to validate their answers.