Avoid These Survey Design Mistakes

Summary: The quality of survey data depends in great measure on the survey design. In the era of DIY research, survey mistakes are too common. This article describes the top two mistakes and how to remedy them.

3 minutes to read. By author Michaela Mora on December 1, 2016
Topics: Analysis Techniques, Market Research, Survey Design

Avoid These Survey Design Mistakes

Start with a good survey design, if you want good survey data. I recently took a brand awareness and positioning survey in the yogurt category that left me wondering how useful my answers were to the brands in question.

First Mistake

After some screening questions, I got one about familiarity with different brands of yogurt, the point at which I started questioning the survey quality. In this case, this question used  the following 3-point scale:

1. Never heard of

2. Heard of, but never used

3. Familiar

Although this scale is not consistent in the terms it uses and doesn’t flow with the terms used in the question, it is still easy to understand what it is after. The issue is that I didn’t know what being “familiar” with each of the brands really meant. Therefore, based on the context of the other two options, and without any others to indicate levels of familiarity, I assumed it equated “familiar” with “have consumed the brand before.”

Second Mistake

The problem with the survey design became pretty clear in the questions that followed. They asked if I agreed or disagreed with certain statements about the brands’ history and engagement in the community. Using a 6-point balance scale with no neutral point or the Don’t Know option made matters worse.

For instance, I like yogurt, have tried many brands, and buy several every week. This makes me familiar with them until a certain point. I look at the label information to check for sugar, fat content, and all the extra fillers.

However, my level of engagement doesn’t go as far as to dedicate time to research the history and community involvement of any of the brands I have consumed. Moreover, many of them were not even on my top-of-mind list.

Consequently, since the scale didn’t have a Don’t Know option, I was forced to select answers that made me feel like a liar because I didn’t really know the answers, probably making the data useless.

Survey Design Remedy

1. If you are going to use a familiarity scale, make sure to define what each of the scale points means. Provide enough of them to capture different levels of familiarity.

2. A better solution yet is to ask specific questions to gauge familiarity with certain items to exclude unfamiliar ones, before you ask participants to agree or disagree with them.

3. Awareness would have been a more appropriate scale for some of the brand statements in the example above. If you use agreement scales, always add a Don’t Know option and a neutral point to the scale. Don’t assume consumers know enough about the subject or have a position formulated about it.

To read more about this, check: Is It Right to Include a Neutral Point in Rating Questions?