Long surveys are a plague. Writing short surveys is an uphill battle with many clients. Whenever there is a chance to do a survey, everybody close to the subject wants to add questions. The thought is, “since we are doing a survey, let’s get as much as possible out of it.”
Unfortunately, the only thing you get out with very long surveys is bad quality data. Why?
Non-Response & Abandonment
As the survey length increases, so do the non-response bias and abandonment rate. In other words, respondents won’t stay too long answering questions. Many won’t even start if they know the survey length. At the same time, it is a best practice to announce the length of the survey in the invitation.
If you try to trick respondents by omitting how long the survey will be, think again. Respondents can always figure it out from the progress bar. They will leave in the middle of the survey if they perceive it as too long (even if no progress bar is shown). Therefore, high abandonment and non-response rates affect sample representativeness negatively.
In an experiment conducted by Galesic and Bosnjac (2003) to prove this point, 3,472 respondents were divided into 3 groups based on an online survey with different lengths (10, 20 and 30 minutes). The chart shows how the number of respondents who completed the survey declined as the survey length increased.
While some respondents are willing to endure a long survey, they are at high risk of experiencing high burden and becoming “satisficers.”
Satisfacing occurs when the respondents select the answer options without giving them too much thought. They go for the most effortless mental activity trying to satisfy the question requirement. Mental work to find optimal answers that best represent an opinion is exhausting after a few minutes.
Consequently, respondents may start selecting the first choice in every question, straight-lining in grid questions (selecting the same across all options), or simply selecting random choices without much consideration. This type of behavior renders the data worthless.
The same experiment by Galesic and Bosnjac was set to test the impact of survey length on data quality. The researchers used a variety of indicators to measure the impact, including response times, item response rate, length of answers to open-ended questions, and variability of answers to questions in grids.
Of all the indicators, the only one that seemed unaffected by survey length was item response rate (defined as the percentage completed from all questions presented in a block).
However, it is unclear if the survey forced respondents to answer before going forward. For the other indicators, the results strongly suggest that survey length affects quality.
Why Do We Write Long Survey?
There are powerful forces pushing clients and forcing research vendors to launch long surveys. Budget, time constraints, and different agendas from internal groups are some of them.
However, when surveys start getting too long, clients and research vendors should take a minute to think about the implications. After all, if we get bad data, we have wasted the little time and money we started with.