Validity and reliability in surveys in often assumed, but a lot of work is required to achieve bothThere are many things to consider if we want to write surveys that gather high quality data, including data collection method, respondent effort requested, question wording, order, format, structure, visual layout behaviors to be measured, accuracy of the elicited information, among others. Although all these issues are important, at the end of the day, what we want is to create surveys that yield results that are valid and reliable.

Validity and reliability are often discussed in the field of psychometrics, but not so much in market research, although it is assumed they are present. 

Validity is concerned with the accuracy of our measurement, and it is often discussed in the context of sample representativeness. However, validity is also affected by survey design since it also depends on asking questions that measure what we are supposed to be measuring.

Most surveys often have what is called face validity, which is a matter of appearances. The questions seem like a reasonable way to obtain the information we are looking for, but are they really? There are other types of validity survey writers should strive for:

• Content validity: This is related to our ability to create questions that reflect the issue we are researching and make sure that key related subjects are not excluded. For example, if we are interested in learning how consumers use hair styling products, and only ask how they have used them in the past week, we are likely to miss information about how these products are used under different weather conditions (given that humidity can give you a bad hair day in a blink of an eye) and end up with an incomplete picture of consumers’ behavior in this category.

• Internal validity: This asks whether the questions we pose can really explain the outcome we want to research. In our hair styling product example, we need to ask questions that help us identify factors that influence the selection of hair styling products. Here we are looking for a relationship between independent variables (e.g. hair type, desired hair style etc.) and the dependent variable (e.g. likelihood to buy the hair styling products).

• External validity: This refers to the extend in which the results can be generalized to the target population the survey sample is representing. As we all know, the way we ask questions will determine the answer we get, so the questions should reflect how the target population talks and think about the issue under research, which often call for the need to conduct exploratory qualitative research. In our example, if we want to estimate the share of preference our hair styling product would get in the hair styling category, we need to include other brands that represent this category, otherwise we can’t extrapolate the results to the category as a whole.

Reliability, on the other hand, is concerned with the consistency of our measurement, that’s the degree to which the questions used in a survey elicit the same type of information each time they are used under the same conditions. This is particularly important in satisfaction and brand tracking studies, as changes in question wording and structure are likely to elicit different responses.

Reliability is also related to internal consistency, which refers to the degree different questions or statements measure the same characteristic. A practical application of this concept can be found in market segmentation studies that try to capture psychographics and construct behavioral or satisfaction segments by asking respondent to rate a list of statements using different rating scales (e.g. agreement/disagreement; likes/dislikes; excellent/poor, etc.). In our example, if we want to identify “lovers of styling products,” the statements used to describe such consumers should provide a consistent description of this group. This can be tested by using correlations, split sample comparisons or methods such as Cronbach’s Alpha.

Validity and reliability are not always aligned. Reliability is needed, but not sufficient to establish validity. We can get high reliability and low validity. This would happen when the wrong questions are asked over and over again, consistently yielding bad information. Also, if the results show large variation, they may be valid, but not reliable. So, don’t forget to think about reliability and validity when writing your next survey and strive for reliable and valid results.