Sample Size Matters
Wednesday, February 16, 2011
The first question I always get from clients interested in conducting a survey is about sample size. Many confuse sample size with representativeness. They are related, but not the same, particularly if convenience samples are used.
In random samples, as we increase sample size the chance each member of the target population has of being selected increases and consequently more segments of the population are likely to be represented. This is based on the assumption that we have a list with all the population members (population frame) and know their probability of being chosen. This could be the case of a customer database/list, if that's our population of interest.
In convenience samples, the population frame becomes the pool of individuals in the sample source (e.g. online panels), which may not include all segments in the target population or only have a few members of certain segments, depending on how the sample source is built. In this case sample quotas, weighting schemes, and mixed mode data collection methods (online/phone/intercepts) are often used in an effort to reach representativeness.
Assuming that we are able to pull a representative sample of the target population by whatever affordable means are available to us, we need to give serious consideration to sample size. This is a case where size matters (pun intended). Why?
It is all about precision, tolerance for risk and cost. For samples smaller than 1000, we always have to think about how confident we want to be that estimates are within a particular range (level of confidence and risk), and how small we want that range to be (level of precision). Unfortunately, they go in opposite directions. Higher levels of confidence require greater ranges (margins of error) in small sample sizes.
For instance, we can be 95% confident that the true estimate for a variable in a sample of 400 is within +/-4.9%, however, if we want a smaller margin of error, in an attempt to gain more precision, with the same sample size, we have to sacrifice certainty and may need to accept a 90% confidence level to get a +/-4.1% margin of error. At the 95% confidence level you are more certain, but less precise as you expand the range to make sure the true value falls in it. At the 90%, you are more precise, but less certain.
If you want to get more precise estimates without sacrificing certainty in the results, then you have to increase sample size, which in turn increases research costs. As the table below shows, as sample increases the differences in margin of error across the different confidence intervals become smaller.
At the end of the day, when it comes to sample size, you need to decide what it is more important to you, certainty or precision, and what your tolerance for risk is, especially if your market research budget is small.
For more help on calculating sample size and margin of error, use our Sample Size and Margin of Error Calculators.