Why Sample Size Matters

The first question I always get from clients interested in conducting a survey is about sample size.

Sample Size & Representation

Many confuse sample size with representativeness. They are related, but not the same, particularly if you use convenience samples.

In random samples, as we increase sample size, the chance each member of the population has of being selected increases. Consequently, more segments of the population are likely to be represented.

Random samples are drawn from a population list (population frame), and the probability of being chosen is known. This could be the case of a customer database/list if that’s our population of interest. 

In the case of convenience samples, the population frame becomes the pool of individuals in the sample source available to us (e.g., online panels).

This population frame may not include all segments in the target population or only have a few members of certain segments, depending on how the sample source is built.

In this case, we use sample quotas, weighting schemes, and mixed-mode data collection methods (online/phone/intercepts) in an effort to reach representativeness. 

Assuming that we are able to pull a representative sample of the target population, we need to give serious consideration to sample size. This is a case where size matters (pun intended). Why?

Why Does Sample Size Matter?

It is all about precision, tolerance for risk, and cost. For samples smaller than 1000, we always have to think about how confident in the method we use to determine that our estimates are within a particular range (level of confidence and risk), and how small we want that range to be (margin of error=level of precision). Unfortunately, they go in opposite directions. Higher levels of confidence require greater ranges (margins of error) in small sample sizes.

For instance, we can be 95% confident that the true estimate for a variable in a sample of 400 is within +/-4.9%. However, if we want a smaller margin of error, in an attempt to gain more precision, we have to sacrifice certainty. This means we may need to accept a 90% confidence level to get a +/-4.1% margin of error.

At the 95% confidence level, you are more certain but less precise as you expand the range to make sure the true value falls in it. At 90%, you are more precise but less certain.

If you want to get more precise estimates without sacrificing certainty in the results, then you have to increase size. On the other hand, this increases research costs.

As the table below shows, as the sample size increases, the margin of error differences become smaller, across the different confidence intervals.

When it comes to determining sample size, you need to decide what’s more important to you, certainty or precision. Also, you need to consider your tolerance for risk, especially if your market research budget is small. There are always trade-offs you will have to make.

For more help on calculating sample size and margin of error, use our Sample Size and Margin of Error Calculators.

(An earlier version of this article was originally published on February 16, 2011. The article was last updated and revised on August 18, 2019.)