Determining the sample size is one of the early steps that must be taken in the planning of a survey. Unfortunately, there is no magic formula that will tell us what the perfect sample is since there are several factors we need to think about:
- ANALYTICAL PLAN: The research objectives and planned analytical approach should be the first factor to consider when making the decision on sample size. For instance, there are statistical procedures (e.g. regression analysis) that require a certain number of observations per variable. Moreover, if comparative analysis between subgroups in the sample is expected, the sample size should be adjusted for it to be able to identify statistically significant differences between the groups.
- POPULATION VARIABILITY: This refers to the target population’s diversity. If the target population exhibits large variability in the behaviors and attitudes of interest being researched, a large sample is needed. If 20% or 80% of the population behaves in certain way, this indicates less variability than if 50% would do so. To be conservative, it is standard practice to use 50% (0.5) as the event probability in sample size calculations since it represents the highest variability that can be expected in the population.
- LEVEL OF CONFIDENCE: This is the level of risk we are willing to tolerate usually expressed as a percentage (e.g. 95% confidence level). Although survey results are reported as point estimates (e.g. 75% of respondents like this product), the fact is that since we are working with a sample of the target population, we can only be confident that the true value of the estimate in that population falls within a particular range or what is called confidence interval. The level of confidence indicates the probability that the true value of the estimate in fact will fall within the boundaries of the confidence interval. How confident can you be? As confident as your tolerance for risk allows you to, knowing that the confidence level is inversely proportional to estimate accuracy or margin of error. The more confident you want to be, the larger the confidence interval that is needed, which leads to lower levels of precision.
- MARGIN OF ERROR: Also known as sampling error, indicates the desired level of precision of the estimate. You have probably seen poll results quoted in the media, saying that the margin of error was plus or minus a particular percentage (e.g. +/-3%). This percentage defines the lower and upper bounds of the confidence interval likely to include the parameter estimate, and it is a measure of its reliability. The larger the sample, the smaller the margin of error and the greater the estimate precision.
Below is a table illustrating how the margin of error and level of confidence interact with sample size. To get the same level of precision (e.g. +/-3.2%), larger samples are needed as the confidence level increases. For example, if we want to be certain that in 95 out of 100 times the survey is repeated the estimate will be +/- 3.2%, we need a sample of 950.
For more help on calculating sample size and margin of error, use our Sample Size and Margin of Error Calculators.
- COST: Sample size cost is often one of the largest items in the budget for market research studies, especially if the target sample includes low-incidence segments or the response rates is low. Many times, our clients have to make a tradeoff between statistical accuracy and research cost. Recently, I received a call from a client who wanted to conduct an online survey with a sample of 1,000 respondents, which would give a statistical accuracy of +/-3.1% at the 95% confidence level, but would cost $8,000 based on certain screening criteria. At the same time, a sample of 400 respondents would give a statistical accuracy of +/-4.9% and cost $3,400. In this case, a 135% increase in sample cost would only yield a 60% gain in statistical accuracy. The client decided to conduct the study on the smaller sample.
- POPULATION SIZE: Most of the time, the size of the total target population is unknown, and it is assumed to be large ( >100,000), but in studies where the sample is a large fraction of the population of interest, some adjustments may be needed.
SAMPLE SIZE CALCULATION CHECK LIST
As a summary, to determine the sample size needed in a survey, we need to answer the following questions:
- What type of data of data analysis will be conducted? Will subgroups be compared?
- What is the probability of the event occurring? – If not previous data exists, use 50% for a conservative sample size estimate.
- How much error is tolerable (confidence interval)? How much precision do we need?
- How confident do we need to be that the true population value falls within the confidence interval?
- What is the research budget? Can we afford the desired sample?
- What is the population size? Large? Small/Finite? If unknown, assume it to be large ( >100,000)
So the answer to the question “What is the right sample size for a survey?” is: It depends. I hope I gave you some guidance in choosing sample size, but the final decision is up to you. To calculate sample size and margin of error, use our Sample Size and Margin of Error Calculators.
Have you wondered, what sample size is needed to get a representative sample, read Does A Large Sample Size Guarantee A Representative Sample?