Posted on August 26, 2010

In-house market research departments are facing great challenges as the market research industry undergo changes, as my colleague Kathryn Korostoff, president of Research Rock Star points out in her recent article Market Research Departments: The Hero of the Market Research Story.

According to Korostoff, in-house market research departments are dealing with:

 1. An unmanageable number of market research-related associations, social networking groups, events and interactive “media”—too many to keep up with, too many to ignore.

  2. Proliferating pockets of unsanctioned research, often done by well-meaning but untrained colleagues.

 3. Increasingly sales-hungry research suppliers, making the risk of answering a phone call almost unbearable.

4. Insufficient time to truly assess the options and merits of social media-related research methods.

 5. Insufficient time, and in some cases authority, to establish and enforce customer research policies (which are of urgent importance because of the unsanctioned research that does take place).

 6. Internal clients becoming aware of numerous “market research” services and solutions, many of which are at best distractions and at worst disruptive.

7. Internal clients who are reading about “free” market research options, when in reality the phrase “market research” is being used very loosely.

Undeniably, the increasing presence of social media, the vast amounts of user-generated content and the proliferation of low-cost online data collection tools are having an impact on the matket research industry, not only for in-house market research departments, but also for research vendors. Some of the challenges research vendors face today are:

• Clients with little training in the fundamentals of research, expecting to get a lot of “free” research just because they have access to cheap online survey tools, social media and information on the Internet.

• Clients’ complacency with “good” enough research and willingness to bend rules that impact data quality due to small budgets.

• Clients’ unwillingness to share critical internal information that would allow research vendors to provide the support clients need and reduce their “vendor management” time.

• Convincing clients that old established ways (“we have always done it this way”) may not be valid anymore for their current situation.

• Time to keep up with demographic and cultural trends that have an impact on how we design research, collect data and analyze it.

• Need to deliver faster results on very small budgets.

I agree with Korostoff, there is a battle out there, but after being on both side of the line of fire as an in-house market research department and as a market research vendor, I can say the battle is equally fierce on both sides. Interestingly enough, we all want the same thing: We want our clients to succeed as much as they want to succeed themselves. 

My hypothesis is that we could win the battle via collaboration since clients and research vendors complement each other. We each have something unique to offer. We can be partners.

To clients I propose: 

1. Let vendors provide you with training and bring you expertise, new approaches and insights when needed.
2. Share your industry expertise with us, tell us about relevant business issues we need to be aware of in order to serve you better.
3. Let’s exchange experiences about the use of emerging technologies and knowledge about new consumer, industry and social trends and their application to market research.
4. Let’s collaborate in testing new approaches and re-evaluate established methods in the light of current market conditions.
5.  Let’s us help lighten your burden and in turn help us grow and keep our industry moving forward.

 If you have other ideas of how clients and research vendors could collaborate to keep the market research community thriving, I welcome your comments.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

Posted on August 13, 2010

A few months ago, I got a call on a Friday at 5:45pm CST from a new client who was referred to me by a colleague. He was trying to help his own client to determine which media buying strategy they should follow. They had tested several positioning approaches and now needed to know who they should target and with which media. They asked me if we could analyze the data and develop a targeting model that could guide their decisions. Of course, we did.

After trying different approaches, we recommended a Tree Analysis-based model, which met sound statistical requirements, but also was very actionable. The client could clearly see what ROI they would get from targeting different segments, which not always meant going for the largest segments, since greater reach also meant higher cost and a greater spillover on segments that were not worth targeting.

The client who called me, had worked with a different research vendor before, someone with a degree in statistics and a professional researcher. When I asked, why did he decide to award us the project, he said “You had the marketing perspective on the issue from the start.” I have to thank that to my previous experience as a market researcher on the client side closer to how business decisions are made.

This case came to my mind, when I recently read the post Why market researchers can never be marketers by Annie Pettit, Chief Research Officer at Conversition Strategies.

She discusses a problem market researchers have been fighting against, which is the use of market research as a disguise for selling. She describes the role of market researchers as guardians and advocates of the rights of research participants. As Pettit says, market researchers should provide a sanctuary for research participants to freely express their feelings and opinions. Then she goes on to say:

“Market researchers do not Sell and do not Market. There is no sanctuary if participants feel they will be targeted based on their opinions. Sugging (selling under the guise of research) and mugging (marketing under the guise of research) tear down the walls of the sanctuary. And once torn down, building them back up is a monstrous task.”

I couldn’t agree more. Market research should not be used to harvest prospects lists or to sell directly to research participants. I get that request now and then and my answer is always NO. That’s a job for telemarketers.

However, I don’t equate marketing with direct marketing (in which telemarketing is a tool). It goes beyond that to encompass strategies companies use to market their products and to do that effectively they should use market research to support business decisions. Here I’m talking about results and insights at the aggregated level, where participants anonymity is guarded. In the above example, the targeting model was never meant to target people who participated in the research, but consumers in the general market that may be like them and would have an affinity with the clients’ products.

Market researchers often complain they are not taken seriously and that too many research reports are used more as dust collectors (the thick ones can support a sofa’s missing leg also) than providers of decision support. I have had the fortune to see the implementation of research insights in action with huge dividends, and all I can say is that after all the correct methodological and data quality considerations has been made, what makes a difference in delivering actionable research is looking at the results from a marketing perspective in light of current market conditions and a client’s financial and capacity resources.

Market researchers, who know the principles of marketing are always in better position to help their clients. They will not do the selling. That’s the clients’ job. But market researchers should provide insights on who to sell, what to sell, why and to some extend how to sell to promote clients’ business growth.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

Posted on July 15, 2010

Qualitative research is going beyond in-person focus groups and experiencing a revolution for the better. Jim Bryson, president of 20/20 Research recently did a great presentation at the Fort Worth monthly luncheon organized by the DFW AMA about the latest online qualitative research techniques.

Thanks to the development of new online platforms, the qualitative research field have seen an explosion of new online qualitative research techniques that makes it possible to collect data in ways we couldn’t before.

Among the new online qualitative research techniques, we now have:

• Bulletin boards
• Immersion IDI’s
• Mobile qualitative
• Quantitative-Qualitative hybrids
• Real time chats
• Research blogs
• Research communities
• Social networking monitoring
• Video journals
• Webcam focus groups
• Web-Intercepts/chats

These new online qualitative research techniques have often made qualitative research better, faster and cheaper (not always). Studies using these online qualitative research techniques can be deployed pretty quickly and provide immediate access to transcripts or videos for review.

As for making qualitative research better, Bryson rightly points out to the advantages of most of these methods:

• Geography: it allow us to reach a wide-range of people across different regions without travel
• Candor: due to the privacy and confidentiality of online, many participants feel more comfortable to discuss sensitive topics
• Convenient access: both clients and participants can participate whenever is more convenient to them and their locations
• Longitudinal capabilities: studies can be extended over time as long as needed to understand the groups of interest

Below are some of the most often used online qualitative techniques, their advantages, disadvantages, and applications according to Bryson:

Mobile qualitative research, according to Bryson, will definitely be part of  market research’s near future. This can be easy and comfortable for the participants, although for now is limited to text only. This technique can be used for reaching to difficult groups, send reminders about “homework” given to study participants, and do research at the point of consumption.

Another approach getting traction is hybrid research, where quantitative and qualitative research are combined in one data collection opportunity. 20/20 Research recently launched a new service called Quallink where participants start in a survey and then are enrolled in a qualitative study. Hybrid research can also be done using SurveyGizmo, which has the capability to integrate online surveys with chat sessions from iModerate.

The main advantages of a hybrid approach are:

• Immediacy: there is no lag between quantitative and qualitative data collection
• Can be very cost effective since cost incurred in recruitment, travel and focus group facilities can be eliminated
• Ability to do a deep-dive on the story behind the numbers
• Ability to segment qualitative data together with quantitative results

There is no question that qualitative research has come a long way and that all these new techniques make this field exciting and promising, but before you get carried away by all the excitement, don’t forget to have clear research objectives and evaluate if these techniques are a good fit for what you want to accomplish.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

As published on July 6, 2010 in the July 2010 issue of the Quirk’s Marketing Research Review.

The advent of user-friendly online survey tools in recent years has created the illusion that anybody can write a survey questionnaire. After all, how hard can it be? It’s like asking questions in a conversation, many think. However, there are many methodological issues to consider when creating a questionnaire if you want to gather high-quality data in a survey. The following are 10 issues that arise in survey design.

• DATA COLLECTION METHOD

Some questions may elicit different answers if asked in an online survey, a telephone interview, a paper survey or a face-to-face interview. While words in phone surveys or in-person interviews are given more importance because of the conversational format, visual design elements have a bigger impact in how questions are read and interpreted in online surveys. Be aware of the types of questions that are a good fit for online surveys.

• RESPONDENT EFFORT
  

There are questions that put a heavier burden on the respondent’s working memory and comprehension or are likely to elicit higher non-response if asked in different data collection modes. Experience tells us that asking a ranking question with 10 items over the phone can overwhelm respondents. In online surveys, rating questions in matrix format with a large number of items increases fatigue and boredom and often leads respondents to adopt a “satisficing” behavior. Satisficing occurs when respondents select the same scale-point to rate all items without giving them too much thought. They go for the most effortless mental activity trying to satisfy the question requirement, rather than work on finding the optimal answers that best represent their opinion.

• QUESTION WORDING

Formulating a question with the right wording so it accurately reflects the issue of interest is one of the hardest parts in writing questionnaires. You may have seen political polls getting different answers depending on how a question is crafted. Data errors can creep into a survey if we use unfamiliar, complex or technically-inaccurate words; ask more than one question at a time; use incomplete sentences; use abstract or vague concepts; make the questions too wordy; or ask questions without a clear task.

Another issue related to question wording is the risk of introducing bias by leading the respondent in a particular direction. I recently received a mail survey sponsored by the Republican Party to represent the opinion of voters in my congressional district and one of the questions was:

“Do you think the record trillion-dollar federal deficit the Democrats are creating with their out-of-control spending is going to have disastrous consequences for our nation?”

Could this question be more biased? The use of adjectives such as “record,” “out-of-control” and “disastrous” makes it really clear what the expected answer is and what the intentions of the sponsor are.

• QUESTION SEQUENCE

Questions should follow a logical flow. Order inconsistencies can confuse respondents and bias the results. For instance if you are measuring brand awareness and ask respondents to recognize brands they are familiar with before asking which brands first come to mind, you are rendering the results from the latter question worthless since respondents can’t avoid thinking of brands they just saw in the first question. This seems basic, but it happens.

• QUESTION FORMAT

Questions can be closed-ended or open-ended. Closed-ended questions provide answer choices, while open-ended questions ask respondents to answer in their own words. Each type of question serves different research objectives and has its own limitations. The key issues here are related to the level of detail and information richness we need, our previous knowledge about the topic, and whether to influence respondents’ answers. For example, for closed-ended questions we need to decide what the answer choices should be and in which order they should appear. This requires we know enough about the topic to provide answer options that capture the information accurately.

• INFORMATION ACCURACY

Questions can be closed-ended or open-ended. Closed-ended questions provide answer choices, while open-ended questions ask respondents to answer in their own words. Each type of question serves different research objectives and has its own limitations. The key issues here are related to the level of detail and information richness we need; our previous knowledge about the topic; and whether to influence respondents’ answers. For example, for closed-ended questions we need to decide what the answer choices should be and in which order they should appear. This requires we know enough about the topic to provide answer options that capture the information accurately.

Some questions yield more accurate information than others. Respondents can answer questions about their gender and age accurately, but when it comes to attitudes and opinions on a particular issue, many may not have a clear answer. Overall, attitudes and opinion questions should be worded in a way that best reflects how respondents think and talk about a particular issue so that we can tease out information that is difficult for the respondent to articulate. However, some questions need to be skipped when they don’t apply to the respondents’ experience or the issue is so irrelevant to the respondent that s/he doesn’t have a formed opinion about it. In the case in which attitude statements appear grouped in a matrix format and some may not apply to a respondents (e.g., a customer satisfaction survey after a phone call to customer support), it is necessary to include a “Not sure/Don’t know/Not applicable” option to avoid introducing measurement error in the data.

For intance, the other day I received an online customer satisfaction survey from BlackBerry after a call I made to its support desk. The survey had a question in which I was asked to rate the representative who took my call on different aspects. One of them was “Timely Updates: Regular status updates were provided regarding your service request.” I wouldn’t know how to answer this, since the issue I called for didn’t require regular updates. Luckily, they had a “Not applicable” option, otherwise I would have been forced to lie, and one side of the scale would be as good as the other.

• MEASURED BEHAVIORS

People tend to have less-precise memories of mundane behaviors they engage in on regular basis, and usually they do not categorize events by periods of times (e.g., week, month and year). We need to consider appropriate reference periods for the type of behavior we want to measure. Asking “Have you purchased any piece of clothing in the last seven days?” will yield a more accurate behavior measure than asking “Have you purchased any piece of clothing in the last six months?”

Measured behavior should be relevant to the respondent and capture his or her potential state of mind. This is valid particularly when we use rating questions and have to decide whether to include a neutral mid-point. A lot of research has been conducted in this realm, particularly by psychologists concerned with scale development, but no definitive answer has been found and the debate continues. Some studies find support for excluding it while others for including it depending on the subject, audience and type of question.

Those against a neutral point argue that by including it we give respondents an easy way to avoid taking a position on a particular issue. There is also the argument that equates including a neutral point to wasting research dollars, since this information would not be of much value or at worst it would distort the results. This camp advocates for avoiding the use of a neutral point and forcing respondents to tell us on which side of the issue they are.

However, consumers make decisions all day long and many times find themselves idling in neutral. A neutral point can reflect any of these scenarios: we feel ambivalent about the issue and could go either way; we don’t have an opinion about the issue due to lack of knowledge or experience; we never developed an opinion about the issue because we find it irrelevant; we don’t want to give our real opinion if it is not considered socially desirable; or we don’t remember a particular experience related to the issue that is being rated.

By forcing respondents to take a stand when they don’t have a formed opinion about something, we introduce measurement error in the data since we are not capturing a plausible psychological scenario in which respondents may find themselves. This is yet another reason to include a “Not sure/Don’t know/Not applicable” option in addition to a neutral point.

• QUESTION STRUCTURE

  Questions have different parts that must work in harmony to capture high-quality data. These are the question stem (e.g. what is your age?), additional instructions (e.g. select one answer) and response options, if any (e.g. Under 18, 19 to 24, 25 +). The wrong combination can leave respondents baffled about how to answer a question. Consider the example below:

  Overlapping answer options

  What is your household income? Select one answer.

  1. Under $25,000
  2. $25,000 to $50,000
  3. $50,000 to $75,000
  4. $75,000 +

  So, which answer should I choose if I my household income is $50,000? Is it option 2 or option 3?

  Conflict in meaning between different parts of the question

  Please indicate the products you use most often. Select all that apply.

  1. Cell phone
  2. Toaster
  3. Microwave oven
  4. Vacuum cleaner

• VISUAL LAYOUT

Using design elements in an inconsistent way can increase the burden put on the respondent in trying to understand the meaning of what is asked. For example, encountering different font sizes and colors across questions forces the respondent to relearn their meaning every time they are used.

Also, presenting scales with different directions (positive to negative or vice versa) in rating questions within the same survey increases measurement error as respondents often assume all rating questions have the same scale direction even when the instructions explain the meaning of the end points of the scale. For instance, if a preference question using a 1-7 scale where 1 means “the most preferred” is followed by an importance question, also using a 1-7 scale, but where 1 means “the least important,” respondents who are not paying attention to the instructions (which is quite common) are likely to assume that the 1 in the importance question means “the most important.” I have seen many examples of this problem, when respondents are asked a follow-up question conditioned on their previous answers and then they realize their mistake and tell us they actually meant to say the opposite.

• ANALYTICAL PLAN

Based on the research object, both the type of information requested and the question format are important for the type of analysis we plan to perform once the data is collected. If you want to develop a customer satisfaction model using linear regression analysis and the dependent variable is an open-ended question, you can forget about modeling anything. This seems obvious, but I have seen non-researchers writing questionnaires without thinking how they will analyze the data and then come to me asking for analyses that are not appropriate for the data collected.

There is also the question of whether we want to replicate the results, track certain events or just run a one-time ad hoc analysis. If the goal is to track certain metrics, time and care should be dedicated to crafting tracking questions, as slight changes in wording can change the meaning of a question and thus its results.

ON YOUR WAY

If you take each of these aspects of survey writing into consideration, you will be on your way to creating surveys that produce valid data and can support with confidence strategic and tactical decisions for your business.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

Posted on June 9, 2010

I recently got an inquiry from a SurveyGizmo user asking about what response rate he could expect from using this online survey tool. Fortunately for any online survey tool, including SurveyGizmo, response rates to online surveys don’t depend on the survey tool you use.

First let’s distinguish between response rates, incidence rates, completion rates and non-response. They are related, but not the same, and some clients use these concepts interchangeably, which lead to confusion in sample size and cost estimations.

Response rates  are usually calculated based on the number of respondents who attempt to participate in a survey, even if they are disqualified after they have been screened with certain questions. If we send a survey invitation to a sample size of 100 people and only 5 attempt to take the survey, then the response rate would be 5%.  Response rates have been used for years as indicators of data accuracy, however recent research has indicated that lower response rates don’t necessarily mean low quality data.

Response rates are affected by:

• Survey topic relevancy: People will not dedicate time to participate in surveys that are perceived as irrelevant.

• Incentives: Sometimes an incentive is needed to motivate respondents, but careful consideration needs to be given to this. Incentives are a tricky subject since we may attract only certain types of respondents and insert selection bias in the sample.

• Survey invitation: Survey invitations should be personalized and provide compelling reasons to participate in the survey. A poorly written invitation can drive respondents away or not catch their attention. Use appealing subject taglines and make the invitation short, clear and persuasive.

• Type of relationship with target survey audience: Depending on the level of relationship respondents have with the brand, organization or company sponsoring the project they will be more or less motivated to participate. For example, customer surveys tend to have higher response rates than those targeted at non-customers. For more on this, check Survey Response Rate Directly Proportional to Strength of Relationship by Jeffrey Henning.

• Privacy protection concerns: People are not comfortable sharing information if they don’t know how it is going to be used. Communication about privacy policy and data security should be clear.

• Reminders: These may be needed to reach busy people or those not available  within a certain time frame when the first invitation is sent out.

Incidence rates are based on the number of respondents that qualify for a study based on certain screening criteria. For example, if we need a sample of females in the general population without any other requirements, the incidence rate is expected to be 50% since half of the population are women. Incidence rates will vary depending on who we are targeting with the study.

Response rates are often used to indicate the number of completed surveys, but I think it is worth to make the distinction between response rates and completion rates since this has methodological and cost implications ( e.g.  when we need to purchase sample from online panel providers).

Completion rates indicate how many people who qualified for the study completed the survey. If they enter the survey, answer some questions and then abandon the survey, they will be counted as incompletes and are usually excluded from the final data. The number of incompletes increases when:

1. The survey is too long
2. Survey flow is confusing
3. There are skip logic errors that show irrelevant questions to respondents who can’t answer them
4. Questions are poorly worded and instructions are unclear
5. Questions are complex and requite a lot of mental effort from the respondent
6. The respondent is not rewarded accordingly based on survey length and amount of effort required
7. The topic and survey format can’t hold the respondent’s interest
8. Privacy protection is unclear or lacking

Non-response occurs when we fail to get a response from the total sample either because respondents refuse to participate in the survey or they start but never complete it. If non-responses follow a pattern that systematically excludes a particular segment of the sample, they introduce what it calls selection bias, which will prevent us from getting a representative sample of opinions in the population of interest. Nonrespondents are often different from respondents, so their absence in the final sample can make it difficult to generalize the results to the overall target population.

In short, regardless of the survey tool you use, you can improve response rates and completion rates if you avoid most of the problems mentioned above.

OTHER RELATED ARTICLES

• Does A Large Sample Size Guarantee A Representative Sample?

• What Is The Right Sample Size For A Survey?

• Testing For Significant Differences In Convenience Samples – What Is The Point?

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

Posted on May 20, 2010

I meet many clients who worry about sample size  trying to ensure they get an enough large sample so that statistically significant differences can be found and inferences to a larger population can be made, but they often don’t know that these statistical tests were meant to work within the probability sampling theory framework.

Since the advent of online panels and the increase of online surveys using panel-provided samples, the issue of testing for significant differences using standard parametric tests has become a moot point in many research studies.

Nowadays many of the surveys conducted online use samples provided by online panels, but these are mostly convenience samples (non-probability). The populations of online panels include respondents who are willing to participate in studies, excluding those unwilling to be part of the panel who may be members of the target population we are after.

In probability sampling, each possible respondent from the target population has a known probability to be chosen. Probability sampling helps us to avoid some of the selection biases that can make a sample not representative of the target population. For more on this read Does A Large Sample Size Guarantee A Representative Sample?

A single probability sample doesn’t guarantee to be representative of a target population, but we can quantify how often samples will meet some criterion of representativeness. This is the notion behind confidence intervals. The probability sampling procedure guarantees that each unit in the population of interest could appear in the sample.

By taking into account all possible random samples that can be taken from a population, we can estimate how often the true value of an estimate can be expected to be within a specific range of values. So, when we  talk about a 95% confidence interval, this really means that the true value of a particular variable is expected to fall within an interval of values 95  out of 100 times we repeat the procedure. When an opinion poll indicates that 50% of people are in favor of a political decision with a +/-3% margin of error at a 95% confidence interval, it is really saying that we can expect that between 47% and 53% of people will be in favor of the decision 95 out 100 times, if we were to repeat the poll. When we test for significant differences, we are looking to see if the value falls outside that range.

Unfortunately, taking a probability sample is hard and costly. For most consumer research studies and social behavior studies, we really don’t know the size of the actual population of consumers behaving in certain ways or consuming certain products, and trying to find out would make the research prohibitively expensive. This is why we often have to settle for convenience samples like the ones offered by online panels. They still can offer valuable insights if designed with care, but again doing statistical testing in a convenience sample is pointless since the assumptions about probability sampling are violated.

Online panels are here to stay, and they will continue to be a source for affordable sample for market research. Research using convenience sample is often better than not research at all if the survey is well designed and screening criteria are used to define the target population.

A more appropriate case for testing statistically significant differences are random samples taken from a customer database, since this is essentially the population frame where we can count all members and estimate their probability to be chosen.

 However, if you don’t have a customer database or are interested in surveying non-customers, then  use a convenience sample, if that is what your research budget can afford or there is no other way to get to the actual population frame (list to pull the sample from), but don’t fret about testing for significant differences. You may feel more confidence if you are able to replicate the results in repeated surveys, but be always cautious about inferences made from convenience samples since there could be a hidden systematic bias in the data.

It is always important that whenever you use convenience samples  you consider the following when analyzing the results:

         1. Who is systematically excluded from the sample?

         2. What groups are over- or underrepresented in the sample?

         3. Have the results been replicated with different samples and data collection methods?

If testing for significant difference gives you peace of mind, even when using convenience samples, do it to confirm the “direction” of the data, but restrain yourself from doing inferences to a larger population.

OTHER RELATED ARTICLES

• Does A Large Sample Size Guarantee A Representative Sample?

• What Is The Right Sample Size For A Survey? 

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

Posted on May 13, 2010

I often get asked “What sample size do I need to get a representative sample?” The problem is that this question is not formulated correctly. 

Sample size and representativeness are two related, but different issues. The sheer size of a sample is not a guarantee of its ability to accurately represent a target population. Large unrepresentative samples can perform as badly as small unrepresentative samples.

A survey sample’s ability to represent a population has to do with the sampling frame; that is the list from which the sample is selected. When some parts of the target population are not included in the sampled population, we are faced with selection bias, which prevent us from claiming that the sample is representative of the target population. Selection bias can occur in different ways:

• Convenience sample: This includes respondents who are easier to select or who are most likely to respond. This sample will not be representative of harder-to-select individuals. Samples from online panels are a good example of convenience samples. These panels are composed by individuals who have expressed interest in participating in surveys, leaving out individuals who may be part of the target population, but are not available for interviewing through the panel.

• Undercoverage: This happens when we fail to include all the target population in the sampling frame. Many online panels work hard at avoiding undercoverage bias, but the fact remains that certain demographics are underrepresented. For example, it is difficult to field online studies targeted at the total Hispanic population in the US without using a hybrid data collection approach that allows us to reach unacculturated Hispanics, who are usually underrepresented in most online panels. Coverage bias is also found in phone surveys that use telephone list sampling frames that exclude households without landline access. As more households substitute cell phones for their landlines, obtaining representative samples of certain demographic groups will soon be difficult without including cell phone lists in the sampling frame.

• Nonresponse: Selection bias also takes place when we fail to obtain responses from all respondents in the selected sample. Nonrespondents tend to differ from respondents, so their absence in the final sample makes it difficult to generalize the results to the overall target population. This is why the design of a survey is far more important than the absolute sample size to get a representative sample of the target population.

• Judgment sample: This is a sample selected based on “representative” criteria based on prior knowledge of the topic or target population. An example would be a study looking for a sample of teenagers, and trying to intercept them at a cross-section near a high school.

• Misspecification of target population: This happens when we use intentionally or unintentionally screening criteria that leave out important subgroups of the population.

• Poor data collection quality: This can introduce selection bias when there are poor quality controls to ensure that we interview the designated members of the sample. An example of this include allowing whoever is available in the household to take the survey instead of the intended member based on certain screening criteria.

So when it comes to getting a representative sample, sample source is more important than sample size. If you want a representative sample of a particular population, you need to ensure that:

1. The sample source includes the whole target population
2. The selected data collection method (online, phone, paper, in person) can reach individuals, with characteristics typical of those possessed by the population of interest
3. The screening criteria truly reflect the target population
4. You can minimize nonresponse bias with good survey design, incentives and the appropriate contact method
5. There are quality controls in place during the data collection process to guarantee that designated members of the sample are reached.

The initial question about sample size versus representative sample is usually followed by “What sample size do I need to get statistical significance? For an answer to this question, check my previous articles What Is The Right Sample Size For A Survey? and Testing For Significant Differences In Convenience Samples – What Is The Point?

For help on sample size calculation use our Sample Size and Margin of Error Calculators.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

Posted on May 6, 2010

Determining the sample size is one of the early steps that must be taken in the planning of a survey. Unfortunately, there is no magic formula that will tell us what the perfect sample is since there are several factors we need to think about:

• ANALYTICAL PLAN: The research objectives and planned analytical approach should be the first factor to consider when making the decision on sample size. For instance, there are statistical procedures (e.g. regression analysis) that require a certain number of observations per variable. Moreover, if comparative analysis between subgroups in the sample is expected, the sample size should be adjusted for it to be able to identify statistically significant differences between the groups.

• POPULATION VARIABILITY: This refers to the target population’s diversity. If the target population exhibits large variability in the behaviors and attitudes of interest being researched, a large sample is needed. If 20% or 80% of the population behaves in certain way, this indicates less variability than if 50% would do so. To be conservative, it is standard practice to use 50% (0.5) as the event probability in sample size calculations since it represents the highest variability that can be expected in the population.

• LEVEL OF CONFIDENCE: This is the level of risk we are willing to tolerate usually expressed as a percentage (e.g. 95% confidence level). Although survey results are reported as point estimates (e.g. 75% of respondents like this product), the fact is that since we are working with a sample of the target population, we can only be confident that the true value of the estimate in that population falls within a particular range or what is called confidence interval. The level of confidence indicates the probability that the true value of the estimate in fact will fall within the boundaries of the confidence interval. How confident can you be? As confident as your tolerance for risk allows you to, knowing that the confidence level is inversely proportional to estimate accuracy or margin of error. The more confident you want to be, the larger the confidence interval that is needed, which leads to lower levels of precision.

• MARGIN OF ERROR: Also known as sampling error, indicates the desired level of precision of the estimate. You have probably seen poll results quoted in the media, saying that the margin of error was plus or minus a particular percentage (e.g. +/-3%). This percentage defines the lower and upper bounds of the confidence interval likely to include the parameter estimate, and it is a measure of its reliability. The larger the sample, the smaller the margin of error and the greater the estimate precision.

Below is a table illustrating how the margin of error and level of confidence interact with sample size. To get the same level of precision (e.g. +/-3.2%), larger samples are needed as the confidence level increases. For example, if we want to be certain that in 95 out of 100 times the survey is repeated the estimate will be +/- 3.2%, we need a sample of 950.

OTHER RELATED ARTICLES

• Testing For Significant Differences In Convenience Samples – What Is The Point?

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

as published on April 30, 2010 by the Dallas Business Journal

Bad economic news has been a daily event for the last 24 months. We have heard about thousands of companies filing for bankruptcy and millions of people losing their jobs. Last year, most companies slashed marketing expenses and reduced market research budgets to almost nothing. It’s survival time, but I would argue many companies are going about it all wrong. Now, more than ever, market research can help companies to invest wisely the few resources they have. Ideally, this is the time to get creative and find cost-effective ways to conduct market research.

So, how can you do good market research on a small budget? Here are points to consider:

• CONTROL FOR RESEARCH SCOPE. Look long and hard at the business issues at hand and separate “nice-to-know” from “need-to-know” information. Focus on the most critical issues to reduce survey length and time needed for data processing and data analysis.

• REDUCE SAMPLE SIZE. ample size requirements should be carefully considered, taking into account the impact on estimate accuracy. As samples get larger, estimate accuracy and sample costs increase, but there is a point where the gains in estimate accuracy don’t justify the increased cost. Recently, I had a client who wanted to conduct an online survey with a sample of 1,000 respondents, which would give a statistical accuracy of +/-3.1%, but it would cost $8,000. On the other hand, a sample of 400 respondents would give a statistical accuracy of +/-4.9% and cost $3,400. As you can see, a 135% increase in sample cost would only yield a 60% gain in statistical accuracy. The client opted for the smaller sample.

• USE CUSTOMER DATABASES. Customers’ feedback often provides invaluable insights given the relationship they already have with a company. What’s more, a customer database is the cheapest source of sample for research purposes, since it is already in-house. However, be aware of its potential limitations given the profile of your customers and the type of data you have gathered about them. Know how the database was sourced. For example, if most of your customers are small businesses or females younger than age 25 and you are interested in understanding how your products will perform in other market segments, then your customer database is not the right sample source for a market segmentation study.

• USE ONLINE DATA COLLECTION TOOLS. In the past few years an explosion of online survey tools has reduced the cost of data collection significantly and shortened the research process time line. However, there are a couple of caveats:

 Market research is the field in which the principle of “garbage in, garbage out” has the most application. Programming and deploying online surveys is easier and cheaper than ever, however the online tools that facilitate this don’t provide the content of the actual survey. You still need expertise in questionnaire design, research methodology, sampling and data analysis to be able to extract useful insights from the survey data.

Online surveys are not always the most appropriate methodology for the research objectives and target sample. For example, consumers in lower income brackets are less likely to have Internet access, therefore they would be underrepresented in an online-only sample. In this case, it would be more appropriate to reach them by phone or in person.

• HIRE SMALL MARKET RESEARCH AGENCIES. There are many small market research agencies with big agency capabilities and experienced market researchers, but low overhead cost. They are often very responsive and flexible and can do the job with the same — and sometimes better — quality than bigger research firms for a fraction of what larger firms would charge.

Market research doesn’t have to be expensive to be well done. Consider these points and find a research partner that is willing to invest in your success. This will allow you to do market research on the smallest budgets.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

Posted on March 12, 2010

In my previous article about how to connect to your customers with the help of research insights, I listed some of the key research questions that any business wanting to grow and succeed should ask. But, how do you do it?

Here we need to make a distinction between data collection methods and types of research based on analytical approach, which are often confused. Data collection methods differ based on whether we want to conduct quantitative or qualitative research.

Qualitative research, which is exploratory in nature, usually uses data collection methods such as focus groups, triads, dyads,  in-depth interviews, uninterrupted observation, bulletin boards, and ethnographic participatory observation.

Quantitative research, which looks to quantify a problem, collects data through surveys in different modalities (online, phone, paper), audits, points of purchase (purchase transactions), and click-streams.

As for types of research, I mean the approaches used to analyze the data collected. Depending of the business objectives, we may decide to gather data to conduct a market segmentation, product testing, advertising testing, key driver analysis for satisfaction and loyalty, usability testing, awareness and usage research, and pricing research, among others.

 When to use each of these data collection methods and types of research depends on the business issues we are dealing with in one or more of four key areas:

• Awareness: Let the market know that the product or service exists
• Targeting: Reach the target segments with the highest profit potential
• Acquisition: Optimize the marketing message, offer,  and price that will close the sale
• Retention: Generate repeat purchases from current customers

The chart below, which we call the Relevant Wheel, shows when it is most appropriate and relevant to conduct different types of research.

Our clients find this chart to be helpful and use it as a reference to determine when a particular type of research is needed. Once this is defined, we discuss the most appropriate qualitative or quantitative data collection methods.

 So next time you wonder what type of research to conduct, I invite you to ask yourself where the particular problem at hand belongs to (Awareness, Targeting, Acquisition or Retention ) and then use this chart to to guide your decision on the most appropriate type of research. I hope you find it useful.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.