Conjoint Analysis myths are spreading fast as the use of conjoint analysis methods has increased over the years in different industries.
During the 2013 Sawtooth Software Conference, Chris Chapman, from Google discussed how this family of research techniques has been successfully used to:
- Determine feature preference
- Predict market share
- Find unmet needs in product portfolios
- Determine likely response from competitors
At the same time, conjoint analysis results don’t always match reality. Clients have expectations that stem from a misunderstanding of what to expect from this research approach.
In an effort to adjust expectations, Chapman discussed the conjoint analysis myths he has encountered.
Many users of conjoint expect to know how many people would buy a product. However, what conjoint analysis tells us is how many people prefer a product compared with other tested alternatives.
To get market share predictions we need a market-based model. This model should include not only preferences but also information about awareness, distribution, marketing, channel effects, etc. Conjoint analysis provides an important piece of the puzzle, but more information is needed.
Good Features / Bad Features
The goal of many conjoint studies is to identify features that drive purchase. Clients sometimes interpret the results as indicators of “good” features in absolute terms.
However, conjoint analysis provides insights into the trade-offs people make among most and least preferred features. The tested features set the context for preferences. There are no absolutes. Results can change drastically if we include or exclude features.
Clients often expect conjoint to tell them what the prices should be. However, conjoint is better suited to learn about price sensitivity, not to set exact price points. Results are very much dependent on the price ranges tested.
New Product Development
We often use conjoint analysis to determine what product we should develop. However, the highest average preference for a particular product configuration may hide heterogeneity in preferences. Consequently, we may end up with a product that doesn’t appeal to anyone in particular.
Preferences alone are not enough to make this type of decision. We need to consider the cost and competition. A competitor may already offer the best product configuration. Consequently, we may find better opportunities in niche areas.
In Chapman’s opinion, conjoint analysis is best to inform product lineup. We require more data and expertise for more precise optimization.
Clients sometimes perceive respondents as belonging to a particular group or type (e.g. brand preferrers) with implicit true states. However, conjoint analysis yields aggregated probabilities. Even when we estimate individual-level utilities, there are no “true types.” We should look at these groups as people with “tendencies” (e.g. they are likely to prefer brand X).
Many clients prefer large samples in trying to get greater statistical power. However, Chapman advocates for using smaller samples (ideally 400). If we use smaller samples, we may afford to replicate the study with different conjoint methods (traditional, adaptive, menu-based, etc.). The goal is to compare and validate results. This will maximize interpretative power by dividing statistical power.
Better than Instincts
Yes, the key is to learn from the conjoint data, even when we make incorrect decisions. In cases in which the data sounds counter-intuitive, we shouldn’t simply dismiss it. Often, going with the data tends to result in losses that are modest in comparison to when wrong decisions are made based solely on instincts.
These issues should not prevent you from doing conjoint analysis. However, having them in mind helps with expectation management, and leads to better use of the insights conjoint analysis provides.