I Was Thinking …

I learned a lot at JupiterResearch, were I was president until we sold to Forrester Research last year. I think I’m going to share some of what I learned on this blog.

One thing I learned is how to design a good survey.  Surveys can be expensive. Some companies spend millions of dollars a year on survey research. With budgets tightening everywhere, it’ s worth learning how to make the most of your survey budget.

It’s not hard to find good basic information on the best practices of survey design and even the design of specific questions. What I found at Jupiter, though, is knowing how to ask a question is not the same as knowing what and whom to ask.

Business Purpose Determines the Relevant Survey Population

Survey data ultimately serves the needs of business people in a few basic ways. Surveys can help identify market opportunities,  define products and services or shape marketing programs. For these purposes, surveys of customers work best. Surveys can reveal customers’ likes and dislikes, their influences, attitudes, behaviors and level of awareness and adoption of products, services or ideas–the raw material of marketing.

Surveys can also help with general business planning. For business decisions on everything from setting revenue targets to defining IT budgets, companies find it help to know what similar companies are doing.  For these purposes, surveys of executives at those companies work best. How much are companies my size spending on e-business infrastructure and applications? How do companies in my industry allocate their online advertising budget between search and display advertising? These questions can be answered by means of an executive survey.

Three Questioning Tactics

Broadly speaking, a survey question can be used in one of three ways. It’s important to be clear which of these ways each question on your survey will be used:

1. hypothesis testing
2. trending and tracking
3. segmentation

Surveys Are Excellent for Testing Hypotheses

Good research is often mobilized by a hypothesis. A good hypothesis is something that is testable and whose proof or disproof has important implications. Suppose you are trying to determine what factors influence the form of payment consumers choose to use when shopping online. (This information would be useful to online retailers, credit card associations and issuers.)  To ask a revealing question on this topic, you need to

start with a good hypothesis. Some hypotheses might be: consumers believe debit cards pose a higher risk of fraud than credit cards; consumers tend to use the same card off line and online; consumers are more likely to use a credit card whose information they’ve already entered and stored at a favorite retailer than some other credit card.

All of those hypotheses can easily be tested with an appropriately worded question. And the answers will have important implications for retailers, card associations and issuers.

Trending and Tracking

Trending and tracking means gathering intrinsically useful information, potentially for the purpose of seeing how that information changes over time. You might survey consumers to measure how widespread concerns about global warming are. If you ask the same question of a similar audience repeatedly over time, you can tell how the sentiment is trending. You can measure the penetration of a consumer product, and how it has trended over time, to deduce how big an opportunity that product presents in the future. Sometimes such data is used as an input into revenue models and forecasts.  Consider how the data will be used, though. Asking consumers how many books they buy in a year could provide data that is useful for building a revenue forecast for a book retailer.  But do consumers prefer to buy books online, at big chain stores, or in specialty stores? That information might be interesting, but probably less useful.

Segmentation Can Make Data More Actionable

Segmentation is used to divide respondents into segments to make it easier to draw actionable implications from the data. Finding that 10 percent of consumers would be interested in pig-leather apparel is one thing. Finding that 30 percent of urban consumers ages 25 to 34 have that interest, is information an apparel marketer can act on more readily. Knowing that online retailers spend about 20 percent of their online revenue on Web site operations is one thing. But knowing that the largest retailers spend just 11 percent of revenues on site operations, while smaller retailers spend 39 percent of revenues on operations, provides a more meaningful basis for benchmarking.

To segment data, you need to have a large enough sample size that you can meaningfully break up the data into groups that are still large enough to be statistically significant. And you need to include segmentation questions. In the pig-leather example, the segmentation questions are age and whether the respondent lives in an urban or suburban area. The online retailer data is segmented by revenues. Designing effective segmentation questions sometimes depends on having good hypothesis about what dimensions will significantly distinguish one segment from another. It may be, for an example, that income level correlates less with interest in pig leather than an urban domicile.

Common Mistakes

It’s not uncommon to get survey data back and find that it’s interesting but not useful. Generally, that’s because the design of the survey did not follow these rules:

1. Devise clear, testable hypotheses with important implications.
2. Survey the right audience: customers for marketing; peers for benchmarking
3. When the sample size permits, include useful segmentation questions.

Some people are tempted to use a survey instrument as a way to learn about a market by gathering a lot of rudimentary data about that market. This is generally a mistake: most of that data will never get used. There are more cost effective ways of getting up to speed on a market than commissioning a survey about it.

Your Thoughts?

Do you have other best practices to share? Would you like to disagree with my on any of the above? I welcome your comments.