Know Thy Skew

I was talking to a friend who had commissioned some web-based market research recently. He is an eminent, big brained technologist whose team is building a service for affluent consumers who own multiple computers and connected devices.

He was kind of amazed at the techniques used to pull together a sample, and how skews in the population were adjusted to compose a group of people that mirrored a hypothetical target audience. If the researchers know the incidence of that audience within the general population, they can perhaps estimate a market size as well.

This got me thinking about how literally every market research carries bias, some of it intentional, as in this case, but more often hidden. I remember one market researcher I worked with sitting me down early in a project and asking “Which sacred cows are you goring this time?” a sly way to get at my agenda and identify the biases I was carrying into the project. There were several.

In my friend’s project, a deliberately skewed sample was constructed to profile a population’s interests and behaviors. It sounded like the first step was to glean a homogeneous population out of a (non-random) group of web surfers, do some profiling on that population and then recruit focus groups out of it to validate their interests in prospective offerings. The precision of the study depends on the existence of a population that one can identify (consumers making >$150,000/year who own 3 or more connected devices) and reach (Adwords? Walt Mossberg?) with a new product or service.

This kind of market research project exists to validate somebody’s ideas or existing plans, i.e. buying courage and resolve. Or in organizations where purse strings need to be loosened, buying putative evidence. (”The research came back and they loved it!”)

A good demo presented with well, by an enthusiastic junior product manager can almost force people to say they love your idea (i.e. can introduce a powerful skew). And how seductive is the focus group ritual! The voyeurism of the one way mirror, the ruthless, puppetmaster culling of possible competitor plants and trouble makers from the participant’s list, the catering, the darkened room and the swanky chairs!

As Seth Godin says in Free Prize Inside “The reason for focus groups, market research, and the like is the continuing mirage that somehow, if we do enough work, we can figure out in advance whether we have the right idea or not.”

Fortune telling seems to always be conducted in a darkened room.

I think market research can be really risky, particularly when desired results are already known. The energy and dollars put into the project almost always exaggerate the perception of the research instrument’s accuracy and precision.

There are a lot of reasons to hire third parties to help manage research projects, but for me the most important one is to identify and document the effects of research biases. Unfortunately epistemological discussions tend to distract from organizational resolve, which is, in fact, what we usually pay research contractors for.

So: know thy skew. Which sacred cows you are turning into hamburger, this time?

Bookmark To