Market Research 101: Who's Taking Your Online Survey?

With anonymity a major feature of online survey takers, market researchers face an endless quagmire of problems related to the authenticity of those taking their online surveys. Are they really who say they are? The problem is particularly acute with technology surveys. In some consumer segments, if an unqualified respondent snuck past the screeners, at least he or she is nevertheless a consumer of say soft drinks, snack foods etc. With technology surveys, the problem is particularly acute. An unqualified respondent attempting and managing to fake their technology knowledge can really taint the data leading to poor insight and decision making.

How can technology marketers ensure that their research efforts are completely valid, accurate and reliable?

Providers of survey and research services should employ the following quality control techniques to combat these problems:

All participants should be recruited through direct marketing at pre-validated targets: Mass recruiting techniques directed at “the universe” are not effective. You must understand and narrowly target your audience before making your participation request.

Participants should be verified utilizing direct profiling questionnaires and mailing address verification: Are they who they say they are and do they meet the requirements you seek in a survey respondent?  

Screening questionnaires must be purposefully designed so they cannot easily be “gamed” by reward motivated respondents: This takes more time and attention to detail at the early stage of the survey process but can prevent unwelcomed surprises or skewing of the results later.

Participants should be subject to “crop dusting” verification: Use identity validation tools such as cross verifying browsers, operating system and internet service provider information as well as identity verification searches on public databases and social networks.

Limit participation in multiple projects: Participants should be prevented from over participation in market research projects to avoid “professional respondents” bias, especially with B2B respondents.

Interview in depth: Probing, in-depth identity interviews of low incidence targets should be conducted to maximize quality control.

Insist on 100% incoming inspection of results: Surprisingly, the research industry standard calls for random inspections. Full manual inspections will identify “straight lining” survey completion, (filling out all the “C” answers, for example), answer racing (lack of careful question consideration) and logical consistency that cannot be trapped by automated inspection tools.

Include questions requiring careful participant consideration:  All surveys should include repetitive factual questions and thoughtfulness tests to ensure participant consistency and attention.   

By taking careful control during the survey participant recruiting and verification stage, you can be confident that the results presented at project completion will be valid. Any holes or inconsistencies that are revealed can be more easily tracked down when you are confident that the survey participants we’re not the problem. Better to know your audience early than question your audience later.