7 Step Message Effectiveness Testing Framework: Part I

An important application of market research is testing market messages for B2B and B2C products. How should messages be tested? What criteria should be used? What makes one marketing message or theme more effective than another? How should we define effectiveness?

The key problem here is that your customers and prospects are not marketing experts. So what are good practices for message testing? Why do it at all? What can and should you expect to get out of it?

The first question is should this be done qualitatively or quantitatively, i.e. should you get in depth insights from a small group of people or should you collect data from a quantitative sample giving you a statistical degree of projectability and accuracy?

So many companies are "quantitatively driven", that is, they are unwilling to make decisions on data unless it is based on large sample sizes. The only problem with that is, as I mentioned earlier, customers are not marketers. And you can only ask broad questions given the limitations of quantitative surveys. What is the value of messaging input from a quantitative sample if you are not able to ask the in-depth questions that reveal not only the message preference but the desires and motivations evoked by your messages and how these move your customer or prospect to action?

In our 17 years of message testing and working with clients to formulate their branding and message architectures, we have found that messaging insights uncovered through in-depth qualitative research were much more actionable and useful to our clients than those derived quantitatively. Agency personnel in charge of creating copy and visuals also had better direction to execute campaigns and reported feeling more empowered with the data. Moreover, some very important and highly public advertising disasters were based on qualitative data that revealed preference between message A or B, but failed to tease out the connotations and emotions that the messages evokes. I will not provide examples here to protect the innocent, but suffice it to say that I am a firm believer in in-depth qualitative techniques for message testing. Good marketers know their markets and are close enough to their customers that they do not need a democratic "quorum" to tell them which way they should head. Instead, they need customers and prospects to become co-creators to help create and refine messages and that is what the qualitative methodologies we propose will accomplish for them.

What then are some of the ground rules for effective message testing that our experience had shown to reveal the most insightful and action oriented data?

1. Set up your research design to be as unstructured as possible-instead of counting on the most important learning to come from questions in your discussion guide that ask: Do you prefer message A, B, or C, count on the most important learning to come from an unstructured stream of consciousness dialogue. To this end, pick a moderator who is comfortable with this style of interviewing and knowledgeable enough about your product to deviate from pre-scripted questions.

2. Decide on which set of customers and prospects you want this feedback. Too often we see qualitative research experiments set up to capture every segment possible. Remember this is qualitative-you do not need to represent every single market segment you cover. Your research partner or vendor can help you determine this.

3. Approach the research project as if you the marketer are integrally part of the discovery process. Develop this expectation with the moderator upfront. Also set the expectation with the moderator/research vendor that the final analysis and report should be delivered in workshop format to the message design team at your end. Seek and engage your external agencies upfront in the process.

What are the actual questions you should ask, what dimensions should message effectiveness be tested on? What are the seven steps referred to in the title of this post? I will cover this off in part two next week.

Alan Nazarelli is President and CEO of Silicon Valley Research Group, a global market research and strategy development firm focused on the needs of technology companies.