A marketer should be testing constantly and should test whatever can be tested, starting with those areas that have the greatest leverage. This may include pricing, positioning, branding, headlines, promotions, channels, content, tone, and even whole environments (e.g. websites, direct mail, catalogs).
Testing OFAT (one factor at a time, also known as A/B testing)is how most testing has been done (or not done). It can be slow, often months or years, and expensive.
The superior option, only recently available to marketers, is multivariate testing (MVT), also known as multivariable testing or factorial design testing. The math for multivariate analysis has been around for over 100 years. But it is complex, intensive, laborious math when done by hand. Only recently has computational power allowed MVT to become mainstream.
The benefits of MVT over A/B testing include:
-smaller sample sizes (=lower costs)
-ability to test multiple variables simultaneously in one experiment (= much faster results and lower costs)
-ability to test combinations of factors ("recipes") rather than just single factors (e.g. a price and a headline)
Here's some data from an MVT supplier for marketers, Conversion Multiplier, that brings the MVT difference to life:
A multitude of selling factors can influence whether your prospects convert to customers. These include headlines, subheads, benefits copy, graphic layouts, prices, discounts, guarantees, privacy assurances, bonus offers, background colors, images, navigation structures, product presentations, calls-to-action, shopping carts, and more.
Just two variations for each of these 15 factors produces 32,768 combinations to test. Three variables produce 14.3 million combinations, and 4 variations -- more than a billion. Because A/B split testing is a sequential process, you're looking at years, if not lifetimes, to identify the winning combinations.
Just two variations for each of these 15 factors produces 32,768 combinations to test. Three variables produce 14.3 million combinations, and 4 variations -- more than a billion. Because A/B split testing is a sequential process, you're looking at years, if not lifetimes, to identify the winning combinations.
MVT is now mainstream in website optimization and a host of tools are available for that niche. In fact, MVT is an integral part of Google's Website Optimizer and they've done a masterful job of keeping the math behind the scenes. Here's a screen shot from Website Optimizer showing how easy it is to interpret results-in this case three combinations of variables were tested on a website vs. the original:
But MVT has little visibility in the remaining universe of marketing in part because there are few tools to enable it with the ease of Google Website Optimizer.
One affordable mainstream tool worth a look is the direct marketing SaaS, Longbow. The Longbow website includes a primer on MVT and also a very useful sample size calculator.
Longbow would be a good tool for performing experiments on direct mail and email campaigns.
Or you may want to hire a consultant or build staff expertise to design and execute your experiment strategy.
Other than those focused on website optimization and a few direct marketers, I have not met a marketer who tests with the breadth, frequency and discipline that is possible. There is no excuse not to test and in fact there are profound benefits-the benefit of the testing itself is measurable and it will also confirm or deny the effectiveness of your marketing components in quantitative terms. Testing should be an ongoing, iterative and strategic part of any marketing plan, ingrained in the marketing culture and a nurtured corporate skill. And with today's computational tools, MVT in particular can give you a quick competitive advantage and financial gain.
