Break Your Bad Ad Testing Habits

"Mistakes were made," is not something you want to hear when it comes to testing paid social.

But, as with trying anything new, they’re bound to happen when moving away from A/B testing and adjusting to better, more granular methods, like multivariate.

But fear not, gentle ad nerds. On the latest episode of Resting Ad Face, Susan and I talk through the four most common faux pas we see new multivariate testers make, and share advice on how to avoid them for more meaningful creative data.

Here are some highlights:

Old A/B testing habits die hard

Marketers are familiar with A/B tests. They’re simple and they’ve been around, in various forms, for ages. So it’s very easy to carry over the practices made popular by that method of testing to multivariate testing, but — sad trombone — they can negatively affect the power and validity of your data.

Here are the two most common habits carried over from A/B testing:

1. Starting your test without a hypothesis. With A/B testing, you don’t need really a hypothesis. You pit a handful of fully realized concepts against each other and an ad either wins, or it doesn’t.

But the foundation of multivariate testing is the individual assets within your ads. To know which assets to include in your test, you have to know what you want to learn.

Let’s say, for example, you want to learn if images of women or men perform better. You now know you’ll need images of men and women to see that hypothesis through.

Here are a few more examples of solid hypotheses for multivariate testing:

• I want to see which generates more conversions: images of my product being used by a model or images of the product by itself.
• I want to see which discount drives CTR the most: BOGO, dollars off, or percentage off.
• I want to see which background colors generate more leads.

Upon reading these, it’s clear which creative elements would be required to run each test — and learn what your customers gravitate toward in your ad creative.

2. Overengineering your test based on assumptions. As creatives and marketers, we often think we inherently "know" what our audience wants to see in our ads. We arbitrarily choose which color combinations to use and which headlines must be paired with which images in our ads, all based on our own bias. (Research shows we’re actually really bad at predicting winning ad creative, by the way.)

In multivariate testing, this typically shows up in the form of conflating variables. For example, let’s say someone on the team decides that only images of plants should be used when mentioning the “all-natural” value prop. If those two creative elements are always only paired together, how will you know whether it was the image or the value prop that prompted the conversion? Answer: you won’t.

Designing modularly is the key to breaking this habit. By separating every creative element within your ad, you can understand which elements — headlines, colors, images, etc. — are the reason people click or purchase.

Continuous testing leads to stacked, incremental gains

Creative testing has traditionally been used as a nice-to-have, sporadically implemented tool. We have a sale or a new product launch coming up, so we test the ad creative once and move on with our lives.

But one of the most important upsides to multivariate testing is the ability to stack performance improvements over time through consistent testing. With each test, you can improve your ad’s ability to convert, little by little, (or sometimes, a lot by a lot).

If your test results show clear positive outliers, you can keep probing subsets of those elements to find the most performant variant. If your test results in no clear winners or losers, you can try something totally different and see if this new direction can help you identify any winners to probe further.