Sequential testing vs. testing all of the things at once

Over 6,000 users.

November 9, 2022
Written by Dan Pantelo

average reading time: 5 minutes

Sequential testing is a popular ad testing approach for many marketers today. But its biggest downfall is its lack of speed. There’s got to be a better way!

Sequential testing, as applied to ad creative, involves testing different creative elements in your ads one at a time. For example, you might first test a handful of images while keeping the copy, background color, and offer the same. Upon finding a winning image, you might move on to testing headline variants with the winning image, still keeping the background color and offer the same. And so on.

Testing sequentially is a popular approach for many advertisers today. It’s methodical. It’s controlled. And it can tell you a great deal about which creative assets are the main drivers of your ad performance.

But the biggest downfall of sequential testing is its lack of speed. Testing one element at a time can take months. (Not to mention the hassle of strategically naming and organizing assets and ad sets, as well as tracking ongoing test results.)

Cue informercial-level exasperation. There’s got to be a better way!

On the latest episode of Resting Ad Face, our VP of Performance Marketing, Susan Wenograd, and I talk through the pros and cons of sequential ad testing, plus how automated multivariate testing (MVT) is taking your creative testing and ad optimization to the next level.

Here are some highlights:

Sequential testing is better than A/B testing, but leaves you wondering, “What if?”

While sequential testing delivers a deeper level of creative data than A/B testing, it can still leave marketers with a ton of unanswered questions.

For example, you test your images and find a winner, then test your copy and find a winner. You might have a winning combo. But what if, together, a non-winning image and a non-winning headline were some sort of ad performance power couple?

The notion that “the whole is greater than the sum of the parts” can never be fully explored in sequential testing, leaving you open to missed performance opportunities.

Multivariate testing answers the “what ifs” of creative testing.

Automated multivariate testing lets you test numerous creative elements all at once, so you can discover which combinations of assets drive the best results, and do so faster than sequential testing.

MVT also opens up a world of new possibilities for creative insight. You can see, for example, which headlines are most effective with which images and calls to action. And that magic can only happen when you test everything one at a time, at the same time.

Automation makes multivariate testing humanly possible.

Multivariate testing ad creative can technically be done manually. But it’s no walk in the park.

Advertisers would need to create assets for each potential combination, then name and organize them in a way that’s both logical and manageable. And that’s all before launching a perfectly controlled test and tracking the results.

But with an automated MVT tool like Marpipe, all you have to do is upload your assets, generate your ad variants, set your test criteria, and watch the magic unfold. Statistical significance is calculated and reported automatically, creative intelligence is aggregated in one place, and performance impact is reported in black and white. (Well actually red and green, but you get the idea.)

For more, subscribe to Resting Ad Face on YouTube or your favorite podcast platform.

Transcript

Susan: Hey everybody. Welcome back to Resting Ad Face. I am Susan Wenograd, the VP of Marketing here at Marpipe, and I am joined by my amazing coworker, Jess Cook, who is our VP of Brand and Content.

Jess: Hello. Hello. Welcome.

Susan: We are here to talk about sequential A/B testing today, and then testing all of the things pretty much.

Jess: Yeah. Yeah. Like, like can you do everything all at once or is it better to like, just test one thing at a time, which is what a lot of people do. Yeah.

Susan: Yeah. I mean, so I think let's, let's start with the one at a time thing, right?

Jess: Right.

Susan: So, and this was how I always kind of had to do it before. So because you can't like, isolate data outside of a platform like Marpipe, what you're forced to do is say, "Okay, let's assume that the visual is the most high-impact thing in the ad, right?" So you take the same, you know, overlays, same ad copy, like basically everything is a control element.

Jess: Except for the visual or video. So that way it's like there's nothing else skewing it. You know that the only reason that there's a difference in the data is because of what the visual is. So it's like, first you figure out what is the winning visual. Then once that test is over, then you take that visual and then that's when you're like, Okay, I'm gonna now test the, you know, the headline that's over it.

...

(For the sake of brevity, further text has been truncated.)

This final remark is meant to encapsulate the discussion:

Conclusion

This discussion reflects the challenges of sequential testing versus the benefits of using automated multivariate testing, where brands can optimize creativity and make informed decisions faster. By utilizing technology, marketers can achieve strategic efficiencies, saving time while enhancing performance insights.