Multivariate Testing
Conversion & UXAlso: MVT · Multivariate Optimisation
Quick definition
Multivariate testing (MVT) is a method of testing multiple page elements at the same time to find which combination performs best. Unlike an A/B test that compares two whole-page versions, MVT isolates the effect of each element, such as a headline, image, and button, across all possible combinations simultaneously.
Each combination needs sufficient traffic to reach statistical significance. Add more elements and the required traffic scales fast. If the total combinations exceed what your page traffic can support in a reasonable timeframe, run a focused A/B test instead.
How it varies across Australia
MVT requires substantially more traffic than A/B testing to reach statistical significance across all combinations. Most Australian mid-market sites lack the volume to run MVT reliably on anything but their highest-traffic pages. The conversion gain from a well-run A/B test on most pages outweighs the theoretical upside of MVT.
See conversion benchmarks across Australian industries →MVT vs A/B testing
Compares two or more complete page variants against each other. Fast to run, low traffic requirement, answers one question at a time.
Best for: most pages, most budgetsTests multiple elements simultaneously across all their combinations. Slower, traffic-hungry, reveals interaction effects between elements.
Best for: high-traffic pages, element interaction questionsEvery possible combination is tested. Most statistically robust. Requires the most traffic.
Combinations = product of all variant countsA statistical subset of combinations is tested. Lower traffic requirement. Less complete picture of interaction effects.
Common in tools like Google Optimize (now sunset)What it actually means
Imagine you want to test three things on a landing page: the headline (two versions), the hero image (two versions), and the call-to-action button colour (two versions). That's two times two times two, which gives eight combinations. MVT runs all eight simultaneously, with real visitors split across all of them, so you can see not just which headline works but which headline works best when paired with which image and which button colour.
That's the promise. The catch is that eight combinations each need enough traffic to reach statistical significance before you can trust the result. If your page gets five hundred visitors a month, you'll be waiting years. If it gets fifty thousand, you might be done in a fortnight.
The other thing MVT reveals that A/B testing can't is interaction effects. Two elements that individually test neutral can produce a significant lift when combined, or two individually strong elements can cancel each other out. MVT is the only method that finds this.
Most conversion teams reach for MVT when they should still be running A/B tests. The traffic maths almost always resolves that question. Run the combinations calculation first.
MVT tells you which combination wins. A/B testing tells you whether your hypothesis was right. Most pages need the second answer more urgently.
How to calculate it
Combinations = variants of element 1 × variants of element 2 × variants of element 3 (continue for each additional element)
Worked example. You want to test a headline with 3 versions, a hero image with 2 versions, and a CTA button with 2 versions. Combinations = 3 × 2 × 2 = 12. Each combination needs enough traffic to reach significance. If you need 500 visitors per combination for 80% power, you need at least 6,000 visitors before you can read the results reliably.
The Australian context
Australian ecommerce and lead-gen sites rarely have the traffic volumes that make full factorial MVT practical outside the top handful of pages. The sites where MVT makes sense tend to be high-volume retail and financial services landing pages, where even a small percentage lift is worth the testing investment. For most mid-market Australian businesses, the better use of conversion budget is disciplined A/B testing on the pages that matter most, before graduating to MVT.
Where people get this wrong
Related terms
Common questions
When should I use MVT instead of an A/B test?
Use MVT when you have a high-traffic page, you want to test multiple elements simultaneously, and you specifically want to understand how those elements interact with each other. If you're testing one clear hypothesis with limited traffic, an A/B test will give you a reliable answer faster.
How much traffic do I need for multivariate testing?
It depends on your number of combinations, your baseline conversion rate, and the lift you want to detect. A rough guide: multiply the number of combinations by the sample size you'd need for a single A/B test. More combinations means substantially more traffic before results are trustworthy.
What tools support multivariate testing in Australia?
VWO, Optimizely, and AB Tasty all support MVT. Google Optimize is no longer available. Most Australian teams run MVT through the same platform they use for A/B testing. The tool matters less than whether your traffic volume can support the test.
Can I run multivariate testing on a mobile app?
Yes. Most major experimentation platforms support in-app MVT. The same traffic constraints apply. Mobile apps with large daily active user bases are often better suited to MVT than equivalent web pages because the session volume is higher and more predictable.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →