Variance

Analytics

Also: Statistical Variance · Data Variance

Variance = average of the squared distance between each result and the mean

What it measuresHow spread out results are

High varianceResults jump around a lot

Low varianceResults cluster tightly

Watch forSmall samples fake certainty

Quick definition

Variance is a statistical measure of how spread out a set of results is around the average. High variance means results bounce around a lot from day to day or test to test. Low variance means results are consistent. Marketers use it to judge whether a change in a metric is real or just noise.

How it varies across Australia

Variance itself doesn't have an industry benchmark, it's a property of your own data. But smaller Australian advertisers, with lower daily traffic and conversion volume than US or UK counterparts, tend to see higher variance in conversion rate and CPA day to day. That makes short test windows riskier here than the same test would be for a larger business.

See data and tracking maturity across Australian industries →

What it actually means

Think of two archers. Both average a bullseye across a hundred arrows. One archer's shots always land near the centre. The other's shots scatter across the whole target, some in the centre, some near the edge, averaging out to the same score. Same average, wildly different variance. Marketing data behaves the same way.

A conversion rate of 3% this week and 3% last week looks stable. But if the daily numbers underneath were 1%, 5%, 0.5%, 6%, 2%, that average is hiding a mess. High variance means individual data points are unreliable, even when the average looks fine. This matters most in A/B testing, where a result that looks like a winner after two days might just be one archer's lucky scatter.

Variance is the reason statistical significance exists as a concept. Significance testing is essentially asking, is the difference between our two results bigger than the variance we'd expect from noise alone? Ignore variance and every experiment becomes a coin flip dressed up as insight. Standard deviation, the square root of variance, is the more commonly reported figure because it's in the same units as the original metric, but they're measuring the same thing.

Variance is why your Tuesday result and your Wednesday result can both be true and still tell you nothing.

How to calculate it

Variance = average of the squared distance between each result and the mean

Worked example. Daily conversion rates over five days: 2%, 4%, 1%, 5%, 3%. Mean is 3%. Squared distances from the mean: 1, 1, 4, 4, 0. Average of those equals variance of 2. Compare that to a week with results of 2.8%, 3.1%, 2.9%, 3.2%, 3.0%, which has a much smaller variance despite a similar average. The second week's average is far more trustworthy.

The Australian context

Australian businesses running paid media or on-site tests often have lower daily traffic than US benchmarks assume, simply because the addressable market is smaller. Lower volume means higher natural variance in daily conversion rate and CPA. A test that would reach statistical confidence in a week for a large US retailer might need three or four weeks here to say anything reliable. Rushing test conclusions is one of the most common mistakes we see in Australian digital teams.

Where people get this wrong

Calling a two-day A/B test result conclusive.Variance in small samples is high by nature. A short test window can easily show a false winner that reverses once more data comes in.

Comparing single days across campaigns without checking spread.One good day and one bad day tell you almost nothing if the underlying variance is high. Look at the range across a full cycle, not a single snapshot.

Assuming a stable average means stable performance.A flat weekly average can hide daily swings that a stakeholder would panic over if they saw them isolated. Report the spread alongside the average, not instead of it.

Common questions

Why does variance matter for A/B testing?

A/B tests compare two averages, but averages alone hide how much the underlying data bounced around. High variance means you need more data before a difference between two versions is trustworthy. Ignoring variance leads to calling winners that are really just noise.

What's the difference between variance and standard deviation?

Standard deviation is the square root of variance. Variance is expressed in squared units, which makes it awkward to interpret directly. Standard deviation converts it back into the same units as the original metric, which is why it's the number usually shown in reports.

How do I know if my variance is too high?

There's no universal threshold, it depends on your traffic volume and what decision you're making. As a rule, if daily results swing wildly around the average, treat any single day or short test window with suspicion and wait for more data before acting.

Does more traffic reduce variance?

More traffic doesn't reduce the underlying variance in behaviour, but it does make your average more reliable and your tests reach confidence faster. Small Australian advertisers with lower volume typically need longer test windows to get the same reliability as larger businesses.

Debrief

Get the next one

No spam. No fluff. Just the next article, straight to your inbox.

Keep exploring

About New Rebellion

New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.

How we think →