Minimum Detectable Effect (MDE)

Analytics

Also: MDE · Minimum Detectable Lift

MDE = smallest lift you can reliably detect given your sample size, baseline rate and desired confidence

Set before testDetermines required sample size

Smaller MDENeeds more traffic to detect

Watch forUnderpowered tests read as 'no effect'

Pairs withStatistical significance, not a substitute

Quick definition

Minimum Detectable Effect (MDE) is the smallest change in a metric, such as conversion rate, that an experiment is sized to reliably detect. Set before an A/B test runs, it tells you whether your traffic and sample size are enough to catch the lift you're actually expecting, or too small to catch anything but a huge swing.

Run the numbers

Baseline conversion rate

Sample size per variant

Minimum detectable effect (relative lift)54.58%

This is a simplified estimate assuming a two sided test at typical confidence and power settings. Purpose-built test design tools give a more exact figure, but the relationship holds. Smaller samples only reveal larger effects.

How it varies across Australia

Australian testing programs often run on smaller traffic volumes than US or UK equivalents, so the MDE a business can realistically detect within a normal timeframe tends to sit higher. Smaller sample pools mean only larger, more obvious lifts show up as statistically significant within weeks rather than months.

See how Australian businesses size and run experiments →

What it actually means

Every A/B test has a floor. Below a certain size of effect, the test simply cannot see it, no matter how long you let it run. That floor is the minimum detectable effect, or MDE. It's the smallest change in a metric like conversion rate that your test is actually sized to catch.

Think of it like listening for a whisper in a noisy room. A louder room needs a louder whisper, a bigger effect, before you can pick it out from the background noise. A quieter room lets you hear something much smaller. Sample size and MDE trade off directly. Set one and you've effectively set the other.

Most teams get this backwards. They launch an A/B test, wait a fixed two weeks because that's the sprint length, and interpret whatever the dashboard says at the end. If the test wasn't sized to detect the lift they actually got, statistical significance never arrives and the result gets read as 'no difference' when it might just mean 'no power'. MDE should be calculated before the test starts, alongside your target sample size and the p-value threshold you'll judge the result against.

An underpowered test doesn't prove there's no effect. It proves you never gave the effect a chance to show up.

How to calculate it

MDE is derived from your sample size per variant, baseline conversion rate and desired confidence and power levels, rather than calculated as a simple ratio

Worked example. A page converts at 5 percent and gets 1,000 visitors per variant per week. Running for four weeks gives roughly 4,000 visitors per variant. At standard confidence and power settings, that sample size can only reliably detect a relative lift of around 20 percent or more. A hoped-for 5 percent lift would need several times that traffic to become detectable.

The Australian context

Australian ecommerce and SaaS businesses typically run experiments on smaller traffic volumes than US or UK equivalents. That means the realistic MDE for most Australian tests sits higher than the aggressive small lifts often quoted in overseas case studies. Chasing a tiny MDE on Australian-scale traffic means either running tests for months or testing far fewer things at once. Most Australian testing programs are better served picking bigger, bolder changes that produce a detectable lift within weeks, and saving fine-grained optimisation for pages with genuinely high traffic, like checkout or the primary landing page feeding a paid acquisition campaign.

Where people get this wrong

Setting a fixed test duration instead of calculating MDE first.Two weeks might be enough sample for a homepage but nowhere near enough for a low-traffic pricing page, so the same duration produces wildly different statistical power across tests.

Reading a non-significant result as proof the change made no difference.An underpowered test can only tell you it didn't detect the effect, not that the effect doesn't exist. Those are very different claims to make to a stakeholder.

Chasing an MDE that's smaller than the traffic can realistically support.A tiny MDE on a page that gets a few hundred visits a week needs a test that runs for the better part of a year. Pick an MDE your traffic can actually deliver within a sensible window.

Common questions

What's a realistic MDE for a small business test?

It depends entirely on your traffic. A page with a few thousand weekly visitors might only realistically detect lifts of a fair size within a month. Lower-traffic pages need to either accept a larger MDE or extend the test well beyond a typical sprint cycle.

How does MDE relate to sample size?

They're two sides of the same trade-off. A smaller MDE needs a larger sample size to detect reliably. Fix your available traffic and you've effectively fixed the smallest lift you can hope to measure within that window.

Can I lower my MDE mid-test?

Not honestly. MDE is set at the design stage based on your planned sample size. Changing it mid-test to match whatever result you're seeing is a form of peeking and undermines the statistical validity of the whole experiment.

What's the difference between MDE and statistical significance?

MDE is decided before the test runs and defines the smallest effect you're sizing the test to catch. Statistical significance is calculated after the test runs and tells you how confident you can be that the observed result is real rather than noise.

Debrief

Get the next one

No spam. No fluff. Just the next article, straight to your inbox.

Keep exploring

About New Rebellion

New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.

How we think →