Confidence Interval
AnalyticsAlso: CI · Confidence Range
Quick definition
A confidence interval is a range of values, built from sample data, that's likely to contain the true result if you repeated the test many times. A 95 percent confidence interval means that if you ran the same test 100 times, roughly 95 of the resulting ranges would capture the true value.
This uses a standard 95% confidence level. Smaller samples and rates near 50% produce wider margins. Treat this as a directional check, not a substitute for a proper testing tool.
How it varies across Australia
Australian businesses generally run smaller sample sizes than US or UK counterparts because the addressable audience is smaller. That means wider confidence intervals by default, and tests that would reach a tight range overseas often stay noisy here for longer.
See how sample sizes vary across Australian industries →What it actually means
Imagine measuring the average height of a crowd by sampling twenty people instead of everyone. You wouldn't claim to know the crowd's exact average. You'd say something like 'probably between 170 and 176 centimetres.' That range is the confidence interval. It's built from sample data and it tells you how much uncertainty is baked into your estimate.
In marketing, confidence intervals show up constantly around conversion rate results, A/B testing outcomes and survey data. A test showing conversion rate improved from 4 percent to 4.6 percent means little on its own. The confidence interval around that 4.6 percent tells you whether the true value could plausibly still be 4.1 percent, in which case you haven't learned much.
The interval width is driven mostly by sample size and by how much the underlying data varies. Small samples produce wide, nearly useless intervals. Larger samples narrow the range and sharpen the estimate. This is why statistical significance and confidence intervals are closely linked, one narrow interval that excludes zero difference is effectively evidence of significance, while a wide interval straddling zero is evidence you need more data.
Most dashboards report a single number and skip the interval entirely. That's how teams end up confidently wrong.
A confidence interval isn't a promise. It's an admission that you don't know the exact number, stated honestly instead of hidden behind a single decimal point.
How to calculate it
Confidence Interval = Sample result plus or minus (Critical value multiplied by Standard error)
Worked example. A landing page test shows a 5% conversion rate from 400 visitors. The standard error works out to roughly 1.1%. For a 95% confidence level, the critical value is about 1.96. Margin of error = 1.96 x 1.1% = 2.2%. The confidence interval is 5% plus or minus 2.2%, or roughly 2.8% to 7.2%. That's a wide range, which tells you 400 visitors isn't enough to trust the 5% figure on its own.
The Australian context
Australia's smaller population means marketing teams here often work with sample sizes that would be considered thin in the United States or United Kingdom. A retailer running an A/B testing programme with Australian-only traffic may need to run tests for weeks longer than a global brand to reach the same confidence interval width. Factor that into test planning rather than cutting tests short because a US-authored blog post said two weeks was enough.
Where people get this wrong
Confidence Interval vs Statistical Significance
| Confidence Interval | Statistical Significance | |
|---|---|---|
| What it answers | What range might the true value sit in | Is the observed difference likely real, not chance |
| Output format | A range, e.g. 3.8% to 5.1% | A yes/no threshold, e.g. p under 0.05 |
| Depends on | Sample size and data variance | Confidence interval width and overlap between groups |
| Common misread | Treated as a probability statement about the true value | Treated as proof the effect is large or important |
Related terms
Common questions
What does a 95 percent confidence interval actually mean?
It means that if you repeated the same sampling process 100 times, roughly 95 of the calculated ranges would contain the true value. It does not mean there's a 95 percent chance the true value falls in this one specific range, a subtle but important distinction.
How do I make a confidence interval narrower?
Increase your sample size. Larger samples reduce the standard error, which directly narrows the interval. Reducing variance in the underlying data, for example by segmenting cleaner audiences, also helps but sample size is the main lever.
Why do two overlapping confidence intervals mean a test result isn't significant?
If the intervals around two groups overlap substantially, the data can't rule out that both groups actually have the same true value. Statistical significance generally requires the intervals to separate enough that chance becomes an unlikely explanation.
Do I need a confidence interval for every marketing metric?
Not every metric needs one reported daily, but any number used to justify a decision, especially A/B testing results or survey findings, should have its confidence interval checked at least once before you act on it. It's the difference between a decision and a guess dressed up as a decision.
Debrief
Get the next one
No spam. No fluff. Just the next article, straight to your inbox.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →