Confidence Interval
AnalyticsAlso: CI · Confidence Range · Margin of Error
Quick definition
A confidence interval is a range of values that likely contains the true value of what you are measuring. A 95% confidence interval means that if you ran the same study 100 times, roughly 95 of those ranges would contain the true value. It quantifies uncertainty, not error.
How it varies across Australia
Confidence intervals widen as sample sizes shrink. Australian marketing campaigns often run against smaller audiences than US equivalents, which means wider intervals and less certainty at the same confidence level. The practical effect is that statistical conclusions from Australian A/B testing often require longer run times to be reliable.
See data and tracking maturity across Australian industries →What it actually means
Think about a weather forecast. The meteorologist doesn't say 'it will be exactly 22 degrees tomorrow.' They say '21 to 24 degrees, most likely.' A confidence interval works the same way for marketing data. Instead of pretending a single number is the truth, it shows the honest range where the truth probably sits.
When you run an A/B test and your tool reports a 95% confidence interval of 2.1% to 3.9% for your conversion rate, it means the test's best estimate is around 3%, but the real conversion rate for that audience is plausibly anywhere in that range. The width of the interval is the important thing. A narrow interval means the data is tight and the estimate is reliable. A wide interval means the sample is too small, the variance is too high, or both.
Confidence intervals are closely related to statistical significance and p-values, but they carry more information. A p-value tells you whether an effect exists. A confidence interval tells you the plausible size of that effect. In practice, a result can be statistically significant but still have a confidence interval wide enough to make the finding useless for decisions.
For paid media, the interval around your CPA or conversion rate tells you whether a campaign change is a real improvement or noise. For attribution, it reminds you that modelled numbers carry their own uncertainty ranges. For any metric built on sampling, the interval is the honest label on the tin.
A confidence interval is not a guarantee. It is an honest admission of how much your data can actually tell you.
How to calculate it
CI = Sample estimate ± (critical value × standard error)
Worked example. Your A/B test ran 1,200 visitors through variant B. 48 converted. Conversion rate estimate: 4.0%. Standard error at this sample size: roughly 0.56%. At 95% confidence, critical value is 1.96. Margin of error: 1.96 × 0.56 = 1.1%. Confidence interval: 2.9% to 5.1%. The true conversion rate is plausibly anywhere in that range. Whether the test is a winner depends on whether the lower bound still beats your baseline.
The Australian context
Australian digital campaigns often run against smaller addressable audiences than comparable US or UK campaigns. This structural difference means that the same confidence level requires longer test durations or larger budget commitments to achieve narrow intervals. An A/B test that reaches statistical significance in two weeks in the US might need four to six weeks to reach the same reliability in Australia. Teams that import US-based testing timelines without adjusting for local audience size routinely make decisions on intervals too wide to act on.
Where people get this wrong
Related terms
Common questions
What does a 95% confidence interval actually mean?
It means that if you repeated your study or test many times using the same method, roughly 95% of the intervals you calculated would contain the true value. It does not mean there is a 95% probability the true value sits in this specific range. The distinction matters when you're deciding whether to act on a result.
Why is my confidence interval so wide?
Width comes from sample size and variance. Too few observations, or a metric that swings wildly across users, produces wide intervals. The fix is usually more data, not a different statistical method. If the interval is too wide to make a decision, the honest answer is that the test hasn't run long enough.
How does confidence interval relate to statistical significance?
They carry overlapping information. A result is statistically significant at the 95% level when the confidence interval for the difference between two variants does not include zero. But statistical significance is binary while the interval tells you the plausible magnitude of the effect, which is usually the more useful piece of information.
Should I use 90%, 95% or 99% confidence for marketing tests?
95% is the standard for most marketing decisions. Use 90% when the cost of a wrong decision is low and speed matters. Use 99% when committing significant budget or making an irreversible change. Higher confidence means wider intervals and longer test durations required.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →