Statistical Significance
AnalyticsAlso: Stat Sig · Confidence Level
Quick definition
A result is statistically significant when the probability it happened by chance is low enough to trust — typically below 5%. Without it, your A/B test winner might be noise.
Where it shows up in the data
What it actually means
Statistical significance tells you whether the difference you observed in a test is likely to be real or could have happened by chance. If you flip a coin four times and get three heads, that doesn't mean the coin is biased — it's a small sample. The same principle applies to marketing tests. Running an A/B test for two days on a low-traffic page and calling variant B the winner because it has a 3% higher conversion rate is almost certainly noise. Statistical significance, usually measured at 95% confidence with a p-value below 0.05, is the threshold that separates a real result from a random fluctuation.
Calling a test early is how marketers manufacture confidence they haven't earned.
The Australian context
Australian e-commerce businesses with traffic under 10,000 monthly sessions often struggle to reach statistical significance on A/B tests in a reasonable timeframe. The practical answer is to test bigger changes — 50% layout differences rather than button colour tweaks — that produce large enough effects to detect with limited sample sizes.
Related terms
Common questions
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →