Simpson's Paradox
AnalyticsAlso: Simpson's Reversal · Yule-Simpson Effect
Quick definition
Simpson's Paradox is a statistical pattern where a trend appears in several separate groups of data but disappears or reverses when the groups are combined. It happens because the groups have different sizes, and the combined total quietly hides that imbalance.
What it actually means
Picture two salespeople. Salesperson A has a higher conversion rate than Salesperson B on cold leads, and a higher conversion rate on warm leads too. Yet when you look at their overall conversion rate across all leads combined, Salesperson B wins. That's Simpson's Paradox. Nothing is broken. The two salespeople just handled very different mixes of cold and warm leads, and the mix, not the skill, is driving the combined number.
This shows up constantly in marketing analytics. A campaign can outperform another on every individual segment, region or device type, then lose on the blended total because one segment carries far more volume than the others. It's a close cousin of confounding variables, the hidden factors that make a correlation look causal when it isn't.
The fix isn't complicated once you know to look for it. Break any headline conversion rate, CPA, or click-through rate (CTR) down by its major segments before drawing a conclusion. If the segment-level story and the total-level story disagree, trust the segments. Attribution reporting, A/B testing readouts and cohort analysis are the three places this paradox does the most damage, because all three compress diverse groups into one clean-looking number.
The paradox isn't in the maths. It's in trusting a total before you've checked what it's made of.
How it shows up
It shows up as a dashboard that tells two different stories depending on whether you filter by segment or look at the top-line number. A retention rate that improves in every customer cohort but declines overall. A landing page that wins the A/B test on desktop and on mobile but loses when the results are pooled. Anywhere group sizes shift between periods or between the groups being compared, the paradox has room to appear.
The Australian context
Australian businesses running national campaigns often see this between metro and regional segmentation. Metro traffic volume can be five to ten times regional volume, so any shift in the metro-to-regional mix between two reporting periods can flip a headline conversion rate even when nothing changed within either segment.
Where people get this wrong
Related terms
Common questions
Is Simpson's Paradox a sign of bad data?
No. The underlying numbers are usually correct. The paradox is a mismatch between what each segment shows and what the combined total shows, caused by uneven group sizes rather than any error in collection or calculation.
How do I check for it in my own reports?
Break any headline metric down by its main segments, such as channel, device or region, before trusting the total. If every segment points one way and the blended number points the other, you've found it.
Does this affect A/B testing results?
Yes. If traffic mix shifts between test variants, such as more mobile users landing on one variant than another, the pooled result can contradict what's happening within each device type. Always check segment-level results before declaring a winner.
Why does this matter for attribution reporting?
Attribution reports blend many channels and campaigns into one view. If one channel's volume grows or shrinks between reporting periods, the blended performance figure can move in a direction no individual channel actually supports.
Debrief
Get the next one
No spam. No fluff. Just the next article, straight to your inbox.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →