Survivorship Bias

Analytics

Also: Survivor Bias

What it isOnly studying the winners

Shows up inCase studies, testimonials, top performers

The fixInclude the failures in the sample

RiskCopying luck, not strategy

Quick definition

Survivorship bias is the mistake of drawing conclusions only from the people, campaigns or businesses that succeeded, while ignoring the ones that tried the same thing and failed. It makes risky or lucky strategies look like reliable playbooks.

What it actually means

Survivorship bias is easiest to see in the classic World War Two story. Engineers wanted to armour the parts of returning planes with the most bullet holes. A statistician pointed out the opposite was true. The planes that got hit in other spots never made it back to be studied. The damage on survivors was showing where a plane could take a hit and still fly, not where it needed protection.

Marketing runs on the same trap. A founder studies five viral TikTok brands and copies their posting schedule, tone and product drops. What they don't see is the thousand brands that ran the identical playbook and got nowhere. Case studies, conference talks and 'what worked for us' blog posts are almost always written by survivors. The failures don't get a keynote.

This matters for attribution, for benchmark reading, and for how you interpret conversion rate or CTR data pulled from a self-selected sample. If you only analyse customers who didn't churn, or campaigns that hit target CPA, you are studying the exception and mistaking it for the rule.

Every viral case study has a graveyard of identical campaigns you never heard about because they flopped.

How it shows up

It shows up whenever a dataset only includes the winners. A/B testing programs that only report the tests that beat control, ignoring the ones that lost or were inconclusive. Influencer case studies that show the one creator who drove sales, not the twenty who didn't move the needle. Retention analysis run only on customers who are still active, which silently excludes everyone who churned and would explain why. Benchmark reports built from businesses willing to share good numbers, which skews the whole sample upward.

The Australian context

Australian small business media loves a growth story. A Bondi skincare brand that scaled fast on paid social gets written up everywhere, and every other founder in the category tries to replicate the exact spend and creative approach. What rarely gets written up is the segmentation, the margin structure or the seed audience that made it work, or the ten competitors running the same playbook that quietly shut down. Read local success stories for the mechanism, not the outcome.

Where people get this wrong

Benchmarking against public case studies instead of your own baseline.Public case studies are self-selected by people willing to publish a win. Your realistic comparison point is your own historical performance, not someone else's best month.

Analysing only active or retained customers to find what drives loyalty.The customers who churned are the ones with the real answer. Excluding them from the analysis removes the exact signal you need to reduce churn.

Copying a competitor's current strategy without asking what they abandoned.What you see today is the strategy that survived internal testing. The failed versions were quietly binned and you have no visibility into what didn't work.

Common questions

Is survivorship bias the same as confirmation bias?

No. Confirmation bias is favouring data that supports what you already believe. Survivorship bias is a sampling problem where the failures are structurally missing from the dataset, so you never even get the chance to consider them.

How do I avoid survivorship bias in A/B testing?

Report every test you ran, not just the winners. Track inconclusive and losing tests in the same log as wins. Over time this gives you an honest hit rate instead of a highlight reel that overstates how reliable your testing process actually is.

Why do case studies suffer from survivorship bias?

Case studies are written by the businesses willing to share a good outcome. Businesses that ran the same campaign and failed rarely publish anything. Reading only published case studies means reading only the winners, which inflates how replicable the strategy looks.

How does survivorship bias affect churn analysis?

If you study only customers who are still subscribed to find out what makes a good customer, you exclude everyone who churned. The churned group often holds the clearest signal about what to fix, so leaving them out produces a distorted, overly rosy picture.

Debrief

Get the next one

No spam. No fluff. Just the next article, straight to your inbox.

Keep exploring

About New Rebellion

New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.

How we think →