Holdout Group
AnalyticsAlso: Control Group · Holdback Group
Quick definition
A holdout group is a portion of your audience deliberately excluded from a marketing campaign so you can measure what happens without it. Comparing the holdout's behaviour to the group that received the campaign shows whether the marketing actually caused a change, rather than just coinciding with one.
How it varies across Australia
Holdout testing is still rare among mid-market Australian businesses, mostly used by larger retailers and finance brands with enough audience volume to make the comparison statistically meaningful. Smaller advertisers tend to skip it because they assume they don't have the audience size to justify it, even when a modest holdout would answer the question they keep arguing about internally.
See data and tracking maturity across Australian industries →What it actually means
Imagine two identical gardens. You water one and leave the other alone. If both grow at the same rate, the watering did nothing. A holdout group is that second garden for marketing. You take a slice of your audience, deliberately withhold the campaign, email, retargeting or discount from them, and watch what they do anyway.
The comparison is the whole point. Attribution models try to divide credit across touchpoints that all happened. A holdout group tests what happens when a touchpoint doesn't happen at all. That's a different and more honest question. It's the closest thing marketing has to a proper experiment, the same logic that runs clinical drug trials, just applied to a retargeting campaign instead of a tablet.
Holdouts get confused with a-b-testing, but they're not quite the same. A-B testing usually compares two active variants against each other. A holdout compares something against nothing. It's the cleanest way to measure incrementality, meaning the lift a channel or campaign genuinely causes rather than the conversions it would have captured regardless.
Most businesses never run one because it feels wasteful to withhold marketing from people who might have bought. That discomfort is exactly why the test is valuable. If withholding marketing changes nothing, you've been spending on people who were converting anyway.
A holdout group answers the one question attribution can't. Would this have happened anyway.
How it shows up
Holdout groups show up most often in retargeting, email retention flows and loyalty programme evaluations. A business will exclude ten percent of an audience from a retargeting campaign for a month, then compare purchase rate and conversion rate between the held-out group and the treated group. The gap between the two, not the raw conversion numbers, is the real result.
It also shows up in incrementality tests run by ad platforms themselves, where Meta or Google will offer a geo-based or audience-based holdout as part of a lift study. The output is usually reported as an incremental lift percentage, which is the number that should actually inform budget decisions.
The Australian context
Australia's smaller population makes holdout testing statistically harder for mid-size businesses. A ten percent holdout of a small email list may not produce enough conversions in either group to draw a confident conclusion, so businesses either need larger audiences or longer test windows than equivalent US advertisers to reach the same statistical power. This is one reason platform-run incrementality studies, which pool data across many advertisers, are often more practical for smaller Australian businesses than running a bespoke holdout internally.
Where people get this wrong
Holdout Group vs A/B Testing
| Holdout Group | A/B Testing | |
|---|---|---|
| What it compares | Something against nothing | One variant against another |
| Primary question | Did the campaign cause any lift at all | Which version performs better |
| Typical use | Proving incrementality of a whole channel | Optimising creative, copy or design |
| Sample discomfort | High, some people get nothing | Low, everyone gets some version |
Related terms
Common questions
How big should a holdout group be?
Big enough to produce a statistically stable comparison, which depends on your baseline conversion rate and audience size. Ten to twenty percent of an audience is a common starting point, but a business with a small list may need a larger share or a longer test window to reach a reliable answer.
Is a holdout group the same as A/B testing?
Not quite. A-B testing compares two active variants against each other, while a holdout group compares an active campaign against no campaign at all. Holdouts answer whether something works. A-B tests answer which version works better.
Why would I deliberately withhold marketing from customers?
Because it's the only clean way to prove a campaign is causing behaviour rather than just coinciding with it. Without a holdout, you can't tell whether customers who saw your retargeting ads bought because of the ads or would have bought regardless.
Can small businesses run holdout tests?
Yes, but the results are less precise with small audiences. Smaller businesses often get more reliable answers from platform-run incrementality studies, which pool larger datasets, than from a bespoke internal holdout on a modest customer list.
Debrief
Get the next one
No spam. No fluff. Just the next article, straight to your inbox.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →