Sequential Testing
AnalyticsAlso: Always-Valid Testing · Sequential A/B Testing
Quick definition
Sequential testing is a method for analysing A/B tests that lets you check results continuously as data comes in, and stop the test early once a statistically valid conclusion is reached. It uses statistical corrections to control the false positive rate that repeated peeking would otherwise cause in a standard fixed-horizon test.
How it varies across Australia
Australian experimentation programs often run with less traffic than US or UK equivalents, which makes fixed-horizon tests slow to reach a usable sample size. Sequential testing tends to matter more here than in bigger markets, because the ability to stop early without inflating false positives directly affects how many tests a team can realistically run in a year.
See conversion and testing benchmarks across Australian industries →What it actually means
Most A/B tests are built like a scheduled exam. You decide the sample size in advance, run the test until you hit it, then check the result once. Sequential testing is built more like a doctor monitoring a patient's vitals. You can check the readings continuously and act the moment you have enough evidence, without waiting for an arbitrary end date.
The catch with standard tests is that checking early counts as cheating, statistically speaking. Every time you peek at a fixed-horizon test before it finishes, you increase the chance of seeing a false positive purely by chance. Run the numbers five times before the planned end date and your real error rate can be several times higher than the 5% you think you're accepting. That's the trap behind most 'we found a winner on day three' stories.
Sequential testing solves this properly. It uses statistical methods, sequential probability ratio tests or alpha spending functions, that adjust the bar for statistical significance as you go, so checking early is mathematically accounted for rather than accidentally free. It doesn't shrink your required sample size, it just makes stopping early honest instead of lucky. Conversion rate and p-value calculations still sit underneath the whole exercise. Sequential testing just changes the rules for when you're allowed to stop looking at them.
Peeking at a test isn't the problem. Peeking without a correction for it is.
How it shows up
Sequential testing shows up inside dedicated experimentation platforms as a 'stop early' or 'always valid' setting, which recalculates the significance threshold every time new data arrives instead of at one fixed point. It also shows up in the reporting: instead of a single p-value at the end, you see a continuously updated confidence interval or a boundary line the effect needs to cross before the test can end. If your testing tool just runs a standard test and refreshes the p-value on a dashboard, checking early is still peeking. True sequential methods are built into the underlying statistics, not bolted on as a dashboard refresh.
The Australian context
Australian testing programs often run into a genuine traffic constraint. A smaller population than the US or UK means fixed-horizon tests take longer to reach a usable sample size, and teams get impatient. That impatience is exactly where sequential testing earns its keep, letting a lower-traffic Australian ecommerce site or SaaS business reach a valid answer without waiting months for a full-powered fixed test.
The risk is teams adopting the language of sequential testing without the maths behind it. Calling a test 'sequential' because you checked it twice a week doesn't make it one. If your experimentation platform doesn't explicitly support always-valid inference, you're running a fixed-horizon test with extra risk, not a sequential one.
Where people get this wrong
Related terms
Common questions
Is sequential testing the same as peeking at a test early?
No. Peeking at a standard fixed-horizon test and stopping when it looks significant inflates your false positive rate. Sequential testing uses statistical methods designed to let you check continuously without that penalty. The difference is the maths underneath, not just how often you look.
Do I need special software to run a sequential test?
Generally yes. Standard A/B testing calculators assume a fixed sample size and a single check at the end. Running a genuine sequential test needs a platform or statistical method, like a sequential probability ratio test or an alpha spending function, built specifically for continuous monitoring.
Does sequential testing need less traffic than a normal A/B test?
Not necessarily less, just used differently. You might finish sooner for a big effect and later for a small one, because the test adapts to the evidence instead of running to a fixed date regardless. Low-traffic sites still need enough volume to detect anything meaningful.
When should I use sequential testing over a standard test?
When speed to decision matters and you want the option to stop early without corrupting your results, particularly for high-traffic pages or urgent decisions. For simple, one-off tests with plenty of time, a standard fixed-horizon test is often simpler to plan and explain.
Debrief
Get the next one
No spam. No fluff. Just the next article, straight to your inbox.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →