CRO · Cluster Guide
How to Run an A/B Test That Actually Means Something
A test can look decisive on day three and mean absolutely nothing by day fourteen. The single most common reason a CRO program stalls out isn't a lack of ideas to test. It's calling winners before the data was ever ready to be trusted.
The Test That Looked Finished on Day Three
It's an easy trap to fall into. A new variant pulls ahead early, the dashboard shows a green checkmark next to "significant," and someone on the team is ready to ship it company-wide by lunch. The problem is that early significance readings in a small sample are frequently just noise dressed up as a signal, and calling a winner before the pre-calculated sample size is reached is the single most common way a CRO program ends up implementing changes that do nothing, or worse, quietly hurt conversion.
A valid A/B test requires a pre-calculated sample size based on your baseline conversion rate and the minimum lift worth detecting, run to completion without stopping early based on interim results. One widely cited industry analysis found that 57% of tests called as winners would not have reached real statistical significance if run to their proper sample size, meaning most reported CRO wins may not be real.
Why "Peeking" Ruins a Test
Checking results daily and stopping the moment a variant looks ahead is called peeking, and it's statistically dangerous precisely because early data is noisy. A variant can appear to be winning by a wide margin in the first three days purely by chance, then regress toward the actual, much smaller or nonexistent difference as more data comes in. The fix isn't complicated: decide the required sample size before the test starts, and don't call a result until you reach it.
“A test stopped early isn't a faster result. It's a coin flip wearing a lab coat.”Chenthil Kumar, Digimarketlabs
Calculating Sample Size Before You Launch
Three inputs determine how much traffic a test needs: your current baseline conversion rate, the minimum lift you actually care about detecting, and your desired confidence level, almost always 95% in conversion optimization work. A free calculator can turn those three numbers into a concrete visitor count and expected test duration, which should be decided before a single visitor sees either variant, not adjusted afterward to match whatever the data happens to show.
Not sure if your current test has enough traffic to mean anything?
We'll calculate the real sample size your test needs and tell you honestly if it's even feasible.
What to Do When Traffic Is Genuinely Too Low
Not every page gets enough visitors to reach formal statistical significance in a reasonable timeframe. For those pages, a structured heuristic audit, reviewing the page against known conversion principles without a live split test, is a more honest approach than running an underpowered test and treating its result as real. See our landing page audit guide for exactly that kind of review.
Reading the Result Correctly Once the Test Finishes
Look at the full confidence interval, not just the headline percentage lift. A reported 10% lift with a confidence interval spanning from negative 2% to positive 22% is not a reliable result, even though the headline number looks appealing. The interval, not the point estimate, tells you how much to actually trust the outcome.
FAQ
A/B testing questions, answered directly
How long should an A/B test run?
Until it reaches the pre-calculated sample size, not a fixed number of days. This varies significantly based on your traffic volume and baseline conversion rate.
Is it ever okay to stop a test early?
Only if a pre-registered stopping rule was built into the test design from the start, or if a serious bug or ethical issue emerges. Stopping simply because a variant looks ahead is not a valid reason.
What if my traffic is too low for a proper test?
Consider a heuristic, non-split-test audit instead, or focus testing resources on your highest-traffic pages where a real result is achievable.
Does a 95% confidence level guarantee the result is correct?
No — it means there's roughly a 5% chance the observed difference is due to random variation rather than a real effect, which is an industry convention, not a certainty.
Want a testing program built on real statistical rigor?
Book a strategy call and we'll review your current testing process for exactly this kind of gap.
Sources & further reading