A team running an A/B test checks the results dashboard daily, and on day six notices the variant has crossed the standard 95% significance threshold, so the test is stopped immediately and the variant declared a winner — a decision-making pattern called peeking, and one that substantially inflates the actual rate of false positives well beyond the nominal 5% the significance threshold is supposed to guarantee, even though nothing about the specific day-six result was calculated incorrectly.
Why a test's significance status fluctuates during data collection
A test comparing two genuinely identical variants, with no real underlying difference between them, will still show a p-value that fluctuates day to day as data accumulates, purely due to ordinary random sampling variation — and across the full course of data collection, that fluctuating p-value will cross below the 0.05 significance threshold at some point considerably more often than 5% of the time, simply because there are many opportunities (every day the test is checked) for the naturally noisy p-value to dip below the threshold at least once, even when there's no real effect present at all.
Why stopping at the first crossing specifically causes the inflation
The standard 5% false positive rate assumes a single, pre-planned test of the data at one predetermined point, not a repeated series of checks with the test stopped the moment any single check happens to cross the threshold — checking repeatedly and stopping at the first significant result is functionally similar to the broader multiple comparisons problem, giving a null effect many separate chances to appear significant by chance, and taking advantage of whichever chance happens to work.
Why this specifically matters given how commonly it happens in practice
Real-time dashboards showing live, continuously updating test results make it easy and natural to check results daily or even more frequently, and the intuitive but statistically incorrect instinct to stop a test as soon as it crosses significance — rather than waiting for the full, pre-planned sample size — is extremely common in practice, meaning a substantial share of "winning" test results declared this way in ordinary business experimentation are likely false positives that a properly completed test would not have confirmed.
What actually addresses this correctly
Determining a fixed sample size or test duration in advance, based on a proper statistical power calculation, and committing to that predetermined stopping point regardless of interim results, avoids the peeking problem entirely by removing the repeated-checking-and-stopping-early behavior that causes the inflation. Sequential testing methods, specifically designed statistical frameworks that allow legitimate early stopping while mathematically controlling the overall false positive rate across the full sequence of interim checks, provide a rigorous alternative for teams that genuinely need the ability to stop a test early without falling into the informal peeking trap.
What this means for running A/B tests correctly
- Determine sample size or test duration in advance using a proper power calculation, and commit to that stopping point rather than checking results informally and stopping at the first significant crossing
- Use a genuine sequential testing methodology if early stopping capability is a real business requirement, rather than relying on informal daily checks and ad hoc stopping decisions
- Treat any test result obtained through informal early stopping with real skepticism about whether it reflects a genuine effect or a chance crossing along a naturally fluctuating path
- Educate teams running experiments on why checking results doesn't cause harm on its own — it's specifically stopping early at the first significant crossing that inflates the false positive rate
The peeking problem is one of the more consequential, easy-to-fall-into statistical mistakes in everyday business experimentation — the fix isn't avoiding dashboards or interim visibility, it's committing to a predetermined stopping rule rather than treating the first significant-looking glance as sufficient grounds to call the test.