An experimentation dashboard reports statistical significance separately across twenty different metrics for the same underlying A/B test — conversion rate, revenue per user, session length, click-through rate on a dozen individual page elements — and one of these twenty metrics shows a statistically significant result at the standard 5 percent threshold, and a team celebrates this finding as real, when a genuinely null experiment testing twenty independent metrics at this threshold has roughly a two-thirds chance of showing at least one falsely significant result purely by chance, a compounding risk called the multiple comparisons problem.
Why testing more metrics compounds the false positive risk specifically
The standard 5 percent significance threshold means a genuinely null result — no real underlying effect at all — has a 5 percent chance of crossing that threshold purely by random chance for any single metric tested. Testing twenty genuinely independent metrics gives twenty separate, independent opportunities for this same chance crossing to occur, and the probability that at least one of twenty independent tests crosses the threshold purely by chance compounds to roughly 64 percent, dramatically higher than the 5 percent rate a team might intuitively expect from the individual per-metric threshold.
Why this specifically matters for how experimentation dashboards are typically built and used
Modern experimentation platforms often report significance testing automatically across a large number of metrics simultaneously, making it easy and tempting for a team to scan the resulting dashboard for whichever specific metric happens to show significance, and then build a narrative around that particular finding — exactly the behavior that the compounding multiple comparisons math makes genuinely risky, since a large enough set of tested metrics makes finding at least one falsely significant result nearly inevitable even when nothing in the underlying experiment actually changed anything.
Why this differs meaningfully from a single, pre-specified primary metric test
An experiment testing a single, pre-specified primary metric at the standard 5 percent threshold genuinely does carry only a 5 percent false positive risk — the compounding problem specifically arises from testing many metrics and then treating whichever one happens to be significant as the meaningful finding, a fundamentally different and much riskier practice than committing in advance to one primary metric as the actual basis for the experiment's conclusion.
What the Bonferroni correction and similar methods actually do about this
The Bonferroni correction divides the standard significance threshold by the number of metrics being tested, requiring each individual metric to clear a considerably stricter bar before being considered significant, directly compensating for the compounding false positive risk that testing many metrics simultaneously otherwise introduces — a genuinely conservative but statistically principled way of keeping the overall false positive rate across the full set of tested metrics near the originally intended level.
Why pre-committing to a small number of primary metrics is often the more practical solution
Rather than applying a statistical correction across dozens of exploratory metrics, many experimentation teams instead pre-commit to a small number of primary metrics — ideally just one or two — as the actual basis for an experiment's success determination, treating every other tracked metric as exploratory and hypothesis-generating rather than as evidence a team should act on directly, a practical discipline that sidesteps the compounding multiple comparisons problem by simply not testing that many metrics as confirmatory evidence in the first place.
What this means for building and interpreting experimentation dashboards
- Pre-commit to a small number of primary metrics before running an experiment, rather than scanning a large dashboard afterward for whichever metric happens to be significant
- Apply a Bonferroni or similar correction when genuinely testing many metrics as confirmatory evidence rather than as exploratory signal
- Treat significant results on non-pre-specified, exploratory metrics as hypothesis-generating leads requiring separate confirmation, not as standalone proof
- Recognize that a dashboard reporting significance across dozens of metrics will nearly always show something significant, even for a genuinely null experiment
The multiple comparisons problem is genuine, compounding math, not a subtle statistical technicality — a team scanning twenty metrics for whichever one crosses the significance threshold is running a fundamentally different, far less reliable test than a team that committed to one primary metric before the experiment ever launched.