A single statistical test at the standard 5% significance threshold has a 5% chance of a false positive — a real, understood, and accepted risk. Run twenty independent tests at that same threshold, and the probability that at least one comes back significant purely by chance climbs to roughly 64%. This is the multiple comparisons problem, and it's exactly what happens, silently, every time a dashboard lets someone slice data across many segments and metrics and highlights whichever combination clears a significance threshold.
The math isn't a corner case — it's the default outcome of testing many things
The 5% significance threshold is a per-test guarantee, not a per-analysis guarantee. It promises that any single, pre-specified test has a 5% false-positive rate. It says nothing about what happens when the same threshold is applied across dozens of tests run against the same dataset — age segments, geographic segments, device types, each crossed against several metrics. With enough combinations tested, finding at least one significant result becomes close to guaranteed, independent of whether anything real is actually happening in the data.
Dashboards that enable this aren't broken — they're doing exactly what they were built to do
A self-serve analytics dashboard that lets a user filter by any segment and see a significance indicator on any metric isn't malfunctioning when it produces a false positive under these conditions — it's behaving exactly as designed, which is itself the problem. Every additional slice a user is free to explore is another chance for a spurious result, and the dashboard has no way to know, or communicate, how many total comparisons a user has implicitly run across a session of exploratory filtering.
The false positives are structurally random, which makes them especially persuasive
A false positive generated this way doesn't come with any tell that distinguishes it from a genuine finding — it looks, statistically, exactly like a real effect, because from the test's perspective it is one; the flaw is entirely in how many tests were run to find it, information the single significant result doesn't carry with it. This is precisely why these findings are so persuasive in practice: a claim like users aged 35-44 on mobile in the Northeast convert 22% better reads as a specific, credible insight, with no visible signal that it's one spurious result among the dozens of segment combinations that were implicitly tested to surface it.
What actually corrects for this
Formal corrections exist — the Bonferroni correction (dividing the significance threshold by the number of comparisons made) and false discovery rate control (a less conservative correction better suited to exploratory analysis with many comparisons) both directly adjust for the number of tests run. Both require knowing, and being honest about, how many comparisons were actually made — which a purely exploratory dashboard session often doesn't track and the user often doesn't consciously count. The more reliable practical fix is deciding in advance which specific segment and metric you're testing, before looking at the data, and treating anything discovered through open-ended exploration as a hypothesis to be validated with a fresh, pre-specified test — not as a finding to act on directly.
What to actually do about it
- Treat any significant result surfaced through open-ended dashboard exploration as a hypothesis, not a finding, until re-tested on fresh, pre-specified terms
- If you must report on many simultaneous comparisons, apply a false discovery rate correction rather than the uncorrected per-test threshold
- Be specifically skeptical of narrow, highly specific segment findings (a triple-crossed segment like age x device x region) — the more specific the slice, the more likely it was one of many implicitly tested
- Decide what you're testing before looking at the data whenever the decision matters enough to justify the discipline
None of this means dashboards or exploratory analysis are useless — it means treating what they surface as a lead to investigate properly, not a result to act on directly, since the discovery process itself already spent the statistical significance before a single confirmatory test was ever run.