A p-value of .03 gets treated as proof. It isn't proof of anything — it's the answer to one specific, narrow question, and most of the trouble with p-values comes from quietly answering a different question with it.
What a p-value actually measures
A p-value is the probability of observing a result at least as extreme as the one you got, if the null hypothesis were true. That's the whole definition. A p-value of .03 means: if there really were no difference between your groups, you'd see a gap this large or larger about 3% of the time by chance alone.
Two things that definition does not say: it doesn't say there's a 97% chance your hypothesis is correct, and it doesn't say anything about how big or meaningful the effect actually is.
Mistake 1: treating p as the probability your hypothesis is true
This is the single most common misreading, and it's an easy one to fall into because the number feels like it should mean that. It doesn't. The p-value assumes the null hypothesis from the start and asks how surprising your data would be under that assumption — it never actually evaluates the probability of your hypothesis versus the null. Confusing the two is called the transposed conditional, and it shows up in write-ups constantly: "there's a 97% chance the new design performs better" is not what p = .03 tells you.
Mistake 2: confusing statistical significance with practical significance
Statistical significance is a function of both effect size and sample size. With a large enough sample, almost any nonzero difference becomes statistically significant, including differences too small to matter for any real decision. A conversion-rate lift of 0.1 percentage points can hit p < .05 with a big enough sample, and reporting only the p-value makes that trivial lift look like a finding.
This is why effect size (how big is the actual difference?) and confidence intervals (what's the plausible range?) matter as much as the p-value itself. A p-value tells you whether an effect is probably not zero. It does not tell you whether the effect is worth acting on.
What to report instead of a bare p-value
- The effect size itself, in the actual units that matter (percentage points, dollars, days) — not just "significant"
- A confidence interval around that effect, so the reader can see the plausible range, not a single point estimate
- The sample size, so a reader can judge how much a large n might be inflating significance
- The p-value last, as a supporting number, not the headline finding
The practical test
Before writing "statistically significant" in a report, ask: would this effect size still be worth acting on if I'd found it in a tiny pilot sample? If the honest answer is no, the significance is coming from sample size, not from a meaningful effect.