A test validation study reports that the test's correlation with subsequent job performance is similarly strong across two demographic groups — evidence of comparable differential validity — and presents this as evidence the test is fair across both groups. This addresses a real but distinctly narrower question than differential prediction: whether the identical test score actually predicts the identical performance level across those same two groups, a question requiring separate, specific statistical evidence that similar overall correlation strength doesn't automatically provide.
What differential validity specifically examines
Differential validity compares the overall strength of a test's correlation with an outcome across separate groups — if the correlation between test score and job performance is similarly strong (a similar correlation coefficient) within Group A and within Group B, the test shows no meaningful differential validity, meaning it appears to be measuring a similarly predictive relationship within each group considered separately.
What differential prediction examines instead, and why it's a genuinely separate question
Differential prediction compares the actual regression lines — both the slope and the intercept — fit separately predicting the outcome from the test score within each group, asking specifically whether an identical test score predicts an identical outcome level across the two groups, not simply whether the general strength of the relationship is similar within each group considered on its own.
Why a test can show no differential validity while still showing meaningful differential prediction
Two groups can each show a similarly strong correlation between test score and performance (no differential validity) while having meaningfully different regression intercepts — meaning the identical test score corresponds to a systematically different predicted performance level in one group compared to the other, a pattern differential validity analysis alone, focused only on overall correlation strength, would completely fail to detect, since it doesn't examine whether the actual predicted outcome level matches across groups at any given specific score.
Why this distinction carries genuine, practical fairness implications
If a test shows meaningful differential prediction — the identical score predicting different actual outcomes across groups — using a single, uniform scoring cutoff across both groups means one group is being systematically over-predicted or under-predicted relative to their actual likely performance, a genuine fairness concern that a differential validity analysis alone, showing only similar overall correlation strength, would not reveal or address.
Why both forms of evidence are necessary for a complete fairness analysis
A complete, defensible test fairness analysis requires examining both differential validity (is the test similarly correlated with the outcome in each group) and differential prediction (does the identical score predict the identical outcome across groups) as separate, necessary pieces of evidence, since a test validation report addressing only one of these two questions has left a genuinely open fairness question unaddressed, regardless of which specific one is reported.
What this means for evaluating test fairness claims across demographic groups
- Ask specifically whether both differential validity and differential prediction have been examined, not just one of the two
- Recognize that similar overall correlation strength across groups (differential validity) doesn't establish that the same score predicts the same outcome across groups (differential prediction)
- Look specifically for regression slope and intercept comparisons across groups as the direct evidence differential prediction analysis requires
- Treat a fairness claim supported only by differential validity evidence as addressing a narrower question than a complete fairness analysis actually requires
Differential validity and differential prediction are genuinely separate fairness questions requiring genuinely separate statistical evidence — a test validation report addressing only the first has answered a real but incomplete part of the fairness question a rigorous analysis actually needs to address in full.