A people analytics team notices that a high share of employees who left in the past year had not received a promotion in over two years, and concludes that promotion stagnation drives attrition. This might be true. It also might not be, and the analysis as described — examining only the people who left — has no way to distinguish between the two, because it never checked how common the same promotion pattern was among the employees who stayed.
The comparison group is not optional — without it, a pattern common among leavers tells you nothing
If 60% of people who left hadn't been promoted in two years, that statistic is meaningless on its own without knowing what share of people who stayed also hadn't been promoted in two years. If that share is also 60%, promotion stagnation isn't actually associated with leaving at all — it's just a common condition across the whole employee population, present in leavers and stayers alike. This is a specific and common form of survivorship-bias-adjacent error: analyzing only the outcome group (leavers) without a proper comparison to the group that didn't have that outcome (stayers) under similar conditions.
Manager quality is a common confound hiding behind more measurable factors
Manager quality is difficult to measure directly and rarely appears as a clean variable in HR data, but it correlates with many things that are easy to measure — team members' engagement survey scores, promotion rates on that team, exit interview sentiment. An attrition model that finds low engagement scores predict attrition may actually be detecting the effect of manager quality, mediated through engagement scores as an easily measured proxy. Acting on the surface finding (raise engagement scores through generic initiatives) without addressing the underlying confound (manager quality on specific teams) tends to produce disappointing results, because the intervention is targeting the proxy, not the actual driver.
Correlation with attrition doesn't mean changing the factor would reduce attrition
Even a properly comparison-adjusted finding — a factor genuinely more common among leavers than stayers — establishes correlation, not that intervening on that factor would causally reduce attrition. Compensation below market rate might correlate with attrition without being the actual reason people left, if it's also correlated with something else genuinely causal (perhaps roles with below-market pay in this dataset also happen to have less growth opportunity, and growth opportunity is the real driver). Treating a correlational finding as a direct intervention target skips the same causal-inference step that any other domain applying data to decisions has to take seriously.
What a more rigorous attrition analysis actually requires
- Always compare against a matched group of employees who stayed under similar conditions — never analyze leavers in isolation
- Look specifically for confounds that could be driving both the measured factor and attrition simultaneously, particularly manager quality
- Treat correlational findings as hypotheses to test with a real intervention and measured outcome, not as confirmed causal levers to pull immediately
- Be skeptical of any attrition finding that hasn't been checked against the base rate of that same factor among people who didn't leave
None of this means attrition data isn't useful — it means the useful version of the analysis requires the comparison group and the causal skepticism that a quick look at exit data alone doesn't provide.