A training program measures participant self-reported competence before and after a training session, and the resulting pre-post improvement score looks impressively large — and a portion of this apparent improvement may be a genuine measurement artifact rather than actual skill gain, since a participant rating their own pre-training competence, before they've yet learned what genuine mastery of that specific skill actually requires, is rating themselves against a less informed, and specifically different, internal standard than the considerably more informed standard they apply when rating their post-training competence, a well-documented problem called response shift bias.
Why a participant's internal frame of reference for self-rating can genuinely change during training
A participant beginning a training program on a skill they're not yet familiar with often has an incomplete or even somewhat inflated understanding of what genuine competence in that skill actually requires, since they haven't yet been exposed to the full scope of what expert-level performance actually looks like — this incomplete understanding means their pre-training self-rating is calibrated against a genuinely different, less informed internal standard than the standard they'll apply once training has given them direct exposure to what real competence actually demands.
How this specific shift can produce an inflated or artifactual improvement score
A participant who initially rates their pre-training competence as reasonably high, based on an incomplete understanding of what the skill actually requires, may after training recalibrate their sense of the skill's genuine demands and realize their pre-training self-rating was actually too generous relative to this new, more informed standard — this recalibration, occurring specifically because of what was learned during training itself, can inflate the apparent pre-post improvement score beyond what represents genuine skill gain alone.
Why this bias specifically affects self-reported measures more than objective performance measures
Response shift bias specifically concerns self-reported competence ratings, which depend directly on a participant's own internal, and potentially shifting, standard for what the skill actually requires — an objective performance measure, testing a participant's actual demonstrated skill against a fixed, external standard applied consistently both before and after training, doesn't share this same vulnerability, since the external standard itself remains constant even as the participant's own understanding evolves.
How the retrospective pretest method specifically addresses this bias
The retrospective pretest method asks participants, after completing training, to retrospectively rate what their pre-training competence level actually was, using their now more informed post-training understanding of what the skill genuinely requires — since both the pre-training retrospective rating and the post-training rating are now made using the same, more informed frame of reference, this method removes the specific frame-of-reference shift that a genuinely separate, actually-collected-before-training pre-rating would otherwise introduce into the comparison.
Why using both a genuinely pre-collected pretest and a retrospective pretest together provides the most complete picture
Comparing a genuinely pre-collected pretest rating, a retrospective pretest rating collected after training, and a post-training rating together allows an evaluator to directly estimate how much response shift bias actually affected the specific training program being evaluated — a meaningful divergence between the genuinely pre-collected and retrospective pretest ratings indicates response shift bias was a genuine factor, while close agreement between the two suggests the bias had comparatively little actual influence in that specific case.
What this means for designing and interpreting pre-post training evaluation
- Recognize that self-reported pre-post improvement scores may be partly inflated by response shift bias, particularly for skills where pre-training understanding is likely to be genuinely incomplete
- Use the retrospective pretest method alongside or instead of a genuinely pre-collected pretest, specifically for self-reported competence measures
- Prefer objective performance measures over self-reported ratings where feasible, since objective measures aren't subject to this particular frame-of-reference shift
- Compare genuinely pre-collected and retrospective pretest ratings directly when both are available, as a way to estimate the actual size of this bias for a specific training program
Response shift bias is a genuine, well-documented measurement challenge specific to self-reported pre-post training evaluation — a reported improvement score partly reflects genuine skill gain and partly reflects a shifting internal standard for self-assessment, and the retrospective pretest method offers a specific, practical way to separate these two genuinely different components.