A performance assessment scored by different raters, across different testing occasions, using different but supposedly equivalent item sets, produces a classical reliability coefficient of 0.75 — a single number indicating a moderate overall level of measurement consistency, and one that provides no direct information about which specific source of inconsistency — differing raters, differing occasions, differing item sets — is actually driving the observed variation. Generalizability theory addresses exactly this limitation, decomposing measurement error into separate, individually interpretable variance components rather than combining every source into one aggregate reliability figure.
Why classical reliability's single-number approach leaves a genuine diagnostic gap
Classical test theory's reliability coefficient answers a single, aggregate question — how consistent is this measurement overall — without distinguishing between genuinely different underlying causes of inconsistency that would call for entirely different, specific remedial actions, meaning two assessments with identical classical reliability coefficients could have completely different actual sources of inconsistency, one dominated by inconsistent raters and the other by inconsistent item difficulty across forms, information the single aggregate number doesn't reveal at all.
What generalizability theory does differently, structurally
Generalizability theory, developed as a direct extension of classical reliability analysis, explicitly models multiple potential sources of measurement variation — raters, occasions, items, and their interactions — simultaneously, using a statistical design (a generalizability study) that separately measures how much of the total observed score variation is attributable to each specific source, producing a decomposed variance breakdown rather than a single aggregate reliability number.
Why this decomposition provides directly actionable diagnostic information
Knowing specifically that rater variance is the dominant source of an assessment's inconsistency, rather than item variance or occasion variance, points directly toward a specific, targeted remedy — improved rater training and calibration — while knowing that item variance dominates instead points toward a different remedy — revising or better equating item sets across forms — a level of specific, actionable diagnostic information a single classical reliability coefficient simply cannot provide on its own.
Why two assessments with identical classical reliability can have entirely different underlying problems
An assessment where inconsistency comes primarily from raters and an assessment where inconsistency comes primarily from item set variation across forms can produce numerically identical classical reliability coefficients while requiring completely different fixes — generalizability theory specifically distinguishes between these two situations, information lost entirely in a classical analysis that reports only the single, combined aggregate figure.
Why this matters directly for assessment programs relying on human raters or multiple test forms
Any assessment program involving human judgment (rater-scored performance assessments, essay scoring) or multiple test forms administered across different occasions is exposed to multiple potential, genuinely distinguishable sources of measurement inconsistency, making generalizability theory's decomposed diagnostic approach particularly valuable for identifying exactly where investment in improving the assessment's actual reliability should be directed.
What this means for evaluating and improving assessment reliability
- Consider a generalizability study, rather than relying solely on a single classical reliability coefficient, for assessments involving multiple raters, occasions, or item forms
- Use the resulting variance component breakdown to direct improvement efforts specifically toward the actual dominant source of inconsistency
- Recognize that a single reliability coefficient can mask genuinely different underlying reliability problems requiring different specific remedies
- Favor generalizability theory particularly for rater-scored assessments, where rater variance is a common and often underappreciated source of measurement inconsistency
Generalizability theory's real advantage over classical reliability is diagnostic specificity — rather than a single number telling you a test is somewhat inconsistent, it tells you precisely where that inconsistency is actually coming from, which is exactly the information needed to fix it efficiently rather than guessing at a general improvement effort.