A new leadership assessment shows a strong correlation with an existing, well-established leadership measure — evidence presented as validating the new assessment. This convergent evidence alone doesn't rule out an important alternative explanation: both measures might be picking up a shared general response tendency or measurement method effect, rather than genuinely measuring the specific underlying leadership trait each claims to assess. The multitrait-multimethod matrix, developed specifically to address this gap, requires demonstrating not just convergent validity but discriminant validity as well, examined together in a single structured framework.
What convergent validity establishes, and its specific limitation on its own
Convergent validity is demonstrated when different measurement methods intended to assess the same underlying trait correlate strongly with each other — a new leadership self-report measure correlating strongly with an established observer-rated leadership measure is genuine, useful evidence the new measure is picking up something real related to leadership. On its own, this evidence doesn't rule out the possibility that both measures are simply picking up a shared, general tendency (such as overall positive self-presentation) rather than the specific leadership construct each claims to be measuring.
What discriminant validity adds, and why it's the necessary complement
Discriminant validity is demonstrated when the same measurement method, applied to genuinely different and theoretically distinct traits, produces correlations that are meaningfully weaker than the convergent correlations found for the same trait measured by different methods — if a leadership self-report measure correlates just as strongly with an unrelated trait like general sociability as it does with an independent leadership measure, this pattern suggests the self-report measure may be picking up a general response tendency rather than something specific to leadership, exactly the kind of problem discriminant validity testing is designed to catch.
How the matrix structure actually organizes this comparison
The multitrait-multimethod matrix systematically arranges correlations across multiple traits (leadership, sociability, conscientiousness) each measured by multiple methods (self-report, observer rating, behavioral assessment), allowing a direct, structured comparison between convergent correlations (same trait, different methods) and discriminant correlations (different traits, same method) within a single, comprehensive framework, rather than examining these comparisons in isolation from each other.
Why this more demanding standard catches problems a single correlation can't
A test validated only through a single convergent correlation with an established measure can pass that specific test while still primarily reflecting a general method effect rather than the specific intended trait — the multitrait-multimethod framework's explicit requirement to also examine and report weaker discriminant correlations directly surfaces this problem, since a measure dominated by a general method effect would show unexpectedly strong correlations even with traits it should be genuinely unrelated to.
What this means for evaluating claims about a psychometric assessment's validity
- Ask specifically whether discriminant validity evidence exists alongside convergent validity evidence, not just whether the test correlates with an established, related measure
- Be specifically skeptical of a validity case built entirely on convergent correlations, without any check on whether the same measurement method also inappropriately correlates with genuinely unrelated traits
- Look for multitrait-multimethod matrix evidence specifically in rigorous test validation research, since it represents a more demanding and complete validity testing standard
- Recognize that a single supportive correlation is meaningfully weaker evidence of genuine construct validity than the fuller convergent-and-discriminant picture this framework requires
The multitrait-multimethod matrix reflects a genuinely more demanding, and more informative, standard for validity evidence — proving a test measures what it claims requires showing not just that it correlates with the right things, but that it doesn't correlate, at least not as strongly, with the things it shouldn't.