Show a hiring manager two versions of a leadership assessment — one with polished, business-sounding items and cleanly designed result pages, another with awkwardly worded items and a plain results table — and they will almost always judge the first one as "clearly the better assessment." This judgment is face validity, and it is nearly unrelated to whether either assessment actually measures leadership ability.
What face validity actually is, and what it isn't
Face validity is simply whether an assessment appears, to a non-expert looking at it, to measure what it claims to measure. It's a surface judgment made without any data about how scores relate to actual outcomes. This makes it categorically different from the validity evidence that actually matters: content validity (does the item set actually sample the relevant domain), criterion validity (do scores correlate with a real external outcome), and construct validity (does the instrument measure the underlying trait it claims to, distinct from other traits it might be confounding with it). An assessment can be strong on any combination of these and still look amateurish, or vice versa — the two are simply answering different questions.
High face validity, low real validity: the common failure mode
A well-designed-looking personality assessment with professional branding, smooth UX, and business-appropriate language can still be measuring almost nothing reliably, if the underlying items were never validated against real outcomes. This is the specific trap: polish is easy to produce and highly persuasive to non-experts, while the actual validity work (piloting against real criteria, checking reliability statistics, confirming the factor structure holds) is invisible in the final product and easy to skip under time pressure. A client or hiring manager evaluating the assessment has no way to see this gap — from their seat, professional-looking and valid are hard to tell apart, because the only cue available to them is exactly the surface quality that face validity measures.
Low face validity, high real validity: the less common but real reverse case
Some of the most criterion-valid assessment items look strange or indirect to a layperson precisely because they aren't asking the obvious, socially-desirable-answer-inviting question. An item that indirectly measures conscientiousness by asking about mundane organizational habits can outperform, in actual predictive validity, an item that directly and transparently asks "are you a conscientious person?" — because the direct item is trivially easy to answer in the flattering direction regardless of the truth, while the indirect item is harder to game. This is why validated psychometric instruments sometimes look, to a first-time reader, oddly indirect or even irrelevant — that indirectness can be doing real measurement work, not failing to look professional.
Why stakeholders over-index on face validity anyway
Face validity is the only property of an assessment a stakeholder without psychometric training can actually evaluate unassisted — real validity requires data most stakeholders never see and wouldn't know how to interpret if they did. This creates a structural, not just a laziness-driven, pressure on assessment builders to over-invest in surface polish relative to underlying validity work, because polish is what gets noticed and validated in the room, while validity evidence has to be actively surfaced and explained to have any influence on the decision at all.
What to actually check, and what to show stakeholders instead
- Ask for reliability statistics (internal consistency, test-retest) before evaluating anything about how the assessment looks or feels
- Ask what the item set was validated against — a real outcome dataset, an expert panel review of content coverage, or neither
- Treat an instrument's polish and professionalism as informative about the vendor's design skill, not as evidence about measurement validity
- When presenting an assessment to stakeholders, lead with validity evidence explicitly, since face validity will otherwise silently do all the persuading on its own
Face validity isn't worthless — an assessment that looks absurd to test-takers can hurt completion rates and perceived fairness regardless of its underlying validity. But it's a UX property, not a measurement property, and treating it as a proxy for the second is where assessment quality quietly goes to die.