A research agency delivers a report proudly emphasizing its 2,000-respondent sample size, implicitly presenting scale as the primary marker of study quality. If that sample was recruited through a convenience panel that systematically over-represents certain demographics, attitudes, or behaviors relative to the actual target population, the resulting estimate is a precise measurement of the wrong group — and no amount of additional sample size fixes that, because sample size and sample bias are answering two completely different statistical questions.
What sample size actually does, and what it doesn't
Increasing sample size narrows the confidence interval around an estimate — it reduces random sampling error, the noise that comes from measuring a subset rather than the entire population, and makes a given estimate more statistically precise. It does nothing whatsoever to address systematic bias — a structural skew in who ends up in the sample in the first place, caused by how respondents were recruited, which platforms or panels were used, or who was more or less likely to respond. A biased recruitment method produces a biased estimate regardless of how many respondents are added, because every additional respondent is being drawn from the same skewed underlying pool.
A concrete way to see why this matters
A study recruiting exclusively through an online panel that skews toward frequent, comfortable internet and survey-platform users will, no matter how large the sample grows, produce an estimate specific to that population — not the broader target population the study claims to represent, some meaningful share of whom don't use that particular panel or don't respond to survey recruitment the same way. Doubling or tripling the sample size within that same biased panel makes the estimate for that specific population more precise; it does not move the estimate any closer to the actual target population's true value, because the bias lives in who was sampled, not in how many of them were sampled.
Why sample size gets over-indexed on as a quality signal anyway
Sample size is a single, easily reported, easily compared number that clients can use to quickly judge a study's apparent rigor without needing to understand or audit the more complicated question of how representative the recruitment method actually was. This creates a predictable incentive for research to emphasize sample size specifically, since it's the dimension of quality that's easiest to communicate and easiest for a client to feel reassured by, even when the recruitment methodology underlying that sample is the more consequential and much harder to evaluate factor.
Why a smaller, well-recruited sample can be genuinely superior
A study with 200 carefully recruited, verified respondents who genuinely match the target population's key characteristics produces a less statistically precise estimate than a 2,000-respondent study, but a less precise estimate of the right population is more useful and more accurate than a highly precise estimate of the wrong one. The two studies aren't comparable on a single quality axis — one is answering a genuinely different, better-targeted question than the other, and the sample size numbers alone obscure this difference completely.
What this means for evaluating or commissioning market research
- Ask specifically how respondents were recruited and how that recruitment method might systematically differ from the actual target population, not just how many respondents were included
- Treat sample size and sampling methodology as two separate quality dimensions requiring two separate evaluations, not a single combined "rigor" judgment
- Be skeptical of a study that emphasizes sample size prominently while saying little about recruitment methodology
- Recognize that a well-recruited smaller sample can be the more accurate choice, even though it will always look less impressive on the single dimension of raw respondent count
Sample size answers "how precise is this estimate." Sample composition answers "an estimate of what, exactly." Both questions matter, but only one of them is fixed by adding more respondents.