← Back to the blog
Epistemic Engineering · October 8, 2026

The Measurement Trap: When the Metric Becomes the Reality

The Measurement Trap: When the Metric Becomes the Reality

A university does not merely read its ranking. Once the ranking becomes consequential, it can begin to influence how the institution budgets, recruits, admits students, allocates resources, and reports its performance. The ranking was introduced as a way to compare universities. It has now become one of the factors shaping the universities being compared.

What happens when a measure becomes consequential?

The question is broader than university rankings. The same problem appears when publication counts influence research funding, when citation metrics affect academic careers, or when AI benchmarks become reference points for investment and model development. A measurement can start as an observation and become an incentive. That transition matters because the behaviour generated by an incentive can later appear in the data produced by the system.

The first mistake: treating measurement as observation

A metric is a reduction. It takes a complex phenomenon and selects some of its properties for measurement. That reduction can be useful. A university ranking cannot describe everything about a university, but it can provide information about selected characteristics. A citation count cannot capture the quality of a research programme, but it can tell us something about the circulation of published work. An AI benchmark cannot measure intelligence in general, but it can provide information about performance on a defined set of tasks.

The problem starts when the metric acquires consequences beyond its original descriptive purpose. A number used only as information has a different role from a number used to allocate money, determine status, select candidates or guide investment.

Once the number affects decisions, the people being measured have reasons to respond to it. This gives us three different questions.

Validity: what does the metric measure?

Reactivity: how does the system change in response to the metric?

Constitution: when does the resulting change become part of the evidence used to justify the metric?

The third question is where the measurement problem becomes more difficult.

Law-school rankings

The history of American law-school rankings provides one of the clearest empirical examples. Sauder and Lancaster studied the effects of the U.S. News & World Report law-school rankings using data from 1996 to 2003. Their analysis examined both prospective students and law schools and found evidence that rankings affected application and admissions decisions. This creates a feedback process. A ranking influences where students apply. It can also influence which applicants schools admit. Those decisions affect the composition of future student cohorts. The characteristics of those cohorts can then affect indicators used in later assessments.

The ranking is therefore no longer external to the process. This does not mean that the ranking creates the quality of a law school from nothing. Nor does it mean that every applicant or every institution reacts in the same way. Later research has questioned the strength of some effects on applicants, which is one reason not to describe the ranking as a simple mechanism controlling student behaviour.

The narrower point is enough. A public measure can become one of the variables that influence the institution it is intended to compare. The contemporary ranking environment also illustrates why the mechanism should not be tied to a single formula. U.S. News has changed its methodology over time, including a substantial increase in the weight given to employment outcomes. Other rankings now emphasize different dimensions of legal education.

The ranking system changes, but the underlying question remains: what happens when an institution has reasons to organize part of its behaviour around the variables that determine its position?

Bibliometrics

Research assessment makes the same problem more consequential because metrics can influence access to funding, employment, promotion and institutional resources. Publication counts, citation counts, journal indicators and other bibliometric measures provide useful information. The difficulty arises when those indicators become embedded in decisions about scientific value. The Australian experience is instructive. Linda Butler studied the effects of a research-funding formula that incorporated publication output. Following the introduction of performance-based funding mechanisms, Australia's share of indexed publications increased. At the same time, the increase was particularly pronounced in lower-impact journals, and the country's relative citation impact declined. The result does not establish that researchers deliberately sacrificed quality for quantity. A much simpler mechanism can explain the behaviour.

When publication output becomes part of the funding formula, increasing publication output becomes institutionally valuable. Researchers and universities can then adapt their strategies accordingly. No misconduct is required. This distinction matters because discussions of research metrics often focus on obvious forms of gaming such as excessive self-citation, citation cartels or paper mills. Those practices are real concerns, but they represent only the most visible end of a much broader process. A researcher who chooses a more publishable question, divides a project into several papers, prioritizes a highly visible journal, or chooses a collaboration partly because it improves measurable output may be responding rationally to the evaluation environment.

The resulting publications are real. The citations are real. The output has genuinely increased. But the composition of scientific production may also have changed because of the metric. Recent research continues to document problems of this kind. A 2025 study of bibliometric anomalies in university rankings identified unusual authorship patterns, rapid growth in publication output and other signals associated with environments where professional or financial incentives are closely connected to bibliometric performance.

Other recent work has examined the relationship between research-assessment practices and Open Science adoption. These studies point to a broader issue: activities that are valuable to science but poorly represented by conventional publication and journal metrics can receive less institutional attention. This does not make bibliometrics useless. It means that what is easy to count can become easier to reward than what is difficult to count.

Replication, mentoring, research infrastructure, negative results, long-term projects, policy contributions and data stewardship can all matter without producing the same type of measurable signal as a publication or citation. A research system does not need to prohibit these activities for them to become relatively less attractive. It only needs to attach stronger consequences to other indicators. AI benchmarks introduce the same problem in a much newer environment. Benchmarks are necessary. Without common evaluation tasks, comparing models would be considerably harder. A benchmark can also be very good at measuring what it was designed to measure. That does not guarantee that its score remains an adequate proxy for broader capability as the surrounding ecosystem changes. Recent research on LLM benchmarks illustrates one part of the problem. A 2026 study of 60 benchmarks found substantial saturation across a large proportion of them, with older benchmarks more likely to lose their ability to distinguish among highly capable systems.

Saturation does not demonstrate strategic behaviour. A benchmark can saturate because models have genuinely improved, because the benchmark has limited resolution, or because the task has become too easy relative to current capabilities. Likewise, contamination is a validity problem before it is evidence of reactivity. Overfitting to a benchmark is more suggestive of adaptation, but still does not by itself demonstrate that an entire research ecosystem has reorganized around the metric. Those distinctions matter. The stronger claim concerns what happens when benchmark scores have consequences.

Leaderboards provide public comparisons. State-of-the-art results can affect peer review, visibility, investment and the allocation of research attention. Recent work has also examined incentives for allocating post-training resources toward improvements that increase leaderboard performance. Optimizing a benchmark is not inherently problematic. If a benchmark measures an important capability, improving performance on it can represent genuine progress. The question is what happens when the benchmark becomes one of the principal ways in which progress is recognized. Research teams have limited time and resources. If one evaluation receives substantial attention from funders, reviewers, companies and the research community, improving that evaluation becomes a rational use of resources. The score can therefore influence what gets researched. That creates a possible feedback process:

> The benchmark identifies a capability as important; resources move toward improving performance on that capability; performance improves; the improvement is reported as evidence of progress; further resources follow.

> The difficulty is not that the score has become meaningless.

> The difficulty is that the score is now part of the process producing the research priorities that it is subsequently used to evaluate.

From reactivity to constitution: This is where I propose using the term metric constitution. I use it here as a synthetic term, not as an established label in the literature. It describes a particular point in the relationship between measurement and action. A metric becomes constitutive when it is embedded in evaluation and allocation systems, changes behaviour and resource distribution, and the resulting changes are later treated as evidence about the phenomenon the metric was intended to measure. The distinction can be summarized simply:

> Metric validity: does the measure provide useful information about the phenomenon?

> Reactivity: do actors change their behaviour because the measure has consequences?

> Metric constitution: does the system later treat the consequences of that adaptation as independent evidence of the underlying phenomenon?

This does not require fraud. It does not even require that the metric be poorly designed. A well-designed measure can become problematic when its institutional importance exceeds its informational role. A benchmark may remain useful for comparing models on a defined task while becoming a poor guide to where research resources should go. A citation metric may remain informative about the circulation of published work while becoming a poor proxy for scientific contribution when used for hiring or funding. A university ranking may contain useful information while influencing the institutions whose performance it reports. The metric and the behaviour it produces can become difficult to separate.

What funders and researchers should ask

Once a metric becomes consequential, evaluating the metric itself is not enough. The surrounding incentive structure also needs to be examined. What behaviour does the metric make rational? If improving the score affects funding, status or access, what strategies become attractive to the people being evaluated? Which resources are now allocated according to it? A metric becomes more consequential as it determines where money, attention, personnel and opportunities go.

What becomes harder to see because it is not counted?

Every measurement system leaves some properties outside its field of view. Those omissions become more important when the measured properties determine rewards. Which outcomes were produced by the metric's own incentives? An increase in a measured outcome may reflect an underlying improvement, an adaptation to the evaluation system, or both.

What evidence could still falsify the metric?

This is perhaps the hardest question. When a metric determines who receives funding, which institutions gain status, which researchers advance, or which technologies attract investment, it can also influence which evidence gets produced and which questions receive attention. A measurement system can therefore become increasingly difficult to challenge precisely because so many decisions depend on it. That is an epistemic risk.

The measurement trap

Metrics are unavoidable. Complex systems require simplification if we want to compare them, allocate resources or make decisions under uncertainty. The relevant question is what happens after the simplification enters the decision process. The ranking can influence the university. The funding formula can influence the research output. The benchmark can influence the direction of model development. The resulting behaviour is not necessarily artificial. It can be the natural response of competent actors to the incentives around them. That is why the problem is easy to miss. The data produced after the intervention may look like confirmation of the original measurement. Publication output increases. A benchmark score improves. An institution moves up the ranking. But those outcomes may partly reflect the fact that the system was organized around producing them.

For funders and researchers, this suggests that metric design should not end with questions of reliability, validity and statistical performance. The downstream effects matter as well. A metric should be evaluated not only for what it measures, but also for the decisions it influences, the behaviours those decisions encourage, the resources they redirect, and the evidence that becomes available as a result. That is the point at which measurement becomes part of the environment it was introduced to observe.

REFERENCES

Butler, L. (2003). Explaining Australia’s Increased Share of ISI Publications: The Effects of a Funding Formula Based on Publication Counts. Research Policy, 32(1), 143-155.

DORA. (2013). San Francisco Declaration on Research Assessment.*

Espeland, W. N., and Sauder, M. (2007). Rankings and Reactivity: How Public Measures Recreate Social Worlds. American Journal of Sociology, 113(1), 1-40.

Meho, L. I. (2025). Gaming the Metrics: Bibliometric Anomalies in Global University Rankings and the Research Integrity Risk Index (RI²). Scientometrics.

Sauder, M., and Lancaster, R. (2006). Do Rankings Matter? The Effects of U.S. News & World Report Rankings on the Admissions Process of Law Schools. Law & Society Review, 40(1), 105-134.

Chen, Y., Zhang, G., and Hardt, M. (2026). Leaderboard Incentives: Model Rankings under Strategic Post-Training.

Akhtar, M., et al. (2026). When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation.

OECD. (2025). OECD Science, Technology and Innovation Outlook 2025. OECD Publishing.