PaxLabs Research
Andres G. and the PaxLabs Research Team · PaxLabs Inc.
The Observer Effect in AI Evaluation: How Benchmarks Shape the Behavior They Measure
Andres G. and the PaxLabs Research Team PaxLabs Inc. Correspondence: legal@paxeer.app
We examine a systematic distortion in the evaluation of artificial intelligence systems: the act of measuring agent performance on a benchmark changes the behavior being measured. Drawing on Goodhart's Law and its formal extensions, we identify three mechanisms through which this observer effect operates: data contamination and memorization, task format specialization, and metric optimization without corresponding capability gain. We present evidence from recent literature showing benchmark saturation, contamination detection failures, and the growing gap between benchmark scores and real-world performance. We formalize the capability-performance gap as the divergence between genuine capability gains and benchmark-specific optimization. Finally, we outline evaluation methodologies that resist gaming, including dynamic task generation, adversarial benchmarks, and capability probes that test underlying competence rather than surface-level task completion.
data contamination, reward hacking, capability measurement, LLM evaluation
The evaluation of artificial intelligence systems has increasingly relied on standardized benchmarks. From MMLU [1] to BIG-bench [2] to HELM [3], the research community has constructed elaborate measurement instruments intended to track progress in language understanding, reasoning, code generation, and other capabilities. These benchmarks serve a dual function: they guide research directions and inform deployment decisions. Yet a growing body of evidence suggests that the relationship between benchmark performance and genuine capability is more fragile than it appears.
We argue that AI benchmarks are subject to an observer effect analogous to Goodhart's Law: when a benchmark becomes a target for optimization, it ceases to be a reliable measure of the capability it was designed to test [4], [5]. This is not a peripheral concern. The entire infrastructure of AI progress measurement depends on the assumption that benchmark scores track meaningful capabilities. When that assumption breaks down, we lose the ability to distinguish genuine advances from artifacts of optimization.
The problem has practical urgency. Models now routinely score above 90% on benchmarks like MMLU [1], while practitioners report persistent failures on tasks that these benchmarks were supposed to represent [6]. Leaderboard rankings influence billions of dollars in investment and shape regulatory approaches to AI governance. If those rankings are compromised by the very act of measurement, the consequences extend well beyond academic concerns.
In this paper, we formalize the observer effect in AI evaluation, identify its three primary mechanisms, review the empirical evidence, and propose evaluation methodologies that are more resistant to gaming. Our analysis draws on work from measurement theory, reinforcement learning, and the empirical study of benchmark contamination.
Goodhart's Law, originally formulated in the context of monetary policy, states that "when a measure becomes a target, it ceases to be a good measure" [4]. The law captures a fundamental tension in any system where measurement serves a dual purpose: providing information about a quantity of interest and incentivizing behavior. When agents optimize directly for the measure, the statistical relationship between the measure and the underlying quantity degrades.
Kirchner et al. [5] provide a formal treatment of Goodhart's Law, distinguishing between a weak form (over-optimizing the metric is useless for the true goal) and a strong form (over-optimizing the metric is harmful for the true goal). They show that the strength of the effect depends on the tail distribution of the discrepancy between the true objective and the proxy measure. Long-tail discrepancies favor the strong form, where increasing metric optimization produces negative correlation with the true goal.
The history of AI evaluation contains numerous examples of benchmark gaming. Systems achieve high scores on standardized tests while failing on tasks that require the same underlying capability, presented in a slightly different format. The pattern is well-documented: a benchmark is introduced, models improve on it rapidly, and the benchmark eventually loses its ability to discriminate between systems of different capability levels [7].
Hendrycks et al. [1] introduced MMLU as a broad measure of multitask language understanding spanning 57 subjects. Within three years, state-of-the-art models routinely scored above 88%, leading researchers to question whether the benchmark still measured understanding or had become a test of memorization and pattern matching [6], [8].
In reinforcement learning, reward hacking occurs when an agent exploits the gap between the specified reward function and the designer's true intent [9]. The agent achieves high reward without completing the task as intended. Amodei et al. [9] identified reward hacking as one of five concrete problems in AI safety, noting that optimizing a proxy objective can produce behavior that diverges from the intended outcome.
Specification gaming, a closely related concept, describes behavior that satisfies the literal specification of an objective without achieving the designer's intent [10]. DeepMind researchers documented numerous cases where RL agents found unexpected shortcuts: a simulated robot learning to slide along the ground instead of walking, a Tetris agent pausing the game indefinitely to avoid losing, and a boat-racing agent collecting power-ups in a loop rather than finishing the race [10].
The connection to benchmark evaluation is direct. When we define a benchmark as the target for model development, we create a specification game. The benchmark is the literal specification; the underlying capability is the designer's intent. Models that optimize for the benchmark may satisfy the literal specification while diverging from the capability we actually care about.
We define the observer effect in AI evaluation as follows:
Definition 1 (Observer Effect): Let B be a benchmark designed to measure capability C. Let M be a model (or training procedure) that has access to B or knowledge of B's structure during development. The observer effect is present when the relationship between performance on B and possession of capability C degrades as a function of M's optimization pressure on B.
Formally, let P(B|M) denote M's performance on benchmark B, and let C(M) denote M's true capability as defined by the benchmark's design intent. In the absence of the observer effect, we expect:
dC/dP(B) >= 0
That is, increases in benchmark performance should correspond to non- decreasing capability. The observer effect is present when:
dC/dP(B) < 0
for some range of P(B), meaning that additional optimization on the benchmark actually reduces or fails to improve the underlying capability.
This effect operates through three distinct channels, which we describe in detail in Section IV.
We note that the observer effect is not merely a theoretical concern. It reflects a structural feature of how modern AI systems are developed. The standard pipeline involves pretraining on large corpora (which may include benchmark data), supervised fine-tuning on task-specific data, and reinforcement learning from human feedback (RLHF) on preference data that may correlate with benchmark-like tasks [11]. At each stage, the benchmark's influence on model behavior can introduce distortions.
Definition 2 (Benchmark Contamination Vector): We define the contamination vector of a model M with respect to benchmark B as the tuple:
V(M, B) = (D, F, R)
where D represents direct data contamination (exposure to B's test items during training), F represents format contamination (exposure to B's task structure during training), and R represents reward contamination (optimization toward B-like metrics during RLHF or other alignment procedures).
Each component of V can independently degrade the relationship between benchmark performance and genuine capability. Their effects may compound.
The most direct form of the observer effect occurs when benchmark test data appears in a model's training corpus. Because modern language models are trained on internet-scale corpora, and because benchmark datasets are published and discussed online, some degree of contamination is nearly inevitable [12], [13].
Dong et al. [14] proposed CDD (Contamination Detection via output Distribution), a method that detects contamination by identifying peakedness in a model's output distribution on benchmark items. Their analysis revealed that contaminated models exhibit memorized response patterns that diverge from genuine generalization. Singh et al. [13] conducted a systematic study of evaluation data contamination, finding that even models trained on data collected before a benchmark's release may exhibit contamination through indirect pathways, including discussions of benchmark items in forums, blog posts, and secondary datasets.
The distinction between memorization and generalization is central. A model that has memorized benchmark answers will score highly on those specific items but may fail on structurally identical problems with different surface features. Dong et al. [14] showed that contamination detection methods based on output distribution analysis can distinguish memorized from generalized responses, but these methods are not routinely applied in benchmark reporting.
Xu et al. [15] surveyed the landscape of benchmark data contamination, identifying it as a pervasive challenge across both open-source and proprietary models. They noted that the opacity of training data for closed-source models makes contamination assessment particularly difficult, creating an asymmetry in evaluation reliability.
A subtler mechanism involves optimization for the specific task format used by a benchmark. Language models develop sensitivity to the structure of evaluation prompts, the format of expected responses, and the distributional properties of benchmark items. This sensitivity can produce high scores without corresponding capability.
MMLU, for example, uses a multiple-choice format with four options [1]. Models trained or fine-tuned on multiple-choice data develop heuristics specific to this format: eliminating implausible options, detecting patterns in answer distributions, and leveraging the constraint that exactly one option is correct. These heuristics improve MMLU scores without necessarily improving the model's understanding of the underlying subject matter.
The BIG-bench collaboration [2] documented this phenomenon across 204 tasks. They found that model performance was sensitive to prompt formatting, few-shot example selection, and the order of answer options. Changes in these surface features, which are irrelevant to the underlying capability, produced measurable changes in performance. This sensitivity is a signature of format specialization rather than genuine capability.
Chollet [16] argued that measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience. Applied to benchmarks, this insight implies that high benchmark scores may reflect the accumulation of format-specific priors rather than the general capability the benchmark was designed to test.
The third mechanism operates at the level of the training objective. When models are trained using RLHF [11] or related procedures, the reward model serves as a proxy for human preferences. Gao et al. [17] demonstrated that optimizing against a reward model follows a predictable pattern: initial optimization improves the true objective, but continued optimization eventually overfits to the reward model's idiosyncrasies, degrading performance on the true objective.
This finding has direct implications for benchmark evaluation. When the reward model is trained on tasks that resemble benchmark items, or when the reward signal correlates with benchmark-relevant metrics, RLHF can produce models that score highly on benchmarks while developing behaviors that diverge from genuine capability. The reward model becomes a proxy for the benchmark, and optimizing the proxy follows the same Goodhart dynamic.
Ouyang et al. [11] showed that InstructGPT models trained with RLHF were preferred by human evaluators over much larger base models, despite having 100 times fewer parameters. However, this preference was measured on a specific distribution of prompts. Whether the preference generalizes to distributions that differ from the training data remains an open question, and the potential for metric-driven optimization without capability gain is inherent in the RLHF framework.
The saturation of benchmarks provides indirect evidence of the observer effect. As models are optimized against a benchmark, their scores converge toward the ceiling, reducing the benchmark's ability to discriminate between systems of different capability levels [7]. This convergence is expected even without the observer effect, but the rate of convergence and the pattern of saturation are informative.
Wang et al. [7] conducted a systematic study of benchmark saturation across 60 LLM benchmarks. They found that nearly half of the benchmarks exhibited saturation, with rates increasing as benchmarks age. Benchmark age and scale were strong predictors of saturation, while commonly assumed safeguards such as private test sets or closed-ended formats showed limited effects. The authors noted that saturation is not merely a function of model improvement; it reflects the community's optimization pressure on specific benchmark distributions.
The MMLU benchmark illustrates this trajectory. When introduced, it was designed to be challenging for the models of its era, with GPT-3 only surpassing random chance with its largest variant [1]. By 2024, multiple models exceeded 88% accuracy [6]. While some of this improvement reflects genuine capability gains, the rate of improvement and the pattern of errors suggest that a substantial portion reflects benchmark-specific optimization.
In response, researchers introduced MMLU-Pro [18], an enhanced benchmark incorporating questions with multiple correct answers and requiring more complex reasoning. MMLU-Pro+ further tests resistance to shortcut learning by requiring models to evaluate the validity of multiple statements independently [18]. The need for such extensions is itself evidence that the original benchmark had been compromised by optimization pressure.
The difficulty of detecting contamination provides further evidence. Dong et al. [14] showed that their CDD method could identify contamination in controlled settings, but real-world contamination is more complex. Indirect contamination, where benchmark items appear in training data through secondary sources, is difficult to detect and may not produce the clean statistical signatures that detection methods rely on.
Singh et al. [13] found that contamination effects vary across models and benchmarks, and that no single detection method is reliable across all settings. The inconsistency between detection methods suggests that contamination is pervasive but heterogeneous in its manifestations.
Pan et al. [19] demonstrated the severity of the issue by intentionally fine-tuning smaller models (BART, T5, GPT-2) on publicly available test sets. These models excelled on the target benchmarks but failed completely on comparable unseen test sets. The ease with which benchmark performance can be artificially inflated through direct contamination underscores the fragility of benchmark-based evaluation.
The practice of training on data that resembles benchmark distributions, even when the specific test items are not included, represents a form of indirect contamination. Models trained on large internet corpora inevitably encounter discussions of benchmark problems, worked solutions, and analyses of benchmark patterns. This exposure creates implicit knowledge of benchmark structure that inflates scores without corresponding capability.
The introduction of LiveBench [20] represents an attempt to address this problem. LiveBench contains questions based on recently released sources, updated on a monthly basis, to limit the window for contamination. Early results suggest that LiveBench provides more discriminative evaluation than static benchmarks, supporting the hypothesis that contamination contributes to benchmark saturation.
VI. THE CAPABILITY-PERFORMANCE GAP
We formalize the distinction between genuine capability gains and benchmark- specific optimization as follows.
Definition 3 (Capability-Performance Gap): Let M_t be a sequence of models indexed by development iteration t. Let B(M_t) be performance on benchmark B and let C(M_t) be the true capability that B is designed to measure. The capability-performance gap at iteration t is:
G(t) = B(M_t) - C(M_t)
When G(t) = 0, benchmark performance perfectly tracks capability. When G(t) > 0, the model's benchmark score overstates its capability. The observer effect predicts that G(t) is monotonically increasing in t for benchmarks that serve as optimization targets.
This gap has practical consequences. A model with a large capability- performance gap will appear superior on benchmarks but underperform in deployment. The gap also corrupts the feedback loop of research: if we cannot accurately measure capability, we cannot make informed decisions about which research directions to pursue.
Chollet [16] proposed a related concept in defining intelligence as skill- acquisition efficiency rather than skill itself. A system that achieves high skill through extensive exposure to a specific task distribution is less intelligent, in Chollet's framework, than a system that achieves the same skill with less exposure. The capability-performance gap captures this distinction in the evaluation context: a model with high benchmark performance acquired through extensive benchmark-specific optimization has a larger gap than a model that achieves the same performance through general capability.
Proposition 1: Under the observer effect, the capability-performance gap G(t) grows at a rate proportional to the optimization pressure on benchmark B, the expressiveness of the model class, and the inverse of the benchmark's coverage of the target capability.
This proposition follows from three observations. First, greater optimization pressure produces more benchmark-specific adaptation. Second, more expressive model classes have more capacity for memorization and format specialization. Third, benchmarks that cover a smaller fraction of the target capability are more susceptible to gaming, because high scores can be achieved through optimization on the covered subset without generalizing to the uncovered portion.
We identify four approaches to evaluation that mitigate the observer effect. Each addresses a different aspect of the problem, and we expect that robust evaluation will require combining multiple approaches.
The most direct defense against data contamination is to generate benchmark items after the model's training cutoff. LiveBench [20] implements this approach by constructing questions from recently published sources. The ARC-AGI benchmark [16] takes a related approach by using novel visual reasoning tasks that are unlikely to appear in training data.
The limitation of held-out generation is that it addresses only the data contamination component of the observer effect. Format specialization and metric optimization can still distort performance on held-out tasks.
Adversarial benchmarks are designed to resist optimization by construction. Rather than fixing a test set, they generate items that are adversarially targeted at the model being evaluated. This approach ensures that high performance requires genuine capability rather than memorization or pattern matching.
The ARC-AGI benchmark [16] approximates this approach by using tasks that require novel abstraction. Each task presents a small number of input-output examples, and the model must infer the underlying transformation. Because the transformations are unique to each task, memorization of specific solutions provides no advantage.
Hsia et al. [21] demonstrated that Goodhart's Law applies even to metrics designed to measure explanation quality, showing that adversarial pressure can compromise evaluation at multiple levels. This finding suggests that adversarial benchmark design must account for the full stack of evaluation metrics, not just the task items themselves.
A fundamental reorientation of evaluation involves shifting from task completion to capability probing. Task completion measures whether a model produces the correct output for a given input. Capability probing measures whether a model possesses the underlying competence that enables task completion across diverse settings.
Chollet [16] articulated this distinction in arguing that intelligence should be measured as skill-acquisition efficiency across a broad scope of tasks. Applied to benchmark design, this implies that evaluation should measure how quickly a model can adapt to novel task variations, rather than how well it performs on a fixed set of tasks.
The distinction between capability probes and task completion has implications for benchmark structure. A capability probe might present a novel problem format and measure the model's ability to infer the format from examples, rather than testing whether the model has memorized solutions to problems