PaperlayerView exact revision on arXiv

Source-linked AI summary

When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark

Junjie Chen

arXiv:2610.16067v1cs.SEcs.AI

TL;DR

Agent benchmarks treat reward functions as definitive, although the harness itself produces the reported scores. This paper audits oracle integrity and examines configuration-dependent measurement; replay correction substantially changes the benchmark’s database-pass labels and measured success.

  • Problem

    Agent benchmarks often treat reward functions as definitive, leaving the software that produces evaluation scores unaudited.

    Source: §1 · p. 3

  • Method

    The paper tests oracle integrity with state invariants, known-answer controls, and injected defects, then analyzes configuration effects on the audited pipeline.

    Source: §1 · p. 3, §3.2 · p. 8

  • Results

    Across 4,320 episodes, replay correction changed database-pass labels from 4,194 to 2,883 and measured success from 94.3% to 65.5%.

    Source: Contributions · p. 4, The correction · p. 13

  • Takeaways & Limitations

    The results show that an evaluation harness can alter measured outcomes, so the harness itself is a relevant object of reliability analysis.

    Source: §1 · p. 3

    The exploratory component contrasts do not establish a definitive ranking because they lack multiplicity correction and have limited power for the reported ordering.

    Source: §4.2 · p. 15

Abstract

from arXiv · show

Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such software for reliability defects that score tables cannot reveal. We introduce an oracle-integrity auditing approach with five state-integrity invariants and three known-answer controls. We evaluate it by injecting seven oracle defect classes across two task domains and a synthetic domain. All injected defect classes are detected, and the repaired oracle has no false positives. Applied to an agent benchmark, the audit reveals a reward-function defect that changes the environment state before post-run scoring. Across 4,320 episodes, replay-based correction changes database-pass labels from 4,194 to 2,883 and reduces measured success from 94.3% to 65.5%. We also study configuration-dependent measurement effects. A component ablation over 3,240 episodes shows that the agent's policy documents, tool descriptions, and tool error strings influence observed trajectory reliability, with tool descriptions showing the largest exploratory effect (-9.2 pp) and error strings the smallest (-5.3 pp). A second task domain reverses the sign of the language effect, so the result is bounded to the tested configuration. Finally, we report a portability boundary: the auditing approach cannot validate a second independently implemented benchmark because the required reference terminal state cannot be reconstructed from its recorded API-call log. The paper releases a replication package containing the auditing suite, fault-injection catalogue, frozen records, deviation logs, and offline analysis entry points without network access, API keys, or model calls.

1 Introduction

Agent benchmarks treat reward functions as definitive, although the harness itself produces the reported score and can make configuration changes look like agent differences. This paper audits the harness as software and distinguishes oracle reliability from measurement-process analyses.

1 Introduction

  • Because benchmark reward functions are treated as defining success rather than as software to test, oracle errors can remain invisible and harness changes can be mistaken for agent changes.The paper treats the harness as the system under test and frames configuration studies as measurement analyses, not claims about intrinsic model behavior.Source: §1 · p. 3

Contributions.

  • The audit combines five state-integrity invariants, three known-answer controls, and seven injected defect classes; every injected class was detected, with no false positives on the repaired oracle.The approach was evaluated on the implementations studied, not claimed as a general-purpose framework.Source: Contributions · p. 3
  • Across 4,320 episodes, replay correction changed database-pass labels from 4,194 to 2,883 and measured success from 94.3% to 65.5%.The reward function changed database state before a separate post-session scorer read it.Source: Contributions · p. 4
  • In a 3,240-episode component ablation, all three localized documents showed protective effects, with tool descriptions largest; a second domain reversed the language effect.The component ordering is exploratory, and the effect is bounded to the tested configuration.Source: Contributions · p. 4
  • The paper releases frozen records, audit materials, and an offline analysis entry point, and reports a portability boundary on a second independently implemented benchmark.The required reference terminal state could not be reconstructed from that benchmark’s recorded API-call log.Source: Contributions · p. 4

What we do not claim.

  • The authors claim a systematic audit evaluated on the reported implementations, not a general-purpose framework.The study includes one agent benchmark, a second task domain, a synthetic domain, and one independently implemented benchmark where the method did not transfer.Source: What we do not claim · p. 4

2 Background and Related Work

The background frames evaluation reliability as a software-testing and measurement-validity problem: an oracle may be stateful, and outcomes can respond to the apparatus rather than the measured property. The paper extends these concerns to agent benchmarks and the reproducibility of their evaluations.

2.1 Test oracles, mutation testing and the oracle problem

  • Mutation testing injects faults to measure whether a test suite detects them; this paper directs that technique at the oracle rather than the program under test.A stateful oracle can return a correct value yet leave the environment in an unexpected state, while an implicit reader contract can make a later read wrong.Source: §2.1 · p. 5

2.2 Flaky tests, measurement validity and configuration sensitivity

  • A measurement that shifts with its apparatus reflects measurement validity, and this paper treats configuration-dependent changes as harness sensitivity rather than agent capability.The paper also considers trajectory degeneration, not only tests that return inconsistent verdicts on unchanged code.Source: §2.2 · p. 5

2.3 Testing and evaluating LLM-based systems

  • Where other work treats the language model as the system under test, this paper tests the evaluation harness, with the model serving as input generator and the agent as the measured system.A reward-function fault can be silent when the reward function is treated as definitional.Source: §2.3 · pp. 5–6

2.4 Agent benchmarks and the reliability of their measurements

  • Prior agent-benchmark work examines noisy scores, shifting rankings, hidden run behavior, and benchmark validity; this paper instead asks whether the score-producing instrument is correct.The passage situates this question alongside work on benchmark validity and reproducibility.Source: §2.4 · p. 6

2.5 User simulators as the measuring instrument

  • User-simulator presentation and language can move measured outcomes without moving capability; this paper complements that literature by localizing language effects to apparatus components and run reliability.The paper considers whether the manipulation changes success rate or only run reliability.Source: §2.5 · p. 6

2.6 Replication and artefact evaluation

  • Replication practice motivates frozen inputs and explicit verification status, while the paper’s figure frames the harness—not the model—as the measuring instrument.The package distinguishes verifiable sources from absent archives rather than treating absence as a pass.Source: §2.6 · pp. 6–7, Figure 1 · p. 7

3 Methodology

The methodology treats the evaluation harness as software under test, then audits the oracle and studies configuration effects using pre-registered episode records. It also specifies paired task-level inference and an offline replication package.

3.1 The evaluation harness as a system under test

  • The benchmark oracle grades a tool-calling agent by combining a terminal database-state comparison with required substrings in its final reply.Because the oracle replays reference actions against mutable state, its state effects matter to later scoring.Source: §3.1 · pp. 7–8

3.2 Oracle-integrity testing

  • The audit uses five state-integrity invariants and three known-answer controls through a four-operation interface, then tests sensitivity with seven one-dimension oracle defects.The checks cover mutation, repeatability, order dependence, exception recovery, session termination, and expected verdict direction.Source: §3.2 · pp. 8–9

3.3 Benchmark, task suite and configuration space

  • Configuration experiments manipulate simulator language and the language of the agent’s policy, tool descriptions, and tool error strings while keeping database contents untranslated.The environment-language manipulation includes single-component and length-padded forms, with a gate checking the translated and untranslated documents.Source: §3.3 · p. 9

3.4 Design and pre-registration

  • The pre-registered main matrix contains 4,320 episodes; later frozen experiments add component ablation, a second task domain, and adjudication, with pilot-driven deviations logged.The second-domain experiment dropped a cross-vendor arm after no collapses in its 120-episode pilot; adjudication became a census of 106 archived episodes.Source: §3.4 · pp. 9–10

3.5 Measurement and statistics

  • Inference pairs conditions within task and uses task-clustered intervals; all scores use corrected grading, and intervals containing zero are reported as no detected difference.Archived state labels were reconstructed by replay while the textual component remained as archived.Source: §3.5 · p. 10

3.6 Replication package

  • The replication package includes audit materials, frozen episode records, preregistration and deviation logs, and offline analysis entry points that require no network, API keys, or model calls.Some benchmark task data and raw provider responses are not redistributed; recomputation is deterministic given the frozen records.Source: §3.6 · pp. 10–11

4.1 RQ1 (primary): How reliable is the evaluation oracle?

Replay-based correction changes database-pass labels from 4,194 to 2,883 and measured success from 94.3% to 65.5%. The section also reports missed injected combinations, which checks are load-bearing, and how the benchmark oracle altered state before post-run scoring.

The suite and its validation.

  • The eight checks were run on retail, airline, and synthetic targets; the shipped repaired copy passed every check, while INV-5 caught the pre-repair reproduction in both benchmark domains.The retail and airline domains use the same scoring implementation; the synthetic domain was written from scratch.Source: The suite and its validation · p. 11

What the class-level summary hides.

  • 54/57 behaviorally distinct variant–target combinations were detected; counting six unresolved combinations instead gives 56/63 on the conservative denominator.The two denominators answer different questions and should not be mixed.Source: What the class-level summary hides · p. 12
  • An exception-swallowing oracle evaded INV-4 on all three targets because its probe path became inapplicable; the class-level table can conceal this miss.The table caption notes that one saturated-oracle variant on each target triggered no check.Source: What the class-level summary hides · p. 12, Table 1 · p. 13

Which checks are load-bearing.

  • Four checks were necessary for at least one defect class (INV-3, CTL-1, INV-4, CTL-3), while the other four had zero unique detections.State invariants miss state-clean but wrong oracles; known-answer controls miss state-dirty but accidentally right ones.Source: Which checks are load-bearing · p. 12

What the suite found on the benchmark.

  • The reward oracle reset the environment database without restoring it, so post-session scoring reread the reference state; INV-5 detects this terminal-state instability.A no-write goodbye episode passed on archived code but not after repair.Source: What the suite found on the benchmark · pp. 12–13

The correction.

  • Under corrected labels, archived database passes decrease from 4,194 to 2,883, while overall success falls from 94.3% to 65.5%; all reported numbers below are computed under the correction.Source: The correction · p. 13

4.2 RQ2 (secondary): How do harness configurations influence observed evaluation outcomes?

The component ablation finds protective effects across the policy, tool descriptions, and error strings, but their effects are non-additive and their ordering is exploratory. Length controls do not resolve a length effect, while the language effect reverses in the airline domain, bounding the result to the tested configuration.

4.2 RQ2 (secondary): How do harness configurations influence observed evaluation outcomes?

  • The component arms localised one document at a time into Chinese while holding the other two in English, across 3,240 episodes.Comparisons were paired within task, simulator, user-language, and repetition cells; the archived fully English and fully Chinese arms were reused.Source: §4.2 · pp. 13–14
  • Localising each component reduced collapse, but the effects did not add; tool descriptions had the largest exploratory effect and error strings the smallest.The tool-description arm was −9.17 pp and the error-string arm −5.28 pp; the component contrasts were exploratory and were not corrected for multiplicity.Source: §4.2 · pp. 14–15
  • The tool-error boundary hypothesis predicted the error strings would carry the effect, but that component had the smallest effect.The authors report the earlier mechanism account and the ablation result as disagreeing.Source: §4.2 · p. 15

Length.

  • Padding the Chinese environment to English prompt length changed collapse by −0.46 pp and success by +0.24 pp, with no detected length effect at this resolution.Individual documents did not receive length-matched controls, so length is not fully excluded as a confound for the single-component arms.Source: Length · p. 15

A second domain reverses the sign.

  • In retail, collapse was 6.48% with the English simulator and 13.15% with the Chinese simulator; in airline, it was 9.67% and 4.33%, respectively.The difference between domain effects was −12.00 pp [−16.33, −7.67]; the authors bound the language finding to the retail configuration and do not attribute a direction to simulator language.Source: A second domain reverses the sign · pp. 15–16

4.3 RQ3 (secondary): How sensitive are the conclusions to execution factors, and where do they stop holding?

The second framework could not be validated because an external harness could not reconstruct the reference terminal state needed for a positive control. A separate language-model adjudication closely agreed with the mechanical collapse criterion, but the evidence remains bounded by its selected sample and non-human judge.

The implementation boundary.

  • The independent framework’s API-call log could not reconstruct the reference terminal state needed for a positive control, so sensitivity and specificity could not be estimated.Its exposed fixture also failed as a positive control: three apparently usable held-out tasks were false positives from the same state-reconstruction defect.Source: The implementation boundary · p. 16

An independent language-model adjudication reproduces the collapse label.

  • A blinded judge returned parseable verdicts for 210 of 212 archived items and agreed with the mechanical criterion on 98.1%.The archive contained collapsed episodes paired with matched non-collapsed controls; Cohen’s κ was 0.962, with sensitivity and specificity each 98.1%.Source: An independent language-model… · p. 17
  • Agreement was 98.6% in the English arm and 97.9% in the Chinese arm, but this is not accuracy against human ground truth.The judge was a language model and the sample consisted of archived positives plus matched negatives, not a deployment-distribution sample.Source: An independent language-model… · p. 17

5 Discussion

The discussion treats the oracle defect as ordinary software state mutation and points to inexpensive checks that can detect it. It also cautions against reading configuration-dependent trajectory reliability as model capability, recommends testing mixed configurations, and describes replay’s narrow preconditions; two negative findings are reported without reconciliation.

An evaluation oracle is software, and it needs the tests other software gets.

  • Repeat-scoring and terminal-state checks would catch the reported state-mutating oracle defect without a reference implementation, model call, or human review.The first check requires the same verdict on repeated scoring; the second checks that the post-session environment still holds the state left by the agent.Source: An evaluation oracle is software, and… · p. 17

What a cross-lingual score table does not say.

  • The retail configuration left aggregate score differences unresolved while trajectory degeneration roughly doubled, and airline showed the opposite direction.The discussion cautions that a language-configuration comparison cannot be read as a model statement or assumed to have a stable instrument-bias direction across task suites.Source: What a cross-lingual score table does… · p. 18

Localisation is a configuration change, and it should be tested as one.

  • Localising only the user-facing side while leaving policy, tool descriptions, and error vocabulary in English creates a configuration distinct from both fully English and fully localised setups.The discussion recommends testing that mixed configuration rather than assuming it lies between the two endpoints.Source: Localisation is a configuration… · p. 18

Replay as the audit primitive.

  • Replay reconstructs the state for scoring from archived tool calls without regenerating model episodes, but requires re-executable calls and a resettable database.The provider offered neither a deterministic seed nor an immutable model version, so episodes could not be regenerated; replay was evaluated on one benchmark.Source: Replay as the audit primitive · p. 18

Reporting a negative result, and reporting a disagreement.

  • The paper reports both a refuted pre-registered mechanism hypothesis and a language effect that reversed in a second task domain.It presents the disagreement between the earlier mechanism account and new ablation data rather than constructing a unified explanation.Source: Reporting a negative result, and… · p. 18

6 Threats to Validity

The paper distinguishes measurement-process observations from claims about intrinsic model capability, then bounds those observations by internal, external, and conclusion-validity constraints. Its portability test also identifies a reference-state contract that another framework must expose for this auditing approach to transfer.

What RQ2 and RQ3 are not.

  • RQ2 and RQ3 characterize measurement on an audited pipeline, not intrinsic language-model capability or behavior.The collapse measure is configuration-specific; its token threshold is not language-neutral, and its agreement was assessed by a language-model judge at ceiling.Source: What RQ2 and RQ3 are not · p. 19

6.2 Internal validity

  • At temperature zero, repeated requests can yield different trajectories, while periodic environment rebuilds leave possible platform drift unexcluded.Comparisons are paired within cells and use task-clustered intervals; a longer-interval drift probe found no evidence of version drift.Source: §6.2 · p. 19
  • The component effects are supported, but their ordering remains configuration-specific: at least a day separates key batches, and the 1.11 pp contrast is most exposed to drift.The provider supplies no immutable model version, and the proposed same-period replication with reversed arm order was not run.Source: §6.2 · pp. 19–20
  • Two second-experiment arms were dropped after their pilot results, making the final grid a post-data design decision.Source: §6.2 · p. 20

6.3 External validity

  • The second benchmark could not be validated because its recorded API-call log did not permit reconstruction of the reference terminal state required by the positive control.No detection rate could be estimated on its held-out task set; portability is bounded to frameworks meeting this reference-state contract.Source: §6.3 · p. 20
  • The results come from one benchmark, two domains, four agents, and six simulator models served by one provider; the trajectory effect was observed for one agent.The Chinese-simulator/English-tool setting and sampled tasks constrain inference, and the second domain did not provide independent-model validation.Source: §6.3 · p. 20

6.4 Conclusion validity

  • Task-clustered intervals support claims about sampled task suites, not episodes, and remain conditional on the tasks sampled.Sensitivity analyses are not independent evidence, and an interval containing zero is reported as no detected difference, not evidence of no difference.Source: §6.4 · pp. 20–21

7 Conclusion

The conclusion treats agent scores as instrument readings and reports an oracle-state defect alongside configuration-dependent trajectory reliability. The declarations describe the archived materials, offline reproduction, and use and limits of the released records.

7 Conclusion

  • The oracle mutated the state it graded, inflating recorded success from 65.5% to 94.3%, while localization effects did not retain their sign across task domains.The paper characterizes language as a measurement configuration parameter, not a property of the model.Source: §7 · p. 21

Declarations.

  • The replication package includes frozen experiment records and offline analysis entry points; the oracle suite and analyses run without network access, credentials, or model calls.Source: Declarations · pp. 21–22
  • The study reports no funding or competing interests and used no human participants; released episode records have separate research-use terms and limits on large-scale redistribution or commercial use.Language models served as agents, user simulators, translator, and adjudicator, as well as assisting with drafting and analysis code.Source: Declarations · pp. 21–22

Related papers

Loading 2610.16067v1…