PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments

Rohith Reddy Bellibatlu, Zichong Wang, Wenbin Zhang

arXiv:2609.37315v1cs.SEcs.AIcs.LG

TL;DR

Benchmarks for tool-using agents assume tools did what their interfaces advertised, while surveyed audit taxonomies do not name that assumption. This paper checks advertised tool contracts against implementations and traces defects to evaluator verdicts. Across four benchmarks it confirms seven tool defects and one evaluator property, while its findings also show important limits in checker recall and audit scope.

  • Problem

    Tool-benchmark scores assume each call did what its interface advertised, but surveyed audit taxonomies have no category for a success signal decoupled from the state it claims to have written.

    Source: §I · p. 1

  • Method

    The paper authors provenance-bound executable contracts from advertised tool surfaces, checks implementations, and traces defects through task and evaluator code to affected verdicts.

    Source: §I · pp. 1–2

  • Results

    Across four shipped benchmarks, the audit confirmed seven tool defects and one evaluator property; at least 5 of AgentDojo’s 25 mutating tools diverged from their advertised surface.

    Source: §I · p. 2, §VI · p. 10

  • Takeaways & Limitations

    Maintainers can ground write graders in stored state, audit tool implementations as well as tasks and gold solutions, and disclose deliberate simplifications where agents can see them.

    Source: §VI · p. 10

    Because benchmarks were selected on known defects or to test generalization, the audits do not support prevalence claims across benchmarks.

    Source: C. Threats to Validity · p. 10

Abstract

from arXiv · show

Tool-using agents are entering settings where a wrong action carries real cost, and the benchmarks certifying them grade what each simulated tool call reports having done, assuming the tool did what its interface advertises. The audit taxonomies we survey publish no category for that assumption, and a defect beneath a score is present on every rerun. We treat a tool's advertised surfaces as an executable contract, check the implementation against it, and trace each score's provenance through the task files and evaluator code to the verdicts that derive from state a defective tool should have written. Across 34 audited mutating tools in four benchmarks we confirm seven tool defects and one evaluator property at pinned commits. On injected defects the checker raised no false positive in 25 flags, flagged 2 of 5 negative controls, and missed most: in 29 of 33 scored misses a clause covered the defect but no probe revealed it. The checker's own static half, run alone, flags 14 of 17 confirmed sites, so on these findings the dynamic half confirms and traces rather than discovers. Twelve further AgentDojo tools, with six held-out tools and the seven audited first, complete its 25-tool mutating surface, on which at least 5 tools diverge from their advertised surface as our contracts read it, a rate for AgentDojo alone. No gold trajectory reaches either tau2-bench defect; on 1,120 paths built to isolate the telecom defect, a number fixed by construction, the evaluator rewards a refuel of a suspended line and fails the repaired tool. The clearest case is a clinical benchmark whose tool tells the agent each write executed under a documented no-write design its interface does not disclose; its grader takes that message as evidence, so its action success rate records whether a request carried the expected payload, not whether any record changed.

I. INTRODUCTION

Tool benchmarks rely on tool reports as evidence of execution, but existing audit taxonomies do not name the gap between a reported success and the state actually written. This paper audits that tool layer and traces defects to affected verdicts.

I. INTRODUCTION

  • Benchmark grades assume tools did what their interfaces advertised, so leaderboard positions and model-selection decisions inherit that assumption.The paper identifies this as consequential because agents are entering settings where mistaken actions carry real cost.Source: §I · p. 1
  • The surveyed audit schemas contain 27 categories but no category for a success signal decoupled from the state it claims to have written.The surveyed audits examine task artifacts, environment configuration, graders, and judges, [3], [4], [5].Source: §I · p. 1
  • MedAgentBench tells agents every POST executed although its documented design performs no write, and its write-task grader treats that message as evidence.The defect is the undisclosed agent-visible success claim, not the deliberate no-write design.Source: §I · p. 1
  • The audit authors executable contracts from advertised tool surfaces, checks implementations, and traces defects through evaluator code to verdicts exposed to them.It distinguishes deliberate simplification from defect by whether the simplification is disclosed to the agent.Source: §I · pp. 1–2
  • Four shipped benchmarks yielded seven confirmed tool defects and one evaluator property; across AgentDojo’s 25 mutating tools, at least five diverged from their advertised surface.The four-benchmark findings were confirmed at pinned commits; the AgentDojo rate applies to that benchmark alone.Source: §I · p. 2

II. RELATED WORK

Related work evaluates benchmark ground truth, audits task and verdict layers, and develops contract-checking methods. This paper places its audit beneath those layers, at the tool’s state transitions.

A. What a Benchmark Treats as Ground Truth

  • Benchmarks grade either post-call state or returned results, but both kinds of scores still depend on the tool layer.Examples of state-based grading include WebArena, AppWorld, OSWorld, AndroidWorld, tau2-bench, and AgentDojo; MedAgentBench [6] grades writes from the request.Source: A. What a Benchmark Treats as Ground… · p. 2

B. Audits of Benchmarks, and the Layer Each Opens

  • Prior audits open task artifacts, verdicts, agent claims, or reported practice, while the closest works leave tool state transitions unopened, [3], [4], [5].Agent-Diff reads a replica-API state delta, whereas Tool-Veritas [4] audits verdicts in the tau2-bench family.Source: B. Audits of Benchmarks, and the Layer… · p. 2

C. Contract Checking, and What Changes to Point It Here

  • Contract checking has established methods,,,,,,,, but agent-tool work does not audit implementations behind their interfaces,,,.ToolFuzz tests argument handling and runtime or response oracles, not state-transition predicates or traces into benchmark evaluators.Source: C. Contract Checking, and What Changes… · p. 2

III. NOTATIONS AND PRELIMINARIES

The paper defines executable defect rules over tool calls and distinguishes undisclosed divergence from advertised simplification. It also separates tool-level defects from an evaluator property and describes which benchmarks use the full audit pipeline.

III. NOTATIONS AND PRELIMINARIES

  • The six defect classes are executable rules over pre-state, post-state, arguments, and results; the classes are a checking discipline, not a partition of all defects.Phantom Effect is checked only when a contract opts into a biconditional success signal, and Finding 5 satisfies both Ignored Argument and Phantom Effect by source reading.Source: §III · p. 2
  • A simplification is benign exactly when advertised; the defect is the undisclosed divergence between the agent-facing interface and implementation.Either side of that divergence may be repaired, including the advertised text.Source: §III · p. 3
  • MedAgentBench’s write graders exhibit Ungrounded Oracle: they decide a state change from the actor’s claim rather than the state.This is an evaluator-layer property, not a seventh tool-boundary defect class.Source: §III · p. 3

IV. METHOD

  • tau2-bench and AgentDojo take all three method stages, while MedAgentBench’s contracts and grader traces are read by hand from source.The stages are contract authoring, checker execution through an adapter, and tracing into evaluators.Source: §IV · p. 3, Figure 2 · p. 3

A. The Contract Specification

The paper turns advertised tool surfaces into per-tool executable contracts and checks implementations against them. Contract-driven probes and static flags produce evidence that people review against the pinned source before calling a defect a finding.

A. The Contract Specification

  • Each tool contract records preconditions, effects, frame clauses, effective arguments, invariants, and a success signal, with provenance for each clause.The refuel_data example derives an active-line precondition from the tool's docstring.Source: A. The Contract Specification · p. 3
  • A suspended-line refuel_data call violates the advertised active-line precondition but adds 40.0 GB and creates an $80.00 draft bill.The enforcing branch is commented out, and the call returns a success message.Source: A. The Contract Specification · p. 3
  • People or annotators write clause predicates; draft placeholders using the constant True are excluded from counts until a person writes the predicate.The validator checks predicate syntax, provenance quotes, and whether cited tool-return lines are returns or raises.Source: A. The Contract Specification · p. 4
  • Contracts distinguish agent-visible, maintainer-annotated, and author-assigned clauses; only agent-visible clauses qualify a benchmark-class cell for headline reporting.Author-assigned clauses have no advertised surface behind them and are marked inferred.Source: A. The Contract Specification · p. 4

B. The Conformance Checker

  • The static half flags five code patterns as candidates for human review, but its flags are not verdicts.Examples include a commented-out guard and a state-writing function with a literal success return.Source: B. The Conformance Checker · p. 4
  • Dynamic probes use fresh environments and snapshots to test preconditions, effects, frames, effective arguments, reset behavior, and invariants.Probe values come from the pre-snapshot; invariants are checked after a legal call sequence.Source: B. The Conformance Checker · p. 4
  • 96 of 180 tau2-bench clause rows were UNTESTABLE because no probe reached the state; any clause violation makes the tool-level verdict VIOLATES.Clause verdicts are CONFORMS, VIOLATES, or UNTESTABLE; untestable clauses remain in every clause denominator.Source: B. The Conformance Checker · p. 4

C. Score-at-Risk as a Provenance Trace

The trace links a confirmed tool defect to the task verdicts whose derivations depend on the state that tool should have written. It distinguishes exposure, trajectory reach, and verdict change rather than treating them as interchangeable.

C. Score-at-Risk as a Provenance Trace

  • The trace runs backward from a confirmed defect to verdicts whose evaluator derivations use fields the defective tool should have written.It matches defect-to-field predicates against evaluator read sites, then enumerates tasks whose derivations include an intersecting field.Source: C. Score-at-Risk as a Provenance Trace · p. 4
  • The ledger records whether an evaluator reads the written attribute, only its collection, a whole-database hash, or an unresolved basis.Rows are tagged exact_field, collection_only, whole_state_hash, or unresolved.Source: C. Score-at-Risk as a Provenance Trace · p. 4
  • Exposure means the evaluator reads a field named by the defect-to-field step; reach means a trajectory calls the defective operation under its trigger condition.A verdict change is the evaluator answering differently under a conforming tool; exposure alone does not establish either reach or change.Source: C. Score-at-Risk as a Provenance Trace · pp. 4–5
  • Unresolved evaluator reads are forced into exposure, so the fallback can widen the population but cannot narrow it.The extractor resolves direct attributes, dotted state paths, and bare-name helpers up to three hops; other read forms are unresolved.Source: C. Score-at-Risk as a Provenance Trace · p. 5

V. EXPERIMENTS

The experiments audit mutating tools across four shipped agent benchmarks, with a defined anchor-audit sample and separate AgentDojo populations. The paper also specifies how findings, coverage, and reproducibility are counted.

A. Experimental Setup

  • The anchor audit covers 34 mutating tools across MedAgentBench, tau2-bench, AgentDojo, and MM-ToolSandbox at pinned commits.Tools enter scope through a benchmark mutates_state flag, a read of registered tool bodies for writes, or a hand-read advertised surface; MedAgentBench’s write path enters by advertisement alone.Source: A. Experimental Setup · p. 5
  • The anchor audit includes all 3 MedAgentBench tools and 19 tau2-bench airline, retail, and telecom tools, but only 7 of AgentDojo’s 25 mutating tools.It includes 5 MM-ToolSandbox tools and does not complete that benchmark’s mutating surface; its eight no-finding tools were hand-chosen rather than sampled by a prior rule.Source: A. Experimental Setup · p. 5
  • AgentDojo’s complete 25-tool surface combines 7 anchor-audit tools, 6 held-out tools, and 12 blind-audit tools.The held-out tools were fixed by suite in a plan before audit; the blind population was the remaining v1 mutating surface, with 11 tools lacking prior knowledge.Source: A. Experimental Setup · p. 5
  • The ledger distinguishes a clause failing on a probe from a confirmed defect instance and from a benchmark-class reporting cell.Its 12 VIOLATES rows span seven tools: six confirmed instances and one unadjudicated suspend_line candidate; MedAgentBench’s two instances have no adapter rows.Source: A. Experimental Setup · p. 5
  • The anchor findings contain 11 scanner sites; held-out and blind audits bring the reported total to 17 sites.The anchor count includes six sites for Finding 5; the held-out second instance adds four, and two blind calendar tools add one each.Source: A. Experimental Setup · p. 5
  • Experiments were registered in a committed plan before producing results, and the artifact supports offline reproduction without API keys or model calls.The paper says the reproduction target re-derives tables and audits numeric claims from the committed score-at-risk ledger and pinned clones.Source: A. Experimental Setup · p. 5

B. Main Results

The main results report headline-eligible defects across the audited benchmarks, with MedAgentBench illustrating a write-success signal disconnected from a state change. The section organizes results around five audit questions.

B. Main Results

  • The results are arranged around five questions: findings, score exposure and verdict change, the checker’s added value, checker reliability, and contract-authoring stability.The questions are addressed in that order.Source: B. Main Results · pp. 5–6

1) Q1, what the audit finds:

  • Eight confirmed instances occupy seven benchmark-class cells, alongside an unadjudicated candidate as an eighth cell; the dynamic checker reproduces three of four headline cells.The three reproduced cells are tau2-bench’s Unenforced Precondition and AgentDojo’s and MM-ToolSandbox’s Ignored Argument; MedAgentBench’s finding is based on hand reading.Source: 1) Q1, what the audit finds · p. 6
  • MedAgentBench’s grader accepts the tool’s success string as evidence, while the write branch leaves clinical records unchanged.The paper reports that Action SR measures whether POST requests carry the expected payload, not whether a clinical record changes, and makes no causal claim about any particular published score.Source: 1) Q1, what the audit finds · p. 6

2) Q2, which verdicts are exposed, and whether any changes:

The trace finds score exposure without showing that shipped gold trajectories reach the defects. Constructed paths isolate the telecom precondition defect and show the shipped evaluator rewarding the defective behavior while failing the repaired tool.

2) Q2, which verdicts are exposed, and whether any changes:

  • Airline exposure includes 50 tasks measured by whole-database hashes, but the constructed booking paths do not produce verdict changes.The hash already differs from the gold database in both arms, so the evaluator’s coarseness, not the defect, explains those failures.Source: 2) Q2, which verdicts are exposed, and… · pp. 6–7
  • Zero action verdicts in MedAgentBench read a write back from FHIR state; its write graders use the transcript instead.Of 150 action tasks, 60 are graded unconditionally from the transcript and 90 gate on live state but still grade the write from the transcript; exposure is undefined or indeterminate, not zero.Source: 2) Q2, which verdicts are exposed, and… · p. 6
  • None of the ten replayed gold trajectories reaches either tau2-bench defect, so none changes verdict.The five sampled trajectories per finding used gold reference solutions because agent trajectories were unavailable offline.Source: 2) Q2, which verdicts are exposed, and… · p. 6
  • 1,135 of 2,285 tau2-bench telecom tasks derive verdicts from fields the defective precondition lets through, including 1,120 exact-field reads and 15 collection-only reads.These counts are exposure, not reach or verdict change; the full generated task file contains 2,285 tasks and differs from the roughly 114-task subset evaluated in the benchmark paper.Source: 2) Q2, which verdicts are exposed, and… · p. 6, Table V · p. 7
  • 1,120 constructed telecom paths earn full reward with the unpatched tool and fail with the one-hunk repair.Each path inserts suspend_line before the gold refuel_data call and is built from a gold-call task; this is an existence result fixed by construction, not an estimate of how often agents take such paths.Source: 2) Q2, which verdicts are exposed, and… · pp. 6–7, Table VI · p. 7
  • Under the alternative construction, 312 shipped verdicts change, while 808 tasks fail even with the defect because suspension trips unrelated user-side assertions.The second construction resumes the line after refueling to remove that confound.Source: 2) Q2, which verdicts are exposed, and… · pp. 6–7

3) Q3, what the checker adds beyond reading source:

Beyond source reading, the checker confirmed a defect in previously unflagged AgentDojo code; on the audited findings, however, its dynamic half confirmed and traced rather than discovered. The completed AgentDojo surface audit found at least 5 of 25 mutating tools diverged from their advertised surfaces, with three untestable.

a) Static scanner

  • Seven of eight Table IV instances surfaced through manual audit; the static half found Finding 8, while the dynamic checker found none.On these findings, the dynamic checker’s role was confirmation against executed state and tracing.Source: 3) Q3, what the checker adds beyond… · p. 7
  • The static scanner raised 23 flags—14 true positives and 9 false positives—and flagged 14 of the 17 confirmed sites.It missed Finding 5’s success return and both blind calendar sites.Source: 3) Q3, what the checker adds beyond… · p. 7

b) Blind audit

  • cancel_calendar_event violated its advertised cancellation effect because listing tools still returned canceled events, the checker’s first defect in previously unflagged code.The docstring promised canceled events would no longer appear, but the implementation only changed status.Source: 3) Q3, what the checker adds beyond… · p. 8

c) Held-out audit

  • Among six held-out tools, update_user_info violated four agent-visible effect clauses, repeating a truthiness-guard defect class rather than adding a new cell.Five held-out tools conformed; three of 43 clause rows were untestable.Source: 3) Q3, what the checker adds beyond… · p. 8

d) The complete AgentDojo surface

  • At least 5 of AgentDojo’s 25 mutating tools diverged from their advertised surfaces; three were untestable, so this is a lower bound, not a prevalence estimate.The remaining 17 conformed under a low-recall procedure, which does not establish absence of divergence.Source: 3) Q3, what the checker adds beyond… · p. 8

4) Q4, how reliable the checker is:

The checker was precise on injected defects but missed most scored cases, chiefly when clauses covered a mutation but probes did not expose it. Open-world escape results are limited to the toy domain and the one rule tested.

Q4 evaluation design

  • The mutation-testing arms evaluate the contract, probes, and checker together, with closed-world results limited to written clauses and open-world results bounding pipeline misses.Recall intervals use design-effect adjustment for clustering by tool; pooled precision and escape rates are unclustered.Source: 4) Q4, how reliable the checker is · p. 8

a) Closed-world recall

  • 29 of 33 real-tool misses had a covering clause, but no probe revealed a behavioral difference; only two misses lacked clause coverage.The other two were a probe gap and a target not found; M-INVAR’s zero means no probe reached a mutated invariant path.Source: 4) Q4, how reliable the checker is · p. 8

b) Precision

  • Across 25 injected-mutant flags, the checker produced 25 true positives and zero false positives.The closed-world precision result was reported alongside 24 semantics-preserving controls.Source: 4) Q4, how reliable the checker is · p. 8

c) Negative controls

  • Two of five negative controls were flagged, both as false positives of the Ignored Argument rule.Whole-post-state equality could not distinguish an effective argument with identical effects across tested values from an ignored argument.Source: 4) Q4, how reliable the checker is · p. 9

d) Open-world escape rates

  • 0.956 was the Wilson lower bound for the missed share of 225 behaviorally live toy-domain mutations; the point estimate was 0.983.Only the mechanical declared-volatile-field rule ran, so this bounds escape under that rule rather than the whole pipeline.Source: 4) Q4, how reliable the checker is · p. 9

5) Q5, whether contract authoring is stable:

Blind contract authoring was highly consistent, but the audit’s prevalence and recall conclusions remain bounded by anchor-driven selection, probe coverage, and unresolved evaluator reads.

5) Q5, whether contract authoring is stable:

  • 98 of 99 blind probe comparisons agreed; the sole disagreement concerned an ISO-string annotation, and a post hoc filter yielded 98 of 98.The blind stratum covered four of five tools; the filtered comparison excluded the reserve_car_rental arg.end_time disagreement.Source: 5) Q5, whether contract authoring is… · p. 9

C. Threats to Validity

  • The six-class taxonomy is incomplete, and absent recorded real calls leave probe escape rates unusable and may explain 29 of 33 closed-world misses.The frame-path grammar cannot exclude a single key, so some advertised effects cannot be fully expressed.Source: C. Threats to Validity · p. 9
  • The checker’s frozen rules leave known false positives and argument gaps unrepaired, while unchecked absorbed crashes bound real-tool recall.The two flagged controls do not establish precision on shipped tools; whether an absorbed crash counts as detected or silent was not checked.Source: C. Threats to Validity · p. 9
  • The authors do not claim that preregistration preceded the decisions it was meant to freeze, because the first commit imported the project as one tree.Later commits establish that the checker freeze preceded scored mutants and that experiment plans preceded their recorded trajectories or results.Source: C. Threats to Validity · p. 9
  • The extractor’s second reader was a model session, not a human; its 20 unresolved rows remain fallback output rather than a measurement.For 22 functions the extractor found a strict superset of the model’s fields, while six travel functions were read only partially.Source: C. Threats to Validity · pp. 9–10
  • Anchor-driven benchmark selection supports existence proofs, not prevalence claims; AgentDojo’s blind audit is the only evidence of a newly found defect.The blind audit found one defect in eleven tools; elsewhere, findings were anchor-driven.Source: C. Threats to Validity · p. 10

VI. CONCLUSION

The audit finds benchmark tool-layer defects and a measurable AgentDojo divergence rate, while treating other audits as existence proofs rather than prevalence estimates. It recommends grounding write graders in stored state and disclosing deliberate simplifications to agents.

VI. CONCLUSION

  • At least 5 of AgentDojo’s 25 enumerated mutating tools diverge from their advertised surface, a rate for that benchmark alone.The anchor-driven audits are existence proofs, not prevalence estimates.Source: §VI · p. 10
  • A CONFORM verdict does not establish conformance because the checker has low recall, and exposure counts flag due diligence rather than correct published scores.The authors distinguish the AgentDojo rate from anchor-driven findings in other benchmarks.Source: §VI · p. 10
  • Maintainers can ground write graders in stored state, audit tool implementations as well as tasks and gold solutions, and disclose simplifications where agents can read them.Source: §VI · p. 10
  • Three teams received findings, reproductions, and repairs; the held-out and blind AgentDojo findings were later filed on its issue thread.The authors report responses and non-response under a rule fixed in advance, without inferring from silence.Source: §VI · p. 10
Loading 2609.37315v1…