Source-linked AI summary
The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean
Yonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. López de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie
TL;DR
Agent benchmarks can mistake scaffold behavior and scorer criteria for model capability, leaving scores difficult to interpret. The paper formalizes this double confound and repairs it by de-scaffolding execution, scoring against seeded ground truth, and reporting reliability beyond the mean. On ComtradeBench, the joint intervention reveals model and robustness differences that the original evaluation obscured.
Problem
Agent benchmark scores may depend on scaffolds and scorers as well as models, leaving unclear when they support claims about model capability.
Method
The paper uses a measurement-theoretic framework and audit protocol that transfers execution-critical decisions to the model, applies seeded ground-truth scoring, and reports reliability statistics.
Results
The joint intervention transforms a nearly flat ComtradeBench leaderboard into a reliability spectrum and reveals reliability differences hidden by mean scores.
Takeaways & Limitations
Benchmark results should be interpreted with their scaffolding level, scoring criterion, and reliability profile rather than as model capability from a single mean.
Takeaways & Limitations
The spectrum rests on five tasks and ten seeds, with two tasks ceiling-saturated and a sixth excluded because its re-emission demand exceeds the submission budget.
Abstract
from arXiv · showhide
Agent benchmarks are increasingly used to compare large language models (LLMs) and guide deployment decisions, yet benchmark scores are meaningful only if they measure model capability rather than properties of the evaluation pipeline. We identify a double measurement confound: execution-critical decisions are performed by a fixed scaffold instead of the model, while the scorer evaluates outputs using criteria that may not reflect task correctness. We unify these issues within a measurement-theoretic framework that characterizes when benchmark scores can be interpreted as evidence of model capability, and instantiate it with an audit-and-repair protocol that (i) transfers execution-critical decisions from the scaffold to the model, (ii) replaces shape-based evaluation with seeded ground-truth scoring, and (iii) reports reliability beyond the mean through worst-case and tail-risk metrics. Experiments on ComtradeBench show that the joint intervention transforms a nearly flat leaderboard into a reliability spectrum that distinguishes both average performance and robustness across seeds. Applying the audit to existing benchmarks further shows that scorer validity is benchmark-specific, whereas scaffold ownership is an uncontrolled axis wherever we probed it. Our results suggest that benchmark scores should be interpreted together with their scaffolding level, scoring criterion, and reliability profile, providing a practical framework for more valid evaluation of LLM agents.
1 Introduction
The paper argues that agent benchmarks can confound model capability with scaffold behavior and scorer validity. Its joint repair de-scaffolds execution, scores against ground truth, and exposes both performance and reliability differences.
- Motivation: 97.5 mean scores for Kimi and Claude nearly match the 96.8 score of a no-LLM rule-based script.Qwen2.5-7B and Llama-3.3-70B also receive byte-identical per-seed scores.
- The confound: The scaffold makes execution-critical decisions while the scorer evaluates output shape and self-reported metadata rather than submitted values.A fabricated record set therefore receives the same 0.987 score as the correct one.
- Joint repair: Jointly transferring execution decisions to the model and applying seeded ground-truth F1 turns eight tied models into a spectrum from 0.974 to 0.000.Neither de-scaffolding nor scorer replacement alone is sufficient.
- Reliability: GPT-4o ranks fifth by mean yet reaches a worst-case score of 0.00, while Claude-Haiku-4.5 never falls below 0.787.The repaired evaluation reveals reliability differences hidden by average performance.
- Audit consequence: The pre-registered stress-test effect disappears when the same episodes are evaluated against ground truth.This result shows that the evaluation criterion changes the reported conclusion.
- Contributions: The paper introduces BenchAudit and applies it to ComtradeBench, τ-bench, and BFCL to produce benchmark-specific validity evidence.Its contributions include a validity card, a joint intervention, and released benchmark artifacts.
2 Related Work
Related work establishes separate concerns about scaffold sensitivity, scorer validity, and aggregate reliability. The paper combines these concerns by treating scaffolding as a controlled variable and seeded tail behavior as a primary measurement target.
- Agent benchmarks: Existing agent benchmarks commonly report means or best-of-k pass@k over few runs without controlling scaffolding level.The paper contrasts this practice with ComtradeBench’s environment-level adversity and decomposed ownership axis.
- Scaffold sensitivity: Prior studies show that harness changes can substantially shift measured success while the model remains fixed.Harness-Bench varies complete harnesses but does not decompose individual harness mechanisms.
- Reliability: The paper imports CVaR, worst-group accuracy, and multi-seed confidence intervals into LLM-agent reliability evaluation.Its position emphasizes worst-case behavior across seeds of a seeded adversary.
- Reliability: The paper’s novelty is to make worst-case behavior a cross-model discriminator rather than only a within-model diagnostic.Worst-case scores can reorder mean rankings up to the frontier.
- Validity: Measurement-theoretic work shows that benchmark scores can fail to track the intended construct, but does not address the paper’s combined confound.The paper connects criterion validity with scaffold ownership and reliability reporting.
- Non-stationary RL: The paper treats a frozen agent facing within-episode adversity as the inverse of non-stationary reinforcement-learning settings.Escalation is driven by agent success rather than elapsed time.
3 The Double Measurement Confound
ComtradeBench exposes a double measurement confound: the production scaffold performs the decisions under test, while the deployed judge scores shape and metadata instead of task correctness. These artifacts jointly flatten model differences.
- Benchmark: ComtradeBench tests adversarial, paginated extraction requiring deduplication, totals filtering, retries, and submission of cleaned records.Faults, duplicates, and distractor rows are injected by a seeded mock environment.
- Scaffold: The production scaffold automatically retries, deduplicates, filters totals, completes pagination, assembles, and submits the payload.The model mainly fills templated slots while the harness owns the adversary-relevant decisions.
- Observed invariance: Kimi and Claude each score 97.5, while a no-LLM rule-based baseline scores 96.8.Qwen2.5-7B and Llama-3.3-70B also produce byte-identical per-seed reward vectors under the production agent.
- Scorer: The deployed judge assigns a deterministic 100-point rubric across correctness, completeness, robustness, efficiency, data quality, and observability.Its correctness dimension checks submitted-line count against self-reported metadata rather than true record identity.
- Scorer failure: An empty submission scores 0.648, while fabricated and correct record sets both score 0.987 despite ground-truth F1 values of 0.0 and 1.0.The probe demonstrates that shape and metadata can substitute for correctness under the deployed judge.
- Joint confound: Only joint removal of scaffold ownership and shape-based scoring recovers model differences that neither intervention reveals alone.Fixing the scorer alone measures scaffold execution, while de-scaffolding alone leaves failures misgraded.
4 Measurement Protocol
The protocol formalizes benchmark scores as functions of model, scaffold, and scorer, then identifies model behavior by returning execution-critical decisions to the model and scoring against seeded ground truth. It complements this repair with reproducible reliability statistics and explicit scope controls.
- Measurement model: A benchmark score is written as S(m, c, j), allowing model, scaffold, and scorer effects to be analyzed separately.A fixed leaderboard configuration can otherwise be mistaken for a property of the model alone.
- Identification: Under the production configuration, scaffold-owned decisions and shape-based scoring make scores invariant across models that fill templates equivalently.The identification strategy moves the scaffold to L2 and the scorer to ground truth.
- Identification: When every execution-critical decision is model-owned and scoring is criterion-valid, matched-seed score differences reflect differences in model behavior.Worst-case and CVaR then estimate execution reliability.
- Scope: At L2, the identified quantity includes the re-emission demand, creating a mechanical-floor caveat for weak models.This scope boundary separates measured execution reliability from a purely decision-level interpretation.
- Audit criterion: Capability claims require Dclaimed ⊆ Dmodel; otherwise the benchmark must return decisions to the model or relabel the result as system-level performance.Model ownership is necessary but also requires a non-degenerate action channel.
- Scaffolding spectrum: L0 assigns all execution-critical steps to the harness, L1 assigns one retry-or-skip decision to the model, and L2 assigns everything to the model.L2 produces the populated reliability spectrum, while L1 diagnoses the mechanical floor.
- Scoring: Ground-truth F1 uses seeded canonical records to compute recall, precision, and F1 deterministically for each task and seed.The generator’s deterministic identifiers make the true unique-record set known exactly.
- Reliability metrics: Reliability reporting includes worst-case score, CVaR@α, Reliability@τ, seeded bootstrap confidence intervals, and paired Cliff’s δ.The default settings are α = 0.2 and τ = 0.9, with τ = 0.5 also reported.
5 Experiments
Jointly removing scaffold ownership and shape-based scoring reveals a broad performance and reliability spectrum that mean scores alone conceal. Across internal and external audits, scaffolding level changes measured outcomes, while scorer validity varies by benchmark.
- 5.1 The De-Scaffolded Reliability Spectrum: Ground-truth scoring with full de-scaffolding opens the flat leaderboard into a populated spectrum spanning Claude-Fable-5 at 0.974 to Llama-3.3-70B and Qwen2.5-7B at 0.000.The L2 evaluation covers five execution tasks with 10 seeds per model-task pair.
- 5.1 The De-Scaffolded Reliability Spectrum: Claude-Fable-5 leads on both mean 0.974 and worst case 0.824, while GPT-4o and GPT-4o-mini combine strong means with a worst case of 0.00.Llama’s zero reflects an empty L2 submission despite approximately 0.83 coverage at L1; Qwen never submits a valid payload.
- 5.1 The De-Scaffolded Reliability Spectrum: GPT-5 and Claude-Haiku-4.5 differ by 0.006 in mean F1 but by 0.079 in CVaR@0.2, showing that tail statistics separate models the mean treats as equivalent.At τ = 0.9, Reliability@τ is 0.720 for GPT-5, 0.600 for Claude-Haiku-4.5, and 0.480 for GPT-4o.
- 5.2 Scaffolding Level as a Hidden Axis: The same environment yields three qualitative verdicts across L0, L1, and L2 because scaffolding changes which execution work and decisions are measured.At L1, SKIP occurs 0 of 120 times and coverage settles near 0.83, while L2 exposes a mechanical marshaling floor.
- 5.3 The Confound Inside the Confirmatory Experiment: a Pre-Registered Stress Test: The 2×2 intervention separates models fully only when both the harness and scorer are changed; the shipped judge compresses the 1.000 ground-truth spread to 0.425 among submitting models.At L0, submissions are SHA-256-identical across all 21 model pairs on all five seeds.
- 5.3 The Confound Inside the Confirmatory Experiment: a Pre-Registered Stress Test: The pre-registered stress-test confirmation disappears under ground-truth scoring because GPT-5’s ground-truth F1 and coverage deltas are approximately zero on identical episodes.The reported GPT-5 result is ∆= +0.011 with CI [+0.002, +0.023], p = .034, δ = +0.25 [−0.15, +0.65].
- 5.3 The Confound Inside the Confirmatory Experiment: a Pre-Registered Stress Test: An exploratory quota squeeze revives SKIP from 0/120 to 38/120, but an always-retry policy produces ∆= +0.096, larger than every model’s penalty.The result is interpreted as resource arithmetic that model decisions only attenuate.
- 5.4 External Validity: Scorer Probed, Scaffold Measured: On τ-bench, changing only the scaffold flag shifts the same model’s reward by up to 0.267 and reorders models, whereas the official scorer distinguishes corrupted from passing outcomes.The audit therefore finds scorer validity benchmark-specific but scaffold ownership uncontrolled wherever probed.
6 Limitations
The evaluation has limited scope: exploratory quota variants, a single stylized trade-data domain, a small task suite, and power primarily for large effects. Several measurements are additionally bounded by output budgets, pinned endpoints, or incomplete ownership interventions.
- B2 and B2-slack are exploratory, while the pre-registered result is the B1 null.The deferred enforced-quota variant uses n = 20 per arm, three models, and one task family.
- Cross-domain transfer remains untested because ComtradeBench is stylized trade-data-shaped extraction and only ComtradeAdapter receives the full ownership intervention.External targets are scorer-audited rather than fully de-scaffolded.
- The spectrum rests on five tasks × 10 seeds, with two tasks ceiling-saturated for frontier models.A sixth task, T7, is excluded because its 750-row re-emission demand exceeds submission budgets and floors every model run.
- B1 has power 0.998 at n = 20 only for large effects, so smaller escalation effects are not excluded.Per-model scores are also point-in-time measurements on pinned API endpoints.
7 Conclusion and Future Work
The paper concludes that valid agent benchmarking requires measuring model-owned decisions, ground-truth correctness, and reliability beyond averages. It leaves open broader ownership tests, stronger scorer audits, and interfaces that support larger submissions.
- When scaffolds own execution-critical decisions and scorers grade shape, leaderboard scores cannot identify model behavior.The repair therefore treats scaffold level, scoring criterion, and reliability as first-class evaluation dimensions.
- The protocol recommends measuring the model, scoring against ground truth, and reporting worst-case, CVaR, and Reliability@τ.It applies on top of existing benchmarks rather than replacing them.
- Future work includes ownership decomposition on external benchmarks, audits for scorers requiring model calls, and protocol variants that lift the L2 interface floor.These directions address semantic preservation, zero-call audit coverage, and larger record sets.
Supplementary Material
The supplementary material provides the complete experimental record, visual pipeline guides, and expanded analyses of the pre-registration deviation, power, and stress-test results. Its tables clarify the exploratory status and interpretation of the reported effects.
- The supplement contains extended analyses and the complete experimental record, regenerated from released result files unless noted otherwise.It includes all results, tables, and figures from the project or appendices.
- Figure A1 summarizes the measurement pipeline, while Figure A7 reports measured per-model scores at each scaffolding level.The scaffolding levels are defined in the main text, Section 4.2.
- The confirmatory B1 run deviated from pre-registration by using the L1 half-scaffold agent with ground-truth fetch coverage instead of the specified slim agent and terminal judge reward.The substituted instrument isolates the retry/skip decision and is reported as exploratory under the pre-registration rule.
- Power was 0.998 for detecting ∆ = 0.209 at n = 20, but medium and small effects were not excluded.The simulation certifies the statistical test, not whether the instrument could detect a behavioral effect.
- Table A1 defines ∆ as paired mean coverage difference and reports seeded bootstrap intervals, sign-flip p-values, Cliff’s δ, Holm-adjusted p-values, deaths, and skips.Only Llama’s B2 penalty is a dominance effect; Claude’s nominal significance is magnitude-driven, with δ = 0.00.
- Under free retries, the always-retry policy makes both arms’ per-seed outcomes identical within each model arm because seeded fault sequences determine them.This identifies the outcome as fixed by the fault sequence rather than model behavior under that condition.
- On the pre-registration-faithful terminal judge reward, Llama has ∆ = +0.005 and Claude ∆ = +0.016, both null, while GPT-5 has ∆ = +0.011 with a confidence interval excluding zero.The three models were evaluated with n = 20 per arm under the frozen decision rule.
A.4 Zero-Slack Decision-Invariance and the Slack Regime (full)
The zero-slack configuration makes retry and skip outcome-equivalent, so escalation penalties reflect resource arithmetic rather than model decision failure. With slack, models diverge in how they use scarce requests, while increasing slack eventually removes decision relevance.
- Zero-slack regime: At zero post-squeeze slack, retry and skip are outcome-equivalent because each fault consumes one page, making the escalating arm decision-invariant.Llama and GPT-5 have identical escalating-arm coverage vectors across all 20 seeds; Claude differs on one seed.
- Slack regime: With two pages of post-squeeze slack, Claude’s penalty vanishes at ∆= +0.010 and GPT-5’s at ∆= +0.019, both non-significant.Claude avoids quota deaths, while GPT-5 has one quota death; Llama remains affected.
- Slack regime: The escalation penalty decays as slack increases: Llama falls from +0.087 to +0.008, and GPT-5 from +0.063 to +0.008.The dose-response configurations use slack values 0, 1, 2, and 4 and are reported as estimation.
- Zero-slack regime: +0.096: a fixed always-retry policy exceeds every model’s escalation penalty under the enforced quota.The penalty is positive for all three models before correction, but survives Holm correction only for Llama and GPT-5.
- Slack regime: At slack 4, the penalty is negligible for every model because four spare requests prevent binding; models are distinguishable only at intermediate slack.Decision relevance is therefore described as an inverted U in slack.
- Reliability beyond the mean: GPT-5 combines perfect deduplication and retry scores with page-drift recall of 0.67, showing that clean outputs can still miss canonical records.On T8, GPT-5 has mean 0.844 and worst-case 0.000, while Claude-Sonnet-4.6 never falls below 0.824 despite a higher mean.
- Measurement boundary: T7 is reported as a scale bound rather than a reliability cell because single-call re-emission budgets cap F1 independently of model ability.The 750-row task caps recall at approximately 0.15/0.30 and F1 at approximately 0.26/0.46 under the two token budgets.
A.10 External Scaffold-Variance Slice: Full Matrix
The τ-bench slice isolates scaffold effects by holding models, tasks, and the user simulator fixed while changing only the scaffold. Scores and rankings shift materially across scaffolds, and the audit protocol formalizes how such ownership and scoring conditions are classified.
- Experimental design: The slice holds the Claude-Haiku-4.5 user simulator fixed and changes only the agent-strategy flag across the four shipped scaffolds.Each model is evaluated on the same 15 tasks with one trial per cell at temperature 0.
- Observed scaffold variance: Changing only the scaffold moves the same τ-bench model by up to 0.267 across 15 fixed retail tasks.The observed averages span 0.733–0.533 for Sonnet, 0.667–0.467 for Haiku, and 0.667–0.400 for GPT-4o.
- Caveat: The few-shot column carries a domain-specific overlap caveat, but the headline 0.267 swing between tool-calling and react does not depend on it.The reported overlap concerns the airline-domain demonstration file, whereas this slice is retail.
- Observed scaffold variance: React is the worst scaffold for all three models, while few-shot triples the Sonnet–Haiku gap from 0.067 to 0.200.GPT-4o crosses between models across scaffolds, so the induced ranking depends on the fixed scaffold.
- Audit protocol: A benchmark licenses model-capability claims only when replay, criterion sensitivity, and ownership completeness all pass.If criterion validity passes but ownership fails, the licensed claim is model–scaffold system performance instead.
- Audit protocol: Unaudited conditions are recorded as not probed rather than treated as passing evidence.The audit records fail or not probed when hooks are unavailable or model-owned channels are degenerate.
- Audit protocol: BenchAudit assigns retry, deduplication, totals filtering, pagination, and submission to the harness or model at each scaffolding level.Its hooks also include a seeded task registry, canonical-truth oracle, and scorer replay function.
B Extended External Audit: a Scorer-Kind Census Across Nine Benchmarks
The nine-benchmark census finds that scorer validity varies by benchmark, while shape-based and LLM-judge failures remain distinct routes to criterion-validity problems. ComtradeBench is unusual because seeded ground truth and a shape scorer enable the full joint intervention.
- Audit scope: The audit uses zero model calls and classifies scorer behavior from released source code or executed canned batteries, labeling the evidence type.Its cross-target coverage includes ComtradeBench, τ-bench, and BFCL through benchmark-specific adapters and shared audit logic.
- Scorer-kind census: Exactly one of nine audited benchmarks, ComtradeBench as shipped, uses a pure shape/self-report scorer.Most shipped scorers are value-grounded; GAIA’s executed scorer spot-check passed correct submissions 5/5.
- Audit scope: BenchAudit validity cards encode each condition as pass, fail, or not probed, with unaudited items never reported as passing.The full cards cover the three main audited benchmarks and retain evidence inline.
- Scorer-kind census: Shape/self-report and LLM-judge scorers can both let well-formed fabrications pass because neither anchors scores to ground truth.The census identifies LLM judging as a fourth scorer kind beyond shape, outcome, and specification scoring.
- Joint intervention: The joint 2×2 intervention is rarely externally reproducible because value-grounded benchmarks lack a broken scorer column and LLM-judge benchmarks lack a truth column.ComtradeBench supplies both through its seeded generator.
- Related benchmark coverage: The positioning matrix reports that τ-bench has passk without seeded worst-case or CVaR estimation, while Harness-Bench varies complete harnesses rather than decomposed ownership.These are qualitative protocol comparisons rather than measured numerical results.
C.3 Per-Task Parameters
The appendix records the seeded task parameters, benchmark aggregates, seed-level grids, and forensic reruns underlying the reported results. These materials expose both output-budget ceilings and mechanically identifiable failure modes.
- Task construction: Each benchmark task page is a deterministic function of task_id and seed, with parameters specifying paging, budgets, rows, rate limits, and fault injection.The complete parameter set is read from the released task implementation.
- Aggregate scope: The main Figure 2 aggregates cover five execution tasks, T3–T6 and T8, with n=10 seeds per cell for eight models.T7 is excluded because its 750-row re-emission demand exceeds output budgets, while partial T10 runs disclose their seed counts separately.
- Seed-level behavior: GPT-4o’s T3 seed-level outcomes are visibly solve-or-fail bimodal in the appendix grid.A forensic rerun expands this analysis to 30 seeds with complete transcripts and judge breakdowns.
- Seed-level behavior: 14/30 GPT-4o T3 episodes achieve F1 = 1.0 and 16/30 achieve F1 = 0.0, for a solve rate of 0.467.The 95% Wilson confidence interval is [0.302, 0.639].
- Failure forensics: Every GPT-4o T3 episode begins with fetch_page(page=0), but only solved episodes vary the page argument after the deterministic HTTP 422 rejection.Failures repeat the identical rejected call and submit an empty, schema-valid payload; solved episodes submit all 25 unique ground-truth records.
- Escalation diagnostics: At zero slack, Llama and GPT-5 have identical escalating-arm coverage vectors across all 20 seeds, while Claude differs on one seed.The appendix exposes the decision-equivalence underlying the main-text interpretation.
- Reliability diagnostics: As adversary dose rises from 0 to 1.0, the within-task mean–worst gap opens from 0 to 0.12–0.21 across models.This degradation curve is the within-task counterpart of the suite-level mean–worst divergence.
C.10 Pre-Registration Digest and Artifact Provenance
The appendix records a disclosed deviation from the frozen Claim B protocol, its pre-registration-faithful rerun, and the provenance of the project’s figures and artifacts. It also documents reliability, adversarial-dose, and harness-by-scorer analyses.
- Pre-registration: The frozen Claim B protocol specified matched-mean arms, 20 paired seeds, three models, bootstrap and sign-flip decisions, and an exploratory enforced-quota variant.
- Protocol deviation: The confirmatory run used an L1 half-scaffold with ground-truth fetch coverage rather than the pre-registered de-scaffolded slim agent with terminal judge reward.
- Protocol deviation: With retries free, explicit SKIP fired in 0 of 120 episodes, leaving no causal path for the pre-registered deduplication/totals channel and yielding a null under the tested instrument.
- Artifact provenance: A pre-registration-faithful rerun used the slim agent and judge-reward outcome, while the corrected artifact map distinguishes the deviating and faithful result files.
- Artifact provenance: The appendix preserves the measurement figures, legacy benchmark artifacts, figure captions, plotted-value provenance, and released result tables.
- Measurement artifacts: The joint intervention figure varies harness and scorer independently, showing collapse under byte-identical scaffold submissions and recovery under model-owned execution with ground-truth scoring.
D.5 Legacy Artifacts: the Original Benchmark Release
The legacy artifacts preserve the original benchmark’s production-scaffold evaluations, whose near-flat scores and training behavior document weak model discrimination. Companion figures expose scorer artifacts, reliability variation, scaffold levels, and adversarial analyses.
- Original benchmark release: The original six-dimension judge under the production scaffold produced near-flat T1–T8 leaderboards, included a no-LLM baseline, and left only T9 visibly discriminative.
- Original benchmark release: The original GRPO reward exhibited under-capacity oscillation, learns-then-collapses behavior, and saturated-at-initialization behavior across the three documented runs.
- Reliability and scaffolding: The scaffolding spectrum compares L0 judge score, L1 coverage, and L2 ground-truth F1, with Claude-Fable-5 observed only at L2.
- Scorer audit: The judge probe floors a contentless submission at 0.648 and assigns fabricated and correct record sets identical totals of 0.987, while ground-truth F1 gives 0.000 versus 1.000.
- Reliability and scaffolding: The E1 reliability spectrum plots mean ground-truth F1 and worst-case performance across 50 runs, with the bar-to-diamond distance representing the reliability gap.
- Supporting analyses: The appendix also preserves raw E1 seed vectors, forensic mechanism counts, B2 zero-slack and slack-2 records, power analyses, and the escalation-penalty forest plot.