Source-linked AI summary

What Does an LLM-Agent Leaderboard Rank Actually Compare?

Wei-Jung Huang

arXiv:2609.07785v1cs.AI

TL;DR

Public leaderboard scores summarize specific evaluations, but whether they support pairwise superiority claims across different targets, label sources, and resource rules is uncertain. The paper introduces an estimand-aware pairwise procedure; across public releases, close ranks often remain unresolved, while labels, repeat rules, and utility rules can change scores or selected systems.

  • Problem

    Public leaderboard scores summarize specific evaluations, leaving the question of when they support broader pairwise superiority claims across differing targets, label sources, and cost rules.

    Source: §I · p. 1
  • Method

    The paper’s estimand-aware procedure specifies each pair’s target and measurement source, checks common support, and evaluates the supported gap using an uncertainty rule and practical margin.

    Source: §I · p. 1
  • Results

    Across public releases, close ranks often do not support a pairwise winner once target, support, and uncertainty are specified; labels, repeat rules, and utility rules can change scores or selected systems.

    Source: §I · pp. 1–2, §VII · p. 7
  • Takeaways & Limitations

    Leaderboard authors can clarify fine-grained orderings by reporting the evaluated configuration, target population, labels, and rules for support, uncertainty, and practical significance.

    Source: §VI · p. 7

    Public releases support pairwise conclusions only for recorded variables and outcomes and do not identify causal effects of repositories, benchmarks, scaffolds, modes, or verifiers.

    Source: §VI · p. 7

Abstract

from arXiv · show

An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ in task mixture, label source, release detail, or cost rule. We study what leaderboard scores estimate and when they justify pairwise superiority conclusions. Our estimand-aware pairwise procedure states the comparison target and measurement source, checks common support, and evaluates the supported difference using a stated uncertainty rule and practical margin. Controlled checks evaluate the decision labels under known finite-sample conditions and show why uncertainty must be included when judging sensitivity to target reweighting. Across SWE-bench, AgentRewardBench, and tau2-bench, close rank differences are often unresolved; proxy labels and utility rules can also change which system is selected. DataAgentBench and Open Agent show what remains estimable from coarser public records. A leaderboard score summarizes a released evaluation, whereas a fine-grained superiority claim additionally depends on the estimand and uncertainty rule used to interpret the difference.

I. INTRODUCTION

A leaderboard rank summarizes a particular agent configuration and evaluation design; it does not by itself establish that one agent is generally superior. The paper introduces an estimand-aware pairwise procedure to determine when public records support a scoped comparison and applies it to detailed and coarser releases.

I. INTRODUCTION

  • A displayed ordering reflects a specific configuration and evaluation design, while an unadjusted success rate estimates success for the observed benchmark mix—not necessarily another target or utility.Configurations can differ in scaffold, tools, action space, prompts, stopping rule, label source, and resource budget; a stable label applies only to the stated comparison fields and does not identify causal effects.Source: §I · p. 1, Figure 1 · p. 2
  • AgentRewardBench’s top agent changes with the label source; SWE-bench Verified has 39 of 45 comparisons underpowered, and tau2-bench pass@4 raises scores 17.7 to 20.4 points over pass@1.For SWE-bench Verified, 5 comparisons are stable and 1 target-sensitive under descriptive pointwise 95 percent intervals and a 2-point rule; a max-deviation simultaneous interval resolves none. On tau2-bench, cost can also change which system a utility rule selects.Source: §I · p. 1
  • The procedure specifies each pair’s evaluated unit and target, checks common support, and assesses the supported gap using a stated uncertainty rule and practical margin.It reports a supported winner only when warranted; otherwise it identifies limits such as missing support, expert–proxy disagreement, insufficient precision, or target sensitivity. Prior work addresses parts of this problem, but the paper says this combination is absent from existing public-log methods,,,.Source: §I · pp. 1–2
  • Controlled tests and public-leaderboard applications examine decision labels, target estimation, and task-conditioned exchangeability, then compare detailed records with coarser exports.The applications cover SWE-bench, AgentRewardBench, tau2-bench, DataAgentBench, and Open Agent; target reweighting and uncertainty leave close comparisons unresolved more often than they reverse the order. Weak verifiers, repeated-attempt rules, costs, and aggregation can change the selected system or measured quantity.Source: §I · pp. 1–2

II. RELATED WORK

Prior work develops agent benchmarks, statistical comparison methods, and leaderboard-robustness tools. This study uses those foundations to assess which pairwise conclusions public logs support.

Agent benchmarks and evaluation documentation

  • Agent benchmarks specify tasks, environments, harnesses, verifiers, and reporting formats, while prior work emphasizes cost control, reproducibility, benchmark design, and structured documentation [1]–[6], [18], [21], [22].This study builds on that documentation to examine which pairwise conclusions reported conditions justify.Source: §II · p. 2

Targets, weak labels, and statistical comparison

  • Survey sampling, estimand specification, weighting, hierarchical and item-response models, preference evaluation, and prediction-powered inference inform analysis of target choice and paired labels [7], [8], –, –.The appropriate analysis depends on the estimand and the information released with the leaderboard.Source: §II · p. 2

Leaderboard robustness, utility, and public-data limits

  • Prior work studies benchmark perturbations, uncertainty-aware scoring, adaptive overfitting, rank-difference inference, robustness, bandit comparisons, and logging overlap –, –.These methods treat rank differences as inferential claims and make overlap and support explicit.Source: §II · p. 2
  • Multi-criteria decision analysis formalizes utility tradeoffs,, while limited public data motivates focusing on score targets and pairwise conclusions estimable from released logs.Most public agent leaderboards do not expose data needed for full causal or off-policy analyses.Source: §II · p. 3

III. A PAIRWISE COMPARISON PROCEDURE

Pairwise leaderboard claims are interpretable only when the score estimand matches the comparison target, the released data support that target, and the difference meets a stated uncertainty and practical-margin rule. The procedure distinguishes observed, standardized, verifier-based, and utility scores rather than treating their rankings as interchangeable.

III. A PAIRWISE COMPARISON PROCEDURE

  • In the running SWE-bench example, raw success rates 0.792 and 0.788 compare 500 released instances, while equal-repository superiority is unsupported if its interval includes zero or the gap misses the 2-point margin.Source: §III · p. 3

A. What Quantity Does the Score Estimate?

  • Scores may target observed-mix success, target-standardized success, verifier-calibrated success, or resource-based utility, and ordinarily describe the full evaluated agent configuration rather than the model alone.A model-only claim requires additional design support; orderings under different estimands answer different questions and should not be treated as interchangeable.Source: A. What Quantity Does the Score… · p. 3

B. Do the Evaluation Data Cover the Target?

  • A target assigning weight outside the compared agents’ common-support cells is not identified without modeling assumptions; the comparison must be unsupported or restricted to a changed target.Direct standardization shows how scores and gaps change under target weights, but does not attribute those changes to causal effects; data without item-level records may only support coarser questions.Source: B. Do the Evaluation Data Cover the… · p. 3

C. When Is a Score Difference Large Enough?

  • A practical winner requires an interval supporting its direction and a standardized gap clearing the margin; ordered labels prioritize unsupported, verifier-sensitive, underpowered, target-sensitive, then stable.The sign-flip radius is the minimum L1 target-weight shift within common-support cells that reverses the gap, with no probabilistic or causal interpretation; family-wide claims require a declared comparison family and error-control rule.Source: C. When Is a Score Difference Large… · pp. 3–4

IV. CONTROLLED BEHAVIOR CHECKS

Controlled checks test whether the procedure recovers known decision labels, reduces estimation error when targets are known, and identifies gaps beyond a task-conditioned null. They show why uncertainty belongs in target-sensitivity decisions and why even a stable top position can coexist with small pairwise margins.

IV. CONTROLLED BEHAVIOR CHECKS

  • In controlled reference cases, simulations assessed label recovery and whether standardization and weak-verifier calibration reduced error under their assumptions; AgentRewardBench permutations supplied a task-conditioned null for balanced gaps.These checks establish how the labels behave before analysis of public releases.Source: §IV · p. 4

A. Decision-Label Recovery

  • The uncertainty-aware target-sensitivity rule recovered the stable scenario in 99.7% of replicates, versus 84.3% for a point-estimate-only rule that overcalled sensitivity in 47 of 300 runs.Across the simulated scenarios, the modal label matched the known state in every scenario, though finite sampling still caused occasional classification errors.Source: A. Decision-Label Recovery · pp. 4–5

B. Known-Target Estimation Check

  • Direct standardization reduced mean absolute error from 0.0184 for the raw observed expert score to 0.0128, while regression standardization gave 0.0139 and cross-fitted weak-verifier calibration gave 0.0144.Uncalibrated weak-verifier estimates had mean absolute error 0.0401 after direct standardization and 0.0804 for the raw weak-verifier score; the full equal-benchmark target was unsupported for the agent missing visual observations, but the common-support target remained estimable.Source: B. Known-Target Estimation Check · p. 5

C. Permutation Check on a Public Release

  • On AgentRewardBench, the observed maximum absolute balanced pairwise gap was 0.1489 versus a permutation 95th percentile of 0.0727 (p < 0.001), and practical gaps of at least 2 percentage points exceeded the reference (p = 0.010).The 1,000 permutations shuffled agent labels within task clusters, preserving task difficulty and benchmark composition while testing exchangeability conditional on task.Source: C. Permutation Check on a Public Release · p. 5
  • Claude remained the top agent in all 1,000 common-support target-weight perturbations, although the top-gap 10th percentile was 0.0165.The top position was stable near the equal-benchmark target, but some pairwise margins remained small.Source: C. Permutation Check on a Public Release · p. 5

V. EMPIRICAL EVIDENCE

The analyses match each leaderboard comparison to the finest public-record resolution available, and the evidence shows that scores alone do not establish broad pairwise superiority. Rankings and selections can depend on target population, label source, protocol, utility rule, and record granularity.

B. SWE-bench: Most Top-10 Pairwise Orderings Are Unresolved

  • 5 of 45 raw-top-10 SWE-bench pairs are stable, 1 is target-sensitive, and 39 are underpowered under the 2-point margin rule.These descriptive labels use pointwise bootstrap uncertainty; accounting for the 45-pair family, simultaneous intervals exclude zero for no pair.Source: B. SWE-bench: Most Top-10 Pairwise… · p. 5, Figure 2 · p. 6

C. AgentRewardBench: Coverage Limits the Target and Label Source Changes the Winner

  • On AgentRewardBench common support, 4 of 6 pairwise comparisons are stable and 2 underpowered; ranks hold, but all benchmark-balanced scores fall.The all-agent, four-group target is unsupported because Llama has no VisualWebArena observations; the common-support target excludes that group.Source: C. AgentRewardBench: Coverage Limits… · p. 6
  • Expert labels select Claude, while GPT-4o-mini labels select GPT-4o; each weak verifier reverses one of six pairwise signs relative to expert labels.The functional evaluator underestimates expert success by 8.6 points (FNR 0.423), while GPT-4o-mini overestimates it by 7.9 points (FPR 0.180).Source: C. AgentRewardBench: Coverage Limits… · p. 6
  • Leave-one-agent-out calibration reduced equal-benchmark Brier loss in all eight verifier-agent folds, but results cover only four observed configurations and observed task distributions.Calibrated-minus-raw differences were −0.037 for the functional verifier and −0.052 for GPT-4o-mini; transport to new models or distributions requires separate evaluation.Source: C. AgentRewardBench: Coverage Limits… · p. 6

D. tau2-bench: Protocol and Utility Choices Change Selections

  • In tau2-bench, equal-domain reweighting shifted scores by at most 2.2 points, yet all six common-support primary-mode pairwise comparisons remained underpowered.The common-support comparison covers four configurations over airline, retail, and telecom.Source: D. tau2-bench: Protocol and Utility… · p. 7
  • In tau2-bench, pass@4 raised common-support scores by 17.7 to 20.4 points over pass@1, while o4-mini telecom pass@1 varied by 56.8 points across modes.The first cost-only utility switch was from Claude 3.7 Sonnet to o4-mini at λ = 0.0514 success-rate units per USD when µ = 0.Source: D. tau2-bench: Protocol and Utility… · p. 7

E. What Coarser Releases Still Support

  • Coarser releases retain some comparisons: DataAgentBench’s top configuration stays PromptQL Claude Opus 4.6, while Open Agent’s remains unchanged under benchmark standardization.DataAgentBench scores shift by up to 9.2 points and 10 of 28 pairs are underpowered; Open Agent success changes from 0.784 to 0.787, with all 66 grouped top-12 comparisons stable, but its export lacks item-level uncertainty and verifier calibration.Source: E. What Coarser Releases Still Support · p. 7

VI. DISCUSSION AND LIMITATIONS

Leaderboard scores support conclusions only about recorded variables and outcomes, and close-comparison claims depend on the stated target, uncertainty, and multiplicity rules. Authors should make these choices and separate decision dimensions explicit.

  • Report the evaluated configuration, score-defining population and labels, and how support, uncertainty, and practical significance govern close comparisons; present cost, latency, repeats, and scaffolds as separate decision dimensions.Source: §VI · p. 7
  • Public releases support pairwise conclusions only for recorded variables and outcomes, not causal effects of repositories, benchmarks, scaffolds, modes, or verifiers; AgentRewardBench enables calibration with paired expert and weak labels.Source: §VI · p. 7
  • State the target distance for sensitivity analyses and comparison family for family-level winners: sign-flip radii vary over common-support cells, while Wasserstein variants depend on task-distance definitions.A named pair may use a pointwise interval; a family-level winner requires a declared comparison family and error-control rule.Source: §VI · p. 7

VII. CONCLUSION

Across public LLM-agent releases, close ranks often do not establish a pairwise winner once the target, common support, and uncertainty are specified. A leaderboard score summarizes one released evaluation; superiority claims also depend on the estimand and decision rule.

VII. CONCLUSION

  • Close ranks often do not support a pairwise winner, and label sources, repeated-attempt protocols, or utility rules can change scores or the selected system.Pairwise superiority additionally requires specifying the estimand and decision rule.Source: §VII · p. 7
Loading 2609.07785v1…