PaperlayerView exact revision on arXiv

Source-linked AI summary

When Pair Count Is Not the Sample Size: What All-Pairs Agent Comparisons Estimate

Wei-Jung Huang

arXiv:2609.33012v1cs.AI

TL;DR

All-pairs comparisons reuse configurations, so pair count alone does not establish independent evidence. The paper distinguishes exact summaries of fixed boards from inference about future configurations, then compares interval widths, coverage, and aggregation choices. Shared-endpoint pair-iid coverage falls far below nominal in the exact model, while matched independent edges remain near nominal.

  • Problem

    All-pairs benchmark comparisons reuse entries, and the interpretation of their mean and uncertainty depends on whether the board is fixed or configurations are sampled from a population.

    Source: §1 · p. 1

  • Method

    The paper contrasts fixed-board summaries with population inference and examines a SWE-bench snapshot, exact binary-model coverage, and pair-inclusion and weighting sensitivity across leaderboards.

    Source: §1 · p. 2

  • Results

    Pair-iid coverage fell from 0.307 to 0.157 with shared endpoints, while matched independent-edge coverage remained 0.950 to 0.951; accounting for shared configurations more than doubled SWE-bench interval width.

    Source: §5.1 · p. 4, §5.2 · p. 6

  • Takeaways & Limitations

    All-pairs analyses should state what is fixed or sampled and how they handle shared entries and pair aggregation.

    Source: §9 · p. 8

    The paper does not establish how often published agent evaluations use pair-iid inference or whether correcting it would overturn conclusions.

    Source: §8 · p. 8

Abstract

from arXiv · show

When an agent benchmark compares every pair of leaderboard entries, the number of comparisons can look much larger than the independent evidence behind them: A versus B and A versus C both reuse A. Whether this reuse affects inference depends on what the analysis is meant to describe. If the board and its outcomes are fixed, the all-pairs mean is an exact summary of those entries, and any interval must come from another declared source of randomness. If the entries are instead treated as iid draws from a population of future configurations and the pair rule is regular and nondegenerate, the same mean is an order-two U-statistic whose first-order uncertainty depends on the number of configurations, not the number of pairs. We use near ties as the running example, but the distinction extends to other symmetric pair summaries when their regularity conditions hold. We examine both interpretations using a fixed SWE-bench Verified snapshot and an exact binary model with known truth. On SWE-bench, intervals that accounted for shared configurations were more than twice as wide as a pair-iid reference that treated the pairs as independent. In the exact model, pair-iid coverage fell far below the nominal level when edges shared endpoints but remained near nominal for matched independent edges. Results on two other fixed leaderboards show that exact summaries also depend on which pairs are included and how they are weighted. An all-pairs analysis must therefore state what is fixed, what is sampled, and how it handles shared entries and pair aggregation.

1 Introduction

All-pairs comparisons reuse configurations, so the number of pairs is not the number of independent configurations. The appropriate interpretation and uncertainty depend on whether the named board or a population of future configurations is the target.

1 Introduction

  • With configurations A, B, and C, comparisons (A,B) and (A,C) both reuse A, so M configurations do not yield an equally large set of independent comparisons.Each entry appears in M −1 pairs.Source: §1 · p. 1
  • When the board and task outcomes are fixed, the near-tie fraction exactly describes the named configurations; intervals require another declared source of randomness.For population claims, configurations instead must be treated as draws from a stated population.Source: §1 · p. 1
  • Under iid configuration sampling and a regular nondegenerate pair rule, the all-pairs mean is an order-two U-statistic whose first-order variance depends on configuration count.The pair count alone does not justify pair-iid inference because comparisons sharing an entry can move together [Hoeffding, 1948, Aronow et al., 2015, Graham, 2020].Source: §1 · p. 2
  • The analysis examines a fixed SWE-bench Verified snapshot, an exact binary model, and two additional leaderboard snapshots to study uncertainty and pair aggregation.The fixed SWE-bench snapshot cannot reveal repeated-sampling coverage because its configurations do not come from a known population.Source: §1 · p. 2
  • The central distinction is whether the all-pairs mean describes a named board or supports inference about future configurations.Each target requires different sampling assumptions and reporting choices.Source: §1 · p. 2

2 Related Work

Prior work studies all-pairs evaluation, dyadic dependence, and dependence among benchmark configurations, but targets quantities or dependence structures distinct from this paper’s all-pairs mean. This paper uses established inference results to distinguish fixed-board descriptions from inference about a sampled configuration population.

Estimands and benchmark uncertainty

  • Existing all-pairs evaluation studies target different quantities and dependence structures from the mean studied here [Wang, 2026, Klabunde et al., 2025, Ashkinaze et al., 2025].The paper frames its question as how interpretation and uncertainty change when pairwise terms reuse sampled configurations.Source: §2 · p. 2
  • Evaluation-noise work estimates task-level noise within model pairs, while this paper studies uncertainty in an all-pairs mean over sampled configurations [Wang, 2026].ReSi tests an all-pairs correlation, and Deep Value Benchmark’s comparison changes from p < 0.001 to p = 0.04 after crossed random intercepts are added for both endpoints [Klabunde et al., 2025, Ashkinaze et al., 2025].Source: §2 · p. 2

U-statistics and dyadic inference

  • U-statistic and dyadic-inference results describe variance for pair means and shared-unit dependence; this paper applies them to separate fixed-board summaries from sampled-population inference [Hoeffding, 1948, Aronow et al., 2015, Graham, 2020].Crossed bootstraps resample row and column entities separately [Owen, 2007].Source: §2 · p. 2

Dependence among evaluated configurations

  • Prior work notes that shared model provenance can pseudoreplicate benchmark entries and that family-balanced resampling can measure sensitivity to model-family representation [Hardy et al., 2026, Patel et al., 2026].Because SWE lacks a validated lineage map, this paper uses normalized names only for pair inclusion and weighting, not lineage-aware inference.Source: §2 · p. 2

3 Fixed-Board and Population Targets

The all-pairs mean has different meanings depending on whether configurations and tasks are fixed or sampled. Its target and uncertainty also depend on the sampling design and the order in which outcomes are aggregated and compared.

3 Fixed-Board and Population Targets

  • A fixed task aggregation produces configuration scores, which a symmetric pair rule then applies to each unordered pair.For the near-tie endpoint, the pair rule compares two scores.Source: §3 · p. 2
  • With a fixed board and fixed outcomes, U_M is the exact fraction of named configuration pairs satisfying the pair rule.The pair count supplies the fraction’s denominator, not uncertainty about future configurations.Source: §3 · pp. 2–3
  • The four target cases distinguish fixed or sampled configurations from fixed or sampled tasks; Table 1 organizes them by whether another configuration set or task panel could change the claim.The uncertainty target must reflect whichever entities are treated as sampled.Source: §3 · p. 3, Table 1 · p. 3
  • When configurations are sampled but task outcomes are fixed, the target concerns a configuration population under an explicit sampling model; when both vary, the target includes both sources of randomness.For fixed configurations, a repeat-panel target uses a common random panel of R task clusters.Source: §3 · p. 3
  • The estimand depends on panel size, cluster design, common-panel sharing, and aggregation order, because a nonlinear pair rule can yield different results when applied before or after task aggregation.Enumerating every edge does not select one of the sampling regimes.Source: §3 · p. 3

4 Dependence from Shared Configurations

When configurations are iid draws and the pair rule is regular and nondegenerate, shared endpoints contribute first-order variance, so uncertainty scales with configuration count rather than pair count. These population claims require a specified exchangeable unit and applicable regularity conditions.

4 Dependence from Shared Configurations

  • Under a regular kernel with ζ1 > 0 and standard moment conditions, the first-order terms yield variance of order M −1 and asymptotic normality.Each first-order term appears in M −1 pairs, giving coefficient 2/M, while the degenerate residual contributes O(M −2) variance.Source: §4 · p. 4
  • Treating N pair values as independent instead gives variance of order N −1 = O(M −2), despite O(M 3) pairs of edges sharing an endpoint.Dyadic sandwich estimators account for correlation among dyads sharing a member [Aronow et al., 2015, Graham, 2020].Source: §4 · p. 4
  • The decomposition addresses future-configuration claims only after specifying the population, exchangeable unit, and regular first-order regime.Shared model families or contributors may mean one configuration is not the appropriate exchangeable unit [Hardy et al., 2026, Patel et al., 2026].Source: §4 · p. 4
  • A positive first-order variance estimate accounting for at least half the total estimated variance diagnoses whether regular first-order formulas are usable for the observed sample.The diagnostic does not establish exchangeability or population regularity, which still depend on the design.Source: §4 · p. 4

5 Interval Width and Coverage

The SWE-bench sensitivity analysis finds substantially wider intervals when shared configurations are accounted for, while exact enumeration shows pair-iid coverage falls when edges share endpoints. These results depend on the chosen sampling interpretation and model conditions.

5.1 SWE-bench Verified: Interval Width

  • The width comparison treats 134 configurations as independent draws under a working model, while the fixed panel’s point estimate is an exact description at the chosen margin.The board supplies no sampling design for future submissions, so the interval comparison is a sensitivity analysis, not a design-based population analysis.Source: §5.1 · p. 4
  • The finite-board near-tie rate is 0.159: shared-configuration interval widths are 0.0358–0.0435, or 2.35–2.86 times the pair-iid width of 0.0152.These widths are compared under a working model of independent configurations; the point estimate itself describes the fixed panel.Source: §5.1 · pp. 4–5, Table 2 · p. 4
  • Across positive near-tie margins, pair-iid remains narrower than the shared-configuration curves, but widths compare different sampled populations rather than interchangeable methods.At the five-point margin, task and repository-block resampling widths are 0.0208 and 0.0238, versus 0.0435 for configuration resampling and 0.0482 for joint resampling.Source: §5.1 · p. 5, Figure 1 · p. 5
  • Task thinning produced mean strict-reversal and unresolved rates of 0.0189 and 0.0101, and deleting one configuration changed the estimate by at most 0.00230.The task-subset calculations condition on observed outcomes and use the same sampled task subset for every pair.Source: §5.1 · p. 5
  • SWE-bench metadata do not support lineage analysis: 43 configurations lack model tags, and proxy grouping does not establish genealogy.A lineage-sensitive summary would require validated family labels.Source: §5.1 · p. 5

5.2 Exact Coverage Under Shared and Independent Edges

  • As M grows from 40 to 160, pair-iid coverage falls from 0.307 to 0.157, while shared-endpoint methods attain 0.928–0.940 coverage on the enumerated grid.The shared-endpoint methods have identical covering sets on this grid, despite differing variance estimates.Source: §5.2 · p. 6, Table 3 · p. 6
  • For matched independent edges, coverage is 0.950–0.951, compared with shared-endpoint pair-iid coverage gaps that grow from 0.642 to 0.794.The control holds the target, pair counts, M values, critical value, and interval formula fixed while changing endpoint reuse.Source: §5.2 · p. 6, Table 3 · p. 6
  • When every pair is marked, intervals equal [1, 1] and cover unconditionally, but zero first-order variance prevents shared-configuration methods from passing the diagnostic.This boundary case does not establish regular calibration.Source: §5.2 · p. 6

6 Sensitivity to Pair Inclusion and Weighting

Exact fixed-board near-tie summaries vary with pair inclusion and weighting. The difference between all-configuration and equal-block aggregation combines excluding within-block pairs with reweighting, not lineage effects.

6 Sensitivity to Pair Inclusion and Weighting

  • A includes every configuration pair, whereas E excludes within-block pairs and gives each pair of distinct model-name blocks equal weight.The difference combines pair exclusion and reweighting, so it is not a lineage effect.Source: §6 · p. 6
  • On the primary snapshots, A − E is 2.92 percentage points for ScienceAgentBench and 4.78 percentage points for Terminal-Bench.These are exact descriptions of named snapshots, without sampling intervals.Source: §6 · p. 7, Table 4 · p. 7
  • Nested and stress-panel differences vary in sign, while the unique-block CORE-Bench Hard control has A = E exactly.The additional panels clarify aggregation behavior but are not independent replications.Source: §6 · pp. 6–7

7 Reporting Implications for Agent Evaluation

An all-pairs mean does not specify its own uncertainty: the target and the entities treated as sampled determine the relevant interpretation. Reporting should make the target, aggregation, sampling assumptions, and uncertainty procedure explicit.

7 Reporting Implications for Agent Evaluation

  • For a fixed-board target, report the configurations and tasks, aggregation order, included pairs and weights, and handling of ties and missing outcomes.The value is exact once those choices are fixed.Source: §7 · p. 7
  • For a population target, state each sampling source, its represented population, the exchangeable unit, and possible exchangeability violations.Match the variance or resampling method to the sampled entities and report its assumptions and diagnostic alongside coverage or width.Source: §7 · p. 7
  • A constant-kernel boundary means neither unconditional coverage nor zero width alone establishes regular first-order calibration.No interval family is valid for every all-pairs analysis; the target and sampling design determine the appropriate interval.Source: §7 · p. 7
  • When available, a matched independent-edge control can test whether a discrepancy remains after endpoint reuse is replaced, holding the target, pair count, critical value, and formula fixed.Within that design, those controls isolate endpoint reuse.Source: §7 · p. 7

8 Limitations

The benchmark evidence supports exact descriptions of the named panels and sensitivity analyses under stated working models, but not population claims without a sampling frame or defensible exchangeability assumptions. The exact coverage findings and broader comparisons also have specific scope limits.

8 Limitations

  • Differences in metrics, configuration support, and observable randomness preclude a pooled cross-benchmark estimate.For SWE, incomplete lineage, missing repeated outcomes, and unavailable judge records also mean a configuration may not be the appropriate resampling unit.Source: §8 · p. 7
  • Exact coverage results are limited to the enumerated binary models, M values, and interval formulas.The equality and independent-edge grids were selected after outcomes were available; the matched control does not trace intermediate dependence levels.Source: §8 · p. 8
  • The paper does not establish how often published agent evaluations use pair-iid inference or whether correcting it would overturn conclusions.Answering either question would require a systematic review of inferential units and interval calculations.Source: §8 · p. 8

9 Conclusion

The conclusion distinguishes exact descriptions of a named board from inference about future configurations. It reports that shared endpoints undermine pair-iid coverage and argues that reports should state their target, sampling assumptions, aggregation, and uncertainty design.

9 Conclusion

  • Pair-iid coverage deteriorates when edges share endpoints but remains near nominal for matched independent edges.On real panels, accounting for shared configurations widened intervals, while changing pair inclusion and weighting altered exact summaries.Source: §9 · p. 8
  • On a fixed board, the all-pairs mean exactly describes named entries under the chosen aggregation.Under iid configuration sampling and a regular, nondegenerate kernel, its first-order variance scales as M −1 rather than with the combinatorial pair count.Source: §9 · p. 8
  • The results support fixed-board descriptions and sensitivity analyses under stated working models, not population claims without a sampling frame or defensible exchangeability assumptions.Source: §9 · p. 8
  • An all-pairs report should identify its target, sampled entities, pair weights, and matching uncertainty design.Source: §9 · p. 8

A.1 Fixed-Panel Construction

The appendix constructs an exact near-tie summary for a fixed SWE-bench panel, then examines interval sensitivities under specified sampling models. It also compares alternative pair weighting and inclusion rules and uses exact enumeration to assess coverage under shared and independent edges.

A.1 Fixed-Panel Construction

  • All 134 eligible configurations on the common 500-task SWE-bench Verified snapshot yielded 8,911 unordered configuration pairs; near ties were pairs with absolute score difference of at most 0.05.Source: §A.1 · p. 10

A.2 Interval and Influence Calculations

  • The joint configuration-and-repository-block sensitivity interval had width 0.0482, versus 0.0238 for repository-block resampling and 0.0208 for individual-task resampling.Individual-task resampling is only a comparator because tasks are clustered within repositories; configuration resampling uses a working independent-configuration model.Source: §A.2 · p. 11

A.3 Half-Task Sensitivity

  • Half-task thinning produced a mean strict-reversal rate of 0.0189, with central 95% sample quantiles [0.0149, 0.0237]; these quantify sensitivity, not latent-parameter uncertainty.Pairing the draws with independent configuration weights gave a joint sensitivity interval of [0.0123, 0.0276].Source: §A.3 · p. 11

A.4 Available Lineage Metadata

  • Neither available proxy supported validated lineage grouping, so the analysis reports no lineage-aware interval or lineage-sensitive outcomes.Normalized model tags covered 91 of 134 configurations, while the hostname proxy grouped 29 configurations together.Source: §A.4 · p. 11

A.5 Alternative Pair Inclusion and Weighting

  • On the observed fixed sets, ScienceAgentBench’s live-page panel (23 configurations, 102 common tasks) had A = 85/253 and E = 27/88, while Terminal-Bench S43 (43 configurations, 89 common task keys) had A = 151/903 and E = 1987/16632.Source: §A.5 · p. 12

A.6 Exact Enumeration and Diagnostics

  • Exact enumeration covered 2,264 method-state rows; for the constant kernel, coverage was 1.000 while the shared-configuration methods had zero probability of passing the diagnostic.The diagnostic requires a positive first-order variance estimate and a first-order share of at least one half; these results concern only the enumerated model.Source: §A.6 · pp. 12–13

A.7 Conditional Coverage and Diagnostics

  • Among passing samples, conditional coverage for the Hoeffding projection, dyadic sandwich, and jackknife was 0.929 at M = 40 and between 0.925 and 0.975 at M = 160, while pair-iid mismatch persisted.The grid was designed after prior simulation outcomes were available, so the analysis is retrospective and exploratory.Source: §A.7 · p. 13

A.8 Exact Comparison with Independent Edges

  • For matched independent edges, exact pair-iid coverage was 0.950, 0.950, and 0.951 at M = 40, 80, and 160, respectively.Expected widths at those values of M were 0.0655, 0.0325, and 0.0162.Source: §A.8 · p. 13

Related papers

Loading 2609.33012v1…