PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer, Sanmi Koyejo

arXiv:2610.00651v1cs.AIcs.LGstat.AP

TL;DR

Agent-evaluation rankings can depend on scaffolds and tasks, so reliability depends on whether the target is a deployed system or an underlying model. The paper uses Bayesian variance decomposition to assess sparse, imbalanced leaderboards and finds that pooling diverse benchmarks can improve cross-task rankings at the same task budget.

  • Problem

    Agent evaluations are increasingly used to compare systems, but it is uncertain whether a ranking reflects performance differences that persist beyond its particular setup or task set.

    Source: §1 · p. 1

  • Method

    The authors apply a Bayesian Generalizability Theory variance-decomposition framework to sparse, imbalanced agent leaderboards, separating claim-relevant signal from irrelevant variation.

    Source: §1 · p. 2

  • Results

    At the same task budget, projected ranking reliability rises from approximately 0.44 within one benchmark to 0.75 across the nine-benchmark battery.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluation design should follow the intended claim, and added effort should target the uncertainty sources that limit its reliability.

    Source: §5.3 · p. 8, §6 · p. 10

    The reliability estimates are conditional on the sampled model pool and scaffold set, and do not establish validity against underlying capability or external ground truth.

    Source: LIMITATIONS AND FUTURE WORK · p. 15

Abstract

from arXiv · show

Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83\%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

1 INTRODUCTION

Agent-evaluation rankings are useful only relative to a specified target, such as a deployed model–scaffold system or an underlying model. This paper analyzes which reliability claims current evaluations support and how evaluation effort can address their uncertainty.

1 INTRODUCTION

  • The measurement target may be the complete deployed system or performance differences attributable to the underlying model, changing what counts as signal or evaluation error.Source: §1 · pp. 1–2
  • A Bayesian Generalizability Theory framework separates signal relevant to the intended claim from irrelevant variation that can still change scores or rankings.It decomposes variation associated with models, tasks, scaffolds, benchmarks, and their interactions in sparse, imbalanced leaderboard designs.Source: §1 · p. 2
  • At the same task budget, projected ranking reliability rises from approximately 0.44 within one benchmark to 0.75 across the nine-benchmark battery.The paper also reports that reliability-aware allocation can reduce projected evaluation cost by up to 83%.Source: §1 · p. 2
  • Even infinitely many similarly constructed tasks improve model-ranking reliability by at most approximately 0.10 under the observed scaffold coverage.The introduction cautions that more tasks do not always make an evaluation more reliable.Source: §1 · p. 2
  • Scaffold choice can change evaluation conclusions, including which tasks a model solves, without improving its overall benchmark score.Source: §1 · p. 2
  • Evaluation design should follow the intended meaning of a score or ranking, with effort directed toward the uncertainty sources that limit that claim.Source: §1 · p. 2

2 RELATED WORK AND BACKGROUND

Prior agent-reliability work focuses mainly on repeatability of a fixed system, whereas this paper considers whether comparative rankings persist across tasks, scaffolds, or benchmarks. Generalizability Theory frames reliability around the measurement object and conditions, with object-by-condition interactions affecting relative standing.

2 RELATED WORK AND BACKGROUND

  • Existing studies primarily test repeatability of a fixed agent, while comparative reliability asks whether model or system rankings persist under new tasks, scaffolds, or benchmarks.A system can be repeatable within one harness while its rank is highly contingent on that harness.Source: §2 · p. 2
  • Generalizability Theory treats evaluation conditions as measurement facets and decomposes score variation into main effects and interactions.Model–scaffold compatibility can count as signal when selecting a deployable system but as error when ranking models independently of scaffolding.Source: §2 · pp. 2–3
  • Facet main effects cancel from comparisons only when objects share conditions and weights; object-by-facet interactions can change relative standing.A Decision study projects reliability before collecting more data.Source: §2 · p. 3

3 RELIABILITY AND GENERALIZABILITY FOR AGENT LEADERBOARDS

The section defines reliability relative to what should generalize—underlying models or deployable model–scaffold systems—and decomposes binary outcomes to project how evaluation design affects rankings. Its framework treats models, tasks, benchmarks, and scaffolds as facets of a sparse, partially crossed leaderboard.

3.1 OBSERVATIONS AND GENERALIZATION UNIVERSE

  • The model universe score is expected to persist across new tasks, benchmarks, and scaffolds resembling those represented in the leaderboard; generalization to benchmark-like tasks does not establish coverage of all real-world uses.The section also treats a model–scaffold pair as the object when selecting a deployable system.Source: §3.1 · p. 3

3.2 A LATENT VARIANCE DECOMPOSITION FOR BINARY OUTCOMES

  • Bayesian Bernoulli–logit mixed models decompose binary task outcomes into facet variation on a latent log-odds scale, avoiding Gaussian approximations that can mislead near floor or ceiling performance.The random-effects decomposition is related to random-item, many-facet Rasch models.Source: §3.2 · p. 3

3.2.1 BENCHMARK-LEVEL MODEL AND SYSTEM RELIABILITY

  • Model–scaffold interactions are rank-relevant because they encode which models benefit from which scaffolds; inter-scaffold reliability measures whether independently sampled scaffolds preserve model ordering on the same tasks.Low inter-scaffold reliability means scaffold changes can alter which model appears strongest, even when each scaffold yields internally stable scores.Source: §3.2.1 · p. 4

3.2.2 PERSISTENT MODEL DIFFERENCES ACROSS POOLED BENCHMARKS

  • The pooled leaderboard model estimates persistent model differences across sampled benchmarks, tasks, and scaffolds, while benchmark-conditioned capability is not assumed to transfer to a new benchmark.Shared models and scaffolds connect benchmarks, but hierarchical priors cannot empirically identify contrasts between disconnected components.Source: §3.2.2 · p. 4

3.3 WHAT ADDITIONAL EVALUATIONS CAN RESOLVE

  • More tasks monotonically reduce task-indexed error, but cannot remove model–benchmark heterogeneity or model–scaffold coupling, so task-only reliability has a ceiling.Distributing a fixed task budget across more benchmarks can increase reliability under the balanced, exchangeable-facet assumptions; breadth reduces condition-specific contamination rather than adding persistent model signal.Source: §3.3 · pp. 4–5

3.4 BAYESIAN ESTIMATION AND DESIGN STUDIES

  • Bayesian estimation with regularizing priors retains uncertainty in variance components when projecting reliability, signal-to-noise ratios, and task-only ceilings for new designs.Partial pooling lets connected observations inform common components while preserving uncertainty where replication is limited.Source: §3.4 · p. 5

3.5 POSTERIOR CAPABILITY, RANKS, AND TRANSPORTABILITY

  • The analysis tests transportability of the shared model component on four contemporaneous benchmarks excluded from model estimation, comparing its association with external performance against an in-panel mean-accuracy aggregate.This is an out-of-panel test of convergent predictive validity, not evidence of universal deployment validity or internal construct validity.Source: §3.5 · p. 5

4 DATA

The data section draws on nine Holistic Agent Leaderboard benchmarks and supplemental Harbor Index and external benchmark data. The HAL rollout design is sparse and imbalanced, but shared models and scaffolds connect its observations.

4 DATA

  • The HAL dataset covers nine benchmarks spanning web navigation, scientific programming, assistance, software engineering, customer service, and competitive programming; four use task subsets from their full benchmarks.The benchmarks are AssistantBench, CoreBench Hard, GAIA, Online-Mind2Web, SciCode, ScienceAgentBench, SWE-bench Verified Mini, τ-bench Airline, and USACO.Source: §4 · p. 6
  • HAL contains 29,923 task-level binary rollouts across 54 models and 13 scaffolds, with incomplete and uneven model–scaffold–benchmark coverage.Nine models appear on every benchmark, each spanning at least 69% of available scaffolds; shared scaffolds connect the benchmark observation graph and permit partial separation of effects.Source: §4 · p. 6
  • Two supplemental datasets provide corroboration: Harbor Index task-level scores for 29 benchmarks, and LLM-level scores for four contemporaneous benchmarks.The authors note that many model–scaffold–benchmark combinations are entirely absent because agent evaluations are expensive.Source: §4 · p. 6

5 RESULTS AND PRACTICAL RECOMMENDATIONS

The results show that evaluation reliability depends on what is being ranked and which conditions the intended claim should generalize across. Task replication alone cannot resolve uncertainty driven by scaffolds, while pooling diverse benchmarks can improve cross-task rankings at a fixed task budget.

5.1 RELIABILITY DEPENDS ON THE MEASUREMENT GOAL

  • Model reliability spans 0.148–0.841, while system rankings count scaffold differences and compatibility as signal, so reliability depends on whether models or model–scaffold pairs are ranked.The figure compares reliability as tasks are added for base models and fixed model–scaffold systems; its dashed lines show task-only asymptotes for model rankings.Source: §5.1 · pp. 6–7, Figure 1 · p. 7

5.2 CHANGING THE SCAFFOLD CAN CHANGE EVALUATION CONCLUSIONS

  • Inter-scaffold reliability ranges from 0.151 on OnlineMind2Web to 0.852 on CORE-Bench Hard, showing that scaffold choice can change model rankings.Task-specific variation across scaffolds exceeds task-specific variation across models with 84% posterior probability, and scaffolds can change which tasks are solved without uniformly improving benchmark scores.Source: §5.2 · p. 7, Figure 2 · p. 8

5.3 MORE TASKS CANNOT ALWAYS RESOLVE EVALUATION UNCERTAINTY

  • Even infinitely many similarly constructed tasks improve model-ranking reliability by at most 0.10; OnlineMind2Web rises only from 0.148 to 0.153.Add tasks when task sampling is the main uncertainty source; otherwise, broaden relevant coverage, such as scaffold coverage, because repeated tasks can leave broader model differences unresolved.Source: §5 · p. 6, §5.3 · p. 8

5.4 POOLING DIVERSE BENCHMARKS CAN MAKE MODEL RANKINGS MORE RELIABLE

  • Pooling the nine-benchmark battery raises projected model-ranking reliability from approximately 0.44 for a single benchmark to 0.75 at the same task budget.A balanced allocation achieves comparable reliability for approximately $19,000 versus more than $47,000 for the full HAL battery; one illustrative allocation estimates an 83% cost reduction. Model-effect estimates also agree more closely with four held-out benchmarks than mean scores do (mean Kendall’s τ: 0.789 vs. 0.463), but this is not proof of a universal one-dimensional agent capability.Source: Figure 3 · p. 8, §5.4 · p. 9, Table 3 · p. 9
  • For cross-task claims, benchmark breadth averages condition-specific model advantages that within-benchmark replication leaves untouched.The authors recommend pooling benchmarks that represent the intended task range while preserving enough task diversity and identifying informative anchors in imbalanced designs.Source: §5.4 · p. 9

6 CONCLUSION

The conclusion emphasizes that reliability is not intrinsic to a benchmark: it depends on the object, score interpretation, and task range of interest. Evaluation design should follow the intended claim, and more evaluation is useful only when it addresses the uncertainty that limits that claim.

6 CONCLUSION

  • Reliability depends on whether practitioners seek model or system rankings, absolute scores, or performance across a broader range of agentic tasks.The pooled estimation correlates more strongly with benchmark-wise observed scores than per-benchmark estimates, but posterior rank intervals remain wide and some neighboring models have ordering probabilities near 1/2.Source: §6 · p. 9
  • Practitioners should specify what a score or ranking should mean, identify the uncertainty that limits that claim, and target additional evaluation effort accordingly.More evaluation is useful only when it addresses those sources of uncertainty.Source: §6 · p. 10
  • The authors report that code and data are available online, with original data sources and computation details provided in the appendices.The paper also states that AI was used during writing and that the code was human-written and validated against paper findings under subsampling.Source: REPRODUCIBILITY STATEMENT · p. 10, AI USE STATEMENT · p. 10

LIMITATIONS AND FUTURE WORK

The reported reliability bounds are conditional on the evaluated models, scaffolds, tasks, and benchmarks, and reliability alone does not establish validity. The appendix describes the datasets and external comparisons used, while identifying limits on scope, scale, and intervention claims.

LIMITATIONS AND FUTURE WORK

  • Reliability bounds describe how consistently a benchmark differentiates ranked objects, not whether rankings match capability or external ground truth.The authors call for complementary validity studies, including external-criterion validation and cross-benchmark transfer analyses.Source: LIMITATIONS AND FUTURE WORK · p. 15
  • The ceiling applies to current leaderboards and their frontier-model competitor set; closely clustered models can lower reliability regardless of benchmark quality.Per-benchmark scaffold counts are limited, and pooled scaffold-variance claims rely on HAL-GENERALIST as a cross-benchmark harness anchor.Source: LIMITATIONS AND FUTURE WORK · p. 15
  • Claims are conditional on the sampled models and scaffolds, and generalize only insofar as those conditions represent the intended evaluation population.Partial pooling does not by itself correct selective evaluation or reporting.Source: LIMITATIONS AND FUTURE WORK · p. 15
  • Reliability is estimated on the latent logit scale, which does not translate one-to-one to observed proportions, so reported values are approximate observed-scale rank-stability bounds.The authors identify comparison across scales as future work.Source: LIMITATIONS AND FUTURE WORK · p. 15
  • The study characterizes reliability bounds rather than validating interventions to raise them; candidate interventions include task selection, scaffold sampling, and task-quality audits.The authors also identify reducing the measurement cost of these recommendations as a direct extension.Source: LIMITATIONS AND FUTURE WORK · p. 15

A.1 HAL DATASET

  • The Holistic Agent Leaderboard is a standardized, cost-aware, third-party evaluation platform and dataset initiative, with cleaned task-level data publicly released.Source: §A.1 · p. 15

A.1.1 BENCHMARKS

  • The benchmark set spans software engineering, customer service, web interaction, scientific coding, and other agent tasks.Examples include SWE-bench, τ-bench, ScienceAgentBench, USACO, GAIA, and SciCode.Source: §A.1.1 · pp. 16–17

A.1.2 SCAFFOLDS

  • HAL scaffolds contrast generalist systems with benchmark-specialist systems, but USACO, SWE-bench Verified Mini, and ScienceAgentBench have limited model–scaffold diversity.USACO has especially little diversity, with large intervals when generalizing across scaffolds.Source: §A.1.2 · p. 17

A.3.1 DIFFERENCES IN CONCORDANCE WITH EXTERNAL PERFORMANCE

The reliability-adjusted latent model effect agreed more closely with aggregated external performance than the mean score in each examined benchmark, though the within-benchmark evidence was descriptive. The appendix also details a multilevel comparison, coverage gaps, and the variance-decomposition models used for reliability estimates.

A.3.1 DIFFERENCES IN CONCORDANCE WITH EXTERNAL PERFORMANCE

  • 0.222–0.533: the latent estimated value’s advantage in Kendall’s τb over mean score ranged from SWE-bench to τ 2-core.Both scoring mechanisms were positively associated with external performance in every benchmark, and the latent estimate showed stronger agreement throughout.Source: §A.3.1 · p. 19, Table 5 · p. 19
  • The direction of the latent estimate’s advantage survived every single-model deletion, though this does not establish stability to broader model-sample changes.For BFCL, a deletion could reduce the difference to zero but did not reverse its sign.Source: §A.3.1 · p. 19
  • Bootstrap intervals did not exclude zero, so the observed within-benchmark advantage is descriptive rather than statistically conclusive.The intervals were exploratory given small model samples; the table caption notes they are marginal and unadjusted for multiple comparisons.Source: §A.3.1 · pp. 19–20, Table 5 · p. 19
  • External agreement is with aggregated observed mean references; the analysis does not propagate reference-score uncertainty or adjust for unequal source coverage.Source: §A.3.1 · p. 20

A.3.2 COMBINED EXTERNAL DATA MULTILEVEL PARTIAL CORRELATION

  • The multilevel partial Kendall correlation compares observed and proxy ranks after accounting for benchmark and leaderboard clustering and observed covariates.The random-effects model retains separate SWE-bench leaderboard observations, increasing the analysis to 52 observations.Source: §A.3.2 · p. 20, Table 6 · p. 20

B HAL COVERAGE MATRIX

C METHODS, ESTIMATION, AND COMPUTATIONAL DETAILS

  • The pooled Bayesian generalized linear mixed models jointly estimate variance components, partially pool sparse cells, and propagate uncertainty into reliability coefficients.The leaderboard model covered 29,923 observations across 9 benchmarks, 1,117 benchmark–task units, 54 models, and 13 scaffolds; variance components use the latent log-odds scale.Source: C METHODS, ESTIMATION, AND… · p. 21, §C.1.1 · pp. 21, 23

C.1.1 LEADERBOARD-LEVEL POOLED MODEL

  • The leaderboard model separates persistent model and scaffold differences from benchmark, task, and interaction variation, including benchmark-dependent model performance and model–scaffold compatibility.The benchmark-level fits distinguish stable model variation from task-, scaffold-, and interaction-driven variation; observation counts alone do not determine ranking information.Source: §C.1.1 · p. 23, §C.1.2 · p. 23

C.1.3 PRIORS AND POSTERIOR COMPUTATION

The appendix details the Bayesian model and robustness checks behind the leaderboard analysis, then compares latent rankings, predictive stability, and an observation-level random-effect ablation. Across these checks, partial pooling supports ecosystem-level estimates, while sparse cells chiefly constrain local prediction.

C.1.3 PRIORS AND POSTERIOR COMPUTATION

  • Weakly informative priors regularize effects supported by few levels without forcing them to equality, and reported convergence diagnostics meet the study’s R-hat threshold.The full fit used six chains and yielded 3,750 retained draws; all reported estimates achieved R-hat < 1.05, with most values rounding to 1.00.Source: §C.1.3 · pp. 23–24

C.2–C.5 ALTERNATIVE ESTIMATION METHODS

  • Sensitivity checks also compare Gaussian mixed-effects, logistic mixed-effects, and nonparametric DISCO decompositions, with DISCO capturing nonlinear dispersion without distributional assumptions.The Gaussian LME is presented as a familiar baseline despite binary outcomes; DISCO dispersion shares sum to one across explained dispersion.Source: §C.3 · p. 25, §C.4 · p. 25, §C.5 · p. 25

C.6 DETAILS ON COMPUTE-USAGE; C.7 POSTERIOR ESTIMANDS

  • The appendix reports computation requirements and identifies the paper’s set of posterior estimands, but the supplied passages do not state their individual values.Main variance-decomposition fits took 8 total hours, and the leave-one-out ablations took 57 hours on an Apple M1 Max.Source: §C.6 · p. 25, §C.7 · p. 25, Table 9 · p. 26

D.1 REFERENCE INDEXED VARIANCE DECOMPOSITION

  • For fixed-condition model comparisons, shared benchmark and scaffold shifts can affect absolute scores while canceling from relative rankings.Consequently, benchmark–scaffold instability need not imply instability in relative model reliability.Source: §D.1 · p. 26

D.2 SENSITIVITY OF MODEL RANKINGS

  • Full-model latent rankings retain substantial agreement with observed leaderboards on six benchmarks, while independently fitted per-benchmark rankings correspond less strongly to raw rankings across benchmarks.Full-model Spearman correlations are at least 0.85 for CORE-Bench Hard, GAIA, Online-Mind2Web, SciCode, SWE-bench Verified Mini, and τ-Bench Airline; shared models and scaffolds connect otherwise isolated benchmark designs.Source: §D.2 · p. 27, Table 10 · p. 27

D.3 POINTWISE PREDICTIVE STABILITY WITH PSIS-LOO

  • Only 21 of 29,923 observations have Pareto shape diagnostic b_k > 0.7, and none has b_k > 1; 19 of those 21 warnings occur in singleton groups.Deleting a singleton’s only observation makes its random-effect estimate approximately prior-predictive, so a large b_k chiefly signals difficulty predicting that isolated group, not that population-level variance components depend on it.Source: §D.3 · p. 28, Table 11 · p. 28

D.3.1 OBSERVATION-LEVEL RANDOM-EFFECT ABLATION.

  • Including an observation-level random effect did not materially change reliability estimates because its variation was attenuated by task replication in the D-study.The term is an observation-level effect in the generally unreplicated HAL cells, rather than an identifiable replicated three-way interaction; multiple runs under the same conditions can instead support measuring score stochasticity.Source: §D.3.1 · pp. 28–29, Figure 6 · p. 29

D.4 LEAVE-ONE-BENCHMARK-OUT SENSITIVITY

Leave-one-benchmark-out refits test whether the estimated variance decomposition and design recommendations persist when an observed measurement instrument is removed. Estimates are generally stable, but CORE-Bench Hard is an unusually informative benchmark and the observed ecosystem has limited redundancy at its high-signal end.

D.4 LEAVE-ONE-BENCHMARK-OUT SENSITIVITY

  • Removing CORE-Bench Hard causes the largest reduction in estimated reliability; without it, the D-study posterior signal-to-noise ratio no longer reaches the prespecified limit-of-detection band.CORE-Bench Hard supplies the most runs, the widest observed accuracy range, and the highest estimated signal-to-noise ratio among the included benchmarks, combining replication with discrimination.Source: §D.4 · p. 29, Table 12 · p. 30
  • Reliability estimates, model variance proportions, and principal interaction facets are generally stable across benchmark-withholding refits, though sensitivity varies with the benchmark removed.A large but noisy benchmark can contribute little to reliability; benchmark count alone is not equivalent to effective measurement information.Source: §D.4 · p. 29
  • These refits test robustness to removing one observed measurement instrument, not prediction of an arbitrary future benchmark with no shared structure.The CORE-Bench Hard sensitivity indicates limited redundancy among the nine observed benchmarks at the high-signal end.Source: §D.4 · p. 30

D.5 LEAVE-ONE-BENCHMARK–SCAFFOLD-OUT SENSITIVITY

Leave-one-benchmark–scaffold and leave-one-LLM-out analyses probe how sensitive variance components and rankings are to individual design elements. The main model-related conclusions are comparatively stable, while benchmark coverage and high-discrimination instruments matter more than a single LLM’s influence.

D.5 LEAVE-ONE-BENCHMARK–SCAFFOLD-OUT SENSITIVITY

  • In leave-one-benchmark–scaffold-out refits, task difficulty accounts for approximately 32.8% of variance, benchmark differences 25.8%, and residual variation 23.7%.Their coefficients of variation are 0.047, 0.060, and 0.035, respectively; the model component ranges from 0.026 to 0.044.Source: §D.5 · p. 30
  • The largest relative variation in benchmark–scaffold deletions occurs in weakly identified scaffold-related effects, but common scaffold shifts cancel from same-condition model contrasts.The task–model component has CV 0.426 but a variance share of only 0 to 0.006, so its high relative variability reflects a small denominator.Source: §D.5 · pp. 30–31
  • No individual benchmark–scaffold cell appears to generate the principal conclusion that bare-model rankings are less reliable than rankings treating observed model–scaffold performance as signal.Removing the CORE Agent benchmark–scaffold combination produces a large increase in uncertainty across the pooled model, consistent with its strong between-LLM discrimination.Source: §D.5 · p. 31

D.6 LEAVE-ONE-LLM-OUT SENSITIVITY

  • Removing an LLM deletes its main-effect level and all observed interactions involving it; the ablations used frequentist fits to reduce the cost of hundreds of refits.Only nine of the 54 models appear in every benchmark, so this also tests sensitivity to highly connected models.Source: §D.6 · p. 31
  • Across leave-one-LLM-out refits, model variance remains between 0.028 and 0.036, with mean 0.033 and CV 0.048.The largest variance components are also stable: task-within-benchmark variance has mean proportion 0.328 and CV 0.006, residual variance 0.236 and CV 0.008, and benchmark variance 0.259 and CV 0.009.Source: Table 13 · p. 30, §D.6 · p. 31
  • Leave-one-model results are more stable than leave-one-benchmark results because models are partially pooled, whereas a benchmark deletion removes a whole instrument and its tasks.The ecosystem therefore has greater redundancy across models than across high-quality benchmarks.Source: §D.6 · pp. 31–32
  • Stable variance components after model deletion do not establish prediction of an unseen model; that generalization additionally assumes exchangeability and connected evaluation conditions.The ablation tests influence on the population variance decomposition, not the ability to predict an unseen model’s performance or rank.Source: §D.6 · p. 32

D.8 RANK STABILITY ACROSS POSTERIOR

  • Posterior capability rankings estimate performance persisting across specified benchmark and scaffold universes, unlike raw percent-correct rankings that credit each observed pairing.The supplied passages describe posterior rank distributions and comparisons with published ranks but do not provide their results.Source: §D.8 · p. 32, Table 14 · p. 32

D.9 IMPLICATIONS FOR BENCHMARK DESIGN

The design implications emphasize that reliable rankings require enough overlap to separate model variation from benchmark, scaffold, and interaction variation. The appendix formalizes how connectivity and observed response patterns support those contrasts, while distinguishing structural identifiability from practical precision.

D.9 IMPLICATIONS FOR BENCHMARK DESIGN

  • Sparse cells chiefly limit local prediction: PSIS warnings cluster in singleton or doubleton interaction levels, while the population-level variance decomposition is stable to those observations.Per-benchmark data are generally insufficient to separate model capability from scaffold and task interactions; cross-benchmark connectivity stabilizes latent rankings, though low-signal benchmarks remain uncertain.Source: §D.9 · p. 33
  • Benchmark quality is not interchangeable with quantity: the analysis identifies CORE-Bench Hard as a high-information anchor and favors multiple discriminating benchmarks with controlled error.Simply adding noisy tasks or near-duplicate benchmarks is not the stated design goal.Source: §D.9 · p. 33
  • Absolute-score and relative-rank reliability are distinct because scaffold and benchmark–scaffold effects can shift accuracies while canceling from same-condition comparisons.Reports should specify whether reliability concerns absolute deployment performance, model ranking, or model–scaffold system ranking.Source: §D.9 · p. 33
  • Once a reasonably diverse model sample is obtained, additional resources may yield greater reliability gains from benchmark quality, task replication, and cross-scaffold connectivity than from sparsely evaluated models.The robustness checks localize rather than eliminate sparse-design consequences: individual-ranking precision still depends on connectivity and a high-signal instrument.Source: §D.9 · p. 33

E IDENTIFIABILITY AND CONNECTIVITY OF THE LEADERBOARD; E.2 OBSERVED INCIDENCE STRUCTURE

  • A fully crossed leaderboard is unnecessary for the studied questions, but sufficient overlap is needed to distinguish persistent model variation from condition-related variation.The observed data include nine benchmarks, 54 LLMs, and 13 scaffolds; nine models appear on every benchmark and one scaffold is shared by eight benchmarks.Source: E IDENTIFIABILITY AND CONNECTIVITY OF… · p. 33, §E.2 · p. 34

E IDENTIFIABILITY AND CONNECTIVITY OF THE LEADERBOARD

  • Connectivity, structural identifiability, and practical estimability are distinct: a proper posterior for a weakly informed component does not show that data identify it precisely.Practical estimability also depends on sample size, outcome variation, and prior regularization, so the paper uses uncertainty and sensitivity analyses.Source: E IDENTIFIABILITY AND CONNECTIVITY OF… · p. 33

E.1 INCIDENCE-GRAPH CONNECTIVITY AND IDENTIFIABILITY; E.2 OBSERVED INCIDENCE STRUCTURE

  • Shared models and scaffolds connect benchmark observations in an incidence graph; contrasts across disconnected components are not identified without added assumptions.The observed graph places all benchmarks in one comparison network, with nine models spanning every benchmark and shared scaffolds connecting the benchmark-specific branches.Source: §E.1 · p. 34, §E.2 · p. 34

E.3 IDENTIFIABILITY OF RANDOM-EFFECT VARIANCES; E.4 HOW THE OBSERVED DESIGN SEPARATES THE REQUIRED COMPONENTS

  • Variance components are structurally identifiable when their observed-design covariance kernels are linearly independent after removing components confounded with terminal residual variation.Complete crossing is sufficient but not necessary: distinguishing components requires observation pairs that share one grouping without always sharing the other.Source: §E.3 · p. 35

E.4.1 MODEL AND BENCHMARK–MODEL VARIATION

The connected leaderboard design uses overlapping models, scaffolds, and model–scaffold pairs to distinguish persistent effects from benchmark-conditioned variation. Some variance combinations and task-only reliability ceilings are identifiable, but sparse replication leaves uncertainty and limits the scope of claims.

E.4.1 MODEL AND BENCHMARK–MODEL VARIATION

  • Nine models appear on every benchmark, distinguishing persistent model variation from benchmark-conditioned model variation.The shared models link observations across benchmarks, allowing the two components to have different support.Source: §E.4.1 · p. 35

F ADDITIONAL TABLES AND FIGURES

  • Additional tables report observed overlap patterns and posterior-mean variance shares for benchmarks and the full leaderboard.The coefficient discussion explains how comparisons across reliability coefficients reveal signal retained when generalizing over tasks, scaffolds, or both.Source: Table 15 · p. 36, F ADDITIONAL TABLES AND FIGURES · p. 38

E.4.2 SCAFFOLD AND BENCHMARK–SCAFFOLD VARIATION

  • A scaffold spanning eight benchmarks and a second shared scaffold connect scaffold effects to benchmark–scaffold interactions.Benchmark-specific scaffolds are less individually informed and rely on hierarchical-model exchangeability assumptions and partial pooling.Source: §E.4.2 · p. 36

E.4.3 MODEL–SCAFFOLD AND BENCHMARK–MODEL–SCAFFOLD VARIATION

  • Recurring model–scaffold pairs help distinguish general scaffold compatibility from benchmark-specific compatibility, while item effects are identified through repeated attempts.Model–scaffold pairs recur less often than models, and item-conditioned interactions are informed by multiple models or scaffold replications within a benchmark.Source: §E.4.3 · p. 36, §E.4.4 · pp. 36–37

E.5 WHAT IS NOT SEPARATELY IDENTIFIABLE

  • Without repeated executions of each benchmark–item–model–scaffold cell, four-way interaction, execution variation, and Bernoulli residual are not separately identifiable.Separating execution stochasticity and grading instability would require repeated runs in the same recorded cell and ideally explicit grading replicates.Source: §E.5 · p. 37

E.6 IDENTIFIABILITY OF THE RELIABILITY ESTIMANDS

  • Connected overlap patterns distinguish the variance combinations used by the leaderboard coefficient, including its task-only ceiling, without resolving every terminal residual source.The stated conditions include cross-benchmark models and scaffolds, recurring model–scaffold pairs, and items attempted by multiple models and scaffolds.Source: §E.6 · pp. 37–38

E.7 BAYESIAN REGULARIZATION AND EVIDENTIAL SCOPE

  • Bayesian posterior uncertainty is retained when components are weakly informed, and large credible intervals indicate too few common evaluation conditions for precise claims.The analysis uses proper, weakly informative priors and sensitivity checks, rather than treating priors as substitutes for disconnected comparisons.Source: §E.7 · p. 38

E.8 CONNECTIVITY SUMMARY

  • The incomplete but connected design supports population-level reliability claims, not a complete factorial decomposition or predictions for every missing cell.Nine cross-benchmark models, shared scaffolds, and recurring model–scaffold pairs anchor the variance combinations used in the analysis.Source: §E.8 · p. 38

F.1 REPEATED TASK SUBSAMPLING

Repeated task subsampling measures how much of the full-data ranking is retained as fewer tasks are evaluated, while latent-scale analyses clarify reliability for binary outcomes. The supplementary analyses also compare estimation approaches and use DISCO as a sensitivity check on interaction-heavy dispersion.

F.1 REPEATED TASK SUBSAMPLING

  • The subsampling procedure uses the same randomly selected tasks for all models and compares each subsample ranking with the complete-data ranking.It averages scores at the benchmark–model level, gives benchmarks equal weight, and repeats the paired sampling 500 times.Source: §F.1 · pp. 38–39
  • 15 tasks per benchmark retain mean Spearman agreement of 0.932 with the complete-data ranking at an estimated 82% cost reduction.Across 500 paired subsamples, 30 tasks increase agreement to 0.976; agreement with the full ranking measures information retention, not ranking validity.Source: §F.1 · p. 39

G.1 LATENT SPACE ESTIMATIONS

  • Latent-scale reliability concerns differences in log-odds, not raw percent-correct, whose variability depends on position on the sigmoid curve.The posterior predictive mapping describes how latent-scale reliability manifests as leaderboard instability in percent-correct.Source: §G.1 · p. 40
  • Latent rank-order reliability estimates exceed observed-scale estimates in proportion to the number of low-scoring models, especially in SciCode.In SciCode, all models have accuracy below 0.1; the logistic mixed-effects approach also handles unbalanced data and near-zero values better than the observation-level linear approach.Source: §G.1 · pp. 40–41

G.1.1 VARIANCE SHARES AND RELIABILITY ON THE LATENT SCALE

  • For binary-response models, variance shares are calculated per posterior draw, yielding credible intervals for the proportions of total latent variance.When sampled facets appear in interactions, their contribution to averaged-score variance shrinks as the number of sampled levels increases.Source: §G.1.1 · p. 41

G.2 DISTANCE COMPONENTS (DISCO) QUASI-RELIABILITY

  • DISCO interaction estimates are an upper bound on non-additive effects because pairwise distances capture nonlinear dependencies that additive random-effects models cannot represent.DISCO has no separate residual term, and independently computed facet proportions do not generally sum to one.Source: §G.2 · p. 45
  • DISCO serves as a sensitivity analysis for whether dominant facets and low model-ranking reliability depend on Gaussian random-effect assumptions.It decomposes dispersion using pairwise distances and between- and within-group terms.Source: §G.2 · p. 45

G.3 WHY ESTIMATION METHOD CHANGES CONCLUSIONS—AND WHAT TO DO ABOUT IT

The section compares estimation methods and shows that their variance decompositions can lead to different conclusions, especially with sparse observations. It also distinguishes reproducible rankings from reproducible score levels and explains how task- and scaffold-related variation limits generalization.

G.3 WHY ESTIMATION METHOD CHANGES CONCLUSIONS—AND WHAT TO DO ABOUT IT

  • The five estimation methods are not interchangeable: sparse observations produce different variance shares and reliability estimates across LMM, Bayesian LMM, GLMM, Bayesian GLMM, and DISCO.The paper recommends communicating uncertainty with Bayesian or nonparametric methods when data are sparse and treating disagreement as evidence that a benchmark is under-instrumented for the inference.Source: §G.3 · p. 45
  • Linear G-theory estimates for pass/fail benchmarks underestimate named-facet variance and overestimate residual noise relative to logit-link models.The Gaussian identity-link model treats binary responses as continuous, so its residual absorbs Bernoulli variance that logit-link models separate.Source: §G.3 · p. 45
  • Bayesian agent-variance posterior means are 2–5× larger than posterior medians, and several benchmarks have 95% model-variance HDIs that include zero.A frequentist GLME point estimate misses posterior tail mass and does not expose this decision-relevant uncertainty.Source: §G.3 · p. 46
  • DISCO attributes 40–57% of total dispersion to task×model interactions, compared with 1–11% from Bayesian estimates.The paper treats DISCO as an upper bound because it captures nonlinear dependencies and has no separate residual term.Source: §G.3 · p. 46

G.4 VARIANCE DECOMPOSITION EXPLAINS THE RELIABILITY CEILING AND GENERALIZABILITY

  • Persistent model variation is smaller than combined scaffold, benchmark-conditioned model, and item-conditioned interaction variation in the leaderboard decomposition.Task-level model and scaffold interactions exceed their respective main effects, so the section describes choosing a model and scaffold for each task rather than relying on a group-level choice.Source: §G.4 · p. 46, Figure 16 · p. 46

G.5 ABSOLUTE RELIABILITY ON LINEAR SCALE

  • Relative reliability can be high while absolute reliability remains low, so stable model ordering does not ensure reproducible score levels.Common shifts in scores can affect absolute decisions even when model rankings remain unchanged.Source: §G.5 · p. 47, §G.5.1 · p. 47
  • The HAL benchmarks do not contain enough signal relative to noise for reliable absolute scoring.Absolute reliability may still matter for other agentic measures when scores are used for threshold or target decisions.Source: §G.5 · p. 47

G.6 FULL ESTIMATION TABLES FOR PROPORTION OF LATENT VARIATION EXPLAINED

  • Full tables provide per-benchmark and full-leaderboard variance proportions, with Bayesian estimates reported as posterior means for cross-method comparability.The section presents these estimates alongside LME, GLME, and DISCO results.Source: §G.6 · p. 48, Table 20 · p. 49, Table 21 · p. 50

Related papers

Loading 2610.00651v1…