PaperlayerView exact revision on arXiv

Source-linked AI summary

One Capability or Many? Testing the Economic Validity of Frontier AI Evaluation

Louis Yiven Zhu

arXiv:2608.29420v1cs.LGcs.CYcs.SE

TL;DR

The paper asks whether economic benchmarks measure a capability distinct from general test-taking, a construct-validity question for frontier-model leaderboards. It tests benchmark structure and held-out prediction using a fixed protocol, finding no distinct economic factor under its pre-specified rule but improved prediction from a multi-factor representation.

  • Problem

    Whether economic benchmarks measure a distinct capability or re-express a general progress axis is an under-examined construct-validity question for leaderboards.

    Source: §1 · pp. 1–2

  • Method

    The study applies a pre-specified construct-validity protocol to a covered leaderboard battery, combining structural tests with leave-one-benchmark-out prediction.

    Source: §1 · p. 2, §3 · p. 3, §4.3 · p. 5

  • Results

    Economic benchmarks form no distinct capability under the pre-specified structure test, yet the multi-factor representation predicts held-out economic scores better on three of four benchmarks; one general factor explains 74.5% of common variance.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Economic benchmarks add predictive information to a general index without yet constituting a distinct capability, and comparisons across release dates should account for the calendar trend.

    Source: §6 · p. 8

    The observational analysis establishes an association between release date and the general factor, not what drives that association.

    Source: §6 · p. 8

Abstract

from arXiv · show

Frontier-model leaderboards now rank systems based on economic benchmarks, tests of how well models carry out professional tasks from software engineering to banking workflows, and those rankings inform what organisations buy, what regulators scrutinise, and expectations of how work will change. Whether such benchmarks measure a capability distinct from general test-taking, or re-express the one axis along which every benchmark rises as models improve, is a question of construct validity that has not yet been studied. We test it on a hash-pinned leaderboard snapshot of 421 model configurations across twelve benchmarks, four of them economic, treating benchmarks as items and models as respondents in a latent-variable model with four hypotheses and their thresholds fixed before analysis. A single factor explains 74.5% of common variance and tracks model release date (R^2 = 0.505), so the leading axis of capability is substantially a time trend; where prior work controls for scale, compute adds little once date is removed. Removing the date trend lowers that share by 14.9 points, and by 24.1 with one row per base model. Under the dimensionality rule fixed in advance the economic benchmarks form no distinct factor, yet a leave-one-benchmark-out test with factors re-estimated inside every fold shows that a multi-factor representation predicts held-out economic scores better than a single general index (pooled Delta-MSE 0.037, 95% bootstrap interval [0.019, 0.055]). Economic benchmarks therefore add incremental predictive information to a largely date-driven general factor, and the evidence does not support treating them as a distinct latent capability. Leaderboards remain a sound guide to overall progress, but most of the gap between models released months apart is calendar, so a small gap between contemporaneous models should be date-adjusted before being read as a capability difference.

1 Introduction

Economic benchmarks for professional work are increasingly included in frontier-model leaderboards, raising the question of whether they measure a distinct capability or reproduce a general progress axis. The paper frames this as a construct-validity test and organizes its analysis around four pre-specified research questions.

1 Introduction

  • Economic benchmarks proxy paid professional work and multi-step tool use, while leaderboards increasingly inform procurement and policy.Examples include banking customer support and agentic coding suites.Source: §1 · p. 1
  • The central question is whether economic benchmarks capture a capability distinct from general test-taking or re-express one dominant axis of model progress.A separate leaderboard column may overstate what the leaderboard measures if the benchmarks only re-express that shared axis.Source: §1 · p. 2
  • The paper treats construct validity as whether a measurement captures the construct it claims to measure, a standard urged for AI benchmarks but found under-examined in recent audits.The passages cite Cronbach and Meehl [1955], Messick [1998], Raji et al. [2021], Jacobs and Wallach [2021], Bowman and Dahl [2021], Bean et al. [2025], and Kearns [2026].Source: §1 · p. 2
  • The analysis frames benchmark scores as a psychometric problem, with benchmarks as items and models as respondents, and maps four research questions to pre-specified hypotheses.The questions address factor dominance, release-date effects, economic-factor distinctiveness, and prediction of held-out economic scores.Source: §1 · p. 2
  • The paper contributes a construct-validity test for economic benchmarks, release-date adjustment, a leave-one-benchmark-out predictive test, and a protocol fixed in advance.The protocol uses hash-pinned inputs and reports failed tests as failed.Source: §1 · p. 2

2 Related work

Prior work treats model scores as psychometric data and audits benchmarks as evaluation artefacts. The paper locates its contribution in testing the economic benchmark column with both structural and out-of-sample predictive evidence.

2 Related work

  • Psychometric studies report a general factor across model tests, multiple capability factors, and prediction from score-based capability spaces.Ilić and Gignac [2024] report a general factor across 591 models and twelve tests; Burnell et al. [2023] resolve three factors across 29 LLMs and 27 tasks; Ruan et al. [2024] use principal components for prediction, including agentic performance from non-agentic benchmarks.Source: §2 · p. 3
  • Kearns [2026] finds a single factor explaining 72% of communal variance on the Open LLM Leaderboard and residualises scores on a scaling law; this paper uses release date as its confound.The passage reports that prior residualisation lowers mean inter-item correlation from 0.64 to 0.48, while this paper finds compute adds little once date is removed.Source: §2 · p. 3
  • Benchmark audits report limited construct-validity evidence and sparse uncertainty reporting, while temporal studies identify saturation related to benchmark age.Bean et al. [2025] report that 53.4% of 445 benchmarks present any construct-validity evidence and 16.0% report uncertainty estimates or statistical tests when comparing models; Akhtar et al. [2026] find nearly half of sixty benchmarks are saturated.Source: §2 · p. 3
  • The cited prior work neither isolates the economic benchmark column nor pairs a structural test with an out-of-sample predictive test.Source: §2 · p. 3

3 Data and design

The study uses a hash-pinned leaderboard snapshot and a pre-fixed coverage rule to define its benchmark battery. Because economic scores cover fewer models, the main hypothesis tests use a complete-case grid, with a larger dense grid reserved for corroborating clustering.

3 Data and design

  • The primary data are a leaderboard snapshot captured on 6 July 2026, containing 548 configurations with benchmark scores, release dates, and provider metadata.The snapshot is pinned by SHA-256 hash; Epoch AI compute figures serve only as a scale robustness check.Source: §3 · p. 3
  • A pre-fixed coverage rule leaves 421 configurations and twelve benchmarks, including four economic benchmarks, after excluding sparse APEX-Agents and demoting MMMU-Pro to sensitivity analysis.The rule retains benchmarks with scores for at least sixty models and models with scores on at least eight of thirteen sufficiently covered benchmarks; one row per base model is a robustness check.Source: §3 · p. 3
  • The 96-model complete-case grid is the only grid with all twelve benchmarks and carries every hypothesis; a 409-model grid of nine near-universal benchmarks supports corroborating clustering.The complete-case grid represents frontier configurations on which the economic benchmarks have been run.Source: §3 · p. 3
  • Benchmarks are standardized, structural missingness is handled by complete-case analysis without imputation, and the study limits its scope to internal validity on one cross-section.Item-level responses, external economic impact, and saturation dynamics are outside scope.Source: §3 · p. 4

4 Methods

The methods combine a latent-structure analysis with a leave-one-benchmark-out prediction test. The analysis specifies factor-count criteria and four quantitative hypotheses in advance, while documenting an exploratory three-factor over-extraction used for the predictive comparison.

4.1 Task 1: latent structure

  • The primary structural analysis uses maximum-likelihood exploratory factor analysis with oblique oblimin rotation, and parallel analysis sets the primary factor count.The first factor’s share of common variance measures dominance; the rotation allows correlated factors rather than imposing independent axes.Source: §4.1 · p. 4
  • Date adjustment regresses each standardized benchmark on release date, then re-fits the factor analysis on residuals to measure how much of the apparent general factor aligns with date.Training compute is included only on a subsample; clustering provides a corroborating structural analysis.Source: §4.1 · p. 4

4.2 Task 2: predictive validity

  • The leave-one-benchmark-out test predicts each of four economic targets from the other eleven benchmarks on the 96-model grid, comparing a multi-factor representation with a single mean-score index.The predictor ladder also includes release timing and scale, the first factor alone, and factors plus model covariates.Source: §4.2 · p. 4
  • Nested five-fold cross-validation supplies out-of-fold errors, with factors re-estimated inside each training fold to prevent leakage.The comparison uses four learners; its test statistic is ΔMSE = MSE(single index) − MSE(k factors), with a paired-bootstrap 95% interval.Source: §4.2 · pp. 4–5

4.3 Pre-specified hypotheses

  • Four hypotheses and thresholds were fixed before analysis; the primary factor count is one, while the three-factor solution is an exploratory over-extraction used for loading patterns and prediction.The predictive conclusion is reported as robust to sweeping the factor count from two to five.Source: §4.3 · p. 5

5 Results

One factor dominates benchmark scores but tracks release date; economic benchmarks do not meet the pre-specified distinct-factor test, yet multiple factors improve held-out economic prediction over a single index.

5.1 Dominance of a single factor (H1)

  • The first factor accounts for 74.5% of common variance, and parallel analysis retains a single factor, supporting a dominant capability axis.The first principal component accounts for 79.4% of total score variance.Source: §5.1 · p. 5

5.2 A date-driven artefact (H2)

  • The leading factor tracks release date (logistic-fit R2 = 0.505); date adjustment reduces its common-variance share from 74.5% to 59.6%, a 14.9-point drop.The primary estimate misses the 15-point threshold; base-model and compute-known grids drop 24.1 and 16.5 points, respectively, while date-plus-compute drops 9.3.Source: §5.2 · pp. 5–6, Figure 2 · p. 6

5.3 Economic distinctiveness (H3)

  • The pre-specified dimensionality rule retains one date-adjusted factor, so economic benchmarks do not meet the planned distinct-factor test.Only three-factor over-extraction separates them: all four load at least 0.40 on F1, with the largest cross-loading at 0.38.Source: §5.3 · p. 6, Figure 2 · p. 6

5.4 Predictive validity (H4)

  • Pooled ∆MSE was +0.037 (95% interval [+0.019, +0.055]), supporting k factors over the mean index for held-out economic prediction.Economic-block test RMSE falls from 0.474 (mean index) to 0.433 (k factors). On GDPval, SHAP ranks agentic factor F1 first; τ 2-Bench’s interval includes zero.Source: Table 1 · p. 7, §5.4 · pp. 7–8, Figure 3 · p. 7

5.5 Robustness

  • Robustness checks preserve the headline structure; collapsing 421 configurations to one row per base model raises the first-factor share to 90.2% and the date-adjustment drop to 24.1 points.The predictor ladder’s ordering holds on all twelve targets, and the H4 verdict is insensitive to regularisation strength and factor count.Source: §5.5 · p. 8

6 Discussion

The discussion finds a largely date-driven general factor, with economic benchmarks adding predictive information but not establishing a distinct latent capability. It draws implications for leaderboard interpretation and benchmark validation while bounding the findings to the study’s evidence and snapshot.

6 Discussion

  • 74.5% of common variance is held by one dominant factor; date adjustment lowers its share to 59.6%, and economic benchmarks form no pre-specified distinct factor despite predicting held-out scores better on three of four benchmarks.The conclusion describes the remaining economic component as small but real and predictive of held-out economic scores.Source: §6 · p. 8
  • A separate economic column is justified by residual economic signal, but comparisons across release dates should be date-adjusted or restricted to one release window.The discussion cautions that rankings of models released months apart largely reflect release date.Source: §6 · p. 8
  • Benchmark builders can test a new suite’s structural distinctiveness after removing release-date trends, then test whether it improves held-out prediction over a single mean-score index.The proposed two-test protocol uses a bootstrap interval excluding zero as the predictive test criterion.Source: §6 · p. 8
  • The claims are bounded by an exploratory economic factor, observational data, uneven benchmark coverage, and a single leaderboard snapshot that may shift as models arrive.The paper calls for confirmation of the exploratory factor on an independent snapshot and does not claim that release date causes the general factor.Source: §6 · p. 8
  • Every test and threshold was fixed before analysis, and the reported numbers are reproducible from pinned inputs.The authors present this as a standard for evaluation claims of this kind.Source: §6 · p. 9

Appendix

  • Appendix sections identify the main-text sections they support, and figures and tables appear with the sections that discuss them.Source: Appendix · p. 13

A Abbreviations and notation

  • Table 2 lists abbreviations and symbols in their order of first appearance in the main text.Source: Table 2 · p. 13

B Leave-one-benchmark-out procedure

The leave-one-benchmark-out procedure guards its predictions against leakage by re-estimating factors within each outer fold and nesting hyperparameter tuning. The accompanying validity discussion specifies factor orientation, the predictive contrast, and limits on generalization.

B Leave-one-benchmark-out procedure

  • Factors are re-fitted using each outer training fold, and hyperparameters are tuned in a nested inner loop so held-out scores do not influence loadings or model selection.The figure presents the complete leave-one-benchmark-out procedure with leakage-safe factor projection and nested tuning.Source: B Leave-one-benchmark-out procedure · p. 13, Figure 4 · p. 13

C Positionality, directionality, and threats to validity

  • Factors are oriented to correlate positively with the mean standardised score; date adjustment regresses scores on release date, and positive ∆MSE favors the k-factor representation.The authors explicitly avoid interpreting the association as evidence that the general factor causes the calendar.Source: C Positionality, directionality, and… · pp. 13–14
  • The sample includes only vendor-submitted models on a public leaderboard, so claims generalise to that leaderboard; newer, sparser economic benchmarks are more sensitive to which models have scores.The paper notes that wide bootstrap intervals quantify this limitation.Source: C Positionality, directionality, and… · p. 14

D Mathematical basis of the two tasks

The section formalizes a common-factor account of benchmark scores and explains how date adjustment and leakage-safe prediction distinguish shared structure from incremental benchmark information. It also describes the factor loadings, benchmark selection, and coverage rules used in the analysis.

D Mathematical basis of the two tasks

  • The common-factor model separates shared variance ΛΦΛ⊤ from benchmark-specific uniqueness Ψ, unlike principal components, which target total variance.Parameters are estimated by maximum likelihood under multivariate normality.Source: D Mathematical basis of the two tasks · p. 14
  • 79.4% of total score variance lies in the first principal component, and Horn parallel analysis retains one factor (9.53 > 1.79; second 0.91 < 1.57).The values compare observed eigenvalues with random-data 95th percentiles.Source: Figure 5 · p. 15
  • Prediction uses factor-score weights estimated on training folds and applied to held-out models, preventing held-out scores from shaping their own projection.The structural factor scores use the oblique form; prediction uses weights built from pattern loadings.Source: D Mathematical basis of the two tasks · p. 15
  • Date residualization lowers the first-factor share from 74.5% to 59.6%, removing the rank-one covariance component aligned with release date.The factor model is refitted on residuals after benchmark scores are regressed on date.Source: D Mathematical basis of the two tasks · p. 15
  • Nested leave-one-benchmark-out evaluation tunes hyperparameters within training folds and bootstraps paired model-level error differences for its interval.Nesting avoids optimistic error estimates from selecting hyperparameters on the same folds used for evaluation.Source: D Mathematical basis of the two tasks · p. 16

E Date-adjusted three-factor loadings

  • All four economic benchmarks load highest on F1, the agentic and work-realistic factor; F2 reflects academic knowledge and F3 physics and hard reasoning.Table 3 reports the date-adjusted three-factor loadings.Source: E Date-adjusted three-factor loadings · p. 16

F Benchmark battery

  • Economic benchmarks score multi-step, tool-using, or human-preference-judged work-like task completion, unlike academic benchmarks’ single-response knowledge or reasoning accuracy.All twelve benchmarks are run by the leaderboard operator under a single harness.Source: F Benchmark battery · p. 16

G Full coverage audit

  • The sixty-model floor ensures the sparsest retained benchmark overlaps every other benchmark on more than 100 models, keeping pairwise-correlation standard errors below about 0.1.The floor is the smallest count meeting that overlap condition.Source: G Full coverage audit · p. 17

H Exploratory data analysis

The benchmark battery is strongly correlated and rises with release date, while the planned dimensionality evidence favors one factor. The section also documents the analysis choices and departures from the pre-specified plan.

H Exploratory data analysis

  • Mean off-diagonal Spearman correlation was ρ = 0.79, every benchmark pair was positive, and scores rose jointly with release date.Economic benchmarks correlated with the academic core almost as strongly as with one another.Source: H Exploratory data analysis · p. 17, Figure 7 · p. 19
  • Several bounded benchmarks were strongly right-skewed, with scores concentrated near the floor, so the analysis retained low-scoring models and checked a logit transform.CritPt had Fisher skewness 3.8, HLE 1.5, and τ 3-Banking 0.9; the logit check confirmed the first-factor share was not an artefact of the raw scale.Source: H Exploratory data analysis · pp. 17, 21, Figure 8 · p. 20
  • The first principal component accounted for 79.4% of total score variance, and only its eigenvalue exceeded the random-data 95th percentile.This is the dimensionality evidence behind H1.Source: H Exploratory data analysis · p. 21

I Hyperparameter grid and selections

  • Modal ridge α = 0.03 was the weakest-regularization choice in the grid, and the H4 verdict stayed unchanged across the k sweep.The reported table summarizes modal selections across the twelve leave-one-benchmark-out targets; H4 rests on fold-averaged out-of-fold errors.Source: I Hyperparameter grid and selections · p. 21, Table 5 · p. 21

J Deviations from the pre-specified design

  • The analysis used 421 configurations rather than the planned one-row-per-base-model unit, and H3 was not supported under the planned parallel-analysis rule.Three-factor economic separation was pre-declared exploratory; Terminal-Bench Hard was also assigned to scientific coding rather than economic, despite date-adjusted loading 0.82 on the economic factor.Source: J Deviations from the pre-specified… · p. 21
  • The fixed 15-point H2(ii) threshold was based on a mistaken reading of Kearns, but the threshold itself was not changed.Two presentational differences left the hypotheses and tests unchanged.Source: J Deviations from the pre-specified… · pp. 21–22

K Clustering of models and benchmarks

Clustering corroborates low-dimensional organization among models and places most benchmarks, including most economic benchmarks, in one correlation block.

K Clustering of models and benchmarks

  • All three clustering algorithms selected two model clusters by silhouette, separating 302 earlier, lower-scoring models from 107 newer, higher-scoring models along the overall-capability axis.The groups had median release dates of 2025-09 and 2026-03 and mean Intelligence Index values of 13.7 and 37.1, respectively.Source: K Clustering of models and benchmarks · p. 22, Figure 9 · p. 22
  • The model clusters corroborate low dimensionality without assuming a factor model, while the gap statistic was inconclusive because it increased monotonically with k.The three clustering algorithms were k-means, Gaussian mixtures, and agglomerative Ward linkage.Source: K Clustering of models and benchmarks · p. 22
  • Ten of twelve benchmarks, including three of four economic benchmarks, formed one correlation-distance block; τ 2-Bench and IFBench branched separately.The benchmark clustering used average linkage on correlation distance 1 −ρ.Source: K Clustering of models and benchmarks · p. 22

L Robustness checks

Robustness checks preserve the predictive gain and strengthen the date-adjustment result after model deduplication, while several planned checks were deferred.

L Robustness checks

  • On the deduplicated grid, the k-factor representation improved pooled economic-block prediction over the single index by +0.038, with a 95% interval of [+0.020, +0.056].This matches the configuration-level +0.037 result; 89 base models carried the complete battery.Source: L Robustness checks · pp. 22–23
  • The deduplicated first-factor share rose to 90.2%, and date adjustment lowered it by 24.1 points; all four economic loadings remained above 0.40.The paper treats the configuration-level structural analysis as the conservative headline.Source: L Robustness checks · p. 23
  • Across delivered checks, the predictive advantage remained positive for k = 2–5, while rank-based and logit analyses raised the first-factor share to 94.5% and 91.8%.FIML/MICE sensitivity, a single-factor-foil replication, reasoning-effort variants, and a taxonomy swap test were not run.Source: L Robustness checks · p. 23

M Explainability and error analysis

The k-factor model predicts GDPval well, and the attribution and residual analyses suggest interpretable sources of prediction error. The section also documents the pinned data and reproducibility checks.

M Explainability and error analysis

  • GDPval was predicted best by a k-factor ridge model, with out-of-fold R2 = 0.88.The regression diagnostics show predictions on the identity line and no systematic residual trend; the largest residuals are described as interpretable outliers.Source: M Explainability and error analysis · p. 23, Figure 11 · p. 24
  • For GDPval, SHAP ranked the agentic factor first, followed by the reasoning flag and hard-reasoning factor; scale and open-weights status contributed little.The largest residuals included Grok 4.3 Non-reasoning at +1.14, which outperformed its economic prediction; the pattern fits GDPval rewarding agentic behaviour captured indirectly by other benchmarks.Source: M Explainability and error analysis · p. 24

N Data provenance and reproducibility

  • The leaderboard snapshot is SHA-256 pinned, and the notebook regenerates selected figures and verifies canonical economic metrics against the reported values.The notebook runs from the pinned snapshots without live network calls; reported mean-index and k-factor ridge fits agreed across tested machines to about 10^-7.Source: N Data provenance and reproducibility · pp. 24–25

Related papers

Loading 2608.29420v1…