PaperlayerView exact revision on arXiv

Source-linked AI summary

A Rosetta Stone for AI Benchmarks

Anson Ho, Jean-Stanislas Denain, David Atanasov, Samuel Albanie, Rohin Shah

arXiv:2512.00193v1cs.AI

TL;DR

Benchmarks saturate quickly, making long-run capability trends difficult to study and limiting comparisons between models evaluated on different benchmarks. The paper stitches benchmark results onto a shared scale, then uses the resulting series for forecasts, algorithmic-progress estimates, and acceleration detection.

  • Problem

    Benchmarks saturate quickly, and models can only be compared directly when evaluated on the same benchmark, limiting study of long-run capability trends.

    Source: §1 · p. 1

  • Method

    The framework stitches benchmark results into a single quantitative scale for model capabilities and benchmark difficulties, without assuming how capabilities evolve over time or with compute.

    Source: §1 · p. 1

  • Results

    The resulting series supports naive future-capability forecasts and yields algorithmic-progress estimates broadly consistent with prior work.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The framework can also be used to spot capability accelerations that might otherwise be hard to detect.

    Source: §5 · p. 14

    The framework inherits benchmark limitations: the selected set may be domain-biased, and benchmark scores may not represent general real-world capabilities.

    Source: §4.2 · pp. 12–13

Abstract

from arXiv · show

Most AI benchmarks saturate within years or even months after they are introduced, making it hard to study long-run trends in AI capabilities. To address this challenge, we build a statistical framework that stitches benchmarks together, putting model capabilities and benchmark difficulties on a single numerical scale. This acts as a "Rosetta Stone", allowing us to compare models across a wide range of abilities and time, even if they are not evaluated on the same benchmarks. Moreover, this works without assuming how capabilities evolve across time or with training compute. We demonstrate three applications of this framework. First, we use it to measure the speed of AI progress over time, and to forecast future AI capabilities. Second, we estimate the rate of improvements in algorithmic efficiency, finding estimates that are higher, but broadly consistent with prior work. Finally, we find that our approach can be used to detect rapid accelerations in AI progress.

1 Introduction

Benchmarks saturate quickly and cover only narrow slices of AI capability, limiting long-run and cross-model comparisons. This paper stitches benchmark results onto a shared scale to compare models across benchmarks and time.

1 Introduction

  • The framework places model capabilities and benchmark difficulties on one quantitative scale, enabling comparisons across benchmarks and long periods.It analyzes existing results rather than requiring models to share an evaluation benchmark.Source: §1 · p. 1
  • It uses minimal assumptions, requiring no postulated relationship between training compute or task length and model performance.Unlike ELO-based approaches, it requires no crowdsourced data collection, making it cheap to implement.Source: §1 · p. 2
  • Stitching benchmarks of different difficulties yields longer capability time series with a linear increase over time, despite using no time or compute information.The framework also estimates relative benchmark difficulties, not only model capabilities.Source: §1 · p. 2
  • The stitched time series supports future-capability forecasts, algorithmic-improvement estimates broadly consistent with prior work, and detection of rapid capability accelerations.The framework is also applied in the Epoch Capabilities Index with continuously updated data and a rescaled normalization scheme.Source: §1 · p. 2

2 Methodology

The methodology fits a statistical model relating benchmark scores to model capabilities and benchmark difficulties, then estimates those quantities from collected evaluation data. It uses regularized least squares and fixes reference values to address identifiability.

2.1 Model

  • A sigmoid relates benchmark scores to model capability and difficulty, with the score midpoint at equal capability and difficulty and the benchmark slope controlling the transition spread.The model assumes a single number can capture capability for simplicity, while acknowledging that capabilities are not truly one-dimensional.Source: §2.1 · p. 2

2.2 Data

  • The model is fit to Epoch AI benchmarking data from internal evaluations and externally collected repositories, whose evaluation setups may vary.Internal evaluations generally have overlap across models; external benchmarks may use different scaffolds for models on the same benchmark.Source: §2.2 · pp. 2–3
  • Filtering out models evaluated on three or fewer benchmarks increases model overlap; the final dataset contains 179 models, 38 benchmarks, and 1,324 scores.The data include benchmarks such as FrontierMath, GPQA Diamond, MMLU, and BIG-Bench Hard.Source: §2.2 · p. 3

2.3 Optimization

  • The model is fit with least squares and L2 regularization, while fixing a benchmark’s slope and difficulty removes scale and shift ambiguities across fits.These identifiability issues arise because multiplicatively rescaled parameters or jointly shifted capabilities and difficulties can yield the same predicted scores.Source: §2.3 · p. 3

3 Results

The Results section presents the fitted framework’s use for interpreting capability trends and supporting forecasts, algorithmic-progress estimates, and acceleration detection. It reports synthetic tests of acceleration detection and analysis of a historical breakpoint.

3.1 Determining model capabilities and benchmark difficulties

  • Estimated capabilities and benchmark difficulties both increase over time, with rankings placing GPT-5 above o3 and FrontierMath Tier 4 substantially above VPCT in difficulty.The most challenging benchmarks are estimated to exceed current state-of-the-art capabilities, but sparse high-performing data for newer benchmarks may overestimate their difficulty and increase uncertainty.Source: §3.1 · p. 3, Figure 1 · p. 4, Figure 2 · p. 4

3.1.1 Interpreting estimated capabilities and difficulties

  • Estimated capability scores correlate with log time horizons at R2 = 0.85, compared with R2 = 0.75 for GPQA diamond performance.Across the 18 benchmarks with enough overlapping data, the median R2 was 0.62; 20 of 38 benchmarks had fewer than five common data points.Source: §3.1.1 · p. 5, Figure 3 · p. 6

3.1.2 Are models optimized for different objectives?

  • Framework predictions average R2 = 0.71 on benchmarks with more than 25 scores and R2 = 0.81 with more than 40 scores.Residuals show Claude models performing better than predicted on coding-heavy SWE-Bench Verified but worse on multimodal GeoBench, while Gemini models show the reverse pattern, weakly suggesting specialization.Source: §3.1.2 · p. 5, Figure 4 · p. 6

3.2.1 Forecasting future capabilities

  • Frontier model capabilities have risen by 0.55 units per year (95% CI: 0.45 to 0.67), implying a naive forecast of around 4.4 units by October 2028.The forecast corresponds to a 17-week time horizon (95% CI: 2.6 to 30 weeks); it may be inaccurate because it does not account for changing drivers of progress, including possible slowdowns or accelerations.Source: §3.2.1 · pp. 7–8

3.2.2 Algorithmic progress

  • The central estimate implies training compute requirements decline about 6× per year, while the algorithmic-quality frontier estimate is around 9× per year.At fixed compute, the central estimate implies capability growth of around 0.3 units per year; the algorithmic-quality frontier estimate ranges from 3 to 40×.Source: §3.2.2 · p. 10, Table 2 · p. 11
  • Algorithmic-progress estimates are highly uncertain, ranging from 1 to 50× lower compute requirements per year, and assume progress is scale-independent.The model cannot test whether progress varies with compute scale because the available data are insufficient; uncertainty is largely driven by estimates of k.Source: §3.2.2 · pp. 10–11

3.3 Detecting rapid performance accelerations

  • Ordinary benchmark saturation can mislead about rapid progress because benchmark slopes and relative difficulties are unknown.The section tests whether the framework can detect a 2× acceleration in capabilities.Source: §3.3 · p. 11, §3.3.1 · p. 11

3.3.1 Synthetic data experiments

  • Synthetic experiments simulate piecewise-linear capability growth with an acceleration against linearly improving benchmark difficulties to test detection.Detection fits a piecewise-linear model with an estimated breakpoint and compares post-breakpoint and pre-breakpoint slopes against a threshold.Source: §3.3.1 · p. 11
  • Under moderate noise, the framework detected a 2× acceleration within 2–3 months when all models accelerated; higher accelerations were detected slightly faster.Figure 8 illustrates a 2× acceleration detected around three months after it began.Source: §3.3.1 · p. 11, Figure 8 · p. 12
  • Noise dominated detection time: increasing it from 0.5× to 4× baseline increased detection time 4.1× on average, while universal acceleration enabled 27% faster detection than sparse acceleration.Acceleration magnitude had only a small effect; 4× was detected only slightly faster than 2× or 8×.Source: §3.3.1 · p. 11
  • The framework’s estimated capabilities depend on benchmark choice, leaving uncertainty about which benchmarks to monitor.Source: §3.3.1 · p. 11
  • Without acceleration, the false positive rate averaged 38% (14%–54%), so detections are monitoring signals for further investigation, not definitive evidence.Rates varied with noise levels and observation windows; the authors recommend combining the approach with other evidence.Source: §3.3.1 · p. 12

3.3.2 Detecting acceleration in the historical data

  • Historical data show an April 2024 breakpoint: the slope rose from 0.352/year to 0.689/year, a 1.95× acceleration that did not cross the 2× threshold.This estimate is similar to Kwa et al. 2025’s potential 1.75× acceleration, inferred from time-horizon doubling times changing from 7 months to 4 months.Source: §3.3.2 · p. 12

4 Discussion

The discussion highlights limits in interpreting the unified scores, in the benchmarks and aggregate data behind them, and in representing capabilities with one number. It also notes that acceleration detection remains unreliable and needs more principled methods.

4.1 Challenges in interpreting model capability and benchmark difficulty scores

  • Capability scores lack a clear intuitive meaning and omit speed and price, while interpretations based on task time horizons remain speculative.The authors note that some applications, such as estimating algorithmic progress, do not require an intuitive interpretation of the scores.Source: §4.1 · p. 12

4.2 Limitations inherited from individual benchmarks

  • The benchmark set may underrepresent economically useful work, emphasize mathematics and reasoning, and reflect evaluation settings that differ from real-world deployment.Benchmark error rates can also cap achievable scores, so estimated capabilities may not fully represent general, real-world capabilities.Source: §4.2 · pp. 12–13

4.5 Future directions for detecting capability accelerations

  • Real-data detection flags a 1.95× frontier-capability acceleration with a March 2024 breakpoint, but the ad hoc detector had many false positives on synthetic data.The authors suggest time-series or sequential-testing methods as a direction for improving detection specificity and reliability.Source: Figure 9 · p. 13, §4.5 · p. 14

4.2 Limitations inherited from individual benchmarks

  • Practitioners can curate or reweight benchmarks, but stitching requires enough evaluated models and overlap across benchmarks.The discussion also notes that recent benchmarks with few high-performing model evaluations may have their difficulties overestimated.Source: §4.2 · p. 13

4.3 AI capabilities cannot be fully described by a single number

  • A single capability score cannot capture domain specialization, although the scores generally predict benchmark results; multidimensional models are a proposed direction.A cited principal-components analysis finds two components explain most benchmark-score variance, with the second resembling a coding-strong, vision- and math-weaker profile.Source: §4.3 · p. 13

4.4 Our model uses average benchmark scores rather than results on individual questions

  • Using aggregate benchmark averages broadens data coverage when item-level results are unavailable, but splitting one benchmark into two can effectively double its weight.The authors propose adapting the framework to question-level results from a wide variety of high-quality benchmarks.Source: §4.4 · pp. 13–14

5 Conclusion

The paper concludes that stitching benchmarks yields a unified scale for model capabilities and benchmark difficulties, enabling longer capability time series. It describes forecasts, algorithmic-progress estimates broadly consistent with prior work, and the prospect of detecting accelerations, alongside related approaches and frameworks.

5 Conclusion

  • The unified capability series supports naive forecasts and algorithmic-progress estimates broadly consistent with prior work, and could help reveal otherwise hard-to-detect accelerations.The conclusion presents these as applications of the stitched benchmark data.Source: §5 · p. 14

A Related work

  • Prior work decomposes capabilities into principal components that scale log-linearly with training compute within certain model families.The passage describes this as an approach to estimate language-model performance without expensive scaling experiments.Source: A Related work · p. 19
  • Time-horizon metrics estimate the human completion times of tasks AI can perform, whereas this framework is more flexible and cheaper but harder to interpret directly.The cited time-horizon metric was estimated on software-related tasks.Source: A Related work · p. 19
  • Unlike ELO-based leaderboards that generally rely on repeated human votes, this approach aggregates existing evaluations and also estimates relative benchmark difficulty.The authors describe it as substantially less costly to implement than ELO-based approaches.Source: A Related work · p. 19
  • Benchmark performance predictability depends on the metric used, and has also been studied in relation to training-compute scaling.Owen 2024 reports moderate predictability from compute scaling; Schaeffer et al. 2025 note its dependence on the performance metric.Source: A Related work · p. 19
  • Unlike Item Response Theory, this framework uses aggregate benchmark scores, allowing it to include more benchmarks and build longer capability time series.The authors connect the longer series to applications including efficiency estimates, acceleration detection, and forecasting.Source: A Related work · p. 19

B.1 Benchmarks

The appendix describes benchmark selection and data collection, model inclusion choices, and additional algorithmic-progress analyses. The benchmark collection prioritizes useful, unsaturated, widely used tasks, while the progress analysis tests alternative modeling choices and an alternative estimation approach.

B.1 Benchmarks

  • Benchmarks come from Epoch AI’s hub, which prioritizes economically valuable, unsaturated, widely used tasks; internal benchmarks must also be easy to run.The authors select metrics that resemble accuracy out of 100% and represent overall benchmark difficulty.Source: §B.1 · p. 19

B.2 Models

  • The data prioritizes highly cited models not heavily optimized for one benchmark, because specialized scaffolds are less likely to span benchmarks and can skew capability estimates.A little over half of the dataset’s models are considered notable under criteria including citations, training cost, usage, state-of-the-art performance, or historical significance.Source: §B.2 · p. 20

C Algorithmic progress

  • The algorithmic-progress analysis estimates compute scaling within model families, then tracks changes in algorithmic quality; alternatives include adding distilled models, changing how scaling is estimated, and checking direct same-capability compute comparisons.The appendix also considers frontier algorithmic progress and possible progress through changes in the compute-scaling parameter.Source: C Algorithmic progress · p. 20

C.1.1 Including distilled models

The appendix checks algorithmic-progress estimates when distilled models are included. It reports substantial overlap with the main analysis, while noting uncertainty in identifying distillation and accounting for other compute sources.

C.1.1 Including distilled models

  • The primary analysis excludes distilled models to focus on the relation between capabilities and compute in final training runs, though many distilled models may be smaller, cheaper-to-serve versions of frontier models.The passage defining the exclusion rationale is truncated; the stated focus and characterization of distilled models are supported by the cited passages.Source: §C.1.1 · pp. 20–21
  • Distillation status may be undisclosed, and some frontier models may be partly distilled, complicating decisions about which models to exclude.The authors also note uncertainty about synthetic-data compute and how much was used in recent models.Source: §C.1.1 · p. 21
  • Including distilled models yields an estimated algorithmic-progress rate around 5× per year, with Δb = 0.3 per year (95% CI: 0.0 to 0.5 per year), substantially overlapping the main-analysis results.This variant adds the Qwen 2.5 family when estimating k, yielding an average k = 0.1625.Source: §C.1.1 · p. 21, Table 5 · p. 22
  • The appendix reports full results across model families in Table 5, whose scope includes models without restrictions on distillation or synthetic-data use.The passage also describes overlap between those results and the main analysis.Source: §C.1.1 · pp. 21–22, Table 5 · p. 22

C.1.2 Estimating k using all models with training compute estimates

The alternative all-model estimate of algorithmic progress uses more training-compute observations but is biased toward higher k and lower progress estimates. Progress that changes k could also vary with compute scale, an effect the available data cannot quantify.

C.1.2 Estimating k using all models with training compute estimates

  • Using all models increases the available observations but likely overestimates k and correspondingly underestimates algorithmic progress relative to family-based estimates.The family-based approach has only three model families and roughly five models per family in the main analysis.Source: §C.1.2 · p. 23
  • With all models, the algorithmic-quality frontier gives k = 0.22, Δb = 0.27 per year, and 3.5× annual algorithmic progress.The 95% confidence intervals are 0.19–0.24 for k, 0.20–0.34 per year for Δb, and 2.6–4.8× per year for progress.Source: §C.1.2 · p. 23
  • At the model-capability frontier, the alternative estimate gives Δb = 0.21 per year and 2.7× annual algorithmic progress.The 95% confidence intervals are 0.02–0.41 per year for Δb and 1.1–6.5× per year for progress.Source: §C.1.2 · p. 23

C.1.3 What happens if algorithmic progress occurs by changing k?

  • If algorithmic progress changes k rather than b, compute savings at a fixed capability may depend on the training-compute scale.Changing k could make efficiency gains larger at larger scales; the supplied figure caption illustrates this possibility.Source: §C.1.3 · p. 23, Figure 11 · p. 23
  • The authors lack frontier-model data spanning substantially different training-compute scales, so they cannot directly estimate this scale dependence.The section also notes that the capability–log-compute relationship may not be linear.Source: §C.1.3 · pp. 23–24

C.2 Directly estimating algorithmic progress

Direct estimates of algorithmic progress vary widely and are considered unreliable because of data constraints. An ordinary least squares approach gives a 20× annual compute-efficiency estimate, but correlated compute scaling makes attribution difficult.

C.2 Directly estimating algorithmic progress

  • The direct-observation approach compares later models at similar capabilities or compute budgets, but constraints needed for estimation sharply limit usable data.The authors describe these estimates as less reliable overall and note that direct estimation is especially sensitive to data limitations.Source: §C.2 · p. 24
  • Direct observations yield annual compute reductions from 2× to 400× (median ~10×) and capability increases of 0.1 to 1.6 units/year (median ~0.4).The authors consider the direct estimates quite unreliable because the data are limited; the tables summarize the two measures.Source: Table 6 · p. 24, §C.2 · p. 24, Table 7 · p. 25
  • The direct estimates are broadly consistent with the core estimates, which increases the authors’ confidence in the core results.Source: §C.2 · p. 25

C.3 Estimating algorithmic progress using ordinary least squares

  • Ordinary least squares estimates compute-efficiency gains of around 20× per year and a capability-trend coefficient b of 0.37.The 95% confidence intervals are 7–40× per year for the gain and 0.29–0.46 for b.Source: §C.3 · p. 25
  • The ordinary least squares estimate is difficult to attribute because algorithmic improvements likely correlate heavily with compute scaling.The authors regard the estimates as useful for sanity-checking because they are broadly similar to the other approaches.Source: §C.3 · pp. 25–26

D Synthetic data generation

The synthetic experiments construct capability and benchmark-difficulty trends, then map them to benchmark scores to test acceleration detection. Capability acceleration occurs after a breakpoint, while benchmark difficulties continue improving linearly.

D Synthetic data generation

  • Synthetic capability trends use an observed baseline improvement of about 0.55 units/year and a single breakpoint after which improvement accelerates by factor N.Benchmark difficulties share the initial improvement rate but do not accelerate.Source: D Synthetic data generation · p. 26
  • The generated trends vary timeframe, total capability increase, starting capability, acceleration factor, and Gaussian noise.The example parameters include 2020–2030, a total increase of 3.5, an acceleration factor of 2×, and noise σ = 0.05.Source: D Synthetic data generation · p. 26
  • The experiment samples model dates and maps capabilities and benchmark difficulties into scores, evaluating each model on 25% of benchmarks and retaining scores between 5% and 95%.The synthetic data use 30 benchmarks; broad and narrow scenarios accelerate 100% and 10% of models, respectively.Source: D Synthetic data generation · p. 26

D.1 Noise structure

The appendix tests Gaussian-noise assumptions and robustness to data selection choices. Estimates remain similar under several variations, though noise tests conflict, fewer benchmark overlaps yield more outliers, and anchor choice produces variation.

D.1 Noise structure

  • Residual tests disagree about Gaussian noise: Kolmogorov–Smirnov gives p ≈ 0.81, while Shapiro–Wilk, Anderson–Darling, and Jarque–Bera reject normality at the same significance level.The authors retain Gaussian noise for simplicity and because residuals appear approximately Gaussian, but identify the assumption as a limitation.Source: §D.1 · p. 26

D.2 Detecting accelerations

  • The detection-data sweep varies post-cutoff acceleration across 2.5 and 5.Source: §D.2 · p. 27

E.1 Varying the amount of overlap

  • Reducing the required overlap leaves estimated capabilities similar but increases the number of models with unexpectedly high or low estimates.The main analysis filters out models evaluated on three or fewer benchmarks to preserve overlap for calibration.Source: §E.1 · p. 28

E.2 Varying the anchors

  • Changing the benchmark anchor produces some variation in model-capability estimates, shown as standard-deviation error bars around the main estimates.Source: §E.2 · p. 28

E Robustness tests

  • The appendix examines robustness to benchmark selection, statistical-model choice, and evaluation setup.Source: §E.3 · p. 28

E.3 Varying benchmark inclusion

  • Benchmark inclusion tests compare random benchmark subsets with benchmarks that labs do or do not optimize for.Source: §E.3 · p. 28

E.3.1 Random selection

  • Across 100 random benchmark selections, the median capability slope is 0.54 units/year (95% interval 0.43–0.64), close to the main estimate of 0.55.Source: §E.3.1 · p. 28

E.3.2 Detecting Benchmark Gaming

The analysis finds little statistically significant evidence that benchmark optimization or strategic reporting inflates estimated capabilities. Checks using an alternate statistical model and earlier data find similar overall results.

E.3.2 Detecting Benchmark Gaming

  • The test compares capability fits on benchmarks publicly reported by frontier AI labs versus others, using recent, similarly difficult benchmarks and multiple anchors.It filters to benchmarks created in 2024 or later, then selects the 9 optimized-for and 10 not-optimized-for benchmarks with the highest model coverage.Source: §E.3.2 · pp. 28, 30
  • 0.12 capability units higher on optimized-for benchmarks—about a sixth of a year’s progress—but the differences are not statistically significant (p-values average 0.22).Across the 20 anchor benchmarks, p-values range from 0.1 to 0.37.Source: §E.3.2 · p. 30

E.4 Varying the statistical model

  • The clipped-linear and sigmoid models perform similarly in cross-validation; optimized-for estimates are slightly higher, but not significantly so.Source: §E.4 · pp. 30, 32, Figure 17 · p. 32

E.5 Analysis over a longer time period

  • Including around 100 earlier benchmark entries produces similar overall findings, although adding earlier data introduces overfitting risk.Earlier years have sparse model coverage and are less representative for forecasting future capabilities.Source: §E.5 · p. 32

E.5.1 Forecasting future capabilities

With earlier data included, the analysis estimates a slower overall progress rate and a possible mid-2024 acceleration. It also finds that GPT-4-to-GPT-5 gains may exceed the earlier GPT-3-to-GPT-4 jump, while benchmark fits are fairly accurate overall.

E.5.1 Forecasting future capabilities

  • 0.42 capability units per year [95% CI: 0.37 to 0.46] is the estimated progress rate, predicting 3.8 rather than 4.5 capability units over three years.Forecasts from this approach underestimated the rate of progress over the last year, even more than the main analysis.Source: §E.5.1 · p. 32
  • 1.7× acceleration was a potential mid-2024 finding, broadly consistent with the analysis restricted to post-2023 data.Source: §E.5.1 · p. 32
  • Earlier-model predictions may be especially unreliable because they rely on sparser data; more notable releases between GPT-4 and GPT-5 may also explain the perceived discrepancy.Source: §E.5.1 · p. 32
  • Closer to 1 capability unit for GPT-4-to-GPT-5 versus around 0.6 for GPT-3-to-GPT-4; the later jump was estimated at a little less than twice the earlier one.Source: §E.5.1 · p. 32, Table 10 · p. 34

F Model fit

  • The framework fits observed scores fairly accurately on most benchmarks, with exceptions such as GeoBench and sparse or low-clustered data on some benchmarks.CommonSenseQA 2.0 has little data, while Cybench scores are mostly clustered at low values.Source: F Model fit · p. 34, Figure 20 · p. 35

Related papers

Loading 2512.00193v1…