PaperlayerView exact revision on arXiv

Source-linked AI summary

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade

arXiv:2602.15327v2cs.LGcs.AIcs.CLstat.ML

TL;DR

For deployment, practitioners need to know what downstream accuracy a pre-training compute budget can attain under contemporary post-training practice, and whether that mapping remains stable as models evolve. This paper estimates high-quantile capability boundaries and finds them comparatively stable for four of six tasks, while math reasoning boundaries improve over time.

  • Problem

    For deployment, the question is what downstream performance a pre-training compute budget can attain under contemporary post-training practice and how stable that mapping is as models evolve.

    Source: §2 · p. 3

  • Method

    The paper estimates high-quantile attainable-performance boundaries from observed post-trained models using monotone, saturating sigmoid functions of log pre-training compute.

    Source: §1 · p. 2, §2.1 · p. 4

  • Results

    Across four of six tasks, boundaries are comparatively stable over time while math reasoning improves; OOD calibration error is 2.2% for the sigmoid boundary versus 3.6% for a compute-agnostic baseline.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The framework offers a data-driven way to translate target accuracy into plausible compute requirements and to identify tasks where attainable boundaries shift.

    Source: §2 · p. 3, §7 · p. 14

    Because the boundary reflects the observed model population, an underrepresented family or recipe that scores higher at fixed compute could place the true attainable boundary above the estimate.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute budget, what downstream accuracy is attainable with contemporary post-training practice, and how stable is that mapping as the field evolves? Using large-scale observational evaluations with 5k existing and 2k newly evaluated model checkpoints spanning 2022-2026 across six benchmarks, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre-training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate temporal reliability by fitting on earlier model generations and evaluating on later releases: across four of six tasks, the out-of-distribution coverage error remains below 2%, while math reasoning exhibits a consistently advancing boundary over time. For instance, at a budget of 10^24 FLOPs, the estimated attainable accuracies are 0.83 on IFEval and 0.54 on MATH Lvl 5. We then extend our approach to analyze task-dependent saturation and to probe contamination-related shifts on math reasoning tasks. Finally, we introduce a balanced I-optimal sampling algorithm that recovers near-full-data frontiers using roughly 20% of the parameter-count-weighted evaluation budget, as low as 5% on some tasks, while maintaining comparable calibration. Together, our work releases Proteus-2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.

1 Introduction

The paper frames prescriptive scaling as estimating attainable post-training performance from pre-training compute using high-quantile capability boundaries, then tests how these boundaries hold up over time. It reports more predictable attainable performance than a compute-agnostic baseline, alongside task-dependent temporal shifts and a lower-cost sampling approach.

Prescriptive scaling

  • Prescriptive scaling estimates high conditional quantiles of post-trained benchmark accuracy from log pre-training compute, using heterogeneous checkpoints and chronological validation.The data combine public leaderboard evaluations with 2.4k newly evaluated open-weight models released from April 2024 through March 2026.Source: Table 1 · p. 2, §1 · p. 2

Contributions

  • 2.2% OOD calibration error for sigmoid capability boundaries compares with 3.6% for a compute-agnostic baseline.The boundaries are modeled as monotone, saturating sigmoid functions of log-compute.Source: §1 · p. 2

Temporal validity and task-dependent ceilings

  • Four of six tasks have comparatively stable capability boundaries over time, while math reasoning exhibits a consistently improving boundary.The paper uses chronological train/validation splits to assess temporal validity.Source: §1 · pp. 2–3

Case studies: saturation and contamination

  • Saturation analysis distinguishes tasks with stable, size-determined ceilings from tasks such as math with ceilings that evolve over time; contamination analysis finds no clear AIME-2025 score inflation.Source: §1 · p. 3

Efficient prescriptive scaling via adaptive sampling

  • The adaptive sampling algorithm recovers sigmoid capability boundaries using typically about 20% of the full parameter-count-weighted evaluation budget, and about 5% on some tasks.Source: §1 · p. 3

2 Estimation of Post-training Capability Boundaries

Prescriptive scaling estimates how pre-training compute relates to attainable post-training performance. The capability boundaries are empirical envelopes shaped by the observed model ecosystem and its prevailing post-training methods, data curation, and evaluation protocols.

2.1 Setting and Modeling Assumptions

  • The study models each task’s capability boundary as the conditional τ-quantile of observed post-trained scores given log pre-training compute, rather than the maximum score.This is an empirical boundary for the observed model population; underrepresented model families or recipes could achieve higher scores, so the mapping can be updated as the evaluation ecosystem changes.Source: §2.1 · pp. 3–4

2.2 Capability Boundary Estimators

  • The estimators compared are a constant baseline, a binwise-constant model, a monotone saturating sigmoid, and a more flexible monotone I-spline.Each task and training period is fit separately to approximate the conditional τ-quantile.Source: §2.2 · p. 4

2.3 Evaluation Metrics

  • Pinball loss evaluates quantile accuracy, while signed coverage error measures whether the fitted boundary attains the target quantile locally across compute bins.Pinball loss uses asymmetric penalties that weigh under-prediction more heavily at high quantiles; coverage error reports empirical bin-level coverage minus τ.Source: §2.3 · p. 5

3 Sigmoid Scaling Laws for Post-training Performance Boundaries

The section uses sigmoid τ-capability boundaries to estimate post-training performance attainable at a given pre-training compute budget, then tests their shape, temporal transfer, and relationship to pre-training scaling.

3.1.1 The Shape of Capability Boundary

  • Sigmoid functions of log-compute approximate post-training capability boundaries, matching I-spline in ID pinball loss and achieving the best OOD calibration error across estimators.The authors use Sigmoid as the default boundary class for the remainder of the paper.Source: §3.1.1 · p. 6

3.1.2 Temporal Stability of the Sigmoid Capability Boundary

  • Coverage error stays within ±2% on BBH, GPQA, MMLU-PRO, and MUSR, while MATH LVL 5—and, to a lesser extent, IFEVAL—shows early-split under-coverage and elevated pinball loss.The evaluation uses three rolling train–test pairs across four chronological periods, restricting evaluation to overlapping compute ranges; systematic under-coverage in a later period could signal a new recipe class or architecture shifting the boundary upward.Source: §3.1 · p. 5, §3.1.2 · p. 6, Figure 2 · p. 7

3.2 From Pre-training Scaling Laws to Post-training Capability Boundaries

  • The pretrain–post-train gap depends on the task: pretrained models lie near the post-trained boundary on MMLU-PRO but substantially below it on MATH LVL 5 and IFEVAL.The comparison uses official pretrained base models and the post-trained τ-capability boundary.Source: §3.2 · p. 7, Figure 3 · p. 7
  • Pretrained accuracies are more often locally nonmonotone in compute, whereas post-trained attainable upper envelopes are much more consistently monotone.The contrast motivates interpreting compute as a more reliable predictor of post-training potential than raw pretrained accuracy.Source: §3.2 · pp. 7–8, Figure 3 · p. 7
  • The top three PCA-derived factors explain ≈95% of variance, but only the first shows a clear monotone increase with pre-training compute.This suggests compute-driven progress concentrates primarily along one dominant latent axis.Source: §3.2 · p. 8

3.3 External Validity: Newly Evaluated Models on PROTEUS-2K

  • Across 2.4k additional open-weight models absent from the leaderboard, its fitted boundary upper-bounds the best observed performance across tasks except mainly MATH LVL 5.On MATH LVL 5, the boundary advances primarily at high compute, with little evidence of systematic uplift at smaller compute.Source: §3.3 · p. 8, Figure 4 · p. 8

4 Capability Boundary Estimation under Limited Budget

Section 4 introduces a budget-constrained balanced I-optimal design for selecting model evaluations while recovering well-calibrated capability boundaries. It reports near-full-data boundaries with about 20% of the evaluation budget, and as little as 5% on GPQA and MUSR.

4.1 Budget-Constrained Balanced I-Optimal Design

  • Under a hard per-period budget, evaluation cost is assumed to grow roughly linearly with parameter count; a greedy gain-per-cost heuristic selects models using metadata and the local Jacobian.The selected subset's total parameter-count cost must not exceed the user-chosen share of the cost of evaluating all candidate models in that period.Source: §4.1 · pp. 9–10
  • The design minimizes average predictive variance across compute-bin midpoints and adds a balance term to distribute selected models across bins.The balance-variance tradeoff is controlled by λ; setting λ = 0 recovers standard I-optimality.Source: §4.1 · pp. 9–10
  • ≈20% of the evaluation budget recovers near-full-data capability boundaries, with GPQA and MUSR near-identical to full-data fits at 5%.Across periods and tasks, error decreases rapidly as the budget increases and stabilizes between 20% and 50%.Source: Figure 5 · p. 10, §4.1 · p. 10

5 Case Studies: Saturation and Contamination Diagnostics

The section applies capability-boundary estimation to diagnose task-dependent saturation and possible evaluation contamination, while testing whether its scaling approach generalizes to closed-source frontier models. Small-model attainable ceilings differ by task, and the cross-benchmark test finds no clear aggregate evidence of post-release AIME-2025 inflation.

5.1 Task-dependent Saturation on Leaderboards

  • On Open LLM Leaderboard v2, MMLU-Pro appears less saturated than MATH Lvl 5, and the size-centric diagnostics complement compute-based boundaries.The diagnostics assess whether small models quickly reach a benchmark’s attainable boundary and whether larger models retain an advantage.Source: Figure 6 · p. 11

5.1.1 Quantifying Saturation with a Size–Time Boundary Model

  • The size–time model fixes τ = 0.98 and uses a late-period size interaction to estimate how model size shifts attainable benchmark performance over time.It includes a smooth time trend and a late-period indicator; the late-period 13B boundary characterizes attainable performance for small models.Source: §5.1.1 · p. 11
  • 13B models reach an estimated late-period 0.98-quantile boundary of ≈0.94 on MATH Lvl 5 but only ≈0.52 on MMLU-Pro, where larger models retain dominance.The comparison indicates that small models approach the attainable boundary on math, while knowledge-intensive MMLU-Pro remains scale-limited.Source: §5.1.1 · pp. 11–12, Table 3 · p. 12

5.2 Contamination or Train-on-test-task Diagnostics on Frontier Benchmarks

  • Frontier-model evaluations test sigmoid-boundary external validity and use cross-benchmark score shifts to probe post-release, benchmark-specific inflation.The section compares the sigmoid estimator with I-splines and examines whether AIME-2025 scores rise relative to MATH-500 predictions after release.Source: §5.2 · p. 12

5.2.1 Frontier-model Scaling on GPQA

  • On frontier models with known compute, sigmoid and I-spline boundaries for GPQA diamond are largely similar, supporting the sigmoid scaling law’s external validity.The evaluation uses Epoch AI frontier-model data.Source: §5.2.1 · p. 12, Figure 7 · p. 13

5.2.2 A Cross-benchmark Shift Test

  • For 90 models in the overlapping MATH-500 score range, the estimated post-release AIME-2025 shift is positive but not significant (p = 0.15), so aggregate inflation is not clearly evident.All models post-date MATH-500, while AIME-2025-specific inflation could affect only models released after February 6, 2025; modest contamination effects cannot be ruled out.Source: §5.2.2 · p. 12

6 Related Works

The paper positions its contribution as estimating attainable downstream-performance boundaries from heterogeneous models, rather than relying on average trends or point forecasts. It also abstracts over post-training pipelines to support engineering decisions.

Scaling laws and downstream predictability

  • Classical scaling laws forecast pretraining loss, but translating them into reliable downstream guidance remains difficult, motivating attainable boundaries rather than average trends.Downstream extensions are noisier and less reliable, while observational studies find that post-training can improve apparent predictability and introduce temporal confounding (Dominguez-Olmedo et al., 2024).Source: §6 · p. 13

Downstream performance forecasting and scaling analyses

  • Prior forecasting methods use power laws, two-stage regressions, task clustering, rectified scaling laws, or trajectory tracking; this work instead estimates high-quantile post-training boundaries and studies temporal stability and data efficiency.Controlled regimes can support reliable FLOP-to-accuracy extrapolation, but emergent abilities, metric noise, and heterogeneous pipelines often disrupt those trends.Source: §6 · p. 13

Post-training and performance variance

7 Conclusion

Prescriptive scaling maps pre-training compute budgets to high-probability downstream performance expectations, with capability boundaries that are stable for most tasks but shift for math reasoning. The framework also supports diagnostics for saturation and contamination, and efficient evaluation using a small fraction of the budget.

7 Conclusion

  • High-quantile capability boundaries map log-compute to attainable post-training performance through simple monotone sigmoid functions and remain temporally stable for most tasks.The framework estimates these boundaries from large, heterogeneous model populations under contemporary post-training practice.Source: §7 · p. 14
  • Math reasoning is a notable exception, with a shifting capability boundary that reflects continued movement in the attainable performance envelope.Source: §7 · p. 14
  • Capability boundaries support practical diagnostics for saturation and contamination and can be recovered near-full with a small fraction of the evaluation budget.The framework is intended to help with budgeting, monitoring, and interpreting language-model progress as scaling regimes evolve.Source: §7 · p. 14

Appendices … B.3 Full I-spline Definition

The appendices justify high-quantile boundaries as estimates of reliably reachable high-end performance and specify the binning and monotone-spline implementations used to construct them.

A.1 Properties of Pinball Loss

  • On synthetic scaling data, smoothed pinball loss at τ=0.98 closely matches the true upper envelope, while squared loss is pulled toward underperforming points and underestimates high-compute performance.The asymmetric loss penalizes boundary estimates below high-performing models much more than estimates above weaker models.Source: Figure 8 · p. 21, §A.1 · pp. 21–22

A.2 Performance Frontier via Quantile Regression

  • Bin-wise maxima vary with sample counts and can confound compute-related performance changes with uneven model counts, hyperparameter search, or pruning; conditional quantiles avoid this dependence and yield smoother boundaries.The passages describe model counts per compute bin as uneven and changing over time.Source: §A.2 · p. 22
  • A high conditional quantile represents performance attainable with high probability, supports bin- and period-level coverage checks, and formalizes reliably reachable top-tier performance rather than a single lucky run.This interpretation is relative to the observed population of reported models; substantial selection effects could shift the implied boundary.Source: §A.2 · pp. 22–23
  • At τ≈0.98, in- and out-of-sample coverage errors are on the order of 1%–2% across tasks and periods, with stability under modest binning and regularization changes.The passage attributes this behavior to a high, but not extreme, quantile that is less sensitive than the maximum to a single outlier.Source: §A.2 · p. 22

B.1 Concrete Illustrative Outlier Example

  • The illustrative Benchmaxx-Llama-3.2-1B-Instruct model scores 0.83 BBH and 0.48 MATH LVL 5 raw accuracy, versus 0.36 and 0.08 for the second-best models with the same base.The example motivates robust high-quantile estimation; the authors do not verify or attribute the cause of the isolated performance spike.Source: §B.1 · p. 23

B.2 Full Bin Construction Algorithm for the Binwise Model

  • Group-aware equal-mass binning keeps identical training z-values together, targets bins of at least mmin samples, and merges undersized bins with an adjacent bin.The procedure sets Beff=min(B,G), targets mass m=max(mmin,⌈N/Beff⌉), and merges with the smaller neighbor when there is a choice.Source: §B.2 · p. 23

B.3 Full I-spline Definition

  • The I-spline estimator combines nonnegative M-spline basis functions into a nondecreasing function g(z), then applies a sigmoid to produce a monotone saturating boundary in [0, 1].The I-spline basis functions are nondecreasing in z.Source: §B.3 · pp. 23–24

C Pre-training vs. Post-training Diagnostics … F Newly Evaluated models

The appendices show that post-training gains vary by benchmark, model-size scaling differs from pre-training behavior on MUSR, and newly evaluated models mostly remain within the estimated capability boundaries. Supplementary analyses describe temporal-validation diagnostics and the evaluation cohorts.

C Pre-training vs. Post-training Diagnostics

D Details and Additional Analyses for Section 3

  • Appendix D includes sigmoid capability-boundary plots over time as a complement to Figure 1.Source: Figure 11 · p. 25

D.1 Omitted Details in Section 3

  • Temporal OOD evaluation fits each estimator on P_t and evaluates on P_(t+1), restricting evaluation to overlapping log-compute ranges.Table 2 reports relative changes against Constant, where more-negative values indicate better performance. Coverage heatmaps reuse training-period bins and show signed empirical-minus-target coverage, with negative values indicating under-coverage.Source: §D.1 · pp. 26–27

D.2 Bin-wise Diagnostics underlying Figure 2

  • The largest bin-wise coverage and pinball-loss deviations concentrate in a few mid-to-high compute bins, while the other benchmarks show mild ID and OOD errors across bins.The supplementary plots cover MMLU-PRO, MATH LVL 5, and IFEVAL; the remaining benchmarks exhibit scaling stability.Source: §D.2 · pp. 27–28, Figure 12 · p. 27, Figure 13 · p. 27

E Scaling Laws for Model Size

F Newly Evaluated models

G Sensitivity to Smoothed-Pinball Hyperparameters … H.3 Latent Capability Factors and Prescriptive Boundaries

The appendix finds that cross-temporal capability-boundary results are robust to reasonable smoothed-pinball hyperparameter choices, while additional analyses reveal task-specific scaling, historical saturation, and a compute-scaling pattern concentrated in one latent capability component.

G Sensitivity to Smoothed-Pinball Hyperparameters

  • Because period shift is the main validation challenge and λ primarily conditions numerics, the authors argue in-period cross-validation can add variance while optimizing the wrong objective.The sigmoid family is low-dimensional and constrained; κ controls smooth approximation near zero residual.Source: G Sensitivity to Smoothed-Pinball… · p. 33
  • OOD pinball loss and coverage remain essentially unchanged with in-period tuned hyperparameters versus the fixed default (κ, λ) = (50, 10−3).Tuned values typically concentrate at κ ∈ {20, 50} and λ ∈ {10−4, 10−3}.Source: G Sensitivity to Smoothed-Pinball… · p. 34, Figure 20 · p. 34
  • OOD performance is stable across a broad reasonable hyperparameter region, but degrades at λ = 10−1; κ has a milder effect once moderate, and coverage can improve slightly at smaller λ.The sweep evaluates OOD pinball loss and absolute coverage error on BBH and MATH LVL 5.Source: G Sensitivity to Smoothed-Pinball… · p. 34, Figure 21 · p. 34, Figure 22 · p. 35

H.1 Public Leaderboards of Frontier Models

  • On the smaller Epoch AI evaluation set, MATH Lvl 5 and Mock ATME show no performance gain with increasing FLOPs, whereas GPQA diamond shows clear scaling.Epoch AI includes many closed-source models compared with the Open LLM Leaderboard.Source: §H.1 · p. 35

H.2 Results for Open LLM Leaderboard v1

  • In Open LLM Leaderboard v1, all six benchmarks are saturated by P3, and beyond MMLU larger FLOPs bring little or no P3 capability-boundary gains.GSM8K and TruthfulQA show large gains from P1 to P2; the authors suggest the benchmarks were too old or severely contaminated.Source: §H.2 · p. 36, Figure 24 · p. 36

H.3 Latent Capability Factors and Prescriptive Boundaries

  • PC1 has a clear compute scaling pattern, while PC2 and PC3 boundaries are almost flat, implying the Open LLM Leaderboard scaling law may largely reflect advances in one component.The subsection provides further details on the PCA results mentioned in Remark 2.Source: §H.3 · p. 36, Figure 25 · p. 37

I Saturation Analysis across Open LLM Leaderboard Versions and Tasks · J Greedy Optimization for the Balanced I-Optimal Design · Algorithm 1: Greedy optimization for the balanced I-optimal design

Leaderboard saturation varies by task and version, while the balanced I-optimal design procedure selects models under per-period budgets using greedy gain-per-cost updates. The appendix also describes computational shortcuts and optional refinement steps for that procedure.

I Saturation Analysis across Open LLM Leaderboard Versions and Tasks

  • On v2, knowledge-heavy or knowledge-plus-reasoning tasks show materially less small-model domination than pure reasoning tasks such as MATH Lvl 5.These are observational diagnostics of submitted models, recipes, post-training, and benchmark targeting, not a controlled parameter-scaling law.Source: I Saturation Analysis across Open LLM… · p. 37, Figure 26 · p. 38
  • Many v1 tasks show strong frontier convergence, indicating a more saturated evaluation regime and demonstrating that conclusions about scaling depend on the benchmark suite.The plots reproduce the core logic of Hooker (2025, Figure 3) on the now-archived v1 suite.Source: I Saturation Analysis across Open LLM… · p. 37, Figure 27 · p. 38

J Greedy Optimization for the Balanced I-Optimal Design

  • The design selects a subset of period-t candidate models under a size budget to approximately maximize Φλ(S) = Φinfo(S) + λ Φbal(S).The implementation uses compute-bin midpoints and their Jacobians, with uniform bin weights in the experiments.Source: J Greedy Optimization for the Balanced… · pp. 38–39
  • For each candidate, the solver uses log pre-training compute, evaluation cost, and compute-bin index, with a p = 4 Jacobian of the high-quantile boundary evaluated at nominal parameters θ0.Bin-midpoint Jacobians are also computed for the design criterion.Source: J Greedy Optimization for the Balanced… · p. 39

Algorithm 1: Greedy optimization for the balanced I-optimal design

  • Sherman–Morrison updates reduce each candidate evaluation to O(p^2) operations, avoiding repeated matrix refactoring.The solver first computes candidate and bin-midpoint Jacobians at θ0 and initializes its information matrix.Source: Algorithm 1: Greedy optimization for… · pp. 39–40
  • The algorithm initializes with an anchor set, repeatedly adds the feasible candidate with the largest gain-per-cost, updates remaining budget and bin counts, and stops when no positive-gain addition remains.The example anchor set uses two extreme-compute models and two near the nominal sigmoid inflection point when available.Source: Algorithm 1: Greedy optimization for… · pp. 39–40
  • An optional Fedorov-style 1-exchange polish swaps selected and unselected models when it increases Φλ, but yields only modest empirical improvements.Figures 28–29 show design quality across budget parameter α using next-period coverage calibration error and pinball loss.Source: Algorithm 1: Greedy optimization for… · p. 40, Figure 28 · p. 41, Figure 29 · p. 41

Related papers

Loading 2602.15327v2…