Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Hanlin Zhang; Jikai Jin; Vasilis Syrgkanis; Sham Kakade

Source-linked AI summary

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade

arXiv:2602.15327v2Feb 17, 2026cs.LGcs.AIcs.CLstat.ML

TL;DR

The paper asks how reliably pre-training compute can predict attainable downstream accuracy for heterogeneous post-trained language models. It estimates high-quantile sigmoid capability boundaries and finds them temporally stable for most tasks, while math reasoning shows a shifting boundary.

Problem
Existing scaling laws provide limited evidence for mapping pre-training compute to attainable downstream accuracy under heterogeneous post-training practices.
Method
The paper estimates high-conditional-quantile capability boundaries as monotone, saturating sigmoid functions of log pre-training compute and tests their temporal validity.
Results
Capability boundaries are comparatively stable for four of six tasks, while math reasoning exhibits a consistently improving boundary over time.
Takeaways & Limitations
The framework provides decision-oriented compute-to-performance expectations while identifying tasks where evolving methods shift attainable capability ceilings.
Takeaways & Limitations
A qualitatively new recipe class or architecture family could raise the attainable boundary at fixed compute, requiring an updated fit.

Abstract

from arXiv · show

Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute budget, what downstream accuracy is attainable with contemporary post-training practice, and how stable is that mapping as the field evolves? Using large-scale observational evaluations with 5k existing and 2k newly evaluated model checkpoints spanning 2022-2026 across six benchmarks, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre-training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate temporal reliability by fitting on earlier model generations and evaluating on later releases: across four of six tasks, the out-of-distribution coverage error remains below 2%, while math reasoning exhibits a consistently advancing boundary over time. For instance, at a budget of 10^24 FLOPs, the estimated attainable accuracies are 0.83 on IFEval and 0.54 on MATH Lvl 5. We then extend our approach to analyze task-dependent saturation and to probe contamination-related shifts on math reasoning tasks. Finally, we introduce a balanced I-optimal sampling algorithm that recovers near-full-data frontiers using roughly 20% of the parameter-count-weighted evaluation budget, as low as 5% on some tasks, while maintaining comparable calibration. Together, our work releases Proteus-2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.

1 Introduction

The paper develops prescriptive scaling laws that predict attainable post-trained performance from pre-training compute using monotone, saturating sigmoid capability boundaries. It shows these boundaries are temporally stable for most tasks but evolve for math, while enabling efficient evaluation through adaptive sampling.

1 Introduction: Although increasing pre-training compute generally improves downstream performance, models with similar FLOPs can show substantial heterogeneity in reasoning, instruction following, and domain-specific question answering.This heterogeneity motivates predicting attainable post-trained performance rather than relying only on average compute trends.
1 Introduction: Post-trained performance is more predictable than pre-trained performance, with sigmoid boundaries achieving 2.2% OOD calibration error versus 3.6% for a compute-agnostic baseline.The boundaries are modeled as simple monotone, saturating sigmoids of log-compute.
1 Introduction: Capability boundaries remain comparatively stable over time for four of six tasks, whereas math reasoning exhibits a consistently improving boundary.Chronological train/validation splits reveal when compute-to-attainable-accuracy relationships remain reliable and when they shift.
1 Introduction: Saturation analysis identifies stable, size-determined ceilings for some tasks but evolving ceilings for math, while contamination analysis finds no clear AIME-2025 score inflation.These case studies apply prescriptive scaling to distinguish task-dependent limits from potential contamination effects.
1 Introduction: Adaptive sampling recovers sigmoid capability boundaries using typically 20% of the full parameter-count-weighted evaluation budget and as little as 5% on some tasks.The method targets accurate frontier recovery under limited computation budget.

2 Estimation of Post-training Capability Boundaries

Prescriptive Scaling asks how pre-training FLOPs translate into downstream performance attainable with contemporary post-training practice. The section formalizes this mapping as an empirical high-quantile capability boundary and evaluates candidate monotone models using quantile accuracy and local coverage.

2 Estimation of Post-training Capability Boundaries: Prescriptive Scaling treats pre-training compute as a controllable design input for forecasting attainable downstream benchmark performance under prevailing post-training practice.The estimated boundaries are conditioned on post-training methods, data curation, and evaluation protocols in the observed model ecosystem.
2 Estimation of Post-training Capability Boundaries: Capability boundaries estimate the conditional τ-quantile of observed scores given log10 pre-training FLOPs, rather than the genuinely maximal observed accuracy.This formulation addresses ubiquitous outliers and represents an empirical attainable boundary for the observed model population.
2 Estimation of Post-training Capability Boundaries: The candidate boundary models comprise constant, binwise-constant, monotone saturating sigmoid, and more flexible monotone I-spline functions.The sigmoid constrains predictions to [0, 1] with nonnegative compute slope and saturation, while the I-spline generalizes its monotone structure.
2 Estimation of Post-training Capability Boundaries: Quantile accuracy is measured with smoothed pinball loss, which applies asymmetric penalties that emphasize under-prediction when τ is near one.The metric is reported for both training and out-of-distribution validation periods but can obscure where errors occur and their direction.
2 Estimation of Post-training Capability Boundaries: Coverage error measures the signed difference between empirical local coverage and the target quantile within each log-compute bin.It tests whether the fitted boundary achieves the intended quantile coverage at different compute levels.

3 Sigmoid Scaling Laws for Post-training Performance Boundaries

Post-training capability boundaries are well approximated by sigmoid functions of log-compute, with strong out-of-distribution calibration across most tasks but temporal drift on MATH LVL 5. Compared with pretrained accuracy, post-trained boundaries more reliably predict attainable performance, with task-dependent gaps between pretraining and post-training.

3.1 Open-weight models: Sigmoid boundaries match I-spline in-distribution loss and achieve the best out-of-distribution calibration across estimators.This motivates Sigmoid as the default boundary class for subsequent analyses.
3.1 Open-weight models: Coverage error stays within ±2% for BBH, GPQA, MMLU-PRO, and MUSR, while MATH LVL 5 shows under-coverage and elevated pinball loss on the earliest split.The departures indicate non-stationary effective boundaries over time; systematic under-coverage can signal a new recipe or architecture family shifting the attainable boundary upward.
3.2 Pre-training versus post-training: The pretrain–post-train gap is task dependent: pretrained models approach post-trained boundaries on MMLU-PRO but remain substantially below them on MATH LVL 5 and IFEVAL.Knowledge-intensive tasks show smaller gaps, whereas reasoning and instruction-following tasks show larger post-training gains.
3.2 Pre-training versus post-training: Post-trained capability boundaries are consistently monotone in compute, whereas pretrained accuracies frequently violate monotonicity across model families.Thus compute predicts potential more reliably than raw pretrained accuracy; among pretrained models, larger-compute bases can score below smaller ones.
3.1 Open-weight models: Leaderboard-fitted boundaries continue to upper-bound the best observed performance across 2.4k held-out open-weight models spanning April 2024 to March 2026, except mainly on MATH LVL 5.The held-out models are absent from the leaderboard, including releases after its retirement on 2025-03-13.

4 Capability Boundary Estimation under Limited Budget

The section introduces a balanced I-optimal design that selects model evaluations under parameter-count-weighted budgets to minimize boundary uncertainty while covering compute regimes. It recovers near-full-data capability boundaries with roughly 20% of the evaluation budget, and with 5% on GPQA and MUSR.

Balanced I-optimal objective: The balanced I-optimal design minimizes average predictive variance while encouraging evaluations to cover all compute bins.Its objective combines predictive-variance and bin-balance terms, with λ controlling the trade-off between boundary uncertainty and coverage.
Cost and budget: Evaluation cost is modeled as linear in parameter count, with selected models constrained by a user-chosen fraction α of each period’s total evaluation cost.The selected subset must remain within the per-period budget while supporting accurate out-of-distribution predictions for the next period.
Implementation: A greedy gain-per-cost heuristic approximately optimizes the design using only model metadata and the local sigmoid-boundary Jacobian.The method uses compute locations, evaluation costs, and local sensitivity information rather than requiring full evaluations of every candidate model.
Empirical behavior: ≈20% budget recovers near-full-data capability boundaries, while GPQA and MUSR achieve near-identical estimates with α = 5%.Across periods and tasks, error decreases rapidly as α increases and stabilizes between 20% and 50%.

5 Case Studies: Saturation and Contamination Diagnostics

The case studies show that attainable performance ceilings depend strongly on both task and model size, while frontier-model scaling remains consistent with the sigmoid boundary and contamination diagnostics find no clear aggregate AIME-2025 inflation. In particular, 13B models approach the late-period MATH Lvl 5 boundary but remain substantially below MMLU-Pro’s boundary.

Motivation: The diagnostics address two time-dependent evaluation concerns: task-dependent saturation as score relationships evolve and artifacts including contamination or training on the test task.The section extends prescriptive scaling beyond capability-boundary estimation as a function of pre-training compute.
Task-dependent saturation: 13B models reach q̂0.98 ≈0.94 on MATH Lvl 5 but only ≈0.52 on MMLU-Pro in the latest period, revealing task-dependent small-model ceilings.Larger models retain dominance on MMLU-Pro, whereas small models approach the top boundary on Math.
Frontier-model external validity: Sigmoid and I-spline boundaries are largely similar on closed-source frontier-model GPQA diamond evaluations, supporting the sigmoid scaling law’s external validity.The boundary is fitted using models with known compute; additional frontier-leaderboard results appear in Section H.
Contamination diagnostics: A positive AIME-2025 shift relative to MATH-500 is not statistically significant (p-value = 0.15), providing no clear aggregate evidence of unusually high post-release AIME-2025 scores.The analysis uses the overlapping MATH-500 score range across release groups, with n = 90.

6 Related Works

Classical scaling laws connect model size, data, and compute to pretraining loss under controlled settings (Hoffmann et al., 2022; Kaplan et al., 2020), while related work seeks actionable guidance for future model development (Hernandez et al., 2021; Kaplan et al., 2020; McCandlish et al.). This work complements those algorithmic approaches by abstracting over heterogeneous post-training pipelines and enabling large-scale observational analysis aligned with engineering decisions.

Classical scaling laws relate model size, data, and compute to pretraining loss under controlled settings (Hoffmann et al., 2022; Kaplan et al., 2020).
Related research translates scaling forecasts into actionable guidance for future model development (Hernandez et al., 2021; Kaplan et al., 2020; McCandlish et al.).
The paper complements algorithmic advances by abstracting over heterogeneous post-training pipelines and treating models with similar pretraining FLOPs as a single domain for large-scale observational analysis aligned with engineering decision-making.

7 Conclusion

The paper introduces prescriptive scaling, a decision-oriented framework for mapping pre-training compute budgets to reliable downstream performance expectations. It finds that attainable post-training performance follows simple monotone sigmoid functions of log-compute and is temporally stable for most tasks.

Conclusion: Prescriptive scaling maps pre-training compute budgets to reliable, high-probability downstream performance under contemporary post-training practice.The framework estimates high-quantile capability boundaries from large, heterogeneous model populations.
Conclusion: Attainable post-training performance is well-approximated by simple monotone sigmoid functions of log-compute.
Conclusion: Capability boundaries are temporally stable for most tasks.

Appendices … B.3 Full I-spline Definition

The appendices justify high-quantile pinball regression as a stable, probabilistically interpretable capability-boundary estimator and specify the binning and monotone I-spline implementation. They also show that this approach avoids outlier-driven maxima while targeting reliably achievable upper-envelope performance.

A.1 Properties of Pinball Loss: At τ = 0.98, smoothed pinball loss closely matches the true upper envelope, whereas squared loss underestimates achievable performance at high compute.The asymmetric loss heavily penalizes boundaries below high-performing models and largely ignores underperforming points.
A.2 Performance Frontier via Quantile Regression: High-quantile regression is more stable than bin-wise maxima because maxima vary with sample count and are driven by occasional extreme models.A quantile such as τ ≈0.98 remains near the top while reducing sensitivity to single outlier runs.
A.2 Performance Frontier via Quantile Regression: High conditional quantiles provide a probabilistic capability boundary, describing performance attainable with high probability at compute C rather than the luckiest observed model.This interpretation supports bin-by-bin and period-by-period coverage checks.
A.2 Performance Frontier via Quantile Regression: The boundary formalizes reliably reachable top-tier performance from a competitive training pipeline without assuming arbitrarily many parallel bets.This differs conceptually from seeking the single best model ever trained at a compute value.
B.1 Concrete Illustrative Outlier Example: Benchmaxx-Llama-3.2-1B-Instruct reaches 0.83 BBH and 0.48 MATH LVL 5, while second-best Llama-3.2-1B derivatives reach only 0.36 and 0.08.The example motivates distinguishing reproducible algorithmic improvements from isolated spikes caused by overfitting or leakage.
B.2 Full Bin Construction Algorithm for the Binwise Model: Group-aware equal-mass binning preserves identical compute values within bins and enforces minimum bin sizes through adjacent-bin merging.The effective bin count is limited by the number of unique compute levels.
B.3 Full I-spline Definition: The I-spline estimator parameterizes a flexible monotone function using nonnegative M-spline bases, making each I-spline basis function nondecreasing.The parameterization is attributed to Ramsay (1988).
B.3 Full I-spline Definition: Constraining the I-spline coefficients makes g(z) nondecreasing, while applying σ yields a monotone saturating boundary qτ(z) in [0, 1].The resulting predictions are bounded and saturate with compute.

C Pre-training vs. Post-training Diagnostics … D.2 Bin-wise Diagnostics underlying Figure 2

Post-training produces the largest gains on IFEVAL and MATH LVL 5, while gains are smaller on the other four tasks. Bin-wise diagnostics show deviations concentrated in a few mid-to-high-compute bins, with mild errors elsewhere indicating stable scaling.

C Pre-training vs. Post-training Diagnostics: The diagnostics compare pretrained checkpoints with a fitted post-trained sigmoid capability boundary using boundary distance and paired post-training lift when matching base identities and compute.Official and pretrained labels from the Open LLM Leaderboard define the pretrained subset, emphasizing popular open-weight families such as Llama, Qwen, and Gemma.
C Pre-training vs. Post-training Diagnostics: Post-training gains are largest on IFEVAL and MATH LVL 5, while remaining smaller on MMLU-PRO, BBH, GPQA, and MUSR across observed pretraining compute.Figure 9 quantifies both pretrained checkpoints’ distance below the post-trained capability boundary and paired post-training lift; Figure 10 overlays the corresponding boundaries.
D.1 Omitted Details in Section 3: The rolling protocol fits each boundary on P_t and evaluates on P_t+1, restricting OOD evaluation to overlapping training and validation ranges in z = log10 C.Base-model tables cover four time periods and include only models used at least ten times.
D.1 Omitted Details in Section 3: Relative changes in Table 2 are computed against the constant baseline, with more negative pinball-loss or coverage-error values indicating better performance.The appendix also defines bin-wise coverage using training-period bin edges reused for OOD evaluation; negative signed deviations indicate under-coverage.
D.2 Bin-wise Diagnostics underlying Figure 2: Supplementary bin-wise diagnostics evaluate MMLU-PRO, MATH LVL 5, and IFEVAL using log-compute bins built on training periods and restricted to train–OOD overlap.Figures 12 and 13 report bin-wise OOD coverage and pinball-loss breakdowns across temporal periods.
D.2 Bin-wise Diagnostics underlying Figure 2: The largest coverage and pinball-loss deviations concentrate in a small subset of mid-to-high-compute bins rather than occurring uniformly across compute.Complete ID and OOD coverage-error and pinball-loss patterns are shown across six tasks in Figures 14 and 15.
D.2 Bin-wise Diagnostics underlying Figure 2: Apart from MMLU-PRO, MATH LVL 5, and IFEVAL, the remaining benchmarks show mild ID and OOD errors across compute bins, implying stable scaling.The conclusion is based on the complete bin-wise diagnostics across the remaining tasks.

E Scaling Laws for Model Size · F Newly Evaluated models

The paper extends capability-boundary analysis from compute to model size, finding temporal stabilization for several tasks, and releases evaluations of newly selected open-weight models in Proteus-2k. The new-model evaluation covers popular Hugging Face-compatible models and recent industry releases, with results organized by model-generation status.

E Scaling Laws for Model Size: Model size is used as a complementary single predictor of sigmoid capability boundaries, indicating what a small model could acquire with unbounded pretraining data.This contrasts with classical scaling laws that use pretraining compute or model size together with pretraining token size.
E Scaling Laws for Model Size: Figure 16 compares sigmoid capability boundaries as functions of model size.The figure provides the section’s model-size scaling-law comparison without supplied numerical cell values.
E Scaling Laws for Model Size: MATH LVL 5 and IFEVAL initially improve at fixed model size, but their capability-boundary curves stabilize in later periods.Figure 17 compares sigmoid performance boundaries across successive periods and tasks.
F Newly Evaluated models: The newly evaluated open-weight set includes popular Hugging Face models compatible with lmeval-harness and recent models manually selected from well-known industry labs.The industry-lab examples include OLMo-3, Nemotron nano, and Nemotron cascade.
F Newly Evaluated models: The evaluation results are released in the Proteus-2k dataset, with a selected subset covering new base models absent from the Open LLM Leaderboard.The selected subset is provided separately as proteus-selected.
F Newly Evaluated models: Results are separated by release era, with earlier models shown in Figure 18 and later models in Figure 19, further divided by whether their base models are new.Table 5 summarizes the counts of newly evaluated models by base-model family.

G Sensitivity to Smoothed-Pinball Hyperparameters … H.3 Latent Capability Factors and Prescriptive Boundaries

Appendix analyses show that the estimator is robust to smoothed-pinball hyperparameters, while older leaderboard data and PCA reveal temporal saturation and compute scaling concentrated in one latent capability factor.

G Sensitivity to Smoothed-Pinball Hyperparameters: In-period cross-validation is unnecessary because the low-dimensional sigmoid family has explicit monotonicity and range constraints, while period shift—not in-period overfitting—dominates the evaluation challenge.κ mainly controls approximation to the nonsmooth check loss, and λ primarily provides numerical regularization.
G Sensitivity to Smoothed-Pinball Hyperparameters: OOD pinball loss and coverage are essentially unchanged between fixed defaults (κ, λ) = (50, 10−3) and in-period tuned hyperparameters.Figure 20 reports the comparison on Pt+1; selected values typically concentrate at κ ∈{20, 50} and λ ∈{10−4, 10−3}.
G Sensitivity to Smoothed-Pinball Hyperparameters: Across the swept grid, performance is stable in a broad reasonable region but degrades for overly large ridge weight λ = 10−1.The sweep covers BBH and MATH LVL 5, where temporal drift is most apparent.
G Sensitivity to Smoothed-Pinball Hyperparameters: Once κ is moderate, it has a mild effect, while smaller λ can slightly improve coverage.Coverage is most sensitive to overly large λ, as shown in the coverage-error sweep.
H.1 Public Leaderboards of Frontier Models: On Epoch AI’s smaller, more closed-source model set, MATH Lvl 5 and Mock ATME show no FLOPs-related gains, whereas GPQA diamond scales clearly with FLOPs.These sigmoid scaling-law results are presented in Figure 23.
H.2 Results for Open LLM Leaderboard v1: Older Open LLM Leaderboard v1 results show large gains for GSM8K and TruthfulQA from P1 to P2, followed by saturation of all benchmarks by P3.The v1 findings differ from those for the newer v2 leaderboard.
H.3 Latent Capability Factors and Prescriptive Boundaries: PC1 scales clearly with compute, whereas PC2 and PC3 have nearly flat capability boundaries, indicating that observed leaderboard scaling may largely reflect one component.The principal-component scaling patterns are shown in Figure 25.

I Saturation Analysis across Open LLM Leaderboard Versions and Tasks · J Greedy Optimization for the Balanced I-Optimal Design · Algorithm 1: Greedy optimization for the balanced I-optimal design

The appendix shows that saturation is task- and leaderboard-version-dependent, while detailing a balanced I-optimal greedy design that selects evaluations using local frontier geometry, information gain, and compute-bin balance. Algorithm 1 implements this procedure efficiently and supports subsequent in-sample and out-of-sample calibration analyses.

I Saturation Analysis across Open LLM Leaderboard Versions and Tasks: Task dependence is pronounced on Open LLM Leaderboard v2: knowledge-heavy tasks show less small-model domination than pure reasoning tasks such as MATH Lvl 5.These diagnostics are observational and reflect submitted models, training recipes, post-training, and benchmark targeting over time.
I Saturation Analysis across Open LLM Leaderboard Versions and Tasks: Many Open LLM Leaderboard v1 tasks show strong frontier convergence, making conclusions about the “death of scaling” sensitive to the benchmark suite.The appendix reproduces Hooker (2025, Figure 3) logic for both archived v1 and current v2 suites.
J Greedy Optimization for the Balanced I-Optimal Design: The balanced design maximizes Φλ(S) = Φinfo(S) + λ Φbal(S) by selecting a candidate subset under each period’s evaluation-size budget.Candidates are characterized by log pre-training compute, evaluation cost, and compute-bin index.
J Greedy Optimization for the Balanced I-Optimal Design: Local Jacobians of the high-quantile sigmoid boundary at nominal parameters provide the candidate and bin-level geometry used for information and balance calculations.The frontier has p = 4 parameters, with bin weights uniform in the experiments.
Algorithm 1: Greedy optimization for the balanced I-optimal design: Sherman–Morrison updates yield a closed-form marginal information gain, reducing each candidate evaluation to O(p^2) operations without matrix refactoring.A small ridge term η > 0 stabilizes inverse-information calculations, and the anchor set ensures well-conditioned local geometry.
Algorithm 1: Greedy optimization for the balanced I-optimal design: Algorithm 1 initializes with range-spanning anchor models, then repeatedly adds the feasible candidate with the largest positive gain-per-cost while updating compute-bin counts.Anchors cover the minimum and maximum observed compute and, when available, the nominal sigmoid inflection point z⋆ = −a0/b0.
Algorithm 1: Greedy optimization for the balanced I-optimal design: Figures 28 and 29 assess how the balanced design affects next-period in-sample and out-of-sample coverage calibration error and pinball loss across evaluation budgets and tasks.Coverage error and pinball loss are each plotted on period t + 1 as functions of budget parameter α using boundaries estimated on period t.