Source-linked AI summary
Observational Scaling Laws and the Predictability of Language Model Performance
Yangjun Ruan, Chris J. Maddison, Tatsunori Hashimoto
TL;DR
Predicting how language-model performance changes with scale matters for benchmark and algorithm development, but compute-based scaling laws require training models across many scales. This paper builds observational scaling laws using a shared low-dimensional capability space and finds complex behaviors—including emergent, agentic, and post-training performance—predictable.
Problem
Understanding how language-model performance changes with scale matters for benchmark and algorithm development, but compute-based scaling laws require training models across many scales.
Source: §1 · p. 1Method
The authors fit observational scaling laws from standardized benchmark performance, mapping models across families into a shared low-dimensional capability space with family-specific compute efficiency.
Source: §1 · p. 2, §3.1 · p. 4Results
Some emergent phenomena follow smooth sigmoids and are predictable from sub-Llama-2 7B models; simpler benchmarks predict GPT-4 agent performance, and the laws predict post-training gains.
Source: §1 · p. 3Takeaways & Limitations
A small set of observable capability measures supports low-cost, broad-coverage predictions of complex downstream capabilities across model families.
Source: §1 · pp. 2–3The approach primarily addresses post-training scaling, not pretraining, and the study mostly examines few-shot or basic prompting and simple agent scaffolding.
Source: §7 · p. 14
Abstract
from arXiv · showhide
Understanding how language model performance varies with scale is critical to benchmark and algorithm development. Scaling laws are one approach to building this understanding, but the requirement of training models across many different scales has limited their use. We propose an alternative, observational approach that bypasses model training and instead builds scaling laws from ~100 publically available models. Building a single scaling law from multiple model families is challenging due to large variations in their training compute efficiencies and capabilities. However, we show that these variations are consistent with a simple, generalized scaling law where language model performance is a function of a low-dimensional capability space, and model families only vary in their efficiency in converting training compute to capabilities. Using this approach, we show the surprising predictability of complex scaling phenomena: we show that several emergent phenomena follow a smooth, sigmoidal behavior and are predictable from small models; we show that the agent performance of models such as GPT-4 can be precisely predicted from simpler non-agentic benchmarks; and we show how to predict the impact of post-training interventions like Chain-of-Thought and Self-Consistency as language model capabilities continue to improve.
1 Introduction
The paper proposes observational scaling laws to make scaling analysis cheaper and broader by using public models and standardized benchmarks. It reports predictable scaling patterns for emergent capabilities, agent performance, and post-training interventions.
1 Introduction
- Standard scaling analyses are uncommon in benchmarking and post-training studies because training across scales is costly and open models span too few scales for reliable predictions.The paper cites 3–5 scales for open models.Source: §1 · p. 1
- Observational scaling laws hypothesize a shared low-dimensional capability space, with model families differing in how efficiently they convert training compute into capabilities.The authors report capability measures with R2 > 0.9 relationships to compute within model families and downstream metrics across families.Source: §1 · p. 2
- Using many public models provides no training-cost scaling analysis, wider scale coverage, and higher resolution than relying on a single model family.The authors say representative families can support predictions by evaluating only 10–20 models.Source: §1 · p. 2
- High-resolution observational scaling laws show that some emergent phenomena follow a smooth sigmoid and are accurately predicted from sub-Llama-2 7B models; the law predicts GPT-4 agent performance from sub-GPT-3.5 models.The scaling laws also predict at-scale gains from Chain-of-Thought and Self-Consistency interventions when fit on sub-Llama-2 7B models, and identify programming capabilities as driving agent performance.Source: §1 · p. 3
- The work contributes a conceptual framework, capability measures for standard benchmarks, predictions of complex capabilities, and a small set of representative model families for low-cost analysis.The framework is intended for benchmark and algorithm development.Source: §1 · pp. 2–3
2 Related Work
Prior work relates compute and loss to downstream performance, examines benchmark correlations, and debates whether apparent emergence is predictable. This paper distinguishes its approach by fitting higher-resolution scaling laws across model families from observable benchmark metrics.
Compute scaling laws
Downstream scaling laws
- Prior downstream work explores transfer and benchmark scaling, but whether benchmark performance is predictable remains debated amid reports of rapid emergence [1] [35] [15] [30] [34] [27] [83] [98].Some studies attribute apparent discontinuities to metric choice or too few data points [60] [38].Source: §2 · p. 3
Correlations between benchmarks
- Benchmark-correlation studies identify low-rank capability structure, while this work uses such structure for scaling predictions aimed at benchmark and algorithm development [40] [14].Prior results include a single factor explaining 85% of performance and three factors accounting for 82% of HELM variation.Source: §2 · p. 4
3 Observational Scaling Laws
Observational scaling laws extract low-dimensional capability measures from benchmark performance, use them as surrogate scale measures, and fit downstream scaling relationships across model families.
3.1 Generalizing Compute Scaling Laws
- Model families share a low-dimensional capability space, while differing in how efficiently they convert compute into capabilities.This generalization addresses family-specific scaling coefficients that often restrict standard analyses to 3–5 models from one family.Source: §3.1 · p. 4
3.2 Identifying a Low-Dimensional Capability Space (Eq. (5))
- The top three principal components explain ∼97% of benchmark variation and represent general, reasoning, and programming capabilities.PC-1 alone explains nearly 80% of variation in LM capabilities.Source: Figure 2 · p. 5, §3.2 · p. 6
3.3 Principal Capability Measures as Surrogate Scale Measures (Eq. (4))
- Within model families, PC-1 correlates linearly with log-training FLOPs, with R2 > 0.9; the pattern generally extends to PC-2 and PC-3.The relationship also appears across varied families, including multilingual and code-trained models, though correlations tend to decrease for lower-ranked PCs.Source: §3.3 · p. 6, Figure 3 · p. 6
3.4 Fitting Observational Scaling Laws (Eq. (3))
- The fitting procedure maps extracted principal components to a scalar capability measure for each normalized downstream error metric.It uses ordinary least squares and constrains the sigmoidal scale h to [0.8, 1.0], yielding h*=1 in most experiments.Source: §3.4 · pp. 6–7
- For cross-family comparisons, f-equivalent FLOPs express a model’s performance as the compute a reference-family model would need to match it.The paper uses Llama-2 as the reference family, so plots can be read in units of Llama-2 FLOPs.Source: §3.4 · p. 7
4 Validating Observational Scaling Laws
The section tests whether observational scaling laws can forecast complex LM behavior from weaker models. Across emergent, agentic, and post-training tasks, the reported predictions are accurate, with capability measures outperforming compute-based baselines in the cited comparisons.
4 Validating Observational Scaling Laws
- Observational scaling laws accurately forecast emergent, agentic, and post-training behavior from weaker models.The section evaluates these predictions with systematic holdouts and robustness checks.Source: §4 · p. 7, §4.1 · p. 9, §4.2 · p. 10
- Validation uses weaker-to-stronger model holdouts, train-only preprocessing, and preregistered predictions assessed on models released after the initial paper release.The standard training-FLOPs cutoff yields 47 training and 30 test models; later-model checks add 20 test models.Source: §4 · pp. 7–8
4.1 Predictability of “Emergent” Capabilities
- Four BigBench tasks labeled emergent show smooth sigmoidal behavior, with transition points often forecast from models only slightly above random.PC measures predict better than training FLOPs, whose extrapolations have much higher MSE; the preregistered predictions also extrapolate to later models including Llama-3.1-405B [24].Source: Figure 4 · p. 8, §4 · p. 8, §4.1 · p. 9
4.2 Predictability of Agentic Capabilities
- Held-out GPT-4 and Claude-2 agent-benchmark performance is accurately predicted from models with over a 10% performance gap.Programming capability (HumanEval) receives significant fitted weight on both benchmarks.Source: Figure 5 · p. 9, §4.2 · p. 10
4.3 Predicting the Impact of Post-Training Techniques
- Observational scaling laws accurately predict stronger-model performance with Naive, CoT, and CoT + SC, while CoT scales more pronouncedly than Self-Consistency with CoT.The fitted weights place greater emphasis on MMLU and HumanEval when moving from Naive prompting to CoT.Source: Figure 6 · p. 10, §4.3 · p. 11
5 Selecting Low-Cost Model Subsets for Practical Scaling Analyses
The section presents V-optimality as a way to reduce the evaluation cost of observational scaling analyses by selecting a small model subset. In the tested post-training prediction tasks, selected subsets approach full-set prediction performance, and 12 models retain overall accuracy.
5 Selecting Low-Cost Model Subsets for Practical Scaling Analyses
- The subset-selection approach is intended to reduce evaluation cost while preserving high prediction accuracy, without requiring knowledge of the downstream task.The selection criteria draw on optimal experimental design.Source: §5 · p. 11
Recommended model series for scaling analysis
- Table 1 lists recommended model sets under different budgets, including a 12-model set spanning stronger Llama-3, weaker Falcon, and programming-specialized DeepSeek-Coder models.The lists include Llama-2 models and are intended to cover diverse capability ranges and dimensions.Source: Table 1 · p. 12, §5 · p. 12
Method
- The method selects at most Mmax models to minimize expected prediction error under the linear-regression design criterion.It searches combinations of whole model families using the capability matrix from the PC analysis.Source: §5 · p. 12
Validation
- V-optimality selection significantly outperforms random selection and quickly approaches full-set prediction performance; 12 of 47 selected models retain overall accuracy.The evaluation averages test MSE across Naive, CoT, and CoT + SC tasks.Source: §5 · p. 12
6 Discussion and Other Applications of Observational Scaling
The discussion presents low-dimensional capability measures as a basis for comparing models and analyzing scaling behavior. It also describes potential applications to compute-efficiency comparisons and to differences in how model families respond to post-training.
Discussion
- The results validate a low-dimensional capability space measured through a low-rank decomposition of existing LM benchmarks, connecting the approach to item response theory.Source: §6 · p. 12
PC-1 as a smooth capability measure with high dynamic range
- PC-1 provides a smooth measure spanning at least five orders of magnitude, enabling comparison of heterogeneous models on one scale where individual benchmarks may saturate or be random.The authors suggest PC-1 may be suitable as a pretraining optimization target.Source: Figure 8 · p. 13, §6 · p. 13
Post-training interventions and their interactions with model families
- LM families benefit similarly from CoT, but Phi benefits earlier and scales less rapidly, while DeepSeek-Coder also transitions earlier with less rapid scaling.These differences point to the importance of pretraining data in model scaling behavior.Source: Figure 9 · p. 13, §6 · p. 13
Training data efficiency measurements using PC-1
- Most models follow a clear training-compute-to-capability tradeoff, while Phi appears unusually compute-efficient, likely because inference FLOPs used to generate its training data are omitted.Source: §6 · p. 13
7 Conclusion, Limitations, and Future Work
Observational scaling laws use a shared, low-dimensional capability space to make low-cost, high-resolution, broad-coverage predictions about complex scaling phenomena. The approach is primarily applicable to post-training analyses, not pretraining in the same way as standard compute-based scaling laws.
7 Conclusion, Limitations, and Future Work
- The approach builds scaling laws across model families using a shared, low-dimensional capability space.It supports predictions about emergent behaviors, agentic capabilities, and post-training interventions.Source: §7 · pp. 13–14
- Observational scaling laws provide low-cost, high-resolution, broad-coverage analyses for predicting complex scaling phenomena.The paper also offers practical prescriptions for applying similar analyses to benchmarks and post-training methods.Source: §7 · pp. 13–14
- The approach is primarily applicable to post-training scaling analyses and does not directly translate to pretraining like standard compute-based scaling laws.Source: §7 · p. 14
Major Changelog
This material records revisions, outlines the fitting procedure for observational scaling laws, and documents collected model metadata and benchmark evaluations.
Major Changelog
- The changelog records clarifications, plot-scale updates, minor numerical revisions, updated preregistered predictions, and added limitations and appendix analyses.Source: Major Changelog · p. 14
A Algorithm
- The fitting algorithm takes benchmark errors, a downstream error metric, compute scales, and a reference family, and returns a fitted scaling law.Source: A Algorithm · p. 22
- The algorithm imputes missing benchmark values, normalizes the target metric, extracts principal capability measures, and fits a sigmoid regression.Source: A Algorithm · p. 22
- It aggregates capability measures, projects them onto a reference family’s compute-equivalent scale, and returns the transformed scaling law.Source: A Algorithm · p. 22
B.1.1 Pretrained Base Models
- The pretrained base-model collection covers 21 families and 77 models; training FLOPs are estimated from parameter counts and pretraining tokens.For continued pretraining, the token estimate includes the additional continued-pretraining tokens.Source: §B.1.1 · p. 23
B.1.2 Instruction-Tuned Models
- The instruction-tuned collection includes open and proprietary models evaluated on AgentBench and AgentBoard, with available metadata and standard benchmark results collected.Pretraining and applicable continued-pretraining tokens are counted, while instruction-tuning and further fine-tuning data are excluded from the token estimate.Source: §B.1.2 · p. 23
B.2 Downstream Evaluation
The appendix describes downstream evaluations and PCA procedures, then reports capability-structure findings for instruction-tuned models and properties of the principal-component measures.
B.2 Downstream Evaluation
- Downstream tasks use likelihood-based two-shot evaluation for emergent capabilities, original-paper protocols for post-training tasks, and sourced leaderboard results for agentic tasks.Source: §B.2 · pp. 23–24
B.3 PCA Analysis
- Missing benchmark values are iteratively imputed using PCA reconstruction, and the top three principal components are extracted after mean-centering the data.For train-test splits, imputation and PCA transformations are fitted on training data and then applied to test data to prevent leakage.Source: §B.3 · p. 25
C.1 PC Analysis of Instruction-Tuned LMs
- The top three PCs explain about 98.6% of instruction-tuned models’ benchmark-performance variance, with patterns similar to pretrained models.Source: §C.1 · p. 26, Figure C.1 · p. 26
C.2 Properties of PC measures
- Lower-ranked PCs generally correlate linearly with log-compute within comparable families, but the correlation decreases at lower ranks.Source: §C.2 · p. 26, Figure C.2 · p. 27
- Aggregated PC measures generally correlate linearly with log-compute within families, with consistently significant correlation for Llama-2, the reference family.Source: §C.2 · p. 26, Figure C.3 · p. 27
- A single benchmark metric can have limited dynamic range, saturating for large models or giving random readouts for weak models.The examples are HellaSwag and Winogrande for saturation, and MMLU and GSM8K for weak-model readouts.Source: §C.2 · p. 26, Figure C.4 · p. 28
C.3 Additional Preregisteration Results
Preregistered observational scaling laws forecast new models and challenging benchmarks. They outperform compute-based scaling laws in the reported extrapolations.
C.3 Additional Preregisteration Results
- Observational scaling laws provide reasonable forecasts for new models on post-training tasks, outperforming predictions based on training FLOPs.For naive prompting, the prediction is somewhat off but follows the general trend.Source: Figure C.5 · p. 29, §C.3 · p. 29
- On GPQA, MATH, and BBH, observational scaling laws provide reasonable forecasts and outperform compute-based laws when extrapolating to larger models.These more challenging benchmarks were selected where existing models and predictions from base benchmarks had non-trivial performance.Source: Figure C.6 · p. 30, §C.3 · p. 30
C.4 Robustness Checks
Across robustness checks, three principal-component (PC) measures generally give the strongest predictions and remain effective under varied holdout cutoffs. For emergent capabilities, model-size predictions are poor, while PC measures retain an advantage across alternative cutoffs and additional tasks.
C.4 Robustness Checks
- Using 3 PCs consistently gives the most robust predictions across post-training, emergent-capability, and agentic tasks.Using more than 2 PCs outperforms compute measures such as FLOPs.Source: §C.4 · p. 31, Figure C.7 · p. 31, Figure C.8 · p. 32
- Varying the FLOPs cutoff changes the test-set share from 60% to 5%; emergent tasks also use an accuracy-based holdout to test weak-to-strong prediction.The study compares test MSE across cutoffs and summarizes overall prediction error with area under the error curve (AUE).Source: §C.4 · p. 33
- PC measures, especially with 3 PCs, reach low prediction error earlier and yield much lower AUE than training FLOPs or model size in most evaluated holdout setups.Across emergent-capability setups, 3 PCs yield lower AUE than compute measures in 7 of 8 setups.Source: §C.4 · p. 33, Figure C.10 · p. 34, Figure C.11 · p. 34
C.5 Emergent Capabilities
- Model size gives significantly worse forecasts than FLOPs and PC measures for emergent capabilities and poorly captures their emergence trend.The authors suggest differing training-data sizes and quality or architectures across model families may explain this result.Source: §C.5 · p. 35, Figure C.12 · p. 35
- With the default arithmetic-task cutoff, many data points lie near perfect performance, making prediction trivial, but PC measures still outperform model size and training FLOPs.The paper had used a different cutoff to make these prediction tasks more challenging.Source: §C.5 · p. 35, Figure C.13 · p. 36
- On additional emergent tasks, PC measures outperform model size and training FLOPs, though IPA Transliterate extrapolation misses the ground-truth trend while still forecasting emergence reasonably.The authors suggest the collected benchmark metrics may not cover the task-specific capabilities well.Source: §C.5 · p. 35, Figure C.14 · p. 37
C.6 Post-Training Method Analysis
PC measures capture post-training scaling trends on GSM8K and BBH, while training FLOPs also predict BBH performance reasonably. For model selection, V-optimality captures scaling trends with more than eight models, whereas random selection performs substantially worse.
C.6 Post-Training Method Analysis
- PC measures capture GSM8K post-training scaling trends and give the best prediction performance across tasks.The analysis compares different scale measures for predicting post-training methods on GSM8K.Source: §C.6 · p. 38, Figure C.15 · p. 38
- On BBH tasks, PC measures capture CoT scaling trends in both setups; training FLOPs predict reasonably but are less tight in the Naive setup and miss the synthetic-data-trained Phi model's behavior.The analysis aggregates task accuracies after normalizing each task by its random-prediction accuracy.Source: §C.6 · p. 39, Figure C.16 · p. 39
C.7 Model Subset Selection
- With more than 8 V-optimality-selected models, fitted scaling curves converge and accurately capture the scaling trend.The paper presents this as evidence of the selection method's efficiency.Source: §C.7 · p. 40, Figure C.17 · p. 40
- Randomly selected models predict substantially worse than V-optimality-selected models, even when 16 models are selected.The comparison uses a fixed random seed.Source: §C.7 · p. 41, Figure C.18 · p. 41
C.8 Additional Analysis
Additional analyses examine non-negative matrix factorization (NMF), broader cutoff choices, single-family predictions, and prediction uncertainty. The paper also preregisters fitted scaling-law forms for testing predictions on future models.
C.8 Additional Analysis
- NMF yields interpretable components with positive, smooth scaling, but its components are not orthogonal or uniquely ordered, so their coefficients may not measure independent capabilities.PCA instead orders components by explained variance, providing an importance measure and a way to trade off representativeness against noise.Source: §C.8 · p. 42, Figure C.19 · p. 42
- Pushing task-specific cutoffs farther back still gives reasonable emergent-capability predictions, while AgentBench extrapolations somewhat underestimate performance but follow the overall trend.The analysis also tests generalization to Llama-3.1 405B (FP8) and uses an 80/20 train/test split for AgentBench.Source: Figure C.20 · p. 43
- Within a single model family, at least 5 models are typically needed for accurate FLOPs-based extrapolation, but results depend on the setup and each tested family predicts one task well but not the other.The examples are Qwen1.5 on non-algorithmic tasks and OPT on arithmetic tasks.Source: Figure C.21 · p. 43
- Observed data fall within the nonlinear-regression predictions' 95% confidence intervals, which may widen when extrapolating from few above-random-guess data points.Persian QA is given as an example of the latter condition.Source: Figure C.22 · p. 43
C.9 Fitted Functional Forms for Preregistration of Predictions
- The fitted scaling-law forms were preregistered as predictions for future models to test the generalizability of the analysis on unseen models.The table presents each form as the logit of normalized accuracy, with benchmark metrics scaled to [0, 1].Source: §C.9 · p. 44, Table C.1 · p. 44