Source-linked AI summary
A Hitchhiker's Guide to Scaling Law Estimation
Leshem Choshen, Yang Zhang, Jacob Andreas
TL;DR
Scaling laws are widely used to predict expensive language-model behavior, but how to estimate and interpret them reliably remains underexplored. This paper analyzes a large dataset of model families and finds that intermediate checkpoints, appropriately sized models, multiple seeds, and cross-family information can improve prediction, while the conclusions remain limited by error metrics and incomparable model families.
Problem
How to estimate scaling laws reliably and efficiently across language-model families remains comparatively little studied despite the cost of training and evaluating large models.
Source: §1 · p. 1Method
Results
Intermediate checkpoints improve fit accuracy, similar-sized models generally improve estimates, multiple smaller models can reduce seed variance, and cross-family scaling information can sometimes predict a target family.
Source: §1 · p. 2Takeaways & Limitations
Efficient scaling-law estimation should jointly consider model count, model size, training tokens, checkpoint availability, and information shared across model families.
Source: §1 · p. 2, §12 · p. 8The primary error metric does not distinguish over- from under-estimation, and incomparable scales and ranges across model families prevent general compute-optimal recommendations.
Source: §11 · p. 8
Abstract
from arXiv · showhide
Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets. This provides an efficient way for practitioners and researchers alike to compare pretraining decisions involving optimizers, datasets, and model architectures. Despite the widespread use of scaling laws to model the dynamics of language model training, there has been little work on understanding how to best estimate and interpret them. We collect (and release) a large-scale dataset containing losses and downstream evaluations for 485 previously published pretrained models. We use these to estimate more than 1000 scaling laws, then derive a set of best practices for estimating scaling laws in new model families. We find that fitting scaling laws to intermediate checkpoints of training runs (and not just their final losses) substantially improves accuracy, and that -- all else equal -- estimates of performance are generally most accurate when derived from other models of similar sizes. However, because there is a significant degree of variability across model seeds, training multiple small models is sometimes more useful than training a single large one. Moreover, while different model families differ scaling behavior, they are often similar enough that a target model's behavior can be predicted from a single model with the same architecture, along with scaling parameter estimates derived from other model families.
1. Introduction
Scaling laws offer a cheaper way to predict large-model behavior, but this paper studies how to estimate them reliably under practical compute tradeoffs. Across a large metaanalysis, it finds that checkpoints, model size, model count, and cross-family information materially affect prediction quality.
Motivation and approach
- Scaling-law estimation is motivated by the high cost of training large language models and evaluating changes to architecture or training data.The paper positions scaling laws alongside small-model, synthetic-task, and theoretical approaches for evaluating pretraining decisions.Source: §1 · p. 1
- A large-scale metaanalysis examines how to obtain reliable scaling-law predictions while trading off the number, size, and training-token budget of preliminary models.The authors collect diverse scaled families to analyze estimation across architectures, datasets, and settings.Source: §1 · pp. 1–2
Key findings
- Different model families have different scaling behavior, yet a target can sometimes be predicted using one same-architecture model plus scaling estimates from another family.This suggests that cross-family information can supplement sparse data from the target family.Source: §1 · p. 2
- Intermediate training checkpoints substantially improve scaling-law fits, often allowing final loss estimation from roughly one-third of training.This finding addresses whether laws must rely only on fully trained models.Source: §1 · p. 2
- Estimates are generally most accurate when derived from models closer in size to the target, although additional smaller models can reduce variance and sometimes outperform a single large model.The paper emphasizes that model-size proximity and the number of models create a practical tradeoff.Source: §1 · p. 2
- The study also considers functional-form behavior and summarizes cost-effective estimation in terms of model count, model size, and training tokens.These factors are analyzed together in Figure 2, while the paper discusses whether scaling laws have fewer degrees of freedom than commonly assumed.Source: §1 · p. 2
2. Defining a scaling law
A scaling law is a parametric predictor of a target model’s held-out loss, fitted from cheaper models that vary systematically in parameter count and training tokens. The paper defines the model families and estimation setup used to study which training data make such predictors reliable.
Scaling-law concept
- A scaling law predicts a costly model’s held-out loss from cheaper models with fewer parameters or training tokens, supporting comparisons among pretraining choices.The same framework can identify compute- or inference-constrained hyperparameter choices.Source: §2 · p. 2
Model families
- A scaled model family contains models differing only in parameter count and training tokens, with size changes systematically altering architectural characteristics such as width, depth, and attention heads.Different seeds and checkpoints count as different models under the paper’s formal definition.Source: §2 · p. 2
- The analysis distinguishes maximal-parameter families from families trained on at least a chosen fraction of the full token budget.The latter groups partially trained and converged subset-trained models together for empirical comparison.Source: §2 · p. 2
Estimation
- The experiments use the Hoffmann et al. (2022) functional form, whose parameters represent family performance and scaling effects for parameter count and training tokens.Parameters are estimated from training models by minimizing reconstruction error against held-out-data loss.Source: §2 · pp. 2–3
- Because a scaling law is an ordinary parametric model, its reliability depends on what training family Ftrain is collected and how accurately the law is estimated.The paper therefore treats data collection as a prerequisite for answering practical estimation questions.Source: §2 · p. 3
3. Data for 1000+ scaling laws and more
The paper releases a large, diverse dataset of model-training behavior and uses it to estimate and evaluate hundreds of scaling laws across model families.
3. Data for 1000+ scaling laws and more
- The released resource aggregates loss and downstream-evaluation data across architectures, datasets, and training settings for future scaling-law analyses.The authors manually extracted reported losses when models or data were not directly released.Source: §3 · p. 3
3.1. Data sources
- The dataset covers 1.9M training steps from 485 pretrained models and more than 40 scaled families, including losses, evaluations, checkpoints, and training metadata.It also records parameters, tokens, architecture, seeds, compute, and training epochs when available.Source: §3.1 · p. 4
3. Data for 1000+ scaling laws and more
- The paper presents findings from estimating hundreds of scaling laws.Source: §3.2 · p. 4
3.2. Scaling law estimation
- Scaling laws are fit on non-maximal-parameter models and evaluated on near-final checkpoints from the 30%-maximal-token family using mean absolute relative error.Families must contain at least three models, and square-loss curve fitting is used for the main estimates.Source: §3.2 · p. 4
4. How well should scaling laws predict?
The paper calibrates useful scaling-law accuracy against meaningful modeling effects and finds that parameter relationships vary across families despite similar size-effect ratios.
4. How well should scaling laws predict?
- 4% is typically the best ARE obtained, while errors up to 20% can still distinguish many modeling choices.Past A/B tests rarely motivated changes below a 4% relative difference, and seed variance can reach 3.5%.Source: §4 · p. 4
- Reported modeling changes span roughly 4% to 50% loss differences, providing concrete effect sizes against which scaling-law errors can be judged.Examples include data deduplication at roughly 5%, doubling training tokens at roughly 4%, and 30× more tokens at 50%.Source: §4 · p. 4
- Model families differ in scaling-law parameters, but the ratios governing model-size and training-set-size influence are surprisingly similar.Figure 3 also distinguishes architecture, scaling family, and per-family intercept in its legend.Source: Figure 3 · p. 5
5. When I train a new model, do I even need a new scaling law?
Scaling behavior differs across model families, yet borrowed scaling parameters can sometimes predict a new family accurately; simpler baselines perform substantially worse.
5. When I train a new model, do I even need a new scaling law?
- Different families show dramatic differences in all five scaling-law parameters, indicating that architecture can change the functional form of scaling behavior.The paper therefore tests prediction reliability separately from whether borrowed parameters are numerically accurate.Source: §5 · p. 5
- A single model from a new family can sometimes support accurate predictions when combined with scaling parameters borrowed from another family.In OLMO, 1B model checkpoints estimate the accuracy of a 7B model with less than 1% error, though larger extrapolations can be worse.Source: §5 · p. 5
5.1. Can I train the target model a bit instead of many small models?
- Training up to 30% of the target run can provide reliable estimates, offering an alternative to training many smaller models.Within the same parameter family, only token-count extrapolation is required because the parameter term is constant.Source: §5.1 · p. 5
- The target-model partial-training approach is presented as a possible alternative to scaling the number of layers, subject to computational constraints.Its usefulness follows from estimating losses within the same parameter family using partially trained target versions.Source: §5.1 · p. 5
5.2. Are even simpler baselines enough?
- Scaling laws can accurately estimate targets that outperform every training model, so extrapolation is not limited to interpolation within observed performance.This motivates comparing fitted laws against pessimistic baselines based on the best or most-trained small model.Source: §5.2 · pp. 5–6
- The best-performance and most-trained-model baselines exceed 15% error in many cases and average 18% ARE across the studied families.They almost never achieve errors below 5%, even when using the full training family.Source: §5.2 · p. 6
6. I have some data; what portions should I use?
Intermediate checkpoints materially improve scaling-law estimation, but the earliest training phase should generally be excluded because it can distort predictions.
6. I have some data; what portions should I use?
- Intermediate checkpoints may be informative despite learning-rate schedules that depend on the eventual training-set size.Recent schedule designs and prior single-parameter-family results motivate testing whether multiple checkpoints improve larger-family estimates.Source: §6 · p. 6
- Including full training curves predicts losses well, whereas relying only on final training checkpoints performs significantly worse across model families.The paper therefore fits subsequent scaling laws using intermediate checkpoints rather than final performance alone.Source: §6 · p. 6
6.1. Should I use all intermediate checkpoints?
- The first roughly 10% of checkpoints is often uninformative and should generally be discarded before fitting scaling laws.Early training can contain loss spikes or increases despite additional training, making it harmful to prediction.Source: §6.1 · p. 6
- Ignoring the first 10B tokens reduces ARE from above 15% to 4–10% for some OPT and Pythia models.Cutting fewer tokens produced noisier results, while cutting more had negligible additional effect in preliminary experiments.Source: §6.1 · pp. 6–7
7. How big a model should I train?
The evidence favors using several models rather than relying only on the largest model, because larger models help but seed variability can make added models harmful or non-monotonic. Five models are a safe baseline, while additional diverse models generally improve robustness.
7. How big a model should I train?
- Larger models usually improve predictions, but extrapolating from smaller models can still achieve less than 10% error in GPT, Gopher, and OPT.In Pythia, the smallest models are not predictive, but the remaining models provide similar fits; predicting 34× larger behavior can still be reliable.Source: §7 · p. 7
- Training additional large models can decrease accuracy when high-variance seeds distort the fit, and cross-validation may fail to identify the problematic model.The paper recommends diversifying model sizes and hyperparameters rather than repeatedly training every model with multiple seeds.Source: §7 · p. 7
- The optimal spacing of model sizes, whether linear, logarithmic, or otherwise, remains unresolved.This optimization is left for future work after fixing the largest model and number of models.Source: §7 · p. 7
- Figure 5 evaluates absolute relative error across the number of models and the subset of early-training checkpoints included in the fit.The vertical axis varies the number of models, while the horizontal axis varies the checkpoints retained from the beginning of training.Source: Figure 5 · p. 7
8. How many models for reliable predictions?
- Five models are a safe baseline, and more models generally improve prediction robustness.Additional models can reduce absolute relative error even when they are not larger, although gains are not guaranteed for every model set.Source: §8 · pp. 7–8
- Adding models does not guarantee better predictions: a harmful model group can raise errors until enough models are added or the problematic model is replaced.The observed effects are non-monotonic across rows, so larger models alone do not ensure better predictions.Source: §8 · pp. 7–8
9. Are all scaling law parameters crucial?
Scaling-law parameters vary substantially across model families, but their relationships suggest that the usual parameterization may contain fewer independent degrees of freedom. Related work broadens scaling-law analysis beyond the loss and standard size–token setting.
9. Are all scaling law parameters crucial?
- Three principal components explain 99.49% of the variance in five learned scaling-law parameters, indicating fewer effective degrees of freedom.Parameters associated with the same training variable are tightly coupled, with mostly linear relationships between A and α and between B and β.Source: §9 · p. 8
- The apparent parameter relationships have exceptions concentrated in T5-Pile and families trained with multiple passes over one dataset.A Pythia continual-training baseline is another outlier, with β > 4.Source: §9 · p. 8
- Future scaling-law forms should account for learning-rate schedules because this analysis assumes their effects are negligible.Ignoring training trajectories may help explain inconsistent fit quality at the beginning of training.Source: §9 · p. 8
10. Related work
- Related work studies scaling laws for costs, data selection, training time, transfer, modalities, mixtures of experts, downstream performance, vocabulary, architectures, and fine-tuning.Other studies also construct scaling laws that generalize across models or loss types.Source: §10 · p. 8
11. Limitations
The paper identifies limitations in its error metric and in aggregating heterogeneous model families, while calling for richer checkpoint data to sharpen its conclusions. It also leaves several directions open, including broader cross-family prediction and alternative parameterizations.
11. Limitations
- ARE does not distinguish overestimation from underestimation, so model-selection error estimates can be unnecessarily conservative.This can occur when two families’ scaling laws are biased in the same direction.Source: §11 · p. 8
- Published model families often cover incomparable scales and ranges, preventing an informative aggregate analysis for general compute-optimal recommendations.This limits broad guidance about which preliminary experiments to run.Source: §11 · p. 8
12. Discussion
- Sharing training losses and evaluation results from multiple checkpoints could sharpen or extend the paper’s conclusions, even without releasing model parameters.The authors frame this as a need for additional information about model training.Source: §12 · p. 8
- Open questions include efficient prediction across many model families, scaling laws for intervention deltas, and architecture comparisons without multiple models.The results also motivate alternative scaling-law parameterizations.Source: §12 · pp. 8–9
Impact Statement
- The paper describes its societal consequences as difficult to specify and generally leaning toward efficient, open, and collaborative outcomes.This statement appears in the impact discussion rather than as an empirical limitation.Source: Impact Statement · p. 9
A. Scale up with 1 model
The experiments examine how scaling-law predictions behave when using one model, partial training data, reverse scaling, and cross-validation for model selection. Intermediate training checkpoints and multiple models improve fitting, while cross-validation does not reliably identify models that should be removed.
C. Is scaling working only upwards?
- As shown in Fig. 8, fitting well requires at least 30–40% of the training, while using more training often improves the fit further; the number of models also plays an important role.Source: C. Is scaling working only upwards? · p. 13
A. Scale up with 1 model
- The one-model setup predicts larger models from a single source model, with the target represented by the final row and predictions beginning early in training.The procedure trades off preliminary model count, model size, and training-data budget.Source: A. Scale up with 1 model · p. 13, Figure 6 · p. 14
B. Loss curves and predictions
- Figure 7 compares predicted and actual loss curves across available training-data percentages for representative scaled families.Prediction points use the available training data on the x-axis and predicted loss on the y-axis, while lines show actual loss.Source: B. Loss curves and predictions · p. 13, Figure 7 · p. 15
D. Can we detect bad models to fit on?
- Cross-validation fails to identify bad models for fitting: removing the flagged model produces the worst actual-target ARE in 58% of cases.Hard-to-predict targets more often indicate that the remaining models are poor predictors than that the target is unusually dissimilar.Source: D. Can we detect bad models to fit on? · pp. 13, 15
E. Huber replication
The paper replicates scaling-law fitting with Huber loss rather than absolute relative error. Using δ = e10−3, it reports overall results that are similar to the primary analysis.
E. Huber replication
- Huber loss is used as an alternative to absolute relative error in the replication.The paper defines Huber loss before reporting the replication results.Source: E. Huber replication · p. 15
- Using Huber loss with δ = e10−3 produces overall results similar to the primary analysis.The setting follows Hoffmann et al. (2022).Source: E. Huber replication · p. 15
- Figure 9 compares models trained with different numbers of models and training-data percentages under Huber loss.The figure marks the most efficient setting to reach an error below 0.05.Source: Figure 9 · p. 17