PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei

arXiv:2001.08361v1cs.LGstat.ML

TL;DR

The paper asks how language-model loss depends on scale, data, compute, and architecture. It empirically measures these dependencies for Transformers and finds power-law scaling with weak dependence on model shape. The resulting relations favor large, sample-efficient models and early stopping when compute is fixed, while indicating eventual overfitting constraints even without data reuse.

  • Problem

    The work investigates how language-model loss depends on model architecture, model size, training compute, and available data across large scales.

  • Method

    The authors empirically study Transformer language models and derive predictive relations for loss, overfitting, training speed, compute allocation, and data requirements.

  • Results

    Power laws describe loss versus model size, dataset size, and optimized compute, while performance depends weakly on many architectural and optimization hyperparameters.

  • Takeaways & Limitations

    Compute-efficient training favors very large models, relatively slow data growth, and stopping significantly before convergence; larger models are more sample-efficient.

  • Takeaways & Limitations

    Compute-efficient training may eventually encounter overfitting even when the process never reuses data, and predictions should use optimized compute Cmin rather than fixed-batch compute C.

Abstract

from arXiv · show

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

1 Introduction

This work empirically investigates how Transformer language-model loss depends on model size, dataset size, compute, and architecture. It finds predictable power-law scaling, weak dependence on model shape, and compute-efficient training strategies favoring large, sample-efficient models.

  • 1 Introduction: Performance depends most strongly on model parameters N, dataset size D, and training compute C, while reasonable architectural shape changes have weak effects.The study focuses on Transformer models and examines trends over more than seven orders of magnitude.
  • 1 Introduction: Power-law relationships describe performance versus N, D, and C when the other factors are not bottlenecks, with trends spanning more than six orders of magnitude.The observed trends show no deviation at the upper end, although loss must eventually flatten before reaching zero.
  • 1 Introduction: Increasing model and dataset size together avoids predictable overfitting penalties; an 8x model increase requires roughly 5x more data to avoid a penalty.The penalty depends on the ratio N^0.74/D, and holding either N or D fixed produces diminishing returns.
  • 1 Introduction: Larger models are more sample-efficient, reaching the same performance with fewer optimization steps and fewer data points.The paper also reports predictable training curves that can be extrapolated to estimate longer-training loss.
  • 1 Introduction: Under a fixed compute budget, optimal training favors very large models, modest data growth, larger batches, and stopping substantially before convergence.The predicted data requirement grows slowly as D ∼ C^0.27, while compute allocation primarily increases model size.
  • 1 Introduction: The scaling relations form a predictive framework for compute allocation, overfitting, early stopping, and data requirements, with weak dependence on many architectural and optimization hyperparameters.The reported numerical scaling values are associated with the Webtext2 training set; the critical batch size follows a loss-dependent power law.

2 Background and Methods

The paper trains decoder-only Transformers on WebText2 to measure cross-entropy loss across model configurations, using non-embedding parameters as model size and an approximate compute accounting.

  • Data and evaluation: The principal metric is autoregressive cross-entropy loss averaged over a 1024-token context.Loss is recorded on WebText2 and several other text distributions.
  • Model parameterization: The Transformer is parameterized by layers, residual-stream dimension, feed-forward dimension, attention-output dimension, and attention heads.Most experiments use context length nctx = 1024.
  • Model parameterization: Model size N denotes the number of non-embedding parameters, excluding vocabulary and positional embeddings.This definition produces cleaner scaling laws than total parameter count.
  • Compute accounting: Training compute is estimated from model size and token processing, with the backward pass making total cost approximately 6N floating-point operations per training token.Context-dependent costs are omitted when dmodel > nctx/12 because they are a relatively small fraction of total compute.
  • Training procedure: The study uses Adam for a fixed 2.5 × 10^5 steps and batch size 512 sequences, while the largest models use Adafactor because of memory constraints.Learning-rate schedules had little effect on results at convergence.
  • Data and evaluation: WebText2 extends WebText with additional Reddit-linked text collected from January through October 2018.The source-selection heuristic retained outbound links receiving at least 3 karma.

3 Empirical Results and Basic Power Laws

Across varied Transformer scales and configurations, performance follows smooth power-law trends primarily governed by non-embedding model size, dataset size, and training compute. Model shape has relatively weak effects, while larger models generalize, use context, and learn from data more efficiently.

  • Experimental scope: The experiments vary model size from 768 to 1.5 billion non-embedding parameters and dataset size from 22 million to 23 billion tokens.They also vary shape, context length, and batch size.
  • Model size and shape: Performance depends strongly on non-embedding parameter count N but only weakly on depth, width, attention heads, and feed-forward dimension within reasonable ranges.Figure 5 finds only a few-percent loss variation across broad model shapes at fixed N.
  • Model size and shape: Excluding embedding parameters makes models with different depths converge to a single performance trend, except for fewer than 2 layers or extreme depth-to-width ratios.Including embeddings makes layer count appear to matter more.
  • Basic power laws: Performance exhibits power-law relationships with model size, dataset size, and compute, with trends spanning more than six orders of magnitude when the other factors are not bottlenecks.The observed trends show no upper-end deviation, although loss must eventually flatten above zero.
  • Generalization: Loss on other data distributions improves smoothly with model size and remains closely linked to WebText2 validation performance, with a roughly constant offset.The transfer relationship does not depend on training phase.
  • Context and recurrent baselines: Transformer performance improves with context position, whereas LSTMs perform comparably early in the context but cannot match Transformers later.Transformers asymptotically outperform LSTMs because they make better use of long contexts.
  • Related architectural comparison: Recurrent Transformers perform slightly better as a function of N but require additional compute per parameter.These models reuse parameters.
  • Compute and sample efficiency: At fixed compute, scanning model sizes identifies the best-performing model and shows that sample efficiency improves with model size.The fixed-batch empirical results are acknowledged as not fully optimal until later adjustment using Cmin.

4 Charting the Infinite Data Limit and Overfitting

The paper proposes and empirically validates a scaling law for test loss as a function of model and dataset size, using it to characterize overfitting and data requirements.

  • Proposed L(N, D) Equation: The proposed L(N, D) scaling law is motivated by vocabulary-rescaling invariance and correct infinite-data and infinite-model limits.Its 1/D expansion is a more speculative assumption, justified by expected dataset-variance scaling.
  • Overfitting: For large datasets, loss follows a power law in model size; for smaller fixed datasets, increasing model size eventually produces overfitting.The extent of overfitting is governed primarily by a specific combination of model and dataset size.
  • Empirical Fit: The L(N, D) equation fits the data well, supporting its use as a compact description of optimally trained test loss.The fit is excellent except for datasets reduced to about 2 × 10^7 tokens, where overfitting occurs very early.
  • Overfitting: The critical batch size approximately doubles for every 13% decrease in loss and depends on loss rather than directly on model size.It is measured empirically and is also roughly predicted by the gradient noise scale.
  • Overfitting: Models smaller than 10^9 parameters can be trained with minimal overfitting on the 22B-token WebText2 dataset, while the largest models show mild overfitting.Dataset size can grow sub-linearly with model size while keeping overfitting controlled, though this is not generally compute-optimal.

5 Scaling Laws with Model Size and Training Time

The paper models training loss as a function of model size and training time using a universal step measure adjusted for batch-size efficiency. These fits describe learning curves and support early-stopping estimates.

  • Batch-Size Adjustment: The analysis defines Smin as the minimum training steps needed to reach a target loss, correcting for inefficient batch sizes relative to Bcrit.The related scaling relation also distinguishes the minimum processed examples, Emin, for the same target loss.
  • Batch-Size Adjustment: Training at the critical batch size provides a roughly optimal time/compute tradeoff, requiring 2Smin steps and 2Emin processed examples.B ≫ Bcrit minimizes steps, whereas B ≪ Bcrit minimizes compute.
  • Critical Batch Size: Bcrit(L) is independent of model size and depends only on the attained loss, confirming the corresponding prediction for Transformer language models.The parameterization assumes divergence as loss approaches zero because the gradient noise scale is expected to diverge near the minimum loss.
  • Learning-Curve Fits: A simple power-law loss model in model size and adjusted training time produces compelling, though imperfect, learning-curve fits.The fits use stable Adam-optimized runs after learning-rate warmup and direct measurements of Bcrit.
  • Learning-Curve Fits: At fixed compute or fixed training steps, each compute budget has an associated optimal model size that maximizes performance.The power-law equation is less reliable very early in training, explaining mediocre fits at small step counts.
  • Early Stopping: The model provides a lower-bound and rough estimate for when early stopping should occur under data limitation.The empirical comparison uses adjusted stopping steps and computes the infinite-data converged loss from the fitted L(N, D) relation.

6 Optimal Allocation of the Compute Budget

After correcting compute for batch-size inefficiency, the paper derives and tests how to allocate compute between model size and training data. Compute-efficient scaling favors much larger models, modest data growth, and few serial steps, but eventually encounters a data-limited breakdown.

  • Compute Correction: Standardizing training to the critical batch size produces cleaner compute-performance trends and enables optimal allocation between model size and processed data.The allocation is determined both empirically and from the fitted L(N, Smin) equation.
  • Optimal Allocation: For a 10× increase in compute, the optimal model size grows 5× while processed data grows only 2×.Batch-adjusted optimization steps grow very slowly, so much of the data increase can support larger batch sizes.
  • Optimal Allocation: Compute-efficient training predominantly increases model size and batch size, with negligible growth in the number of serial optimization steps.The paper suggests that improving early training dynamics may therefore be especially valuable.
  • Predictions: The predicted compute and model-size power laws agree closely with empirical scaling, supporting a predictive framework for language-model performance.The optimal allocation is obtained by minimizing the fitted loss at fixed compute.
  • Contradictions and a Conjecture: Compute-efficient training eventually becomes inconsistent with the dataset-size scaling law because productive data growth is limited to roughly one epoch.This indicates that the scaling laws must break down before the predicted intersection, even without data reuse.
  • Contradictions and a Conjecture: The estimated intersection occurs near 10^4 PF-days, 10^12 parameters, 10^12 tokens, and 1.7 nats/token, but each value is highly uncertain.The uncertainty can span about an order of magnitude in either direction because it depends sensitively on fitted power-law exponents.

7 Related Work

The paper situates its scaling laws among prior work on dataset and model-size scaling, architecture scaling, and generalization in overparameterized models.

  • Dataset and Model Scaling: Earlier and recent studies reported power-law relationships involving dataset size, while this paper finds sub-linear rather than super-linear data growth with model size.The authors identify prior work on joint model-size and data-size scaling as especially close to their results.
  • Architecture Scaling: Unlike EfficientNet’s architecture-scaling prescription for image models, this paper finds that language-model performance depends far more on overall scale than precise width and depth choices.For language models, width and depth should remain roughly fixed in proportion when scaling up.
  • Generalization: Prior work on overparameterized models reported a jamming transition near equal model and dataset sizes, whereas this paper observes no such transition under early stopping.The paper instead finds that required training data scales sublinearly with model size.

8 Discussion

The paper finds that language-model loss follows predictable scaling relations with model size, data, and compute, while depending weakly on many architectural details. These relations support predictions about overfitting, compute allocation, and future model-parallel training strategies.

  • Loss scales consistently as a power law with non-embedding parameters N, dataset size D, and optimized training compute Cmin, with diminishing returns at larger scales.The relations extend beyond observation by providing a predictive framework for compute scaling, overfitting, early stopping, and data requirements.
  • Performance depends most strongly on N, D, and C, while reasonable changes in architectural shape such as depth versus width have weak effects.
  • The scaling laws remain conjectural outside language modeling, because the authors do not know which results depend on natural-language structure.They identify testing on images, audio, and video, and developing a theory of the laws, as important directions.
  • Continued loss improvement may translate into seemingly qualitative capability changes, so loss alone may not reveal progress on relevant language tasks.
  • Larger models are expected to improve performance and sample efficiency, motivating further work on model parallelism, sparsity, branching, and networks that grow during training.

Appendices

The appendices summarize the paper’s key trends and set up the derivation of compute-efficient training. They define the optimization problem in terms of model size, updates, batch size, and training compute.

  • Summary of trends: The appendices provide a reference summary of the paper’s key empirical trends and fitted values.
  • Notation: The appendix adjusts C, S, and αC for training at the critical batch size, omitting the adjustment label to simplify notation.
  • Derivation setup: Compute-efficient training is derived by expressing the number of parameter updates S in terms of compute C, model size N, and critical batch size Bcrit.The compute estimate is C ≈ 6NBS, with B the batch size and S the number of training steps.
  • Derivation setup: The compute-efficient frontier is determined by the equations obtained from optimizing the loss under a fixed compute budget.

B.2 Efficient Training

The efficient-training analysis derives how model size and training duration should be chosen for a fixed compute budget. It concludes that compute efficiency favors larger models trained for fewer updates and stopped before convergence.

  • Compute-efficient frontier: The efficient-training frontier follows from combining the optimality conditions obtained after substituting the fixed-compute relation for S.
  • Compute-efficient frontier: Compute-efficient training should stop at a fixed percentage αN/αS ≈ 10% above the converged loss.
  • Comparison to inefficient training: For the same target loss, compute-efficient training uses 7.7x fewer parameter updates, 2.7x more parameters, and 65% less compute than the comparison setup.
  • Sensitivity to model size: A model between 0.6x and 2.2x the optimal size requires only a 20% increase in compute budget.Using a smaller model can help when inference cost matters, while a larger model can reduce the number of training steps.
  • Sensitivity to model size: A 2.2x larger model requires 45% fewer steps at a cost of 20% more training compute, within the power-law region after initial transients.

C Caveats

The caveats qualify the scaling-law analysis through theoretical uncertainty, limited experimental regimes, and possible omissions in compute and hyperparameter accounting. Additional results examine early stopping, parameter reuse, critical batch size, and context-dependent loss.

  • Caveats: The scaling laws lack a solid theoretical understanding, especially for model-size and compute scaling, making their trustworthy range difficult to determine.
  • Caveats: Predictions for Bcrit(L) are uncertain far outside the explored loss range, potentially affecting trade-offs between data parallelism, serial steps, and training time.
  • Caveats: The compute estimate omits contributions proportional to context length, so compute scaling may be confounded when nctx ≳ 12dmodel.
  • Architectural comparisons: Recurrent Transformers perform slightly better at equal parameter count but slightly worse when parameter reuse is accounted for per FLOP.
  • Sample efficiency: Larger models reach fixed losses in fewer serial steps and improve sample efficiency, with Figure 19 reporting nearly a 100-fold sample-efficiency improvement across model sizes.
  • Context dependence: For fixed model size, loss scales as a power law with context position T, while training first learns short-range information and later learns longer-range correlations.
  • Context dependence: Models trained with nctx = 8 can outperform the largest nctx = 1024 models on very early tokens, while nctx = 1024 models improve steadily with size on all but the first token.

D.6 Learning Rate Schedules and Error Analysis

Learning-rate schedules have limited impact when they satisfy broad conditions, while power laws provide substantially better qualitative fits for scaling trends. Learning-rate choice depends on model size, and the fitting procedures exclude specific small or incompletely trained models.

  • Learning-rate schedules: Learning-rate schedule choice is mostly irrelevant when the summed learning rate is sufficiently large, warmup is included, and the final rate decays near zero.Schedule variations appear comparable to statistical noise across runs.
  • Learning-rate schedules: Performance does not depend strongly on learning rate when it is not too small and does not decay too quickly.These experiments compare cosine, linear, and other decay schedules on a 3-million-parameter model without decaying to zero.
  • Learning-rate schedules: Larger models require smaller learning rates to prevent divergence, whereas smaller models tolerate larger learning rates.
  • Learning-rate schedules: The learning-rate formula breaks down for N > 10^10 parameters and may depend on network width, although it works sufficiently well for the considered models.
  • Scaling-law fits: Power-law fits for L(N), L(C), and L(D) are qualitatively more accurate than logarithmic fits.The parameter-count trend L(N) is also fit better by a power law than by a logarithm.
  • Scaling-law fits: Fits exclude certain one-layer models and the largest models not trained to convergence, although including them changes parameters only marginally and preserves extrapolation.For L(C), the one-to-two-layer transition creates a noticeable lump in the data; L(N) excludes very small one-layer models and incompletely converged large models.

D.8 Generalization and Architecture

At fixed total parameter count, generalization to other data distributions does not depend on network depth. Instead, it appears to depend on performance on the training distribution, with one dataset showing an unusual 12-layer overfitting result.

  • Generalization and architecture: Generalization to other data distributions does not depend on network depth when total parameter count is held fixed.This conclusion comes from evaluations across a series of datasets using models with approximately 1.5 billion parameters.
  • Generalization and architecture: Generalization performance appears to depend primarily on training-distribution performance rather than depth.
  • Generalization and architecture: The 12-layer model overfit the Internet Books dataset, so its early-stopped performance is shown.The paper reports not seeing this surprising result in other experiments.
Loading 2001.08361v1…