PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Autoregressive Generative Modeling

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, Sam McCandlish

arXiv:2010.14701v2cs.LGcs.CLcs.CV

TL;DR

The paper asks whether predictable scaling laws extend beyond language and what they imply for model distributions and downstream capabilities. It studies autoregressive Transformers across multiple modalities and tasks, fitting loss and related performance measures as functions of model size, compute, and data. The results show smooth power-law-plus-constant scaling across domains, nearly universal compute-optimal model-size exponents, and continued downstream gains even when generative loss levels off.

  • Problem

    The paper investigates whether scaling laws apply across data modalities and whether loss improvements translate into representation quality and downstream performance.

  • Method

    The authors evaluate autoregressive decoder-only Transformers across image, video, multimodal, mathematical, and referenced language settings while varying model size, compute, and dataset size.

  • Results

    Scaling laws hold across domains, with optimal model size scaling as Nopt(C) ∝C^0.7 and compute-optimal dataset size suggesting D ∝N^0.4.

  • Takeaways & Limitations

    The results suggest that generative-loss scaling can predict continued improvements in practical downstream capabilities and provide estimates of remaining model-distribution information.

  • Takeaways & Limitations

    Interpreting irreducible loss as entropy and reducible loss as KL divergence requires fully converged models, but the largest studied models were not fully converged.

Abstract

from arXiv · show

We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as $S($True$) + D_{\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an $8\times 8$ resolution, and we can forecast the model size needed to achieve any given reducible loss (ie $D_{\mathrm{KL}}$) in nats/image for other resolutions. We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question "Is a picture worth a thousand words?"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.

1 Introduction

The paper tests whether autoregressive Transformers exhibit precise, broadly shared scaling laws across modalities and tasks, and examines what those laws imply for model capability, data complexity, and downstream performance. Across domains, loss follows power-law-plus-constant trends, optimal model size scales with compute, and several downstream or modality-specific measures improve smoothly with scale.

  • Cross-domain scaling: Scaling laws apply across language, image, video, multimodal, and mathematical problem-solving domains using a common autoregressive Transformer architecture.The models require only minimal changes to hyperparameters such as width, depth, or learning rate.
  • Cross-domain scaling: Loss scales as a modality-dependent power law in model size, compute, and sometimes dataset size, plus an irreducible constant.The reducible component often follows the power law closely even when it is much smaller than the irreducible loss.
  • Information-theoretic interpretation: The information-theoretic interpretation treats irreducible loss as an estimate of true-distribution entropy and reducible loss as an estimate of KL divergence.This interpretation requires assuming that an infinitely large Transformer could model the data distribution exactly after dataset size and compute also become infinite.
  • Information-theoretic interpretation: Billion-parameter models capture all but a few nats/image for YFCC100M images downsampled to 8x8, while larger practical images require much larger models.Scaling laws allow forecasting the model size needed to reach a specified reducible loss at other resolutions.
  • Compute and model size: β ∼0.7: optimal model size follows nearly the same power law with compute across all domains.The corresponding compute-optimal dataset-size relation is D ∝N 0.4, suggesting sub-linear dataset growth with model size.
  • Downstream and modality-specific results: ImageNet classification loss continues following a power law with pretrained model size even after generative loss approaches its irreducible component.This supports the paper’s broader conclusion that downstream performance can continue improving when generative loss appears to level off.

2 Central Empirical Scaling Laws in Each Domain

Across modalities, decoder-only Transformers trained with autoregressive cross-entropy exhibit smooth scaling with model size and compute, while domain-specific trends reveal important architectural and evaluation differences.

  • Decoder-only Transformers use autoregressive cross-entropy across language, multimodal, image, video, and mathematical problem-solving domains.Sparse attention is used for many models, while math models use dense attention.
  • Mathematical Problem Solving: Larger models overfit two extrapolation generators, achieving worse loss but higher accuracy, while training-loss trends are less consistently power-law-shaped.The affected generators are probability__swr_p_level_set_more_samples and probability__swr_p_sequence_more_samples.
  • Model Size Scaling and Aspect Ratios: Image and math models perform optimally near aspect ratio 5, favoring deeper, thinner models than optimized language models.The study reports at least a 10x smaller aspect ratio than language models, while another setup describes approximately 10 for math, image, and video models.
  • Compute Scaling and Optimal Model Sizes: Compute-constrained optimal model size follows Nopt(C) ∝ C^β with β ∼0.7 across domains, implying sub-linear dataset scaling during compute-optimal training.The paper gives a median estimate of D ∝ N^0.4 and notes that critical-batch-size adjustments were not made.
  • Position-Dependent Loss: Context-position loss is modality-dependent: language and video follow power-law-plus-constant trends, whereas images have highest loss at the first pixels and near the center.Video’s per-frame oscillations are attributed to video encoding, and the first image pixel can fall outside the displayed color range.

3 Image and Video Modeling, the Reducible Loss, and Downstream Tasks

Image and video modeling exhibit smooth scaling of reducible loss with resolution, compute, and model size, while downstream ImageNet classification continues improving even near generative irreducible loss.

  • Image Modeling: Image models were trained across 8x8–32x32 pixels and 64x64 VQ encodings to study how representation complexity affects scaling.The study measured reducible loss against compute and model size across resolutions and encodings.
  • Image Modeling: A few-nat reducible loss per image is reached for 8x8 images by models approaching one billion parameters.The paper interprets this as near-perfect modeling of the downsampled YFCC100M distribution.
  • Image Modeling: Larger images require enormous compute to approach perfect distributional modeling, with requirements growing steeply with image resolution.The authors describe the extrapolated requirement as a steep power law or possibly exponential.
  • Video Modeling: Video reducible loss also follows a power law, forecasting about 10^13 parameters and 10^4 PF-days for a few nats/frame on a final frame.The forecast concerns the 16th frame of this video type, relevant to continuation generation.
  • Downstream Tasks: ImageNet fine-tuning scales smoothly in both classification loss and error rate, even when generative loss approaches its irreducible level.The result suggests that semantic information remains in the last few generative-loss bits.

4 Multimodal Models and Information Gain

The paper uses multimodal models to quantify information shared between captions and images, finding asymmetric information transfer and slow infogain growth with model size.

  • Information Measures: Mutual information measures how much one modality reveals about the other, while infogain normalizes mutual information by text entropy and lies between 0 and 1.The measurements require finetuning on data with and without captions or images to avoid blank inputs being out of distribution.
  • Information Measures: Mutual information is estimated by subtracting losses with paired versus blank conditioning inputs.This is done separately for text-to-image and image-to-text models.
  • Scaling Results: Billion-parameter decoder-only Transformers extract about 8 nats of image information from an average test caption.The paper observes model-size trends for mutual information and infogain in both directions.
  • Scaling Results: For text-to-image models, reaching 20% of the text information would require approximately 3 trillion parameters, while the threshold remains far out of reach for image-to-text models.The paper cautions that extrapolating to infogain = 1 is unrealistic because infogain grows very slowly and captions are not always image-correlated.
  • Scaling Results: Text-to-image models have much larger mutual information and infogain than image-to-text models, which the authors speculate reflects the greater processing needed for image semantics.The comparison is empirical; the processing explanation is presented as a speculation.
  • Scaling Results: A 32x32 image is worth only about 2–3 words to the best image-to-text models under the paper’s mutual-information comparison.The estimate compares approximately 8 nats of image-to-text mutual information with 3.4 nats per word.

5 Mathematical Problem Solving and Extrapolation

Mathematical problem-solving performance extrapolates beyond training distributions with scaling trends, but model size matters primarily through the training-distribution loss it achieves.

  • Extrapolation Setup: Mathematical problems enable extrapolation tests by varying numerical ranges, operations, or recursive and compositional depth beyond training.The paper treats extrapolation as generalization to a larger or more diverse distribution.
  • Scaling Trends: Loss and accuracy show smooth power-law-plus-constant trends across mathematical difficulty levels, with difficulty-dependent exponents and offsets.The test sets are indexed by numerical levels corresponding to generation entropy.
  • Scaling Trends: Extrapolation performance improves with model size, but this improvement is mediated by better performance on the training distribution.Increasing size does not automatically improve extrapolation beyond its effect on training loss.
  • Scaling Trends: Models of different sizes that attain the same training-distribution loss perform about equally across the various test distributions.Thus interpolation and extrapolation performance depend on model size primarily through training loss.
  • Interpretation: The information-theoretic interpretation differs for these problems because answers are deterministically related to questions, so answer entropy should vanish.The paper notes that more detailed mathematical results appear in the appendix.

6 An Inconsistency in Compute and Datasize Scaling Laws

Power-law projections for compute and dataset-size scaling eventually conflict, revealing that at least one extrapolated trend must fail; optimization inefficiency is offered as a possible explanation.

  • Motivation: The section investigates an inconsistency between dataset-size and compute scaling laws using low-resolution image models.The authors expect the qualitative phenomenon to extend across the covered datasets.
  • Dataset-Size Scaling: Learning curves lie above the dataset-size trend because optimization fails to reach the minimum loss within a single epoch.With perfect optimization, the dataset-size trend would coincide with the learning curve under the stated assumptions.
  • Trend Projections: For 16x16 images, the compute-optimal model trend is Nopt(C) ≈ (2.8×10^8)C^0.74, implying C(D) ≈ (5×10^-42)D^3.9.Compute is measured in petaflop-days and dataset size in tokens.
  • Inconsistency: The extrapolated compute trend eventually intersects and falls below the dataset-size trend, so L(D), L(C), or Nopt(C) must break down by that point.The intersection is structurally inevitable when the trends become straight lines on a log plot.
  • Possible Resolution: The authors hypothesize that increasingly efficient optimization causes the compute frontier to merge with L(D), potentially changing the relative scaling of optimal model and dataset sizes.They explicitly state that the resolution of the inconsistency is uncertain.
  • Consistency Check: Irreducible-loss estimates are similar across dataset-size and compute trends: about 2013 versus 2023 nats/image at 16x16, and 599 versus 602 at 8x8.The authors present this agreement as a consistency check on the entropy estimates.

7 Related Work

Prior work established predictable power-law scaling across models, datasets, and architectures, while theoretical explanations and broader modality coverage remained limited.

  • Recent research examined scaling relations across many model architectures and datasets, with language modeling providing the closest precedent for this study.
  • Theoretical explanations for precise scaling relations remained scarce, including proposals linking scaling exponents to data-manifold dimension and model-size expansions.
  • Comparisons across studies require accounting for differing data distributions and context lengths, estimated here to introduce errors below about 2% of the loss.
  • Transformers had already been applied to video, multimodal, and mathematical problem-solving tasks, while autoregressive image modeling followed earlier PixelRNN-based work.

8 Discussion

The discussion argues that common scaling laws extend across modalities and reveal consequences for model sizing, information theory, and how capabilities should be understood.

  • A single Transformer architecture exhibits common loss scaling with model size and compute across images, videos, multimodal data, math, and language.
  • Nopt ∝ C^0.7 is approximately universal across data distributions, implying additional compute should favor larger models over substantially longer training.
  • Scaling laws motivate studying machine learning across broad hierarchies of model, data, and compute scales rather than focusing only on particular architectures or algorithms.
  • The loss decomposes into the true distribution’s entropy plus KL divergence from the model, enabling entropy identification through extrapolation of precise trends.

Contributions

The paper’s work spans image, video, multimodal, mathematical, and language-modeling experiments, with supporting analyses of data generation, figures, and implementation choices.

  • Contributors: The research involved dedicated contributions to image, video, multimodal, math, and codebase development, alongside dataset curation and research leadership.
  • Image modeling: Image experiments analyze compute trends across pixel resolutions and VQ-VAE encodings, individual-image losses, image improvement patterns, and conditional-completion quality.
  • Mathematical problem solving: Mathematical data were generated procedurally across sampled modules and entropy levels, with separate procedures and official files used for extrapolation evaluation.
  • Mathematical problem solving: Performance degrades smoothly as mathematical problems move beyond the training distribution.
  • Mathematical problem solving: Dataset-size experiments for math evaluate minimum training loss in a regime where N ≫ D, making overfitting rather than model size or compute the constraint.

B.3 Additional Math Results

Additional results examine mathematical performance by dataset size, module, difficulty, and interpolation or extrapolation setting, alongside multimodal mutual-information analyses.

  • Math performance: Math analyses report performance by module and difficulty level, including official interpolation and extrapolation test-set accuracies.
  • Dataset size: The math dataset-size study fixes nlayer = 64 and dmodel = 512 across dataset sizes.
  • Math difficulty: Levels ≤10 belong to the training distribution; performance shows a kink at level 10 but varies smoothly on more difficult extrapolated problems.
  • Math difficulty: Power-law trends for math model size are distorted on some in-distribution difficulty levels, possibly because of memorization and an implicit data curriculum.
  • Multimodal modeling: Mutual-information analyses measure empirical caption-image mutual information and Infogain, defined as mutual information divided by empirical text entropy.
  • Multimodal modeling: Learning curves mix data with and without captions or images to reduce confounding from blank or out-of-distribution inputs.

C Additional Multimodal Results

The multimodal experiments compare training strategies and track how mutual information changes with model size. The language-model arithmetic progression shows smooth improvement alongside qualitative shifts in question understanding.

  • Multimodal experiments: The two multimodal training strategies produce very similar final mutual information and infogain.The comparison covers training from scratch on captioned and blank-caption data and finetuning on mixed multimodal data.
  • Language experiments: Increasing GPT-3 parameter count smoothly improves the loss assigned to correct arithmetic answers.The arithmetic evaluation measures answer probabilities across model sizes.

D Additional Language Results

Language-model capabilities improve smoothly with parameter count, while qualitative learning phases emerge as models move from misunderstanding questions to answering them correctly.

  • Language results: Smooth improvement in correct-answer loss accompanies increasing parameter count across arithmetic and simple factual-question evaluations.The largest models correctly answer the evaluated questions, whereas smaller models struggle to understand them.
  • Language results: Larger GPT-3 models progressively shift probability toward correct answers for simple factual questions.The evaluation asks for the first and second presidents of the United States.
  • Information-theoretic measures: Empirical mutual information is bounded by the cross-entropy loss on either modeled variable, yielding an InfoGain bound below one.The mutual information is maximized for deterministic relations and minimized when the variables are independent.
  • Information-theoretic measures: Evaluations can apparently violate InfoGain < 1 when language models are not precisely translation invariant across token-position formulations.The paper gives a hash-sequence example in which contextual and standalone probabilities can differ.

E.1 Approximate Derivation of Scaling Relations

The derivation connects mutual-information scaling to token-position loss laws and interprets cross-entropy on ImageNet through entropy and KL divergence. The appendix also records implementation settings and a limitation on estimating the ImageNet–YFCC100M divergence.

  • E.1 Approximate Derivation of Scaling Relations: The language-model derivation estimates mutual information between the first T tokens and the following T tokens using token-position power-law loss scaling.The approximation assumes a position exponent p < 1 and uses the unigram entropy LU.
  • E.1 Approximate Derivation of Scaling Relations: The T-dependent harmonic factor is identical across model sizes, so the model-size dependence enters through L(N).For language models, αN ≪ 1 permits the approximation N^-αN ≈ 1 − αN log(N).
  • E.1 Approximate Derivation of Scaling Relations: InfoGain is bounded by a ratio determined by p and T and therefore does not actually approach 1.This follows because L(N) remains between zero and the unigram entropy LU.
  • E.2 Estimating DKL Between Real-World Distributions: Cross-entropy on ImageNet is interpreted as DKL(ImageNet||YFCC100M) + S(ImageNet) for models trained on YFCC100M.The evaluation tests whether loss on the new distribution follows a power-law plus constant trend.
  • E.2 Estimating DKL Between Real-World Distributions: The ImageNet–YFCC100M KL divergence cannot be identified from the observed loss trend alone because S(ImageNet) also requires estimation.The paper notes that dataset-size scaling might provide this estimate in future work.
  • Implementation details: Training uses a 3000-step linear warm-up followed by linear decay to one-tenth of the maximum learning rate.The implementation section also reports approximate non-embedding parameter counts and model-specific batch settings.
Loading 2010.14701v2…