PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Compute-Optimal Large Language Models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre

arXiv:2203.15556v1cs.CLcs.LG

TL;DR

Choosing model size and training-token count under a fixed compute budget matters because large-model training is costly. The authors develop three predictive approaches from over 400 training runs and test them with Chinchilla. Using Gopher’s compute budget, Chinchilla is 4 times smaller yet outperforms Gopher and even larger models on nearly every measured evaluation task.

  • Problem

    How should model size and training-token count be chosen under a fixed compute budget, given the substantial cost of training large language models?

    Source: §1 · pp. 1–2

  • Method

    The authors develop three predictors of compute-optimal model size and training duration from over 400 training runs, then test their predictions by training Chinchilla.

    Source: §5 · p. 15

  • Results

    Chinchilla outperforms Gopher and even larger models on nearly every measured evaluation task, using Gopher’s compute budget with a model 4 times smaller.

    Source: §1 · p. 2, §5 · p. 15

  • Takeaways & Limitations

    Chinchilla’s reduced model size considerably lowers inference cost and facilitates downstream use on smaller hardware.

    Source: §1 · p. 2

    Validation includes only two comparable large-scale training runs, Chinchilla and Gopher, with no additional tests at intermediate scales.

    Source: §5 · p. 15

Abstract

from arXiv · show

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

1. Introduction

Compute-optimal training scales model size and training tokens in equal proportions. At Gopher’s compute budget, Chinchilla outperforms Gopher with a four-times-smaller model trained on four times more tokens.

1. Introduction

  • Predetermined compute budgets and typically only one feasible large-model training run make accurate model-configuration selection critical.Training incurs substantial compute and energy costs that increase with model size.Source: §1 · p. 1
  • Model size and training-token counts should scale equally for compute-optimal training, rather than favoring parameter growth as recommended by Kaplan et al. (2020).For a 10× compute increase, Kaplan et al. (2020) recommend 5.5× larger models and 1.8× more tokens. Both analyses conclude that compute-optimal models need not reach their lowest possible loss.Source: §1 · p. 1
  • The optimization minimizes final pre-training loss L(N, D) over model parameters N and training tokens D, subject to FLOPs(N, D) = C.The analysis uses smoothed training loss as an unbiased estimate of test loss, assuming an infinite-data regime where training tokens are fewer than corpus tokens.Source: §1 · p. 2
  • Over 400 models, spanning under 70M to over 16B parameters and 5B to over 400B training tokens, underpin estimates of compute-optimal model sizes and token counts.Each model configuration was trained for several different training horizons.Source: §1 · p. 2
  • 70B-parameter Chinchilla, trained on 1.4 trillion tokens at Gopher’s compute budget, outperforms Gopher with a model four times smaller.It uses four times more training tokens, while its smaller size considerably reduces inference costs and facilitates downstream use on smaller hardware.Source: §1 · p. 2

2. Related Work

The section contrasts earlier scaling analyses with training schedules matched to token budgets and an analysis predicting similar growth in model size and training data. It also reviews data requirements, hyperparameter selection, and alternatives to dense transformers.

Large language models

  • Larger language models have improved task performance, but rising training and inference costs and the need for larger, high-quality datasets constrain further scaling.The landscape includes dense transformers and mixture-of-expert models; computational requirements increase with model size (Rae et al., 2021).Source: §2 · p. 3

Modelling the scaling behavior

  • Unlike Kaplan et al. (2020), this analysis predicts that model size and training data should scale at roughly the same rate.Matching learning-rate schedules to training-token counts improves final loss; intermediate losses from a fixed 130B-token schedule overestimate loss at smaller token budgets, contributing to Kaplan et al. (2020)'s different conclusion. The analysis also includes models up to 16B parameters to capture slight FLOP-loss frontier curvature, with most models above 500 million parameters rather than the predominantly smaller models in Kaplan et al. (2020).Source: §2 · p. 3
  • Clark et al. (2022) find diminishing scaling gains from additional experts as model size increases, but fixed training-token counts may underestimate branching improvements.Their loss model depends on model size and expert count, without varying the training-token budget.Source: §2 · pp. 3–4

Estimating hyperparameters for large models

  • The study focuses on model size and training steps, using prior research and experimental heuristics to select other hyperparameters.Relevant work addresses learning rate and batch size, weak dependence of optimal batch size on model size, and larger feasible batches (Shallue et al., 2018; Zhang et al., 2019). Models are slightly shallower than proposed by Levine et al. (2020) for better wall-clock performance on the authors' hardware.Source: §2 · p. 4

Improved model architectures

  • Conditional computation gives MoE models large effective sizes with relatively fewer training and inference FLOPs, although computational benefits diminish for very large models (Clark et al., 2022).Examples include Switch transformer (Fedus et al., 2021) and GLaM (Du et al., 2021).Source: §2 · p. 4
  • Approximately 10 times more data tokens are effectively seen during training with retrieval in Borgeaud et al. (2021), suggesting greater importance for training-data scale.Explicit retrieval augments transformers through an approach orthogonal to conditional computation.Source: §2 · p. 4

3. Estimating the optimal parameter/training tokens allocation

Three empirical approaches estimate how to allocate training compute between model size and training tokens, consistently favoring approximately equal scaling of both. Their projections suggest that many existing large language models should have been smaller and trained on more tokens.

3.1. Approach 1: Fix model sizes and vary number of training tokens

  • Power-law fits yield compute-optimal model size N_opt ∝ C^a and training-token count D_opt ∝ C^b, with exponents a = 0.50 and b = 0.50.For each parameter count, four training horizons span a 16-fold range; smoothed, interpolated losses identify optimal allocations at 1,500 logarithmically spaced FLOP values. At 10^21 FLOPs, the predicted model size has a clear advantage over that recommended by Kaplan et al. (2020).Source: §3.1 · p. 5

3.2. Approach 2: IsoFLOP profiles

  • 0.49 for a and 0.51 for b are Approach 2's fitted exponents for compute-optimal model size and training-token count, respectively.Unlike Approach 1's use of full training trajectories, IsoFLOP profiles compare final losses at fixed compute budgets. Parabolic fits locate each profile's loss-minimizing model size, followed by power-law fits across budgets.Source: §3.2 · pp. 5–6, Figure 3 · p. 6

3.4. Optimal model scaling

  • 40 billion parameters is Approach 3's projected optimum for Gopher's compute budget, reflecting its tendency to favor smaller models at larger budgets.Observations at C ≤ 10^21 have larger residuals, so Huber fitting downweights them and emphasizes higher-compute observations. Combined with negative curvature in the compute-to-optimal-size frontier, this produces smaller predicted models than the other approaches.Source: Figure 4 · p. 7, §3.4 · p. 7

3.3. Approach 3: Fitting a parametric loss function

  • 0.46 for a and 0.54 for b are Approach 3's fitted exponents for compute-optimal model size and training-token count, respectively.It models final losses from Approaches 1 and 2 using terms for text entropy, finite model capacity, and incomplete training. Minimizing this loss under FLOPs ≈ 6ND (Kaplan et al., 2020) balances the model-size and data-dependent terms.Source: §3.3 · pp. 6–7

3.4. Optimal model scaling

  • Three approaches independently predict that compute-optimal model size and training-token count should increase in approximately equal proportions as compute grows.Despite different fitting methods and training runs, their estimated scaling relationships are comparable.Source: §3 · p. 4, §3.4 · p. 7, Table 2 · p. 8
  • 175 billion parameters require an estimated 4.41 × 10^24 FLOPs and over 4.2 trillion training tokens to lie on the compute-optimal frontier.A 280 billion parameter Gopher-like model would require approximately 10^25 FLOPs and 6.8 trillion tokens. Despite substantial uncertainty in distant extrapolations, these estimates suggest existing models are oversized for their budgets and underscore the importance of collecting more training data.Source: Table 3 · p. 8, §3.4 · p. 8
  • Additional IsoFLOP analyses on C4 (Raffel et al., 2020a) and GitHub code (Rae et al., 2021) also support equal scaling of model size and training tokens.These experiments reproduce the scaling conclusion on two additional datasets.Source: §3.4 · p. 8

4. Chinchilla

Chinchilla trains 70B parameters on 1.4T tokens using the same FLOPs as Gopher, improving performance across language modelling and downstream evaluations. Its smaller size reduces memory and inference costs, but gender-bias improvements are uneven and unprompted toxicity remains largely unchanged.

4.1. Model and training details

  • 70B parameters trained on 1.4T tokens test compute-optimal scaling at Gopher’s FLOP budget; Chinchilla’s fourfold smaller size reduces memory footprint and inference cost.Training adjusts MassiveText sampling, replaces Adam (Kingma and Ba, 2014) with AdamW, removes NFKC normalisation from SentencePiece (Kudo and Richardson, 2018), and combines bfloat16 passes with float32 optimiser weights (Rajbhandari et al., 2020). AdamW surpasses Adam’s training performance only around 80% through the cosine cycle but finishes notably better; removing normalisation particularly helps mathematics and chemistry representation.Source: §4 · p. 9, §4.1 · p. 9

4.2.1. Language modelling

  • 7.16 perplexity on Wikitext103 (Merity et al., 2017) improves on Gopher’s 7.75; Chinchilla also beats Gopher across every Pile subset (Gao et al., 2020).Chinchilla outperforms Jurassic-1 (178B) Lieber et al. (2021) on all Pile subsets except dm_mathematics and ubuntu_irc. Training on four times more data could increase train/test leakage, so the authors emphasize evaluations less susceptible to leakage.Source: Figure 5 · p. 10, §4.2.1 · pp. 10–11

4.2.2. MMLU

  • 67.6% average 5-shot MMLU accuracy places Chinchilla 7.6% above Gopher despite its smaller size.Chinchilla exceeds the 63.4% June 2023 expert forecast and achieves greater than 90% accuracy on four tasks. It improves on Gopher on 51/57 tasks, ties on 2/57, and underperforms on 4/57.Source: Table 6 · p. 11, §4.2.2 · p. 11, Figure 6 · p. 12

4.2.3. Reading comprehension

  • 77.4% LAMBADA accuracy for Chinchilla exceeds Gopher’s 74.5% and MT-NLG 530B’s 76.6%.Chinchilla also improves accuracy over Gopher by more than 10% on both RACE-h and RACE-m (Lai et al., 2017). Different prompt formats prevent direct RACE comparisons with GPT-3 and MT-NLG 530B.Source: §4.2.3 · p. 11, Table 7 · p. 12

4.2.4. BIG-bench

  • 65.1% average BIG-bench accuracy versus Gopher’s 54.4% represents a reported 10.7% improvement, with Chinchilla underperforming on only four of 62 tasks.The four exceptions are crash_blossom, dark_humor_detection, mathematical_induction, and logical_args.Source: §4.2.4 · pp. 11–12, Figure 7 · p. 13

4.2.5. Common sense

  • Chinchilla matches or outperforms Gopher and GPT-3 across the evaluated common-sense benchmarks, outperforming MT-NLG 530B on all but one.On TruthfulQA (Lin et al., 2021), Chinchilla achieves 43.6%, 58.5%, and 66.7% accuracy at 0-shot, 5-shot, and 10-shot, respectively; Gopher achieves 29.5% 0-shot and 43.7% 10-shot. The authors interpret these gains as evidence that better modelling of pre-training data alone can substantially improve this benchmark.Source: §4.2.5 · p. 12, Table 8 · p. 13

4.2.6. Closed-book question answering

  • 31.5% 5-shot and 35.5% 64-shot accuracy establish new closed-book Natural Questions records for Chinchilla, versus Gopher’s 21% and 28%, respectively.Chinchilla substantially outperforms Gopher on both TriviaQA evaluation sets (Joshi et al., 2017) and outperforms GPT-3 on the unfiltered set. On filtered TriviaQA, it trails the open-book state of the art by only 7.9%.Source: §4.2.6 · p. 13, Table 9 · p. 14

4.2.7. Gender bias and toxicity

  • 3.2%, 8.3%, and 9.2% pronoun-resolution improvements over Gopher for male, female, and neutral pronouns show uneven zero-shot Winogender gains (Rudinger et al., 2018).Female stereotype-contradicting examples improve by 10%, but these gender-bias and toxicity evaluations are not comprehensive. Across 25,000 unprompted samples, toxicity differs negligibly: mean PerspectiveAPI scores are 0.087 for Chinchilla versus 0.081 for Gopher.Source: §4.2.7 · pp. 14–15, Table 10 · p. 15

5. Discussion & Conclusion · Appendix

Chinchilla outperforms Gopher at the same compute budget, supporting predictions favoring smaller models trained on more data. Limited validation and possible scaling-law curvature qualify these predictions, while dataset growth demands attention to quality, train–test overlap, and ethical risks.

5. Discussion & Conclusion

  • 70B-parameter Chinchilla outperforms Gopher and even larger models on nearly every measured evaluation task, while matching Gopher’s training compute budget.The comparison validates predictions that a smaller model trained on more data can perform better at the same compute budget.Source: §5 · pp. 15–16
  • Over 400 training runs support three predictive approaches that all identify Gopher as oversized and favor smaller models trained on more data.Most existing large models were trained on around 300 billion tokens despite increasing model sizes, motivating reconsideration of compute allocation.Source: §5 · p. 15
  • Two comparable large-scale training runs—Chinchilla and Gopher—and no additional intermediate-scale tests limit validation of the predictions.The cost of training large models restricts the available experimental evidence.Source: §5 · p. 15
  • Concavity at high compute budgets challenges the assumed power-law frontier and suggests that optimal large-model sizes may still be overestimated.The assumed frontier relates compute budget, model size, and the number of training tokens.Source: §5 · p. 15
  • Less than one epoch of data was used in every analysis run, leaving multiple-epoch training for future work.Source: §5 · p. 15
  • Larger datasets may help only when high-quality, and require responsible collection and train–test overlap checks for language modelling and downstream evaluation.The claim that dataset scaling benefits depend on high quality is explicitly speculative.Source: §5 · p. 16
  • Web-dataset growth increases the quantity, not necessarily frequency, of toxic, biased, and private material, making dataset introspection increasingly important.Chinchilla exhibits bias and toxicity but appears less affected than Gopher. Understanding the relationship between model performance and toxicity remains future work.Source: §5 · p. 16
  • The authors expect similar model-size–data trade-offs in other modalities and describe their methods as easy to reproduce for selecting model sizes and training steps.The demonstrated methodology applies to autoregressive language models; its extension to other modalities remains an expectation.Source: §5 · p. 16

A. Training dataset · B. Optimal cosine cycle length

Chinchilla and the scaling runs use a MassiveText mixture that repeats MassiveWeb and Wikipedia for more than one epoch. The main analysis calibrates cosine cycle length to the compute-limited training duration because overly long cycles produce suboptimally trained models.

A. Training dataset

  • Chinchilla and all scaling runs use the training dataset makeup documented in Table A1.The table identifies this dataset as MassiveText.Source: A. Training dataset · p. 22, Table A1 · p. 22
  • More than one epoch is used for both the MassiveWeb and Wikipedia subsets during training.These subsets are therefore reused rather than traversed only once.Source: A. Training dataset · p. 22
  • The MassiveText sampling distribution differs slightly from Rae et al. (2021), whose proportions appear parenthetically in Table A1.The supplied caption does not provide the individual sampling proportions.Source: Table A1 · p. 22
  • 1.4 trillion tokens defines the training horizon for Table A1’s epoch counts, alongside subset disk sizes, document counts, and sampling proportions.These fields describe the mixture’s composition and how often its subsets are used.Source: Table A1 · p. 22

B. Optimal cosine cycle length

  • 10× learning-rate decay is used with the cosine schedule, following Rae et al. (2021).The analysis treats cosine cycle length and its corresponding learning-rate drop as a key assumption.Source: B. Optimal cosine cycle length · p. 22
  • Setting the cosine cycle length too far beyond the target training duration produces suboptimally trained models.The main analysis therefore calibrates cycle length to the maximum number of training steps permitted by the FLOP budget.Source: B. Optimal cosine cycle length · p. 22

C. Consistency of scaling results across datasets · D. Details on the scaling analyses

C4 and GitHub show scaling behaviour very similar to MassiveText, suggesting dataset independence when training does not exceed one epoch. Learning-rate decay and cosine-cycle length also affect performance.

C. Consistency of scaling results across datasets

D.1. Approach 1: Fixing model sizes and varying training sequences … D.4. Small-scale comparison to Kaplan et al. (2020)

These appendices specify learning-rate schedules, a decomposed loss model, and compute-optimal parameter/token projections. At 10^21 FLOPs, Approach 1’s smaller model outperforms the model selected using Kaplan et al. (2020).

D.1. Approach 1: Fixing model sizes and varying training sequences

  • 25% cosine-cycle overshoot is the threshold beyond which performance noticeably degrades, supporting matching the learning-rate cycle to the training duration.Maximum learning rates range from 2 × 10^-4 for the smallest models to 1.25 × 10^-4 for the largest, with 10× cosine decay. Training curves use Gaussian smoothing with a 10-step window.Source: §D.1 · p. 24

D.2. Approach 3: Parametric fitting of the loss

  • Loss decomposes into irreducible Bayes risk, model-size-dependent functional approximation error, and stochastic approximation error from empirical-risk optimization and single-epoch training.Fitting uses LBFGS with a grid of initializations and Huber-loss δ = 10^-3. Larger δ overfits small-compute runs and predicts larger held-out runs poorly; smaller δ leaves predictions unchanged.Source: §D.2 · p. 25
  • N^-1/2 approximation scaling is expected for two-layer networks (Siegel and Xu, 2020); stochastic convergence is lower-bounded by D^-1/2.The stochastic convergence rate is expected to be dimension-free and depend on loss smoothness, motivating a data-dependent stochastic term (Bubeck, 2015).Source: §D.2 · p. 25
  • 1.69 (E), 406.4 (A), and 410.7 (B) are fitted constants; both parameter/data scaling exponents are below 0.5.The authors identify increasing these scaling exponents as a goal for future models and training approaches.Source: §D.2 · p. 25

D.3. Predicted compute optimal frontier for all three methods

  • All three approaches provide optimal parameter/token projections across FLOP budgets; Table A3 tabulates Approaches 2 and 3, while Figure A3 compares all three.The supplied captions identify the projections’ scope but do not provide numerical frontier values.Source: Table A3 · p. 26, Figure A3 · p. 26

D.4. Small-scale comparison to Kaplan et al. (2020)

  • 2.80-billion-parameter Approach 1 outperforms the 4.74-billion-parameter model selected using Kaplan et al. (2020) at 10^21 FLOPs.The predicted optimal sizes were 2.86 billion parameters for Approach 1 and 4.68 billion for Kaplan et al. (2020). The tested transformers shared the same depth-to-width ratio to reduce confounding.Source: Figure A4 · p. 27, §D.4 · p. 27

E. Curvature of the FLOP-loss frontier

The FLOP-minimal loss frontier curves as model size increases, making projections depend on the model sizes used. The analysis does not account for this curvature, which suggests that even smaller models may be optimal at large compute budgets.

E. Curvature of the FLOP-loss frontier

F. FLOPs computation

The paper includes embedding costs in training FLOPs and assumes the backward pass costs twice the forward pass. Differences from the common C = 6DN approximation (Kaplan et al., 2020) are very small and do not affect the analysis.

F. FLOPs computation

  • Embedding matrices count toward both training FLOPs and total parameters, although their contributions are small for large models.Multiply-accumulate operations are counted with a factor of 2.Source: F. FLOPs computation · p. 27
  • Attention FLOPs include key, query, and value projections, key-query logits, softmax, attention reductions, and the final linear projection.The logits, softmax, and reduction terms scale quadratically with sequence length.Source: F. FLOPs computation · pp. 27–28
  • 2 times the forward-pass FLOPs are assumed for the backward pass, following Kaplan et al. (2020).Total forward-pass FLOPs are embeddings + num_layers × (total_attention + dense_block) + logits.Source: F. FLOPs computation · p. 28
  • The detailed FLOP calculation differs very little from C = 6DN (Kaplan et al., 2020), without affecting the analysis.C denotes training FLOPs, D the number of training tokens, and N the parameter count; Table A4 compares the calculations across model sizes.Source: Table A4 · p. 28, F. FLOPs computation · p. 28
  • 6.3 × 10^23 rather than 5.76 × 10^23 FLOPs is reported using a slightly more accurate calculation than Rae et al. (2021).Source: F. FLOPs computation · p. 28

G. Other differences between Chinchilla and Gopher … H.4. BIG-bench

Chinchilla outperforms Gopher on every Pile subset, while optimiser comparisons favor AdamW and Chinchilla’s training setup. Additional material reports MMLU and BIG-bench subset evaluations and describes the Winogender coreference protocol.

G. Other differences between Chinchilla and Gopher

H.1. The Pile

H.2. MMLU

  • Table A6 reports Chinchilla and Gopher performance on each MMLU subset.The supplied passage does not provide subset scores or comparisons.Source: §H.2 · p. 30

H.3. Winogender Setup

  • Winogender tests coreference by comparing sentence-role completion probabilities for female, male, and gender-neutral pronouns.Examples specify the correct referent; an unbiased model would identify it correctly regardless of pronoun gender.Source: §H.3 · p. 30

H.4. BIG-bench

  • Table A7 reports Chinchilla and Gopher performance on each considered BIG-bench subset.The supplied passage does not provide subset scores or comparisons.Source: §H.4 · p. 30

I. Model Card · J. List of trained models

Chinchilla’s model card documents restricted research use, broad evaluation without repeated training runs, and toxicity, bias, and language-related limitations. The accompanying inventory records model sizes and hyperparameters, including variants trained with different schedules or token counts.

I. Model Card

  • Chinchilla is intended for DeepMind researchers, will not be publicly released, and requires further safety, fairness, and application-specific analysis before downstream use.Harmful or deceitful language generation is explicitly out of scope.Source: §I · p. 32
  • Chinchilla is trained on English data; analysis of models trained on the same corpus in Rae et al. (2021) found unequal performance across some dialects.It uses the dataset from Rae et al. (2021), with sampling differences documented in Table A1. Preprocessing uses a 32,000-token SentencePiece vocabulary and, unlike Gopher, omits NFKC normalization.Source: §I · pp. 32–33
  • Training costs prevented repeated Chinchilla runs; broad task evaluation estimates overall performance, while Gopher, trained on the same dataset, provides a comparison.The principal comparison with Gopher concerns text likelihood prediction.Source: §I · p. 32
  • Evaluation spans language modelling, knowledge, reasoning, question answering, reading comprehension, commonsense understanding, toxicity, and gender bias.Measures include perplexity, bits per byte, task accuracy, exact-match accuracy, generation toxicity, and toxicity-classification accuracy. Gender-bias tests examine gender-term probabilities and coreference resolution.Source: §I · pp. 32–33
  • Chinchilla can generate toxic language, reproduce gender stereotypes, and produce differing sentiment toward racial, religious, or national groups.Toxic outputs are particularly likely following toxic prompts. The authors suggest that group sentiment differences likely reflect the training text.Source: §I · p. 33

J. List of trained models

  • Table A9 inventories all trained models’ sizes and hyperparameters, with many configurations trained using multiple learning-rate schedules or training-token counts.Many models were trained multiple times with different numbers of training steps.Source: J. List of trained models · p. 34, Table A9 · p. 36

Related papers

Loading 2203.15556v1…