PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, Geoffrey Irving

arXiv:2112.11446v2cs.CLcs.AI

TL;DR

Language modelling offers a route toward intelligent communication, but evidence is needed on how scaling Transformer models affects diverse capabilities and behaviour. This paper trains and analyzes models from tens of millions to 280 billion parameters across broad benchmarks, finding strong but uneven gains and examining toxicity, bias, and safety. The results show broad improvements over existing language-model state of the art alongside limited benefits for some mathematical and logical reasoning tasks.

  • Problem

    The paper examines how Transformer language-model scale relates to performance across diverse capabilities and to model toxicity and bias.

  • Method

    The authors train and compare Transformer language models across scales, evaluate them on 152 tasks, and analyze dataset quality, toxicity, bias, dialogue behaviour, and safety.

  • Results

    Gopher outperforms current language-model state of the art on roughly 81% of comparable tasks, while larger scale benefits some reasoning categories less and changes toxicity-related behaviour.

  • Takeaways & Limitations

    Scale and data quality yield broad but nonuniform performance advances, while mathematical and logical reasoning may require more than scale alone.

  • Takeaways & Limitations

    Automatic toxicity and bias classifiers can themselves encode social biases, limiting reliance on their evaluations.

Abstract

from arXiv · show

Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.

1. Introduction

Language modelling uses prediction over abundant text to study intelligence-related capabilities, and this paper tests how Transformer model scale affects broad task performance and model behaviour. Gopher improves over existing language-model results on most comparable tasks, while scale also changes toxicity and bias-related properties.

  • Autoregressive language modelling predicts future text from past context, providing a general objective with access to internet, books, articles, code, and other writing.The objective approximates specific applications because it predicts all sequence aspects rather than only task-relevant aspects.
  • The paper presents Gopher, a 280 billion parameter Transformer, and evaluates language-model performance across 152 diverse tasks.The study describes training methods, infrastructure, and the MassiveText dataset alongside benchmark analysis.
  • 81% of comparable tasks show Gopher outperforming current state-of-the-art language models, notably in fact checking and general knowledge.The reported comparison covers 124 tasks with comparable results and identifies knowledge-intensive domains as especially strong.
  • Scaling increases toxic responses to toxic prompts, while larger models more accurately classify toxicity.The paper examines toxicity and bias because harmful content appears in training data and downstream applications.
  • The paper discusses ethical deployment, mitigation of undesirable behaviour before and after training, and applications to AI safety.It also considers how language models might accelerate research toward safer intelligent technology.

2. Background

The paper frames language modelling as probability modelling over tokenized text, usually trained autoregressively with Transformers. It situates this approach within a progression of increasingly large neural language models and complementary instruction-tuning and sparse-model developments.

  • Language modelling represents the probability of text by tokenizing sentences, paragraphs, or documents into sequences of vocabulary tokens.Tokenization may be open-vocabulary, such as byte-level modelling, or closed-vocabulary, such as word lists.
  • Autoregressive sequence modelling factorizes token-sequence probability so each future token is predicted from preceding context.The paper focuses on this objective because of its strong performance and simplicity.
  • Transformers are the architecture studied because they have demonstrated state-of-the-art language-model performance.The paper connects their progress to scaling training data, model parameters, and computation.
  • Language-model development progressed from BERT at 345 million parameters through GPT-2, Megatron, T5, and GPT-3 at 175 billion parameters.These models successively supported stronger classification, zero-shot modelling, transfer learning, and closed-book question answering performance.
  • Recent alternatives include sparse mixture-of-experts models and instruction-tuned FLAN and T0 models, whose ideas are complementary to building a powerful base language model.The paper compares these systems where possible despite their different training approaches.

3. Method

The Gopher family comprises six autoregressive Transformer models spanning 44 million to 280 billion parameters, trained on curated, diverse text with scale-aware optimization and distributed infrastructure. The method combines data-quality filtering, non-uniform dataset sampling, and parallelism strategies designed for TPU training.

  • The Gopher family contains six Transformer language models ranging from 44 million to 280 billion parameters, with the largest named Gopher.
  • The models use an autoregressive Transformer with RMSNorm and relative positional encodings, enabling evaluation on longer sequences than used during training.
  • All models are trained for 300 billion tokens with a 2048-token context window using Adam, warm-up, cosine decay, scale-dependent learning rates, and gradient clipping.For the 7.1B model and Gopher, the clipping value is reduced to 0.25 for improved stability.
  • Training uses JAX and Haiku with data and model parallelism on TPUv3 chips, while optimizer-state partitioning, rematerialization, and model parallelism address Gopher’s memory demands.Gopher’s half-precision parameters and single-precision Adam state occupy 2.5 TiB, exceeding the memory of one TPUv3 core.
  • Within TPU pods, data and model parallelism incur only 10% overhead for Gopher; pipelining is used across pods rather than within smaller TPU configurations.
  • MassiveText combines web pages, books, news articles, and code, then applies quality filtering, repetition removal, deduplication, and test-set-overlap removal.The dataset contains about 2.35 billion documents and 10.5 TB of text, with training sampling tuned across subsets.

4. Results

Gopher is evaluated against prior language-model, supervised, and human results across 152 diverse tasks. Its gains are broad but uneven: knowledge-intensive tasks benefit most from scale, while several reasoning-heavy areas show smaller improvements or regressions.

  • Comparisons with State of the Art: Gopher outperforms current language-model state of the art on 100 of 124 comparable tasks, or 81%.Baselines include GPT-3, Jurassic-1, and Megatron-Turing NLG, with the exact baseline varying by task.
  • Comparisons with State of the Art: Gopher improves on 11 of 19 language-modelling datasets, with particularly strong gains on books and articles.It underperforms on 8 tasks, including Ubuntu IRC and DM Mathematics; the authors suggest tokenizer representation for numbers may contribute.
  • Comparisons with State of the Art: RACE accuracy rises from 47.9% to 71.6% for high-school comprehension and from 58.1% to 75.1% for middle-school comprehension.The high-school result approaches human-rater performance, but remains below supervised state of the art and the estimated human ceiling.
  • Comparisons with State of the Art: Scale improves fact checking with claims alone and with gold evidence, mainly by separating SUPPORTED from REFUTED rather than REFUTED from NOTENOUGHINFO.The authors interpret this pattern as improved factual knowledge rather than deeper understanding of misinformation.
  • Comparisons with State of the Art: Gopher achieves 60% average accuracy across MMLU’s 57 tasks, exceeding GPT-3’s 43.9% and UnifiedQA’s 48.9%.The table-level comparison still places Gopher far below human-expert performance.
  • Performance Improvements with Scale: Across the 152-task scaling analysis, 79 tasks show relative gains above 25%, while 16 show zero or no gains.The largest benefits occur in medicine, science, technology, social sciences, and humanities; mathematics, logical reasoning, and common sense benefit less, with some smaller-model wins.

5. Toxicity and Bias Analysis

The analysis examines how model scale relates to toxicity, toxicity detection, and several forms of bias. Larger models better match toxic prompts and classify toxicity, while scale produces inconsistent or context-sensitive bias patterns.

  • Toxicity generation: Larger models produce continuations whose toxicity more closely tracks prompt toxicity, with the effect plateauing near 7.1B parameters.When input toxicity increases, larger models respond with greater toxicity.
  • Toxicity generation: Unprompted toxicity remains low and does not increase with model size, staying slightly below toxicity levels in the training data.The evaluation finds no amplification of training-data toxicity for unprompted samples.
  • Toxicity detection: The largest model reaches an AUC of around 0.76 in the 20-shot setting for toxicity classification, outperforming smaller models but remaining below specialized classifiers.The state of the art for few-shot toxicity detection is not well established.
  • Bias: Few-shot toxicity classification with the 280B model is prone to subgroup bias, so higher classification accuracy does not guarantee fair outcomes.The analysis finds that the model exhibits bias against subgroups in different ways.
  • Bias: Gender-bias measurements show no consistent correlation with model size and vary when templates or gender terms change.Changing “was” to “is,” or using only “male” and “female,” substantially alters measured bias.
  • Bias: Models show disparate ability to model dialects, which could create allocational harms for users with different dialects.The analysis measures perplexity on African American-aligned and White-aligned Twitter corpora.

6. Dialogue

The paper probes Gopher through dialogue prompting, finding conversational successes alongside factual unreliability. Prompting and dialogue fine-tuning receive similar human preference, while dialogue prompting changes the observed toxicity-scaling pattern.

  • Dialogue prompting: Dialogue-Prompted Gopher can emulate a conversational format at a decent quality through conditional sampling from a dialogue prompt.The paper presents example transcripts to illustrate capabilities and limitations.
  • Dialogue prompting: Naively prompted Gopher produces first-person narratives, blog-like text, and generic existential-question lists consistent with its training distribution.These responses reflect that language models reproduce their input distribution rather than being trained specifically for conversation.
  • Dialogue capabilities: Dialogue prompting can keep Gopher on topic and elicit technical discussion and a correct citation, but responses can mix factual and non-factual content.The paper describes this mixture as a source of subtle misinformation.
  • Dialogue limitations: Prompting conditions the model’s response prior but does not produce a consistently reliable or factual dialogue model.The paper reports both successes and failures as common.
  • Dialogue training: Human raters showed no significant preference between Dialogue-Tuned and Dialogue-Prompted Gopher: preference was (50 ± 0.04)% from 1400 ratings.The authors describe this as an initial result requiring more rigorous comparison of fine-tuning and prompting.
  • Dialogue safety: Dialogue-Prompted Gopher does not follow the toxicity increase with scale seen for Gopher, with toxicity tending to slightly decrease as scale grows except for the most toxic prompts.The reported decrease begins from the 117M-parameter model.

7. Discussion

The discussion argues that scaling improves many capabilities but leaves important safety, measurement, efficiency, and deployment challenges. It emphasizes careful evaluation and downstream mitigation while acknowledging that mitigation choices remain empirical.

  • Efficiency: Scaling further requires more energy and compute for larger Transformers or more efficient architectures.The paper presents scaling as a path that faces increasing computational demands.
  • Efficiency: Sparse-parameter training did not yet yield an overall efficiency boost, while retrieval reached GPT-3-level performance with a 7B model and over a 10× reduction in training compute.The paper suggests Transformer scaling may be a transitory stage as more efficient architectures develop.
  • Evaluation challenges: Toxicity and bias metrics have important limitations because classifiers can encode social bias and automatic evaluation can introduce unintended biases.The Perspective API is capable but may assign higher toxicity to innocuous mentions of particular identity groups.
  • Evaluation challenges: Distributional-bias evaluation is brittle: changing a template verb can alter observed trends, while naturalistic high-quality datasets are difficult to collect.The paper therefore treats measurement design as an open challenge.
  • Context: Toxic outputs may help toxicity-detection applications but be problematic in other applications, and equalizing sentiment could erase historical or political context.The discussion frames desired behavior as application- and context-dependent.
  • Mitigation: The paper does not explore mitigation strategies, emphasizing instead that robust metrics and clear criteria are needed for effective mitigation.It argues that reliable analysis tools are as important as mitigation methods.
  • Safety strategy: Many harms may be better addressed downstream through fine-tuning, monitoring, stakeholder engagement, controlled release, and application-specific guidelines.Downstream interventions can support faster correction than infrequent, expensive pre-training.
  • Safety strategy: Claims that a harm is best mitigated downstream remain empirical, because failed downstream mitigation can lock mistakes in until retraining.The paper assigns responsibility for ensuring mitigations occur in deployed applications.

8. Conclusion

The paper finds that data quality and scale improve language-model performance, but gains are uneven: mathematical and logical reasoning benefit little up to Gopher’s scale. It connects this pattern to possible limits of the language-modelling objective and calls for stronger analysis and interpretability tools.

  • Data quality and scale yield performance advances over existing work, but benefits are nonuniform across tasks.Tasks requiring more complex mathematical or logical reasoning show little benefit up to Gopher’s scale.
  • The authors suggest that language modelling may compress associative facts more easily than mathematics, potentially limiting reasoning gains.They also leave open the possibility that new reasoning capabilities could emerge beyond the scale reached here.
  • The paper advocates broad analysis and interpretability tools to understand model behaviour and fairness, mitigate harms, and inform alignment efforts.

10. Contributions

The paper combines a large collaborative effort spanning model development, dataset construction, infrastructure, evaluation, analysis, and safety documentation. Its contributions include a curated and processed training corpus together with broad methodological and behavioural analyses.

  • The project documents contributions across model architecture, optimisation, infrastructure, dataset development, serving, fine-tuning, evaluation, and model analysis.
  • MassiveText is developed from diverse sources including web pages, books, news articles, and code, with documented collection and processing choices.The paper includes a datasheet for MassiveText and discusses filtering decisions intended to preserve coverage across dialects and sociolects.
  • The data pipeline applies text quality filtering, repetition removal, document deduplication, and test-set-overlap filtering to training subsets.MassiveWeb additionally receives custom text extraction and extra explicit-content filtering.

A.2. Dataset Analysis

The dataset analysis characterises MassiveText by document length, toxicity, language, domains, tokenizer compression, and sampling choices. Ablations show that preprocessing and subset weighting affect downstream performance, while the analysis also records tokenizer and evaluation limitations.

  • Dataset statistics: 120,000 tokens is the average Books document, whereas most MassiveWeb, C4, News, and Wikipedia documents average fewer than 1,000 tokens.The longest book exceeds 1.3M tokens, and GitHub documents average 2,946 tokens.
  • Dataset statistics: 0.8% of sampled texts exceed a toxicity score of 0.5, compared with 4.3% reported for GPT-2 training data.Across subsets, mean and median toxicity scores are 0.10 and 0.07, respectively, with a 95th-percentile score of 0.3.
  • Dataset statistics: 99% of MassiveText is English, while the remaining text is mostly Hindi followed by European languages.Chinese and Japanese account for 5% and 4% of the non-English tokens, respectively.
  • Dataset statistics: Four of the six highest-token-count MassiveWeb domains are academic or scientific despite no explicit scientific-content bias.The top domains also include presentation, question-answering, and social-media websites.
  • Sampling and preprocessing ablations: The 10% C4, 50% MassiveWeb, 30% Books, and 10% News weighting performs well across tasks and best on Curation Corpus.These weights are selected for the main Gopher training experiments.
  • Sampling and preprocessing ablations: Successive MassiveWeb preprocessing stages improve downstream performance, although the effect of deduplication may be underestimated in the subsampled ablation.The ablation trains 1.4B-parameter models on 5GB subsamples and uses validation loss on three downstream tasks as a quality proxy.
  • Tokenizer analysis: The 32,000-tokenizer’s NFKC normalization is not fully lossless and can reduce model expressivity while changing evaluation and test datasets.The authors plan to use lossless normalization forms in future work.

B. Gopher Model Card

The model card defines Gopher as a Transformer language model intended primarily for research and evaluated across language modelling, understanding, reasoning, safety, and bias measures. It warns that downstream use requires further safety and fairness analysis, while large-scale training has optimisation and evaluation constraints.

  • Model and intended use: Gopher is a Transformer language model whose primary intended uses include NLP research, AGI-related investigation, fairness and safety research, and studying LLM limitations.
  • Model and intended use: The model card places harmful or deceitful language generation out of scope and advises against downstream applications without further safety and fairness mitigations.
  • Evaluation: Evaluation covers language modelling, completion, reading comprehension, MMLU, BIG-bench, fact checking, question answering, toxicity, gender and occupation bias, dialect performance, and sentiment bias.Some evaluations rely on classifiers known to include biases.
  • Evaluation: The evaluation principally measures text prediction likelihood rather than long-range generation, treating generation quality as dependent on both the language model and decoding approach.
  • Limitations: Because training costs prevent multiple Gopher runs, broad evaluation across task types provides a reasonable estimate of overall performance rather than repeated-run variability.
  • Training considerations: Adafactor reduces memory usage but becomes less stable and lower-performing than Adam at large scales, with minor loss divergences observed for a 7.1B-parameter model.

C.2. Lower-Precision Training with bfloat16

Gopher’s lower-precision training exposed performance risks when parameters were stored and updated directly in bfloat16. Keeping a float32 parameter copy for optimizer updates matched full-precision performance while reducing memory and improving speed.

  • Training configuration: A float32 parameter copy in the partitioned optimizer state matches full float32 training performance with lower memory use and faster updates.The copy is used for optimizer updates, then cast to bfloat16 for the forward pass.
  • Training configuration: The tested configurations varied whether parameters and activations used fp32 or bfloat16 and whether a float32 optimizer-state copy was retained.Randomized rounding was tested for bfloat16 parameter updates in some configurations.
  • Results: 1.4× speed improvement was achieved with bfloat16 parameters and a float32 optimizer-state copy, while performance remained indistinguishable from the fp32 baseline.This configuration also offers a reduced memory footprint.
  • Results: bfloat16 parameters without random rounding performed worst among the tested methods.The comparison used fp32 everywhere as the precision baseline.

D.8.3. BIG-bench 5-Shot Results

BIG-bench results show that scaling improves many task categories, with especially large gains in alignment, social bias, memorization, and numerical response, while mathematical and multi-step reasoning benefit little. Gopher also performs strongly in comparisons involving T0 models, TruthfulQA, reading comprehension, and fact-checking, though gains vary by task and setup.

  • Overall BIG-bench performance: 41/51 task categories show a step change between the 7.1B model and Gopher, while Gopher outperforms other models on 41 evaluated tasks.The category averages include between 1 and 49 tasks, and tasks may belong to multiple categories.
  • Category-level scaling: Mathematical reasoning, multi-step decomposition, paraphrasing, summarization, and negation remain difficult across model scales.For mathematical and multi-step categories, scale appears to make little difference; some language tasks are hard regardless of scale.
  • Category-level scaling: The largest improvements occur on Alignment, Social Bias, Memorization, and Numerical Response tasks.These gains are associated with understanding implicit human preferences and recalling factual or numerical information.
  • Comparisons with T0: Gopher achieves the best overall zero-shot performance among the evaluated Gopher and T0 family models, but trails T0 models on Novel Concepts.Gopher leads on Hindu Knowledge and Known Unknown, is similar to T0++ on Misconceptions, and performs worse than all but T0 on Novel Concepts.
  • TruthfulQA: 10-shot prompting raises TruthfulQA performance from 29.5% to 43.7% for Gopher, while few-shot evaluation is not consistently beneficial for smaller models.The result is consistent with few-shot learning emerging at particular model scales for different tasks.
  • Reading comprehension: 71.6% accuracy on RACE-h exceeds GPT-3’s 46.8% and Megatron-Turing’s 47.9%, but remains below ALBERT-XXL’s 90.5%.The paper notes a substantial remaining gap from the estimated 94.2% human-accuracy ceiling.
  • Fact-checking: Gopher reaches 89% macro-F1 on the reported binary few-shot fact-checking comparison, an 18% absolute improvement over the cited 1.5B GPT-2 result.The comparison combines REFUTED and NOTENOUGHINFO into one class, unlike the three-way setup elsewhere.

D.13. Common Sense: PIQA, WinoGrande, SocialIQA, HellaSwag

Common-sense performance generally increases with model size, but the largest models perform similarly on several benchmarks and remain well below fine-tuned and human performance. Social common sense is especially challenging, while evaluation uses zero-shot multiple-choice scoring across physical, temporal, and social knowledge.

  • Evaluation setup: Gopher-family models evaluate physical, temporal, and social common sense through zero-shot multiple-choice benchmarks.The model assigns probabilities to answer choices conditioned on the context and question, selecting the highest-scoring response.
  • Scaling results: HellaSwag and PIQA differ by less than 1.5% across Gopher, GPT-3, and Megatron-Turing, suggesting limited gains beyond current largest models.The paper qualifies that further investigation is needed before firmly establishing this conclusion.
  • Benchmark comparisons: Gopher outperforms GPT-3 on PIQA and is nearly on par with Megatron-Turing, while its advantage over GPT-3 is smaller on HellaSwag and WinoGrande.These comparisons indicate that model size does not produce uniform differences across benchmarks.
  • Remaining gaps: Zero-shot large-language-model performance remains substantially below fine-tuned state-of-the-art and human performance across common-sense datasets.The paper identifies a large remaining opportunity for improvement and notes the difficulty of acquiring broad common-sense knowledge without benchmark-specific fine-tuning.
  • Scaling results: Common-sense reasoning increases with model size, but the performance gap among Gopher, GPT-3, and Megatron-Turing is small.The compared models span 175B to 530B parameters, a roughly 3× size difference.
  • Remaining gaps: 50.6% accuracy on SocialIQA leaves Gopher far behind fine-tuned state-of-the-art performance, indicating that social common sense is especially difficult.The paper recommends evaluating SocialIQA alongside HellaSwag, PIQA, and WinoGrande.

E.1.2. Results

Larger models respond more faithfully to toxic prompts, while unprompted generations do not amplify training-data toxicity. Toxicity classification improves with scale, but bias evaluations remain sensitive to templates and demographic term choices.

  • Toxicity generation: Larger models produce continuations whose toxicity tracks prompt toxicity more closely, with the trend plateauing near 7.1B parameters.Continuation toxicity remains consistently below prompt toxicity.
  • Aggregate toxicity metrics: As models scale, unprompted expected maximum toxicity and toxicity probability decrease, whereas prompted metrics show no clear scaling trend.These metrics estimate the largest toxicity across 25 samples and the probability of at least one toxic continuation, respectively.
  • Dataset effects: The C4-trained model records lower toxicity than MassiveText-trained models, while GPT-2 records the highest scores, indicating dataset construction affects toxicity substantially.The authors suggest training-data toxic content may explain part of the difference.
  • Toxicity generation: Unprompted toxicity is low, decreases slightly relative to training data, and does not increase with model size.For the 280B model, mean toxicity is 0.08 versus 0.1 for the training distribution.
  • Distributional bias: Gender-bias conclusions vary with prompt wording and gender-term selection, and no consistent correlation with model size appears across these choices.Using “was” versus “is” changes the observed trend, while using only “male” and “female” yields substantially lower bias.
  • Distributional bias: Using one gender-and-occupation template, 34.2% of occupations are biased toward women and the remainder toward men, with average biases of 0.52 and -0.74 respectively.The most male-associated occupations include carpenter and sheriff; the most female-associated include dietitian and receptionist.
  • Distributional bias: Bias analyses depend strongly on demographic term and comparison-class choices, so swapping terms without considering markedness can change conclusions.The paper notes that majority attributes may be left unspecified in language.

F. Compute Usage

The paper measures practical compute across training, evaluation, fine-tuning, and emissions, finding substantial costs and several opportunities for efficiency improvements. Fine-tuning the entire model generally gives the best performance for a compute budget, but resource and overfitting trade-offs remain.

  • Compute accounting: Reported FLOPs include rematerialisation, padding, and repeated computation, and represent a best-effort lower bound excluding several inefficiency sources.Rematerialisation increases compute by 33%.
  • Inference cost: Removing repeated processing of common prefixes could reduce inference FLOPs by 4-100×, depending on the evaluation.The paper identifies more efficient evaluations as important for future deployment.
  • Training time: Attention consumes 39% of accelerator time for the 417M model but 8% for Gopher, with over 70% of attention time spent on relative positional encodings.Large batch sizes also improve efficiency at large scales by reducing pipelining and data-parallelism costs.
  • Environmental cost: Training Gopher emitted an estimated 380 net tCO2e after 920 hours, compared with 552 net tCO2e for GPT-3.The estimate uses Google’s Georgia datacentre measurements and an estimated 283W per chip.
  • Efficient training: The paper reports modest success in compression and concludes that none of the investigated efficiency methods has entirely satisfactory results.The methods include distillation, pruning, sparse training, and reverse distillation.
  • Fine-tuning: Fine-tuning the entire model with an appropriate learning rate gave the best performance for a given compute budget, while smaller tuning scopes trade performance for lower cost and memory.Bias-only tuning uses 66% and final-40%-layer tuning 60% of full-model fine-tuning FLOPs.
  • Fine-tuning: Wikitext103 and Curation Corpus caused overfitting, whereas Python GitHub did not overfit after more than 4.5 million sequences.The Python GitHub experiments were sometimes stopped early because the study emphasized trends over exact performance numbers.
  • Fine-tuning: The learning rate balances performance, compute, and tuning method, with optimal rates decreasing as the number of trained parameters increases when overfitting occurs.For the largest models, minor improvements can require substantially more FLOPs.

G.2.1. Distillation

The paper evaluates distillation, pruning, sparse training, and model expansion as routes to cheaper language-model training or inference. Compression improves over similarly sized models trained from scratch, but substantial teacher gaps and limited efficiency gains remain.

  • Distillation: A 7.1B-to-1.4B distillation student reaches 28 C4 perplexity versus 30 for a same-sized model trained from scratch, but remains behind the teacher at 22.A 1.4B-to-785M distillation also preserves a substantial teacher gap.
  • Distillation: The teacher-to-student size ratio affects distillation efficacy: a 1.4B teacher reduces C4 evaluation loss by 2.7% more than a 7.1B teacher for a 417M student.The distilled model still remains nearly 20% worse than the 1.4B teacher in perplexity.
  • Distillation: Logit and attention distillation slightly outperform vanilla distillation, but the optimal schedule depends strongly on student and teacher sizes and no general recipe emerges.Matching model depths constrains the compression setup.
  • Pruning: Models at different sparsity levels have approximately the same scaling coefficient, while higher sparsity lowers the log-log intercept; 90% sparsity needs about 2.5× fewer parameters for a given evaluation.Increasing sparsity can also increase in-training loss.
  • Pruning: Pruning achieves only about 20-30% compression without an appreciable accuracy drop, and unstructured sparsity is difficult to exploit efficiently on most accelerators.A 2.5× parameter reduction may not offset sparse-computation inefficiency on GPUs.
  • Sparse training: Dynamic sparse training with RigL yields minimal reduction in FLOPs needed to reach a particular performance in large language models.This contrasts with reported benefits in some computer-vision settings.
  • Model expansion: Depth expansion can warm-start a 9B model from a 4.5B model to comparable from-scratch performance with a 40% compute reduction.A 417M model was also expanded by more than 3× while maintaining performance above an equivalent fresh model.
  • Overall compression: Across methods, compressed models underperform models trained from scratch, and the paper characterizes unconditional autoregressive-model compression as especially challenging.The authors note that computer-vision compression results do not transfer well to large-scale language modelling.

H. Dialogue-Prompted Gopher Details

Dialogue-Prompted Gopher uses a hand-authored conversational prompt and filtered dialogue data to structure interaction, then is compared with a supervised dialogue-tuned model. Human preferences do not differ statistically, while transcripts illustrate both useful contextual behavior and recurring failures.

  • H.1. Construction: The Dialogue-Prompted Gopher model is built from raw Gopher using a conversational prompt and a uniform input-output template.The prompt uses alternating User and Gopher turns.
  • H.1. Construction: Responses are sampled after appending “Gopher: ” to the dialogue history and are truncated when Gopher begins a new User turn or reaches the maximum length.The construction uses nucleus sampling with p=0.8.
  • H.1. Construction: The hand-authored prompt consumes roughly 800 of Gopher’s 2048-token context, leaving room for subsequent dialogue.The prompt conditions the model toward desirable responses for both parties.
  • H.2. Dialogue Dataset Filtering: The dialogue dataset filters consecutive paragraph blocks into strictly alternating two-person exchanges using repeated, distinct speaker prefixes.Blocks must contain at least six paragraphs, and the procedure is reported to yield high-quality dialogue.
  • H.3. Comparison Methodology: Participants compare Dialogue-Prompted and Dialogue-Tuned Gopher moves in ongoing dialogues, while each dialogue proceeds independently of the participant’s selection.This design is called the move selector.
  • H.3. Comparison Methodology: There is no statistical difference in human preference between prompted and tuned models under either move-selector setting.The prompted model theoretically has an advantage when the selector always chooses prompted responses.
  • H.4. RTP in a Dialogue Setting: The dialogue toxicity evaluation samples 500 questions from each of four RTP toxicity buckets and generates 25 continuations per question, up to 100 tokens.This produces 2,000 questions evaluated with Perspective API scores.
  • H.5. Selected Transcripts: Selected transcripts show factual recall mixed with misinformation, toxic or harmful outputs, nonsensical responses, reasoning failures, refusals, useful pointers, and an illusion of creativity.One semi-factual dialogue is mostly correct but includes a misleading response despite technically correct surrounding answers.
Loading 2112.11446v2…