Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Motivation

• Because natural language expresses many intelligence tasks as language input and output, language communication provides a general interface for intelligent systems.The paper concludes by discussing application-driven safety and mitigating undesirable behaviour before and after training.Source: §1 · pp. 3–4
• Autoregressive language modelling predicts future text from its past, supports many cognitive tasks, and exploits abundant textual data, but only approximates specific goals.The authors therefore argue that such models should be treated with appropriate caution.Source: §1 · p. 3

Related work

• Neural language-model progress has been driven by scale and architecture, with GPT-3 demonstrating strong generation and few-shot capabilities across many NLP tasks.Power laws relating cross-entropy loss to model size were reported for recurrent and Transformer language models; GPT-3 used 175 billion parameters and 300 billion tokens (Brown et al. (2020)).Source: §1 · p. 4

Contributions

• The authors describe a training protocol for Gopher’s architecture, optimisation, infrastructure, and MassiveText dataset, then analyse its performance across 152 tasks.Gopher contains 280 billion parameters.Source: §1 · p. 4
• Gopher improves performance over current state-of-the-art models on roughly 81% of tasks with comparable results, especially fact checking and general knowledge.The evaluation spans 152 tasks examining diverse aspects of intelligence.Source: §1 · p. 4

Safety and behaviour

• Larger models generate more toxic responses to toxic prompts but classify toxicity more accurately, motivating analysis of scale, bias, and downstream harms.The paper also examines dialogue interactions through prompting and presents qualitative capabilities and limitations.Source: §1 · p. 4

• Language modelling represents text as token sequences and models their probability, using open-vocabulary BPE with UTF-8 byte backoff in this paper.Open-vocabulary tokenization can uniquely represent any string, unlike closed-vocabulary schemes.Source: §2 · p. 5
• The paper focuses on autoregressive next-token prediction, where each token is predicted from preceding context, because of its performance and simplicity.Alternative objectives include masked-token and permutation-based sequence modelling.Source: §2 · p. 5
• Transformers (Vaswani et al., 2017) have achieved state-of-the-art language-model performance, motivating this paper’s focus on scaling data, parameters, and computation.Prior milestones include BERT, GPT-2, Megatron, T5, and GPT-3, establishing the term Large Language Models.Source: §2 · p. 5
• Recent related models extend language-model scaling through diverse data, larger vocabularies, sparse mixture-of-experts architectures, and instruction fine-tuning (Wei et al., 2021).The paper compares with these developments where possible, while treating instruction tuning as complementary to building a powerful base language model.Source: §2 · p. 5

3.1. Models

• The Gopher family comprises six autoregressive Transformer language models ranging from 44 million to 280 billion parameters, with the largest called Gopher.The models use RMSNorm, relative positional encodings, 32,000-token SentencePiece vocabulary, and byte-level backoff.Source: §3.1 · pp. 5–6

3.2. Training

• All models are trained for 300 billion tokens with a 2048-token context window, using Adam, learning-rate warm-up and cosine decay, and size-dependent batch and gradient-clipping settings.The maximum learning rate decreases and batch size increases with model size; Gopher’s batch grows from three to six million tokens, while clipping is reduced to 0.25 for Gopher and the 7.1B model.Source: §3.2 · p. 6

3.3. Infrastructure

• Gopher training uses model and data parallelism within TPU pods and pipelining across pods, incurring only 10% overhead from data and model parallelism.The topology addresses Gopher’s 2.5 TiB state by combining parallelism, optimizer-state partitioning, and rematerialization.Source: §3.3 · pp. 6–7

3.4. Training Dataset

• MassiveText contains about 2.35 billion documents and 10.5 TB of English text from web pages, books, news, and code, filtered, deduplicated, and sampled for training.Gopher uses 300B tokens, or 12.8% of the dataset; sampling proportions are tuned for downstream performance, and MassiveWeb improves performance over C4.Source: §3.4 · p. 7

4.1. Task Selection

• Task selection spans mathematics, common sense, logical reasoning, general knowledge, science, ethics, reading comprehension, and conventional language modelling, using probability-based evaluation with BPB for language modelling and accuracy otherwise.The study filters documents similar to older test sets but cautions that training-set test leakage may remain.Source: §4.1 · p. 8, Table 3 · p. 8

4.2. Comparisons with State of the Art

• Gopher outperforms current language-model SOTA on 100 of 124 comparable tasks, with more uniform gains in reading comprehension, humanities, ethics, STEM, and medicine than in reasoning-heavy categories.The strongest improvements are knowledge-intensive, while several logical and mathematical tasks show smaller gains or deterioration.Source: §4.2 · pp. 8–9, Figure 1 · p. 9
• On RACE, Gopher raises high-school reading-comprehension accuracy from 47.9% to 71.6% and middle-school accuracy from 58.1% to 75.1%, approaching human-rater performance at the high-school level.The smaller Gopher models perform worse, indicating that scale combined with data is important; all models remain below roughly 95% human-ceiling and over 90% supervised performance.Source: Table 4 · p. 9, §4.2 · pp. 9–10
• Gopher improves on 11 of 19 language-modelling tasks, especially books and articles, but underperforms SOTA on 8 tasks including Ubuntu IRC and DM Mathematics.The weaker numerical-task performance may reflect poor tokenizer representation for numbers, while gains on books and articles may relate to MassiveText’s 27% book-data sampling proportion.Source: §4.2 · p. 9, Figure 2 · p. 10
• Gopher achieves 60% average accuracy across MMLU’s 57 tasks, exceeding GPT-3’s 43.9% and UnifiedQA’s 48.9% but remaining below estimated human-expert performance of 89.8%.This establishes a substantial improvement over prior language-model and supervised baselines without reaching expert-level performance.Source: Table 5 · p. 11, §4.2 · p. 11

4.3. Performance Improvements with Scale

• Scaling from at most 7.1B to 280B parameters yields significant improvements over 25% on 79 of 152 tasks, small improvements on 57, and no gains on 16.The models share the same dataset and token budget, isolating parameter-scale effects; the largest benefits occur in medicine, science, technology, social sciences, and humanities.Source: §4.3 · pp. 11–12, Figure 4 · p. 12
• Scale produces reduced benefits in mathematics, logical reasoning, and common sense, with Gopher sometimes below smaller models on Abstract Algebra, Temporal Sequences, and High School Mathematics.Language-modelling tasks show the smallest average relative gains partly because BPB limits the possible improvement range, while common-sense gains are constrained by strong smaller-model performance.Source: §4.3 · p. 12

5. Toxicity and Bias Analysis

• The analysis studies toxic outputs, toxic-text recognition, group-related distributional bias, and subgroup dialect modelling across model scales, while warning that its metrics have important limitations.The authors retain these evaluations to identify challenges and future work, not to establish best practice.Source: §5 · p. 13, §5.1 · p. 13

5.1.1. Generation Analysis

• Larger models align continuation toxicity more closely with prompt toxicity, with responses becoming more toxic as input toxicity rises before plateauing near 7.1B parameters.Unprompted toxicity remains low, does not increase with model size, and is slightly below training-data toxicity.Source: §5.1.1 · pp. 13–14, Figure 5 · p. 14

5.1.2. Classification Analysis

• At 280B parameters, few-shot toxicity classification reaches an AUC of around 0.76 in the 20-shot setting, but remains below task-specific state-of-the-art classifiers.Smaller models perform comparably or worse than a random classifier with AUC 0.5.Source: §5.1.2 · p. 14, Figure 5 · p. 14

5.2. Distributional Bias

• Increasing model size does not remove distributional bias: standard cross-entropy training causes models to reflect biases present in their training data.The analysis covers gender–occupation associations, social-group sentiment, and dialect perplexity.Source: §5.2 · pp. 14–15

5.2.1. Gender and Occupation Bias

• Gender-bias measurements show no consistent relationship with model size, while changing templates or gender-word sets substantially changes the measured bias.These evaluation choices therefore materially affect reported results.Source: §5.2.1 · p. 15, Figure 6 · p. 15
• Coreference accuracy increases with scale, but performance on stereotype-opposing “gotcha” examples remains considerably lower and differs significantly between male and female pronouns.Thus, larger models still exhibit gender and occupation bias despite improved overall task performance.Source: Figure 6 · p. 15, §5.2.1 · p. 16

5.2.2. Sentiment Bias towards Social Groups

• Sentiment distributions show no clear trend with scale, while some attributes receive notably lower mean sentiment and reflect historical and contemporary discourse about specific groups.The unclear downward trend is particularly relevant for some race and religion attributes.Source: §5.2.2 · pp. 16–17, Figure 7 · p. 16

5.2.3. Perplexity on Dialects

• Scaling improves both dialects’ perplexity at roughly the same rate, but African American-aligned text remains higher-perplexity than White-aligned text, so the gap does not close.This disparate modelling ability could produce allocational harms for users of different dialects.Source: §5.2.3 · p. 17, Figure 8 · p. 17

6.1. Prompting For Dialogue

• Naively prompted Gopher reproduces training-distribution patterns rather than conversation, generating first-person narratives, blog-like text, and generic existential-question lists.This behavior appears when the model is prompted with a question without a dialogue-specific role prompt.Source: §6.1 · p. 18, Table 6 · p. 18
• Dialogue-Prompted Gopher can sustain topical, technically detailed conversations, but it also produces confident factual errors, contradictions, harmful text, and unreliable claims of search.The prompt conditions response priors without creating consistently reliable or factual dialogue.Source: §6.1 · p. 18, Table 8 · p. 20

6.2. Fine-tuning for Dialogue

• Human raters showed no significant preference between Dialogue-Tuned Gopher and Dialogue-Prompted Gopher: preference was (50 ± 0.04)% across 1400 ratings.Dialogue-Tuned Gopher was fine-tuned for ∼5 billion tokens on a curated MassiveWeb dialogue dataset.Source: §6.2 · p. 18

6.3. Dialogue & Toxicity

• Dialogue prompting prevents continuation toxicity from increasing with model scale, with toxicity slightly decreasing from 117M parameters except for the most toxic prompts.For highly toxic prompts, prompted models remain largely near 44M toxicity levels, unlike unprompted models’ upward trend.Source: Figure 9 · p. 19, §6.3 · p. 19
• Adversarial attacks can still induce Dialogue-Prompted Gopher to recite discriminatory jokes, insult users, and elaborate on inappropriate desires despite safety instructions.The model sometimes states that instructions prohibit a behavior immediately before exhibiting it.Source: §6.3 · p. 19

7.1. Towards Efficient Architectures

• Scaling this Transformer architecture further requires more training energy and compute, motivating more efficient architectures because sparse-parameter training did not yet provide an overall efficiency boost.Conditional experts and retrieval offer alternatives: Switch Transformer has 1.7T parameters at lower compute than Gopher, while retrieval reached GPT-3-level performance with 7B parameters and over 10× less training compute.Source: §7.1 · pp. 20–21

7.2. Challenges in Toxicity and Bias

• Toxicity and bias evaluation is limited by classifier social bias, brittle templates, unclear context, and metrics that may erase historically meaningful differences.The paper therefore emphasizes reliable metrics, desirable-behaviour specifications, analysis tools, and interdisciplinary data collection as prerequisites for effective mitigation.Source: §7.2 · pp. 21–22

7.3. Safety benefits and safety risks

• Language models could support safe AI through natural-language communication, explanations, and recursive schemes such as iterated amplification, debate, and recursive reward modelling.These benefits depend on models that can follow human discussion and reasoning, while experimental work remains nascent.Source: §7.3 · p. 22
• Large language models also risk memorization, high training costs, distributional shift, amplified bias, and toxic-language generation, so these harms require mitigation.Possible pre-training measures include privacy-preserving algorithms and broader multilingual datasets, but many harms may be better addressed downstream.Source: §7.3 · p. 22
• Language-model safety depends on application-specific downstream mitigation, because the same model may need to generate no toxic language yet recognize toxic text accurately.The paper favors technical and sociotechnical downstream measures, while noting that claims about downstream mitigation remain empirical and require shared deployment responsibility.Source: §7.3 · pp. 22–23

Conclusion

• Data quality and scale yield performance advances over existing language-model work, but scaling benefits are nonuniform across tasks.Complex mathematical and logical reasoning shows little benefit up to Gopher’s scale.Source: §8 · p. 23
• The limited gains on reasoning tasks may reflect the language-modeling objective’s difficulty compressing mathematics compared with learning associative facts.Alternatively, substantially larger models might overcome this bottleneck and develop new reasoning capabilities.Source: §8 · p. 23
• The paper advocates broad development of analysis and interpretability tools to understand model behavior and fairness, mitigate harms, and guide AI toward societal benefit.These tools can also inform how increasingly powerful language models are used for scalable alignment.Source: §8 · p. 23

10. Contributions

• Contributions span model design, training infrastructure, results and analyses, efficient training and inference, project management, resources, and research advising.Source: §10 · p. 24, Implementation of training… · p. 24, Results and analyses · p. 24

A. MassiveText

• MassiveText combines web pages, books, news articles, and code to promote dataset diversity, while documenting collection procedures, design choices, and a datasheet.The authors use simple heuristics rather than classifier-based filtering to avoid demographic, dialect, and sociolect bias, identifying quality filtering as future work.Source: A. MassiveText · p. 39

A. Dataset Pipeline

• The dataset pipeline filters, normalizes, deduplicates, test-set-filters, and extracts fixed-size training chunks from MassiveText documents.Source: §A.1 · p. 39, §A.1.1 · p. 39

A.1.1. Pipeline stages

• Pipeline stages remove non-English and explicit-content pages, extract coherent HTML text for MassiveWeb, and apply simple heuristics targeting low-quality documents.Filters include document length, mean word length, symbol ratios, bullet and ellipsis patterns, alphabetic-word coverage, and stop-word presence.Source: §A.1.1 · p. 40
• Documents with excessive repeated lines, paragraphs, or n-grams are removed using multiple duplication measures and thresholds detailed in Table A1.Likelihood-based ranking is considered costly and potentially biasing because high-likelihood text can still be repetitious and low quality.Source: §A.1.1 · pp. 40–41, Table A1 · p. 41
• Exact duplicates and near-duplicates are removed using 13-gram MinHash Jaccard similarity above 0.8, while training documents resembling test documents are filtered similarly.Wikipedia validation and test pages receive additional removal, but later-created datasets such as the Pile may remain leaked.Source: §A.1.1 · p. 41

A.1.2. Constructing Token Sequences

• Training sequences sample and crop documents, tokenize with BOS and EOS, concatenate ten crops, split into 2048-token sequences, discard short final chunks, then mix and shuffle subsets.This procedure improves shuffling and avoids wasting computation on PAD tokens.Source: §A.1.2 · p. 42

A.2. Dataset Analysis

• The dataset’s documents vary sharply in length, with most major text subsets averaging under 1,000 tokens, GitHub averaging 2,946, and books averaging 120,000.The longest book exceeds 1.3M tokens, while most documents from MassiveWeb, C4, News, and Wikipedia fit within the models’ 2,048-token sequence length.Source: §A.2 · p. 42, Figure A2 · p. 43
• Toxicity is generally low: mean and median scores are 0.10 and 0.07, 0.8% of spans exceed 0.5, versus 4.3% reported for GPT-2 training data.The comparison may reflect different training-data selection principles; Wikipedia has the lowest toxicity, while GitHub’s higher scores may reflect classifier uncertainty out of domain.Source: §A.2 · pp. 42–43
• MassiveText is predominantly English, with academic or scientific domains prominent in MassiveWeb despite no explicit scientific-content bias.Over 99% of MassiveText is English, and four of MassiveWeb’s top six domains are academic or scientific.Source: Figure A3 · p. 44
• The 32,000-token BPE vocabulary is only 1%–3% less compressive than GPT-2/3’s 50,000-token vocabulary on text, but over 13% less compressive on GitHub.Compression is measured in UTF-8 bytes per SentencePiece token; GitHub is least compressible and C4 most compressible.Source: Table A2 · p. 44, §A.2 · p. 44

A.3.1. MassiveText Subsets Weighting

• The ablation selects sampling weights of 10% C4, 50% MassiveWeb, 30% Books, and 10% News because they perform well across tasks and achieve the best Curation Corpus performance.Wikipedia and GitHub are fixed at 2% and 3%, respectively, while the remaining 95% is swept across four text subsets.Source: §A.3.1 · p. 45, Figure A4 · p. 45, §A.3 · p. 45

A.3.2. Iterative Refinement of MassiveWeb

• Iterative MassiveWeb processing improves downstream performance: quality filtering and deduplication substantially help, and the final dataset outperforms OpenWebText and C4 on all three evaluated datasets.The ablation uses 1.4B-parameter models and 5GB subsamples; deduplication’s effect may be underestimated because larger datasets likely contain more duplicates.Source: §A.3.2 · pp. 45–46, Figure A5 · p. 46

A.4. Text normalisation

• The tokenizer’s NFKC normalization is not lossless, changing expressions such as 2^5 to 2 5 and potentially reducing model expressivity; the authors recommend lossless normalization in future work.Because normalization also changes evaluation and test data, the paper treats this as a limitation of the current preprocessing.Source: §A.4 · p. 46

A.5. MassiveText Datasheet

• The MassiveText datasheet describes a text-only, large-scale dataset created for language-model pretraining and broader NLP tasks, collected through extraction, cleaning, filtering, deduplication, and random web subsampling.The dataset includes web pages, Wikipedia, news, books, and permissively licensed source code; it was collected over multiple months in 2020.Source: §A.5 · pp. 46–48

B. Gopher Model Card

• Gopher is a Transformer language model intended primarily for language-model research, including NLP applications, AGI investigation, and fairness and safety research.Source: B. Gopher Model Card · p. 49
• The model should not be used for harmful or deceitful generation or downstream applications without additional safety and fairness analysis.Source: B. Gopher Model Card · pp. 49–50
• Because Gopher was trained on English data, it has unequal performance on some dialects, including African American English, requiring downstream factor analysis.Source: B. Gopher Model Card · p. 50
• The model card evaluates language modeling, knowledge, reasoning, question answering, reading comprehension, fact checking, common sense, toxicity, and demographic bias.Source: B. Gopher Model Card · pp. 50–51
• Evaluation emphasizes text-likelihood prediction rather than long-form generation, since high-quality generation also depends on decoding methods such as search or reward models.Source: B. Gopher Model Card · p. 50
• The broad evaluation provides a reasonable overall performance estimate, although training Gopher multiple times was infeasible because of its cost.Source: B. Gopher Model Card · p. 51

C.1. Adafactor

• Adafactor reduced memory requirements but, unlike at 1.4B parameters, reduced performance and increased instabilities at larger scales, with 7.1B training showing loss divergences despite a lower learning rate.Fine-tuning with Adafactor was also prone to divergence and sensitive to learning rate and batch size, although it was used for Gopher fine-tuning to reduce hardware requirements.Source: §C.1 · p. 53, Figure A6 · p. 53

C.2. Lower-Precision Training with bfloat16

• A float32 optimiser-state copy makes bfloat16 training indistinguishable from full float32 performance while reducing memory use and improving speed by 1.4×.Without this copy, small updates and low precision made many layers stale; all bfloat16 configurations used fp32 for attention softmax and loss computation.Source: §C.2 · pp. 53–55, Figure A7 · p. 54

D.1. Overview

• Gopher generally equals or outperforms language-modeling state of the art across task categories, although human performance often remains substantially higher.The overview compares raw Gopher results with language-modeling SOTA, supervised SOTA, and human expert performance.Source: §D.1 · p. 55, Figure A8 · p. 56

D.2. Pile

• 10/18 Pile tasks reach language-modeling SOTA, with the largest relative gains on Gutenberg, GitHub, PubMed, arXiv, and Stackexchange.Against Jurassic-1, Gopher performs better on 8/16 tasks, ties one, and performs worse on 7/16; GPT-3 is best on OpenWebText2.Source: §D.2 · p. 60, Table A7 · p. 62

D.4. Filtering Test-Set Documents

• Filtering high n-gram similarity reduces train-test leakage for selected older benchmarks, while newer Pile, MMLU, and BIG-bench evaluations are reported without retraining because re-training is expensive.The paper notes that paraphrased contexts can enable memorisation in question answering, translation, and arithmetic even when n-gram filters miss the overlap.Source: §D.4 · p. 61

D.5. Scaling Curves

• 417M→7.1B parameters follow an approximately linear log-log scaling fit, but 280B Gopher deviates from the power law, suggesting under-training or a changed trend at that scale.The scaling law holds for PG-19 but is substantially less reliable on datasets such as Curation Corpus summarisation.Source: §D.5 · p. 62, Figure A10 · p. 63

D.6. Scaling Context Length

• Longer evaluation contexts improve articles and code after clamping positions at 2048, but provide no gain for PubMed Abstracts and smaller gains for books.Naive relative-position extrapolation degrades sharply; clamping makes distant tokens share relative times beyond 2048, limiting their distinguishability.Source: §D.6 · pp. 62, 64, Figure A11 · p. 64

D.7. MMLU

• 60.0% 5-shot MMLU accuracy for Gopher exceeds GPT-3’s 43.9%, forecaster expectations of 57.1%, and human-rater performance of 34.5%, while remaining below human experts at 89.8%.Gopher improves over GPT-3 on 55 of 57 tasks, but Abstract Algebra and High School Mathematics show no positive scaling trend.Source: §D.7 · pp. 65–66, Figure A14 · p. 67

D.8. BIG-bench / D.8.1. Task Selection

• BIG-bench evaluation uses 62 English multiple-choice tasks selected to probe semantic capability without fine-tuning, excluding non-English, long-context, tokenisation, programmatic, and inappropriate tasks.Multiple-choice scoring selects the answer with highest log-probability, avoiding confounds from open-ended decoding methods.Source: §D.8 · p. 69, §D.8.1 · p. 69

D.8.2. Multiple Choice Evaluation

• Models receive five examples followed by the current input and choices, then select the choice with the highest summed token log-probability.Accuracy is computed from the selected choice.Source: §D.8.2 · pp. 73–74

D.8.1. Task Selection

• The evaluation covers 62 English-language multiple-choice tasks organized into broader categories.The selected tasks span categories listed in BIG-bench’s constituent-task table.Source: Table A10 · p. 73, Table A9 · p. 74

D.8.4. Relative vs absolute accuracy

• Normalising accuracy by each question’s random-chance baseline preserved the broad results pattern and correlated with model size better than unnormalised accuracy.The normalised score accounts for varying numbers of answer choices across questions.Source: §D.8.4 · p. 76, Figure A16 · p. 78, Table A12 · p. 78

D.8.3. BIG-bench 5-Shot Results

• Gopher outperformed other models on 41 of 62 BIG-bench tasks, with a step change from the 7.1B model to Gopher across 41 of 51 task categories.The largest improvements occurred on Alignment, Social Bias, Memorization, and Numerical Response tasks, whereas mathematical, multistep, decomposition, paraphrasing, summarization, and negation tasks showed limited scaling benefits.Source: §D.8.3 · p. 74, Figure A15 · p. 77

D.8.5. Comparing Gopher family models to models from the T0 family

• Gopher achieved the best overall 0-shot BIG-bench performance among the evaluated models, outperforming T0 models on Hindu Knowledge and Known Unknown tasks.It performed similarly to T0++ on Misconceptions and worse than all but T0 on Novel Concepts.Source: §D.8.5 · p. 77, Table A13 · p. 78

D.9. TriviaQA & NaturalQuestions

• Performance on TriviaQA and Natural Questions increased smoothly with model size, indicating that capacity supports factual memorisation.The largest model was comparable to GPT-3 in few-shot closed-book question answering but slightly below GPT-3 on Natural Questions, possibly because GPT-3 used more Wikipedia examples.Source: Table A15 · p. 80, §D.9 · p. 80

D.10. TruthfulQA

• 43.7% accuracy with 10-shot prompting improved over 29.5%, and this consistent few-shot gain occurred only for Gopher at 280B on TruthfulQA.Unlike prior baselines, Gopher-family zero-shot accuracy also improved with scale across the tested formulations, though prompting and answer-choice setup affect comparisons.Source: §D.10 · pp. 81–82, Figure A17 · p. 82, Table A16 · p. 83

D.11. Reading Comprehension: RACE

• The RACE evaluation used few-shot multiple-choice prompts with answer options, and omitting options or changing completion scoring affected smaller and largest models differently.Gopher’s predictions showed a consistent trend of over-confidence while remaining otherwise reasonably calibrated.Source: §D.11 · pp. 82, 84
• 71.6% accuracy on RACE-h exceeded GPT-3’s 46.8% and Megatron-Turing’s 47.9%, but remained below ALBERT-XXL’s 90.5% and the estimated 94.2% human ceiling.Gopher advanced state-of-the-art performance among autoregressive language models without fine-tuning, although supervised methods using far fewer parameters performed better.Source: §D.11 · p. 83

D.12. Fact-Checking: FEVER & MultiFC

• 50%: closed-book FEVER performance reaches this level for the largest model, while SUPPORTED-versus-REFUTED classification reaches 78%.Performance improves monotonously with model size, but REFUTED-versus-NOTENOUGHINFO performance plateaus after 1 billion parameters.Source: §D.12 · pp. 85–86
• 88%: models above 1 billion parameters perform comparably to the trained Decomposable Attention baseline on FEVER with gold or sampled evidence.Gopher performs this entailment-recognition task without fine-tuning using only 15 demonstrations, five per class.Source: §D.12 · p. 86
• 89% macro-F1: Gopher improves absolute performance by 18% over the 71% reported for 1.5B GPT-2 in binary few-shot fact-checking.REFUTED and NOTENOUGHINFO are combined into one class, so this is a binary rather than three-way classification comparison.Source: §D.12 · p. 86

D.12.1. MultiFC

• 64% and 67% macro-F1: Gopher achieves these scores on MultiFC in claim-only and claim-and-evidence conditions, respectively, exceeding a random baseline.The 165 website-specific labels are remapped to SUPPORTED or REFUTED, so Augenstein et al. (2019)'s 49.2% macro-F1 and 62.5% micro-F1 are not directly comparable.Source: §D.12.1 · p. 86

D.13. Common Sense: PIQA, WinoGrande, SocialIQA, HellaSwag

• The evaluation covers physical, temporal, and social common sense using zero-shot multiple-choice scoring, while generative non-multiple-choice evaluation is left for future work.Each answer choice is scored conditional on the context and question, with the highest-scoring choice selected.Source: §D.13 · p. 87, Table A17 · p. 87
• Gopher outperforms 175B GPT-3 on PIQA and performs nearly on par with 530B Megatron-Turing, while its advantages are smaller on HellaSwag and WinoGrande.Comparisons use zero-shot multiple-choice evaluation across common-sense benchmarks.Source: §D.13 · p. 87
• Zero-shot models remain substantially behind state-of-the-art and human performance on every common-sense dataset, leaving considerable room for improvement.The analysis emphasizes that supervised common-sense data are difficult to curate because common-sense knowledge is broad and varied.Source: §D.13 · pp. 87–88
• Less than 1.5% separates 175B–530B models on HellaSwag and PIQA, suggesting limited gains beyond current largest models.Figure A20 shows common-sense ability increasing with model size, while performance differences among Gopher, GPT-3, and Megatron-Turing remain small.Source: §D.13 · p. 87, Figure A20 · p. 88
• Gopher achieves 50.6% zero-shot accuracy on SocialIQA, particularly lagging fine-tuned state of the art and indicating its greatest difficulty with social common sense.The authors recommend including SocialIQA alongside HellaSwag, PIQA, and WinoGrande in future evaluations.Source: §D.13 · p. 88

E.1.1. Methodology

• Toxicity experiments sample 25k unconditional continuations per model and 25 continuations for each selected prompt, using up to 100 tokens and nucleus sampling with p=0.9.The conditional setting uses 10% of the 100k RealToxicityPrompts prompts for efficiency, and incomplete sentences are truncated.Source: §E.1.1 · p. 89
• The Perspective API assigns TOXICITY scores from 0 to 1, with analyses using both the conventional 0.5 threshold and four toxicity bins.The bins are very low, low, medium, and high, enabling clearer trend decomposition.Source: §E.1.1 · p. 89

E.1.2. Results

• Larger models more closely preserve prompt toxicity, although continuations remain consistently less toxic than their prompts.Continuation toxicity rises with prompt toxicity more steeply for larger models, suggesting greater faithfulness to toxic inputs.Source: §E.1.2 · p. 89
• The 1.4B C4 model has the lowest toxicity metrics and GPT-2 the highest across Table A18, while prompted metrics show no clear scaling trend.Unprompted expected maximum toxicity and toxicity probability decrease with model size, whereas dataset construction appears to affect toxicity more than scale.Source: Table A18 · p. 90, §E.1.2 · p. 90

E.2.2. Subgroup Bias Metrics

• Few-shot toxicity classification exhibits subgroup bias: low BPSN AUC marks Muslim-related text as disproportionately toxic, whereas low BNSP AUC marks Atheist-related text as disproportionately non-toxic.The evaluation uses subgroup AUC, BPSN AUC, and BNSP AUC to characterize disparities in classifier behavior.Source: §E.2.2 · p. 93, Figure A23 · p. 94

E.3.1. Gender and Occupation

• Gender–occupation bias has no consistent relationship with model size, and its measured magnitude changes substantially with the prompt template and selected gender terms.Using “male” and “female” alone produces substantially lower bias than summing across all listed gendered terms.Source: §E.3.1 · p. 95
• The models reproduce stereotypical occupation associations: 34.2% of occupations are biased toward women, while the remainder are biased toward men under one template and term set.Only 13 occupations have absolute bias below 0.1, with average bias -0.74 for male-biased and 0.52 for female-biased occupations.Source: §E.3.1 · pp. 95–96
• Winogender coreference accuracy increases with model size, with higher performance for neutral “they” pronouns than gendered pronouns.The evaluation measures whether the model assigns higher probability to the correct coreference resolution.Source: §E.3.1 · p. 96

E.3. Distributional Bias

• Sentiment-bias evaluation finds no strong correlation between parameter count and continuation sentiment, while group-fairness scores compare sentiment distributions across demographic attributes.Completions are sampled with nucleus sampling and scored after excluding the prompt; Wasserstein-1 distances define the fairness metric, where zero indicates no bias.Source: §E.3.2 · pp. 97–98

E.3.2. Sentiment Bias

• Sentiment and toxicity conclusions depend on carefully chosen prompt templates, demographic terms, and comparison classes, especially when marked attributes such as “White” are contrasted with no attribute.Low sentiment appears for “Atheist,” “White,” “Black,” “a sheriff,” and “a guard”; co-occurrence patterns also differ across Syria, Iran, Iceland, Black, White, and unmarked templates.Source: §E.3.2 · p. 100

F. Compute Usage

• Reported FLOP totals include rematerialisation, padding, and repeated computation, but are best-effort lower bounds that omit development, pre-emption, and other inefficiencies.Rematerialisation alone increases compute by 33%.Source: F. Compute Usage · p. 102
• 4–100× fewer FLOPs could be used in evaluations by eliminating repeated processing of common prefixes, making more efficient evaluations important for future work.The reported inference costs are higher than necessary because many evaluations repeat computation.Source: F. Compute Usage · p. 102
• Attention consumed 39% of accelerator time for the 417M model but 8% for Gopher, with more than 70% of attention time spent on relative positional encodings.Large batch sizes improve efficiency at scale by reducing pipelining and data-parallelism costs, while rematerialisation imposes a major tax on Gopher.Source: F. Compute Usage · pp. 102–103, Table A27 · p. 103
• Training Gopher emitted 380 net tCO2e after 920 hours, compared with 552 net tCO2e for GPT-3.The estimate uses a 1.08 datacenter PUE, 0.33 net tCO2e per MWh, and 283W per chip.Source: F. Compute Usage · p. 103

G. Reducing Inference and Training Costs

• Compression via distillation and pruning, along with sparse training and reverse distillation, produced only modest improvements and small shifts in scaling curves.The methods were not remarkably successful for language modelling over a diverse corpus, despite success in some application-specific compression settings.Source: G. Reducing Inference and Training Costs · p. 103

G.1. Efficient Fine-tuning

• Fine-tuning experiments used Wikitext103, Curation Corpus, and Python GitHub to compare bias-only, final-40%-layer, and entire-model strategies under performance, FLOP, and memory trade-offs.Wikitext103 and Curation Corpus overfit, whereas Python GitHub did not overfit after more than four and a half million sequences.Source: §G.1 · pp. 103–104
• Full-model fine-tuning consistently achieved the best performance and FLOP efficiency, while final-layer tuning was not FLOP-efficient and bias-only tuning depended on dataset domain.Bias-only tuning worked relatively well on overfitting-prone in-domain datasets but had little impact on Python GitHub; full tuning remained most expensive.Source: Figure A27 · p. 104, §G.1 · p. 105
• A 5× smaller learning rate reduced Wikitext103 perplexity by 0.04 but required three times as many FLOPs, illustrating the compute–performance trade-off.For overfitting models, the optimal learning rate decreased as the number of trained parameters increased.Source: §G.1 · p. 105

G.2.1. Distillation

• A 5× distillation from 7.1B to 1.4B and an approximately 2× reduction to 785M beat similarly sized models trained from scratch, but remained more than 10% worse than their teachers.For 7.1B→1.4B, C4 perplexity was 28 versus 30 for the baseline and 22 for the teacher; for 1.4B→785M, the student improved evaluation perplexity by 7% over the base model but remained 10% behind the teacher.Source: §G.2.1 · p. 106, Figure A28 · p. 106, Table A29 · p. 106

G.2.2. Pruning

• 90% pruning required approximately 2.5× fewer parameters for a given evaluation loss, but compression without an appreciable accuracy drop was only about 20–30%.All investigated sparsity levels had approximately the same scaling coefficient, while greater sparsity lowered the intercept; sparse computation also remained difficult to exploit efficiently on GPUs.Source: §G.2.2 · pp. 108–109, Figure A30 · p. 108

G.3.1. Dynamic Sparse Training

• Dynamic sparse training with RigL produced minimal reduction in the FLOPs required to reach a given performance, despite dynamically growing and dropping sparse weights during training.RigL uses dense backward passes to activate zero-valued weights with large gradients and drops weights with low magnitude, leaving most training to sparse updates.Source: §G.3.1 · p. 109

G.3.2. Reverse Distillation

• Reverse distillation accelerated the beginning of training but delivered small gains over a full pre-training run and performed worse than warm starting.For a student twice the teacher’s size, teacher probabilities were useful for only the first 5 million sequences before interpolation to one-hot targets; distillation also incurred storage or runtime memory and compute costs.Source: §G.3.2 · pp. 109–110, Figure A31 · p. 110

G.3.3. Warm starting

• Warm starting expanded a 4.5B model to 9B while achieving scratch-model-comparable performance with a 40% reduction in compute, although gains diminished for larger or width-heavy expansions.A 417M model could be expanded by more than 3× while outperforming an equivalent fresh model at convergence, whereas expanding 3B to 7.1B produced a slightly worse converged model.Source: §G.3.3 · pp. 110–111, Figure A31 · p. 110

G.3.4. Alternative Warm Starting Methods

• Alternative warm-starting schemes underperformed models trained from scratch, whereas replicating layers in depth and tiling weights in width worked best, with slight gains from adding noise.Tiling preserves the original weight magnitude distribution and matrix structure, helping integrate newly initialized parameters.Source: §G.3.4 · pp. 111–112, Figure A32 · p. 112

G.4. Future Work for Efficient Training

• Efficient training remains an open problem because the investigated techniques either fail to improve results or provide minor gains with substantial code or operational complexity.Suggested directions include architecture search, mixture-of-experts models, quantization, hardware-accelerated sparsity, and semi-parametric approaches.Source: §G.2.1 · p. 107, §G.2.2 · p. 109, §G.4 · p. 112

H.1. Construction

• The model is constructed from raw Gopher with a conversational prompt and template that standardize alternating User and Gopher utterances.The hand-authored prompt consumes roughly 800 of Gopher’s 2048 context tokens, leaving room for subsequent dialogue.Source: §H.1 · pp. 112–113, Table A30 · p. 114
• Responses are sampled from the full dialogue history with nucleus sampling at p=0.8 and truncated when Gopher begins the next User turn or reaches maximum length.User messages are appended with a User prefix, and generation begins after an appended Gopher prefix.Source: §H.1 · p. 113

H.2. Dialogue Dataset Filtering

• The dialogue dataset is filtered from MassiveWeb by retaining at least six-paragraph, strictly alternating two-person exchanges with consistent speaker prefixes.This interview-transcript-style heuristic is described as reliably yielding high-quality dialogue.Source: §H.2 · p. 115

H.3. Comparison Methodology

• Dialogue-Prompted Gopher showed no statistical difference in human preference from Dialogue-Tuned Gopher under either move-selector setting.Participants compared model moves turn by turn, while each dialogue continued according to the selected model independently of their choices; the prompted model could theoretically benefit when its own moves were selected.Source: §H.3 · p. 115, Table A31 · p. 115

H.4. RTP in a Dialogue Setting

• The RealToxicityPrompts dialogue evaluation samples 2,000 questions across four toxicity buckets and 25 continuations per question, with aggregate results reported in Figure 9.Questions are formed by truncating RTP prompts after a question mark and sampling continuations up to 100 tokens.Source: §H.4 · p. 115

H.5. Selected Transcripts

• Open-ended transcripts show that Dialogue-Prompted Gopher can answer trivia and recall facts, but also produces false, nonsensical, toxic, or misleading responses.Examples also include reasoning failures, declined reasonable requests, useful pointers without detail, and apparent creativity in conversation.Source: §H.5 · p. 115, Table A32 · p. 116, Table A33 · p. 116
• The transcripts additionally illustrate that toxic questions may be evaded inconsistently, while dialogue can mix contextual conditioning with factual recall and produce mostly technically correct semi-factual answers.Some examples show useful but incomplete pointers, reasonable requests being declined, and factual precision undermined by a misleading response.Source: Table A34 · p. 117, Table A39 · p. 118, Table A38 · p. 118