PaperlayerView exact revision on arXiv

Source-linked AI summary

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel

arXiv:2204.02311v5cs.CL

TL;DR

Few-shot language modeling still required clearer evidence about how scaling affects performance across tasks. The paper trains and evaluates PaLM, a 540-billion-parameter dense Transformer, finding state-of-the-art few-shot results broadly and breakthrough performance on reasoning tasks.

  • Problem

    Existing language models often require substantial task-specific data and parameter updates for finetuning, motivating clearer evidence about scaling in few-shot learning.

  • Method

    The paper trains PaLM, a 540-billion-parameter dense Transformer language model, and evaluates its few-shot capabilities across language, reasoning, multilingual, and code benchmarks.

  • Results

    PaLM achieves state-of-the-art results on 28 of 29 widely evaluated English NLP tasks, exceeds average human performance on BIG-bench, and matches or exceeds finetuned state of the art on reasoning tasks.

  • Takeaways & Limitations

    Few-shot improvements generally continue with scale, while some capabilities show discontinuous gains at the largest model size, especially with chain-of-thought prompting for reasoning.

  • Takeaways & Limitations

    Fairness analyses use only English data and may miss biases arising in other languages and sociocultural contexts.

Abstract

from arXiv · show

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.

1 Introduction

PaLM extends scaling-based language-modeling improvements with a 540-billion-parameter densely activated autoregressive Transformer trained using Pathways. The paper reports continued scaling gains across language, code, mathematical reasoning, multilingual, bias, and toxicity evaluations, including discontinuous improvements on some tasks.

  • Motivation: Large language models have achieved strong results across many language tasks, with encoder-based models typically adapted through task-specific finetuning.BERT and T5 are described as using infilling pre-training objectives before finetuning on specific tasks.
  • Motivation: Few-shot autoregressive language models use natural-language task descriptions and optional exemplars, achieving strong results without large task-specific datasets or parameter updates.GPT-3 established this evaluation approach for decoder-only models trained with a left-to-right next-token objective.
  • PaLM: PaLM is a 540-billion-parameter densely activated autoregressive Transformer trained on 780 billion tokens using Pathways across thousands of accelerator chips.Pathways enables efficient training of very large neural networks across multiple TPU v4 Pods.
  • Efficient scaling: 6144 TPU v4 chips enabled efficient training of PaLM 540B at a scale and efficiency level not previously reached for models of this size.The introduction presents this as the first large-scale use of Pathways for training a single model across thousands or tens of thousands of accelerator chips.
  • Continued improvements from scaling: PaLM achieves few-shot state-of-the-art results on the vast majority of hundreds of natural-language, code, and mathematical-reasoning benchmarks, including 28 of 29 widely evaluated English understanding benchmarks.The introduction states that these results typically exceed prior results by significant margins and indicate that scaling improvements have not plateaued.
  • Discontinuous improvements: Scaling from 62B to 540B usually yields gains similar to scaling from 8B to 62B, but certain tasks show discontinuous improvements with drastic accuracy jumps.The comparison is framed against the power-law behavior commonly observed in neural-network scaling.
  • Multilingual understanding and bias: PaLM evaluates multilingual translation, summarization, and question answering despite only approximately 22% non-English training data, and analyzes distributional bias and toxicity.The bias analysis reports improved Winogender accuracy with scale and a new state-of-the-art result for PaLM 540B in 1-shot and few-shot settings.

2 Model Architecture

PaLM is a decoder-only standard Transformer with architectural modifications targeting quality, efficiency, long-context performance, and multilingual tokenization. The study evaluates dense models at 8B, 62B, and 540B parameters using shared training data and vocabulary.

  • PaLM uses a standard decoder-only Transformer in which each timestep attends only to itself and preceding timesteps.
  • SwiGLU activations replace standard MLP activations, improving quality in compute-equivalent experiments despite requiring three rather than two matrix multiplications.
  • 15% faster training speed results from parallel Transformer layers that fuse MLP and Attention input matrix multiplications; ablations found no degradation at 62B scale.The authors extrapolated this quality neutrality to the 540B scale.
  • Multi-query attention shares key/value projections across heads while retaining per-head queries, preserving model quality and training speed while reducing autoregressive decoding cost.Standard multi-headed attention is inefficient during decoding because key/value tensors are not shared between examples and only one token is decoded at a time.
  • RoPE embeddings, shared input-output embeddings, bias-free dense kernels and layer norms, and a 256k-token SentencePiece vocabulary complete the architecture.The vocabulary was chosen to support many languages without excess tokenization and is lossless and reversible, preserving whitespace.
  • 540B, 62B, and 8B dense models were trained identically except for batch size, with FLOPs per token approximately equal to parameter count.

3 Training Dataset

PaLM was pretrained for one epoch on a 780-billion-token corpus spanning diverse natural-language sources and source code. The dataset incorporated quality and licensing filters, language-specific code processing, deduplication, and contamination analysis.

  • Dataset composition: 780 billion tokens comprise PaLM’s pretraining corpus, mixing filtered webpages, books, Wikipedia, news, source code, and social-media conversations.All three models were trained for exactly one epoch on identically shuffled data.
  • Source code: 196GB of source code from open-source GitHub repositories covers 24 common programming languages after excluding copyleft-licensed files.Filename extensions restricted the corpus to the selected languages.
  • Data filtering: Webpages were sampled proportional to classifier-assigned quality scores, increasing the frequency of higher-quality pages without eliminating lower-quality pages.The quality classifier was trained on pages resembling known high-quality webpage collections.
  • Source code: Duplicate source-code files were removed using Levenshtein distance because duplicate files are common in repositories.The code corpus was filtered by repository license and filename extension before deduplication.
  • Data analysis: The authors check for evaluation-data contamination in Section 8 and provide overlap analysis, while Appendix D supplies a dataset datasheet and language breakdown.Table 2 documents source proportions for the final dataset mixture.
  • Dataset composition: Over 100 languages are represented in the multilingual corpus, with language proportions reported in Appendix Table 29.Table 2 reports the proportion contributed by each data source.

4 Training Infrastructure

PaLM 540B was trained across two TPU v4 Pods using Pathways, combining pod-level data parallelism with within-pod model and data parallelism. The section also introduces MFU as an implementation-independent efficiency metric and contextualizes PaLM’s accelerator utilization against prior systems.

  • Training infrastructure: PaLM 540B used two TPU v4 Pods, each containing 3072 chips and 768 hosts, scaling training to 6144 TPU v4 chips.The system was described as the largest TPU configuration at the time and used model and data parallelism.
  • Within-pod parallelism: Each Pod replicated model parameters while partitioning weight tensors across 3072 chips with 12-way model parallelism and 256-way fully sharded data parallelism.Weights were all-gathered during the forward pass, and activations were selectively saved and rematerialized during backpropagation.
  • Pathways execution: Pathways enabled two-way pod-level data parallelism by splitting each batch across Pods, exchanging gradients, and applying updates after local and remote gradient accumulation.A single Python client dispatched the batch, while each Pod performed forward and backward computation using standard within-Pod parallelism.
  • Cross-pod communication: Cross-Pod gradient transfer at 6144 chips and 1536 hosts required only 1:1 communication between corresponding hosts because each core needed remote gradients for its model-sharded parameters.The system addressed the throughput challenge over the data-center network connecting the Pods.
  • Efficiency measurement: MFU measures observed token throughput relative to theoretical peak throughput while avoiding HFU’s dependence on implementation-specific operation counts.PaLM’s corresponding hardware FLOPs utilization was 57.8%; reported MFU values were 21.3% for GPT-3 and 32.5% for Gopher.

5 Training Setup

PaLM used standard large-Transformer training practices with variance-scaled initialization, Adafactor optimization, fixed-length sequences, scheduled batch growth, and no dropout. Training the largest model nevertheless produced irregular loss spikes that required checkpoint-based mitigation.

  • Weight initialization: Kernel weights used fan-in variance scaling, embeddings used unit-variance initialization, and shared output logits were scaled by 1/√n.Specifically, W ∼N(0, 1/√nin) and E ∼N(0, 1).
  • Optimization hyperparameters: Adafactor training used learning rate 10−2 for the first 10,000 steps, momentum β1 = 0.9, global-norm clipping 1.0, and dynamic weight decay of lr2.The second-moment schedule β2 = 1.0 −k−0.8 was chosen for greater stability than β2 = 0.99 in large language models.
  • Loss function: The loss combined average token log probability without label smoothing with z loss = 10−4 · log2 Z, which increased training stability.The auxiliary term encourages the softmax normalizer log(Z) to remain close to 0.
  • Sequence and batch size: All models used sequence length 2048, while the largest model increased batch size from 512 to 1024 to 2048 at steps 50k and 115k, finishing at step 255k.These batch sizes corresponded to 1M, 2M, and 4M tokens, respectively.
  • Training instability: The largest model experienced roughly 20 irregular loss spikes despite gradient clipping, whereas smaller models did not.Restarting from a checkpoint roughly 100 steps before a spike and skipping roughly 200–500 batches prevented recurrence at the same point.

6 Evaluation

PaLM 540B achieves broad state-of-the-art or near-state-of-the-art performance across language understanding, generation, BIG-bench, and reasoning evaluations. Scaling and chain-of-thought prompting produce especially large gains on some challenging tasks, although substantial gaps remain on others.

  • Language understanding and generation: PaLM 540B outperforms prior SOTA on 24 of 29 tasks in 1-shot and 28 of 29 tasks in few-shot settings, including gains exceeding 10 points on some Reading Comprehension and NLI tasks.It also outperforms the similarly sized Megatron-Turing NLG 530B on all benchmarks.
  • Language understanding and generation: PaLM 540B improves average NLU and NLG scores by more than 5 points, with category averages improving with scale; PaLM 62B outperforms GPT-3 175B in both categories.On MMLU, PaLM 540B improves the average score by ≈2 points and outperforms Chinchilla on every category except Other tasks.
  • Finetuning and transfer: On SuperGLUE, PaLM achieves competitive close-to-SOTA performance, while the results show that scale can help bridge decoder-only models’ gap with encoder-decoder models.A significant gap remains between few-shot and finetuned results.
  • BIG-bench: On BIG-bench, PaLM 540B approaches the best human performance on some tasks, while discontinuous scaling gains indicate that certain capabilities emerge only after sufficient scale.English proverbs improves from 25% for PaLM 62B to 87% for PaLM 540B, and goal step wikihow and logical args follow log-linear scaling.
  • BIG-bench limitations: BIG-bench performance remains uneven: average human performance exceeds PaLM 540B on 35% of individual tasks, and some BIG-bench Lite tasks remain far from human performance.For navigate and mathematical induction, scaling from PaLM 62B to 540B yields only modest improvements, with both models still far from the best human score.
  • Reasoning: With 8-shot chain-of-thought prompting, PaLM 540B achieves SOTA accuracy on 4 of 7 reasoning datasets and close-to-SOTA accuracy on the remaining 3.With an external calculator, it reaches 58% on an arithmetic reasoning task, exceeding the prior SOTA of 55%; scaling also fixes many PaLM 62B errors.

7 Memorization

PaLM’s memorization of training examples increased with model size and strongly with example duplication, while most memorized content was formulaic but some stories, news articles, and facts were also reproduced. The authors argue that memorization risks depend on the dataset and application, and suggest filtering verbatim training sequences at generation time.

  • Evaluation method: The evaluation prompted 100 randomly selected training sequences with their first 50 tokens and measured exact 50-token continuation matches under greedy decoding.The methodology follows Carlini et al. (2022), who used prompt lengths from 50 to 500 tokens.
  • Scale effects: 1.6% of examples were exactly reproduced by the 8B model, compared with 2.4% for the 540B model.Held-out examples from the same distribution also showed memorization above 0%, partly because some were very similar to training examples.
  • Duplication effects: 0.75% of examples seen exactly once were memorized by the largest model, versus over 40% of examples seen more than 500 times.Training was deduplicated only on full documents, allowing high duplication rates for evaluated examples.
  • Data characteristics: Memorization was driven mainly by exact or near duplication and templating, with code containing boilerplate and copied snippets while books primarily contained unique strings.The authors also conclude that larger models memorize at higher rates, with PaLM’s log-linear fit having slope 0.002 and R2 = 0.976.
  • Risks and mitigation: Most memorized instances were formulaic, but the models also memorized stories, news articles, and facts; risks depend on dataset content and downstream consequences.A proposed mitigation is a bloom filter that prevents generation of sequences occurring verbatim in the training data.

8 Dataset Contamination

The paper assesses dataset contamination using dataset-construction knowledge, overlap statistics, and manual examination rather than high-order n-gram matching alone. Across English NLP and translation benchmarks, clean-subset comparisons indicate limited result inflation, while De-En shows a scale-consistent clean-set penalty.

  • English NLP benchmarks: The authors manually examined examples from 29 primary English NLP tasks alongside dataset-construction knowledge to identify contaminated benchmarks.This approach supplements overlap statistics rather than treating every high-order n-gram match as contamination.
  • English NLP benchmarks: The 29 tasks are divided into four categories: wholesale contamination, constructed from web, context on web, and no significant overlap.Datasets in the first two categories are considered contaminated, whereas those in the latter two are not.
  • English NLP benchmarks: An equal number of sets had positive and negative accuracy deltas on clean subsets, suggesting contamination did not meaningfully inflate reported results.Positive deltas count against inflation, while negative deltas can indicate inflation.
  • Machine translation: De-En results were roughly 3.0 BLEU worse on the clean set, and this delta was consistent across model scale.Because 8B has less memorization capacity than 540B, a primarily memorization-driven difference would be expected to be smaller on 8B.

9 Exploring Explanations

This section showcases PaLM’s explanatory language generation with chain-of-thought prompting on preliminary tasks requiring logical inference, world knowledge, abstract language understanding, and lexicographic understanding. Qualitative examples suggest remarkable deep language understanding, while the authors acknowledge the analysis is not a thorough quantitative evaluation.

  • Task design: PaLM’s explanatory-generation evaluation used two preliminary tasks requiring logical inference chaining, world knowledge, abstract language understanding, and lexicographic understanding.The tasks were prompted with 2-shot exemplars demonstrating the desired output style.
  • Evaluation setup: All predictions used the same 2-shot exemplars, which were style-related but content-independent and fixed before evaluating examples.The exemplars were written by the authors and were not modified after examining model outputs.
  • Evaluation setup: All outputs used greedy decoding, making each result PaLM’s canonical 1-best prediction rather than a temperature-sampled alternative.The authors describe this distinction as critical because sampling could produce many possible outputs.
  • Interpretation: The authors characterize the examples as demonstrating a truly remarkable level of deep language understanding, while acknowledging that they do not constitute a thorough quantitative analysis.The section emphasizes that its greatest value comes from reading the examples themselves.
  • Examples: Figure 19 presents PaLM 540B explanations of jokes involving wordplay and expectation reversal, including kleptomania and borrowing money from a pessimist.The figure states that evaluated jokes and inference-chaining examples were written by the authors.

10 Representational Bias Analysis

PaLM’s representational-bias evaluation finds improved Winogender performance with scale and reduced slice gaps with more shots, while revealing stereotypical associations and toxicity patterns. The authors caution that these findings are limited by brittle, English-centric, non-comprehensive evaluations and may vary across downstream applications.

  • Winogender: PaLM 540B reaches 79.4% Winogender accuracy with multiple-choice scoring, 69.7% with generative scoring, and 84.7% with 4-shot generative scoring.The 1-shot multiple-choice result exceeds GLaM’s 71.7% accuracy, and accuracy improves with model scale.
  • Winogender: Accuracy is higher on stereotypical than gotcha examples, with the lowest performance on gotcha examples involving female gender.The stereotypical–gotcha gap improves from 14.1 to 10.1 percentage points in 1-shot settings and from 18.3 to 9.2 in 4-shot settings.
  • Identity co-occurrence: Co-occurrence analyses associate Islam with charged terms including terrorist, violent, and radical, while identity-term analyses require careful interpretation because terms such as Indian, Black, and White are not disambiguated.The top-10 co-occurring words are the same for PaLM 62B and 540B across race, religion, and gender, suggesting training data influences results more than model size.
  • Toxicity: Continuation toxicity probability increases with prompt toxicity but is generally below prompt toxicity and the human baseline, with a visible increase between 8B and larger models.PaLM 62B and 540B have very similar toxicity profiles; lower reported toxicity partly reflects measuring only the first full sentence.
  • Limitations: The fairness and toxicity analyses are not comprehensive: they use only English data, lack standardized and broadly identity-covering benchmarks, and may not predict downstream impacts across applications.The authors emphasize that prompt-framing variance makes template-based evaluations brittle and call for robust benchmarks and metrics.

11 Ethical Considerations

PaLM’s scale creates broad application potential but also risks involving stereotypes, privacy, downstream harms, misuse, and software-development safety. The paper emphasizes that its bias and capability evaluations are limited, context-dependent, and incomplete.

  • Bias and toxicity: PaLM’s training data and outputs reflect social stereotypes and toxicity associations around identity terms, while removing them is non-trivial.Automated toxicity filtering may disproportionately exclude content about or authored by marginalized subgroups.
  • Evaluation limitations: The fairness analyses are narrowly scoped to gender, race, ethnicity, and religion in English, excluding other social-disparity axes.The paper specifically notes that sexual orientation and disabilities were not examined.
  • Context-dependent risks: Downstream harms depend on how PaLM is used; exposure to toxic content may nevertheless be important for applications such as online content moderation.The same training exposure can be undesirable abstractly yet useful for detecting toxic content.
  • Evaluation limitations: Most benchmarks, including all fairness evaluations, are in English, limiting assessment of non-English performance, biases, and non-dominant English dialects.The authors call for more robust non-English evaluations before PaLM is used in those languages.
  • Evaluation limitations: Because PaLM reflects language use through late 2021, it may perform poorly on contemporary casual language, slang, and shifting language norms that benchmarks do not assess.The paper states that these capabilities cannot be evaluated with standard benchmarks.
  • Misuse and deployment: Large language models can enable misinformation campaigns and targeted harassment, while PaLM-Coder raises unresolved concerns about suggestion correctness, robustness, safety, security, and developer confidence.These misuse risks are general to large language models, not specific to PaLM.

12 Related Work

Related work traces modern language-model progress to large-scale self-supervised training and the Transformer architecture, while surveying techniques and architectural variants that improve model scaling. It concludes that future Pathways models could combine advances from these research directions.

  • Language modeling: Large-scale language modeling uses next-token or masked-span prediction over vast corpora to advance language understanding and generation.The cited corpora include internet data, books, and forums.
  • Scaling architectures: The Transformer became the de-facto language-model architecture, while model size and total computation grew by several orders of magnitude in four years.The passage identifies BERT as an early major success of scaling for language understanding across classification tasks.
  • Distributed training: Because larger models exceed single-accelerator memory and training efficiency, researchers split tensors across accelerators or pipeline layers across stages.Other work seeks to increase model scale while limiting communication overheads.
  • Architectural variants: Retrieval models reduce model size by embedding large text collections for later access, while Mixture-of-Experts sparsity scales model size through example-dependent parameter subsets.These are presented as architectural variants for more efficient scaling.
  • Future directions: Future Pathways language models could combine improvements from these research directions.The referenced directions include retrieval, sparsity, and other efficient-attention or scaling approaches listed in the related work.

13 Open Questions in Scaling

The section identifies four scaling axes—model size, training tokens, corpus quality, and sparse capacity—and highlights unresolved trade-offs among them. It also discusses evidence that token allocation can matter substantially, while noting computational, data-repetition, and efficiency constraints.

  • Scaling factors: Four scaling axes are model depth and width, number of training tokens, training corpus quality, and increased capacity without increased compute through sparsity.The paper primarily studies model depth and width, while acknowledging that the other factors also affect quality.
  • Compute allocation: A critical open question is how smaller models trained on more tokens compare with PaLM 540B trained on 780B tokens at roughly the same total training cost.Examples include 62B parameters with 7T tokens, 120B with 3.6T tokens, and 240B with 1.8T tokens.
  • Compute allocation: 70B-parameter Chinchilla, trained on 1.4T tokens, outperforms 280B-parameter Gopher, trained on 300B tokens, by a large margin across a wide array of language tasks at similar training cost.Chinchilla and Gopher used the same training corpus and general training setup.
  • Compute allocation: Chinchilla moderately outperforms PaLM’s interpolated scaling curve on BIG-bench but slightly underperforms it on 9 English NLP tasks, while Gopher significantly underperforms both curves.The comparison covers 58 BIG-bench tasks and 9 English NLP tasks, with aggregate results plotted against total training FLOP count.
  • Data scaling: Data begins repeating in some PaLM subcorpora after 780B tokens, leaving unclear how repeated data compares with unseen data for large-scale training.Appendix ablations found no benefit from repeated data sources after 7 passes.
  • Future directions: Future work will study trade-offs among the four scaling factors and additional variables including model architecture, pre-training tasks, and optimizer configuration.Strong conclusions require experiments at scale, which carry large computational costs; smaller models may also increase wall-clock time or require larger batch sizes.

14 Conclusion … Authorship Order

PaLM demonstrates that scaling and chain-of-thought prompting drive strong few-shot language, reasoning, and multilingual capabilities, while its conclusions identify continued scaling potential and open architectural questions. The paper also documents contributions across the project lifecycle and clarifies authorship roles and ordering.

  • 14 Conclusion: PaLM 540B achieved state-of-the-art results on 28 of 29 widely evaluated English NLP tasks.The model was trained on 780B tokens of high-quality, diverse text.
  • 14 Conclusion: Few-shot reasoning performance matched or exceeded the finetuned state of the art through scale combined with chain-of-thought prompting.Chain-of-thought prompting explicitly elicits natural-language logical inference chains.
  • 14 Conclusion: Scaling improvements typically remained log-linear and had not plateaued, while some benchmarks showed discontinuous gains at larger scales.These findings suggest the scaling curve had not yet reached its apex.
  • A Contributions: The PaLM effort was organized into Preparation, Model Training, Post-training, and full-project-lifecycle contributions.The contribution record covers proposing, designing, implementing, derisking, training, evaluation, analysis, and paper writing.
  • Preparation: Preparation contributions included architecture and optimizer selection, scaling validation, training strategy and efficiency, and dataset and vocabulary development.These activities supported the model recipe and its scaling implementation.
  • Model Training: Model Training contributions covered large-scale training, including debugging instability, improving architecture and optimizers, refining training strategy, and resolving infrastructure bottlenecks.Infrastructure improvements and downstream performance validation were also assigned within this phase.
  • Post-Training: Post-training contributions included few-shot evaluation infrastructure, English, BIG-bench, reasoning, code, translation, multilingual evaluations, ethical analysis, environmental impact, and model documentation.The listed work also covered bias, contamination, memorization, explanations, serving, product management, and paper review.
  • Full Project Lifecycle; Authorship Order: Full-project-lifecycle roles included project leadership, responsible AI and safety leadership, resource management, and advising; authorship order reflected contribution levels and advisory roles.The first three authors contributed equally, the next four contributed across multiple phases, and the last five advised the overall project.

B Compute Usage and Environmental Impact

This section defines model FLOPs utilization (MFU), reports compute usage and training-efficiency comparisons, and describes PaLM-540B’s renewable-energy-backed environmental accounting. It also notes that carbon-intensity figures are based on 2021 annual grid levels and may change after review.

  • Compute Usage: MFU measures achieved token throughput relative to a system’s theoretical peak FLOPs throughput for dense Transformer language models.The theoretical peak throughput is R = P (6N + 12LHQT).
  • Compute Usage: 29.7% without attention and 30.2% with attention were reported for Megatron–Turing NLG 530B, compared with PaLM 540B’s 238.3K tokens/sec average training throughput.Megatron–Turing used A100 GPUs with 312 peak matmul TFLOP/s and achieved 65.43K tokens/second on 2240 GPUs.
  • Compute Usage: PaLM 540B uses re-materialization during training, whereas PaLM 62B and 8B do not except in self-attention.Including re-materialization, 75% of non-attention forward-pass FLOPs and all attention forward-pass FLOPs are recomputed.
  • Environmental Impact: PaLM 540B training occurred in Google’s Oklahoma datacenter, which has a PUE of 1.08 and is substantially powered by wind and other renewable sources.The paper reports net tCO2e emissions using the methodology of Patterson et al. (2021).
  • Environmental Impact: All energy use and emissions from PaLM training and the paper’s experiments were compensated with renewable energy sources.The section distinguishes this accounting from broader strategies for reducing greenhouse-gas emissions.
  • Environmental Impact: Carbon-intensity values use 2021 annual grid levels for each datacenter, and reported numbers are subject to change pending annual review.Google’s carbon-intensity estimates rely on ElectricityMap.org hourly grid estimates and life-cycle emission factors.

C Dataset Analysis

The dataset analysis documents potential personally identifying information, hierarchical topics, and associations involving gender pronouns, religions, and racial or ethnic identities. It reports detected data proportions and examines co-occurring occupations and tokens.

  • Descriptive Statistics: Addresses appeared in 1.75% of data points, emails in 0.7%, people’s names in 32%, and phone numbers in 1.5%.These detections are not necessarily personally identifying information; 10% of phone numbers were 1-800 or 1-888 numbers, and 11% of names came from the books corpus.
  • Topic Modeling: A hierarchical topic-modeling classifier was applied to characterize information contained in the dataset across its varied sources.The analysis used hierarchical rather than standard flat topic modeling.
  • Association analysis: The analysis compared word-topic associations involving gender pronouns, religions, and racial or ethnic identities with other words and occupations.Religious mentions and a subset of ethnicities and races were annotated with knowledge-graph tools, while the identity selections followed Brown et al. (2020) as closely as possible.
  • Gender pronouns: 41% of data points contained they/them pronouns, compared with 30% containing he/him pronouns and 14% containing female pronouns.The pronoun categories were defined from their corresponding sets of gendered forms.
  • Gender pronouns: Top co-occurring occupations and TF-IDF tokens were reported for each gender-pronoun set.TF-IDF values were computed after restricting the dictionary to the 50,000 most frequent tokens and retaining 20 tokens per data point, with punctuation, numbers, and URL pieces filtered out.

D Datasheet · E Model Card

The datasheet describes PaLM’s publicly sourced, modified multilingual, conversational, and code training data, along with preprocessing, splitting, and known content risks. The model card specifies PaLM’s inputs, outputs, research applications, model sizes, evaluation coverage, and the need for downstream safety and fairness assessment.

  • D Datasheet: D Datasheet: PaLM’s training data combines filtered webpages, books, Wikipedia, and public-domain social-media conversations from publicly available sources.The dataset was adapted from Du et al. (2021) and Adiwardana et al. (2020).
  • D Datasheet: D Datasheet: Dataset modifications adjusted component proportions, added deduplicated license-filtered GitHub code, and included conversation date markers for conditional generation.The proportion changes aimed to avoid repeated examples and reduce unstable training or overfitting; date conditioning can help avoid outdated facts.
  • D Datasheet: D Datasheet: The training mixture covers 124 languages, with English comprising approximately 78% of training tokens.Multilingual Wikipedia and conversation data were included to improve multilingual capabilities and increase token counts.
  • D Datasheet: D Datasheet: The dataset likely contains offensive, insulting, or threatening material because such content is prevalent on the web and potentially in old books.The model should not be used for unacceptable language-model use cases such as generating toxic speech.
  • D Datasheet: D Datasheet: Web-page boilerplate and HTML markup are removed, conversations are extracted algorithmically, and training and development sets use random splits.Boilerplate removal uses proprietary software and conversation extraction uses a special-purpose algorithm.
  • E Model Card: E Model Card: PaLM takes text as input and generates text as output, primarily supporting language-model and NLP research, fairness and safety research, and study of LLM limitations.Within Google, it is used for open-ended text and code generation research, including reasoning and code synthesis and understanding.
  • E Model Card: E Model Card: The largest PaLM model has 540 billion dense parameters, alongside 8 billion and 62 billion parameter models trained from random initialization.The model is described as standalone, with no upstream dependencies reported.
  • E Model Card: E Model Card: PaLM is evaluated on English NLP, BIG-bench, reasoning, code completion, multilingual generation and question answering, translation, and bias and toxicity tasks.The model card identifies these evaluation categories across the PaLM family.

F Training for longer … H.3 Additional Multilingual NLG results

The supplied passages describe longer PaLM 62B training with refreshed data, comparisons across model scales, and supplementary benchmark, bias, toxicity, memorization, and multilingual NLG results. They show that additional training improves PaLM 62B but does not close its gap to PaLM 540B, while scaling reduces GSM8K errors.

  • F Training for longer: Refreshed data produced much better few-shot performance than repeated subcorpora when PaLM 62B continued training.The refreshed dataset avoided training on any data more than once, motivating continued training to 1.325 trillion tokens.
  • F Training for longer: With training extended to 1.325T tokens, PaLM 62B improved across several tasks but did not bridge the gap to PaLM 540B.The comparison included MMLU, BIG-bench, GSM8K, MBPP, HumanEval, and average English NLP scores.
  • G.1 Reasoning: Scaling from 62B to 540B parameters substantially reduced PaLM’s GSM8K errors.The examples include corrections of semantic-understanding errors and one-step-missing errors.
  • G.2 Bias and Toxicity: The supplied bias and toxicity analyses examine gender, race, and religion associations alongside human-versus-model continuations and toxicity probabilities.The tables report descriptive-word distributions and continuation examples for PaLM 540B.
  • G.3 Memorization: The memorization analysis presents hand-picked PaLM 540B training examples that exactly reproduced the true continuation.These examples showcase types of text memorized by the model.
  • H.1 English NLP tasks on smaller models: Across 29 NLP benchmarks, supplementary results compare PaLM 8B, 62B, and 540B, including 62B evaluations at 795B and 1325B tokens.Few-shot results generally use 5-shot evaluation unless otherwise specified, while the 62B model uses 1-shot results in the listed training conditions.
  • H.2 Additional BIG-bench results: Additional BIG-bench results report PaLM 8B, 62B, and 540B performance on 24 BIG-bench Lite tasks and preferred metrics for tasks favoring either PaLM 540B or average humans.The Lite results use 5-shot evaluation, while the top-task tables provide raw and normalized preferred metrics.
  • H.3 Additional Multilingual NLG results: Additional multilingual NLG results expand GEM evaluations with granular ROUGE-2, granular ROUGE-L, and BLEURT-20 scores.The ROUGE tables separate precision, recall, and f-measure for data-to-text and summarization datasets.
Loading 2204.02311v5…