PaLM: Scaling Language Modeling with Pathways

Efficient scaling

• PaLM is a 540B-parameter densely activated autoregressive Transformer trained on 780B tokens using Pathways across 6144 TPU v4 chips.Pathways enabled highly efficient training across multiple TPU v4 Pods, extending prior approaches based on model, data, dataset-quality, and sparse-capacity scaling.Source: §1 · p. 3

Continued improvements from scaling

• PaLM 540B achieves few-shot state-of-the-art results on 28 of 29 widely evaluated English language understanding benchmarks, often by significant margins.The comparisons use the best per-task results from GLaM, GPT-3, Megatron–Turing NLG, Gopher, Chinchilla, and LaMDA.Source: §1 · pp. 3–4

Breakthrough capabilities

• Combining model scaling with chain-of-thought prompting lets simple few-shot evaluation outperform or match finetuned state of the art across many multi-step reasoning tasks.PaLM also shows breakthrough performance on BIG-bench’s 150+ language understanding and generation tasks and can explicitly interpret and explain complex reasoning chains.Source: §1 · p. 4, Figure 1 · p. 4

Discontinuous improvements

• Scaling from 62B to 540B produces drastic accuracy jumps on roughly 25% of BIG-bench tasks, indicating discontinuous capability improvements beyond previously studied scales.Results compare 8B, 62B, and 540B models; typical scaling follows the observed power-law pattern, but some tasks improve sharply only at the largest scale.Source: §1 · p. 4

Multilingual understanding

• With approximately 22% non-English training data, PaLM 540B bridges prior finetuned state-of-the-art performance in non-English summarization and exceeds prior state of the art in translation.The evaluation covers multilingual machine translation, summarization, and question answering across many languages.Source: §1 · p. 4

Bias and toxicity

• PaLM 540B improves Winogender accuracy with scale and reaches a new state of the art, while stereotype associations persist and toxicity is slightly higher for 62B and 540B than 8B.Generated toxicity correlates highly with prompt toxicity, unlike human continuations; the authors propose broader multilingual and risk evaluations.Source: §1 · p. 5

2 Model Architecture

• PaLM uses a standard decoder-only Transformer architecture (Vaswani et al., 2017) with SwiGLU, parallel layers, multi-query attention, RoPE, shared embeddings, and no biases.These modifications target model quality, training speed, autoregressive decoding cost, long-sequence performance, and large-model training stability.Source: §2 · p. 5
• SwiGLU activations improve quality over standard ReLU, GeLU, or Swish in compute-equivalent experiments, despite requiring three rather than two MLP matrix multiplications (Shazeer, 2020).The compute-equivalent comparison gives standard ReLU proportionally larger dimensions.Source: §2 · p. 5
• Parallel layers fuse MLP and attention input projections, yielding roughly 15% faster large-scale training, with no observed quality degradation at 62B and extrapolated neutrality at 540B.Ablations found a small quality degradation at 8B but none at 62B; the formulation follows Wang & Komatsuzaki (2021).Source: §2 · p. 5
• Multi-query attention shares key/value projections across heads while retaining per-head queries, preserving quality and training speed while significantly reducing autoregressive decoding cost (Shazeer, 2019).Sharing addresses poor accelerator efficiency in standard multi-headed attention when decoding one token at a time.Source: §2 · p. 5
• PaLM uses a lossless, reversible 256k-token SentencePiece vocabulary (Kudo & Richardson, 2018a) generated from training data to support many languages efficiently while preserving whitespace and Unicode coverage.Numbers are split into individual digit tokens, and out-of-vocabulary Unicode characters are represented as UTF-8 byte tokens, supporting code and exact reconstruction.Source: §2 · p. 6

2.1 Model Scale Hyperparameters

• The study compares dense Transformer models with 540B, 62B, and 8B parameters, whose FLOPs per token approximately equal parameter count because they are standard dense architectures.All three models use the same data and vocabulary and are trained identically except for batch size; their hyperparameters are listed in Table 1.Source: §2.1 · p. 6, Table 1 · p. 6

2.2 Model Card

• The PaLM Model Card summarizes the model’s architecture, training setup, training data, and intended usage (Mitchell et al., 2019).It is provided in Appendix E.Source: §2.2 · p. 6

Dataset composition

• The 780-billion-token pretraining corpus combines filtered webpages, books, Wikipedia, news, source code, and social media conversations, with proportions chosen to avoid repeating data.All three models use exactly one identically shuffled epoch of the data; webpages are sampled proportional to quality scores rather than fully filtered by a threshold.Source: §3 · p. 6

Code data

• The code component contains 196GB from open-source GitHub repositories, restricted to files in 24 common programming languages and excluding copyleft-licensed files.The languages include Java, HTML, Javascript, Python, PHP, C#, XML, C++, and C.Source: §3 · pp. 6–7
• Duplicate code files are removed using Levenshtein distance because duplicate files are common in source-code repositories.This preprocessing targets redundancy in the GitHub-derived code corpus.Source: §3 · p. 7

Dataset composition

• The final training mixture includes text from over 100 languages, with source proportions documented in Table 2 and language distributions in Appendix Table 29.The paper also directs readers to Section 8 for contamination checks and Appendix D for a broader dataset datasheet.Source: Table 2 · p. 7, §3 · p. 7

4 Training Infrastructure

• PaLM 540B trains on 6144 TPU v4 chips using 12-way model parallelism and 256-way fully sharded data parallelism without pipeline parallelism.Each pod contains a full model copy, with weights partitioned across 3072 chips; activations are selectively saved and rematerialized during backpropagation.Source: §4 · pp. 7–8
• Pathways implements two-way pod-level data parallelism by dispatching batch halves, exchanging gradients, and applying identical updates on both pods.Its sharded-dataflow execution and asynchronous gang-scheduling mask dispatch latency and amortize data-transfer management across thousands of chips.Source: §4 · p. 8, Figure 2 · p. 8
• 1.95x single-pod throughput, equivalent to 97% perfect weak scaling, is achieved across two TPU v4 Pods.Each training step transfers approximately 1.3 GB per host pair, producing an aggregate 81 Tbps burst; chunked routing over diverse links mitigates congestion.Source: §4 · p. 8

4.1 Training Efficiency

• MFU measures observed tokens-per-second against peak throughput for required forward–backward operations, excluding rematerialization, enabling implementation-independent comparisons.The paper argues HFU is inconsistent because hardware FLOP counts depend on system, implementation, compiler choices, rematerialization, and measurement methodology.Source: §4.1 · p. 9
• 45.7% MFU without self-attention and 46.2% with it, versus 21.3% for GPT-3, 32.5% for Gopher, and 29.7–30.2% for Megatron–Turing NLG 530B.PaLM 540B reaches 238.3K tokens/sec at batch size 2048; its hardware FLOPs utilization is 57.8%.Source: Table 3 · p. 9, §4.1 · p. 9

5 Training Setup

• Kernel weights used fan-in variance scaling, embeddings used unit-variance initialization, and shared embeddings required scaling pre-softmax logits by 1/√n.Source: §5 · p. 10
• Adafactor without factorization used parameter scaling, with a 10^-2 initial learning rate, inverse-square-root decay, β1 = 0.9, adaptive β2, gradient clipping at 1.0, and dynamic weight decay.The adaptive β2 schedule was chosen for greater stability than β2 = 0.99 because rare embedding tokens can have poorly estimated second moments over shorter windows.Source: §5 · p. 10
• Training used a standard token-averaged language-modeling loss plus z loss, 2048-token sequences, no dropout, and a staged batch schedule reaching 2048 sequences, or 4M tokens.The batch schedule favored smaller batches early for sample efficiency and larger batches later for gradient quality and TPU efficiency; finetuning generally used dropout of 0.1.Source: §5 · p. 10

5.1 Training Instability

• The largest model exhibited roughly 20 irregular loss spikes despite gradient clipping, whereas smaller models did not, and no principled mitigation strategy was identified.Source: §5.1 · p. 11
• Restarting about 100 steps before a spike and skipping 200–500 batches prevented recurrence, indicating spikes depended on both the data batches and the model parameter state.Using the same surrounding batches from an earlier checkpoint did not reproduce the spike, arguing against bad data alone as its cause.Source: §5.1 · p. 11

6.1 English NLP tasks

• PaLM 540B improves average NLU and NLG scores by more than 5 points, with category averages increasing with scale and PaLM 62B exceeding GPT-3 175B in both.Table 5 averages eight NLG benchmarks and the remaining benchmarks as NLU tasks under 1-shot evaluation.Source: §6.1 · p. 12, Table 5 · p. 13
• PaLM 540B outperforms prior SOTA on 24 of 29 tasks 1-shot and 28 of 29 few-shot, exceeding Megatron-Turing NLG 530B on all benchmarks.It surpasses prior SOTA by more than 10 points few-shot on some Reading Comprehension and NLI tasks, implicating pretraining data, training strategy, and token count alongside scale.Source: §6.1 · p. 11, Table 4 · p. 12

6.1.1 Massive Multitask Language Understanding

• PaLM 540B improves MMLU’s average score by ≈2 points and outperforms Chinchilla in every category except Other tasks.MMLU covers 57 multiple-choice tasks spanning mathematics, history, law, and other topics, evaluated 5-shot on test sets.Source: §6.1.1 · p. 13, Table 6 · p. 13

6.1.2 Finetuning

• Finetuned PaLM achieves competitive close-to-SOTA SuperGLUE performance and widely outperforms the best decoder-only autoregressive leaderboard model.Scale helps bridge the gap with encoder-decoder models, although finetuned results remain significantly above few-shot results.Source: §6.1.2 · p. 13, Table 9 · p. 14

6.2 BIG-bench

• On 58 common BIG-bench tasks, PaLM significantly outperforms GPT-3, Gopher, and Chinchilla; PaLM 540B 5-shot beats average human performance and prior SOTA on 44 tasks.Performance appears log-linear with scale, while the 150-task textual collection shows similar characteristics.Source: §6.2 · pp. 14–15, Figure 3 · p. 15
• BIG-bench reveals discontinuous scaling: 25% of tasks exceed +10% discontinuity and 15% exceed +20%, while PaLM 540B reaches 87% on English proverbs versus 25% for PaLM 62B.Other tasks show limitations: PaLM 540B remains far below best human performance on navigate and mathematical induction, and average humans outperform it on 35% of tasks.Source: §6.2 · pp. 16–17

6.3 Reasoning

• The evaluation includes reasoning tasks requiring multi-step arithmetic or commonsense logical reasoning.The supplied passage introduces this subsection but does not provide its results.Source: §6.3 · p. 20
• The reasoning evaluation covers arithmetic tasks requiring natural-language-to-equation transformations and commonsense tasks requiring chained world-knowledge inferences.Chain-of-thought exemplars contain manually written intermediate steps, after which the model generates reasoning for test examples while only final answers are scored.Source: §6.3 · p. 20

6.3.1 Results

• On GSM8K, PaLM 540B achieves 58% with 8-shot chain-of-thought prompting and an external calculator, outperforming Cobbe et al. (2021)’s 55% prior SOTA.Scaling from PaLM 62B to 540B fixed a large amount of reasoning errors across semantic understanding, one-step missing, and other categories.Source: §6.3.1 · p. 21, Figure 9 · p. 21
• PaLM 540B with 8-shot chain-of-thought achieves state-of-the-art accuracy on four of seven reasoning benchmarks and near-state-of-the-art accuracy on the other three.Scaling and chain-of-thought prompting both substantially improve performance; without them, PaLM would achieve state of the art on only SVAMP.Source: §6.3.1 · pp. 21–22, Figure 10 · p. 22

6.4 Code Tasks

• PaLM outperforms LaMDA across all code tasks, matches Codex 12B on HumanEval, and achieves strong code and language performance in one non-code-specific model using far less Python training data.PaLM used approximately 2.7B Python tokens versus Codex’s reported 100B, with the authors attributing the result to transfer across programming languages and natural language plus scale-related sample efficiency.Source: §6.4 · p. 24, Table 12 · p. 25
• PaLM-Coder 540B reaches 88.4% pass@100 on HumanEval, 80.8% pass@80 on MBPP, and an 82.1% DeepFix compile rate versus 71.7% from prior work.Relative to non-finetuned PaLM, it improves HumanEval pass@100 by 12 absolute points and MBPP pass@80 by 5 points; performance improves at every tested scale through 540B.Source: §6.4 · pp. 25–26, Table 12 · p. 25, Figure 12 · p. 25
• On DeepFix, PaLM produces the smallest edits, whereas PaLM-Coder has the highest success rate under small normalized edit distances and prompt changes raise compile rate from 82.1% to 83.4%.The revised prompts also increase success for at most five changed lines from 66.8% to 70.2%, while functional correctness remains potentially overestimated by limited tests.Source: §6.4 · pp. 26–27, Table 13 · p. 27

6.5 Translation

• The translation evaluation considers English-centric pairs across high-, mid-, and low-resource regimes, direct French–German translation, and extremely low-resource English–Kazakh translation.PaLM was not explicitly trained on parallel text; French and German have roughly 24B and 26B training tokens, compared with approximately 134M for Kazakh.Source: §6.5 · p. 28
• PaLM outperforms translation baselines by up to 13 BLEU on English-centric language pairs and surpasses supervised baselines for German-English and Romanian-English, though it only matches supervised French-German performance in extremely low-resource translation.On German-French and Kazakh-English, PaLM still provides strong performance despite the more challenging setting.Source: Table 14 · p. 28, §6.5 · pp. 29–30, Table 15 · p. 29
• Scaling PaLM from 62B to 540B produces drastic translation gains, including 13 BLEU for English-German and 17 BLEU for English-French, beyond the projected power-law trend.The results compare 0-shot translation across 8B, 62B, and 540B models.Source: §6.5 · p. 29, Figure 15 · p. 29

6.5.1 Further findings and analysis

• Translation quality is generally better into English than out of English, while language-name prompts often outperform 1-shot or few-shot exemplars and large self-supervised models can match smaller specialized systems.The authors suggest increasing multilingual pretraining data may mitigate directionality and non-English generation weaknesses; most dedicated baselines are under 1B parameters.Source: §6.5.1 · p. 30

6.6.1 Results

• Finetuned PaLM 540B closely matches or exceeds prior results on all English summarization tasks, while achieving new finetuning state of the art on 4 of 6 summarization tasks that generate English text.Non-English MLSum summarization does not reach state of the art, and few-shot performance remains farther from finetuning for non-English generation.Source: Table 16 · p. 32, §6.6.1 · p. 32

6.7 Multilingual Question Answering

• PaLM 540B achieves competitive multilingual question-answering performance, outperforming mT5 XXL but remaining below ByT5 XXL, while few-shot quality has a large average gap from finetuning that is smaller for Swahili and Finnish.PaLM uses approximately 22% non-English training data, compared with 6× and 1.5× as much for mT5 and ByT5, respectively.Source: Table 17 · p. 33, §6.7 · p. 33

6.8 Analysis

• Few-shot performance usually improves with more examples and model scale, but Trivia QA peaks at 1 shot, while Web Questions shows substantial checkpoint variance and can worsen after an earlier state-of-the-art checkpoint.Across the evaluated tasks, the 540B model achieves the best performance across model sizes; StoryCloze and Natural Questions vary relatively little across checkpoints.Source: Figure 16 · p. 34, §6.8 · p. 34, Figure 17 · p. 34

Evaluation method

• The evaluation sampled 100 training sequences, prompted each with 50 tokens, and used greedy decoding to test exact 50-token continuations.This methodology follows Carlini et al. (2022), who used prompt lengths from 50 to 500 tokens.Source: §7 · p. 35

Memorization rates

• PaLM reproduced exact 50-token continuations in 1.6% of data for the 8B model and 2.4% for the 540B model.Held-out memorization rates exceeded 0% because some examples closely resembled training examples, such as boilerplate licenses differing only in year.Source: §7 · p. 35, Figure 18 · p. 35

Duplication effects

• Memorization strongly correlates with duplication: examples seen once had a 0.75% rate, whereas examples seen over 500 times exceeded 40% for the largest model.Full-document deduplication still permits repeated 100-token spans.Source: §7 · p. 35, Figure 18 · p. 35

Corpus differences

• Exact duplication, near duplication, and templating differentiate memorization across corpora: code contains boilerplate and copied snippets, whereas books primarily contain unique text.The corpus breakdown is presented in Figure 18(c).Source: §7 · p. 35, Figure 18 · p. 35

Scale effects

• Larger models memorize more, with PaLM’s log-linear fit having slope 0.002 and R2 = 0.976, closely matching Carlini et al. (2022)’s GPT-Neo slope and R2.Carlini et al. (2022) reported slope 0.002 with R2 = 0.965 for GPT-Neo.Source: §7 · pp. 35–36

Risks and mitigation

• Most memorized instances were formulaic, but stories, news, and facts also appeared; problematicness depends on dataset contents and application consequences, not extractability alone.Extractable content also depends on model size and attackers’ prompt knowledge; longer prompts can reveal more memorization. Bloom filters could block verbatim training sequences, but not near-matches.Source: §7 · p. 36

English NLP benchmarks

• The analysis classified 29 English benchmark tasks using overlap statistics, manual example inspection, and knowledge of dataset construction, identifying wholesale or web-extracted datasets as contaminated but web-context-only datasets as clean.Ten of the 29 tasks fell into the first two contamination categories, though only a portion of their evaluation examples appeared in training; examples include SQuADv2 and Winograd, Web Questions, ReCoRD, and Lambada, versus BoolQ, Multirc, and ANLI.Source: §8 · pp. 36–37
• The contamination analysis used a 70% 8-gram-seen threshold to divide partially contaminated datasets into clean and contaminated subsets, reporting clean-versus-full results in Table 18.The threshold was applied to the question, prompt, or target, and the training corpus contained about 200 billion sampled or filtered English-web words versus many trillions of web tokens.Source: §8 · p. 37, Table 18 · p. 37
• Equal positive and negative accuracy deltas on clean versus full English benchmark subsets, with similar negative-delta counts for 8B and 540B, provide little evidence that contamination inflated results.The comparison is not perfectly apples-to-apples because subsets differ and some test sets are small; greater memorization by 540B would otherwise predict more negative deltas.Source: §8 · p. 37, Table 18 · p. 37

Machine translation

• Five of six machine-translation datasets had very similar BLEU scores on clean and full sets, while De-En was roughly 3.0 BLEU worse on the clean set consistently across model scales.Clean subsets removed examples whose target reference sentences had high n-gram overlap with training; the analysis found no wholesale contamination.Source: §8 · p. 37, Table 19 · p. 38

• The explanations are motivated as potentially useful for judging whether predictions rely on correct reasons, calibrating user confidence, or directly providing the desired output.The authors distinguish these uses from the accuracy gains previously demonstrated for chain-of-thought prompting.Source: §9 · p. 38
• The section demonstrates explanatory generation on “Explaining a Joke” and “Logical Inference,” requiring chained reasoning, world knowledge, abstract language understanding, and lexicographic understanding.Each task used 2-shot exemplars specifying the desired output style.Source: §9 · p. 38
• The demonstrations used fixed author-written 2-shot prompts and greedy decoding, with prompts prepared before evaluation and outputs representing canonical 1-best predictions.The authors argue these choices reduce content overlap, output cherry-picking, and the likelihood that fully correct explanations arise from lucky guesses or memorization.Source: §9 · p. 38
• PaLM 540B produced coherent explanations of jokes involving wordplay, literal interpretations, and reversal of expectations.The showcased outputs explain why kleptomaniacs “take things literally” and why a pessimist will not expect borrowed money back.Source: §9 · p. 38, Figure 19 · p. 39

10.1.1 Gender and occupation bias

• PaLM 540B achieves 79.4% 1-shot Winogender accuracy with multiple-choice scoring and 69.7% with generative scoring, rising to 84.7% generative accuracy at 4 shots, but remains below human performance of 95.9%.The model sets a new state-of-the-art in 1-shot and few-shot settings, although task-adapted models and humans perform better.Source: §10.1.1 · p. 40, Figure 20 · p. 41
• Winogender accuracy is higher on stereotypical than gotcha examples, with the lowest accuracy on female-gender gotchas; the gap changes from 14.1 to 10.1 points at 1 shot and 18.3 to 9.2 at 4 shots.The stereotypical–gotcha gap measures reliance on occupation-gender statistical shortcuts, while pronoun frequencies may partly explain slice differences.Source: §10.1.1 · p. 41

10.1.2 Toxicity and bias

• Identity co-occurrences reveal charged associations, including terrorist, violent, and radical for Islam and potentially violent terms for Latinx, while small prompt changes can drastically alter outcomes.The analysis uses 800 top-k samples per prompt and descriptive-word counts from the first full sentence; toxicity scores are higher for Islam and Atheism prompts.Source: §10.1.2 · pp. 42–43

10.2 Toxicity in open-ended generation

• TPC increases with prompt toxicity and is generally below prompt and human toxicity, but 62B and 540B have similar profiles; model continuations follow prompt toxicity more consistently than human continuations.TPC rises visibly from 8B to the larger models, while reported toxicity is constrained to first complete sentences and therefore is not directly comparable across continuation lengths.Source: Figure 23 · p. 44, §10.2 · p. 44

10.3 Limitations

• The evaluations are limited to English and selected proxy tasks, lack standardized and comprehensive fairness coverage, and may not predict downstream impacts across applications, cultures, or fine-tuned systems.The authors recommend developing benchmarks for other languages and sociocultural contexts and assessing fairness gaps in the intended application before deployment.Source: §10.3 · p. 45

Bias and toxicity

• PaLM’s training data and outputs reflect social stereotypes and toxicity associations, while removing them is non-trivial and requires contextualized fairness evaluations and protections.Automated toxicity filtering may disproportionately exclude content about or authored by marginalized subgroups (Dodge et al., 2021).Source: §11 · p. 46

Fairness limitations

• Fairness analyses cover only English gender, race, ethnicity, and religion biases, so evaluations must address application-relevant disparities and sociocultural contexts.Untested areas include sexual orientation, disabilities (Hutchinson et al., 2020), and caste in non-Western contexts (Sambasivan et al., 2021).Source: §11 · p. 46

Downstream use

• Downstream harms depend on model use, so content undesirable in general may be necessary for applications such as toxic-content detection (Blodgett et al., 2020; Hutchinson et al., 2020).The section argues that deployment context should guide harm assessment and mitigation.Source: §11 · p. 46

Evaluation limitations

• Benchmark scores may not capture full language complexity or generalize across real-world contexts because training data and benchmark construct validity are limited (Raji et al., 2021).The evaluations assess scientific progress but leave gaps between intended and measured capabilities.Source: §11 · p. 46

Language coverage

• Most benchmarks and all fairness evaluations are English-only, while filtering and dated training data may limit dialectal, multilingual, casual-language, and contemporary-slang capabilities.The training data spans texts through late 2021, and standard benchmarks do not assess these dimensions adequately.Source: §11 · p. 46

Malicious and software-development risks

• Large language models also risk misinformation and targeted harassment, requiring scalable guardrails, while PaLM-Coder must produce suggestions that are correct, robust, safe, and secure.Developers must also be confident that PaLM-Coder suggestions have these properties.Source: §11 · p. 47

Scaling language modeling

• Self-supervised language modeling on large corpora has produced advanced understanding and generation, with predictable power-law improvements from scaling data, parameters, and computation (Devlin et al., 2019; Raffel et al., 2020; Kaplan et al., 2020).Objectives include next-token and masked-span prediction over web, book, and forum data.Source: §12 · p. 47

Related model advances

• Transformers became the dominant efficient architecture, supporting progression from BERT and GPT to 175B GPT-3, larger dense and sparse models, code and dialogue advances, and instruction-following improvements (Vaswani et al., 2017; Brown et al., 2020).The cited progression includes 178B Jurassic-1, 280B Gopher, 530B Megatron-Turing NLG, trillion-parameter sparse models, and 137B LaMDA.Source: §12 · p. 47

Distributed scaling

• Because frontier models exceed single-accelerator memory, prior work splits tensors or layers across accelerators and reduces communication overhead; PaLM combines data and model parallelism through Pathways (Shazeer et al., 2018; Huang et al., 2019; Barham et al., 2022).Other approaches target scalable distributed training through communication-efficient parallelism.Source: §12 · p. 47

Efficient architectural variants

• Architectural alternatives improve scaling through retrieval, Mixture-of-Experts parameter sparsity, and sequence-length sparsity, with future Pathways models potentially combining these research directions (Borgeaud et al., 2021; Shazeer et al., 2017).Retrieval reduces model size by externalizing text, while sparsity supports larger models or efficient training on extremely long sequences.Source: §12 · pp. 47–48

Data scaling

• Data quality is a major scaling factor: careful filtering produces extremely significant few-shot gains, while PaLM’s repeated data after 780B tokens showed no benefit for PaLM 62B.Training longer on a refreshed dataset did improve performance, but the value of repeated versus unseen data remains unclear.Source: §13 · pp. 48–49

Future work

• Future work will study trade-offs among the four scaling factors and additional variables including architecture, pre-training tasks, and optimizer configuration.The four factors are model depth and width, number of training tokens, corpus quality, and increased capacity through sparse models.Source: §13 · pp. 48–49

Open scaling questions

• The key unresolved question is how models with different parameter–token allocations compare with PaLM 540B when total training cost is held roughly constant.Examples include 62B parameters trained on 7T tokens, 120B on 3.6T tokens, and 240B on 1.8T tokens; such comparisons could reveal whether smaller models achieve comparable downstream performance at lower inference cost.Source: §13 · p. 48

Scaling comparisons

• A 70B-parameter Chinchilla trained on 1.4T tokens outperforms the 280B-parameter Gopher trained on 300B tokens by a large margin across many language tasks at similar training cost.The models used the same training corpus and general training setup, supporting the conclusion that Gopher was under-trained for its size.Source: §13 · p. 48

Experimental constraints

• Large-scale size–token experiments are difficult because they require substantial computation, may increase wall-clock time or batch-size demands, and could reduce TPU efficiency.PaLM 540B already uses a 4M-token batch, and maintaining sample efficiency with larger batches is unclear.Source: §13 · pp. 48–49

Scaling comparisons

• Chinchilla moderately outperforms PaLM’s interpolated scaling curve on BIG-bench but slightly underperforms it on nine English NLP tasks, while Gopher significantly underperforms both.Figure 24 aggregates performance by total training FLOP count; PaLM, Chinchilla, and Gopher differ in training corpus, model size, and token count, complicating causal conclusions.Source: §13 · p. 48, Figure 24 · p. 49

14 Conclusion

• PaLM 540B achieves state-of-the-art results on 28 of 29 widely evaluated English NLP tasks, exceeds average human performance on BIG-bench, and attains additional state-of-the-art results in code, multilingual NLP, and translation.The model was trained on 780B tokens as a 540B-parameter dense Transformer.Source: §14 · p. 50
• Few-shot reasoning results match or exceed the finetuned state of the art because scale combined with chain-of-thought prompting improves multi-step inference and prediction quality.Explicit inference chains also help explain predictions, jokes, and complex scenarios.Source: §14 · p. 50
• Scaling improvements are typically log-linear but can become discontinuous at 540B, suggesting some capabilities emerge only at sufficient scale and that future generations may reveal more.Comparisons use identically trained 8B, 62B, and 540B variants.Source: §14 · p. 50

A Contributions

• The project contributions covered proposal and preparation, model architecture and optimization, scaling validation, training efficiency, dataset and vocabulary development, and infrastructure implementation.Preparation included selecting the model and optimizer, validating scaling, improving training strategy, and building supporting systems.Source: A Contributions · p. 64, Preparation · p. 64

Model Training

• Model Training contributions included large-scale training, infrastructure improvements, and validation of model performance on downstream tasks.Large-scale training involved debugging instability, improving architecture and optimization, refining training strategy, and resolving infrastructure bottlenecks.Source: Model Training · p. 64

Post-Training

• Post-Training contributions covered evaluations and analyses spanning English NLP, BIG-bench, reasoning, code, translation, multilingual tasks, bias, contamination, memorization, explanations, ethics, compute impact, serving, and documentation.The section also records paper writing and reviewing, with all authors contributing to writing and reviewing.Source: Post-Training · pp. 64–65

Full Project Lifecycle

• Full-project-lifecycle roles included overall project leadership, responsible AI and safety leadership, resource management, and advising.These responsibilities were assigned across the project lifecycle.Source: Full Project Lifecycle · p. 65

Authorship Order

• The first three authors made equal contributions, the next four contributed significantly across multiple phases, the last five advised the overall project, and remaining authors were listed randomly.This defines the stated authorship-order convention.Source: Authorship Order · p. 65

MFU calculation

• MFU is achieved throughput divided by theoretical peak throughput, with dense Transformer computation requiring 6N non-attention matmul FLOPs per token plus 6LH(2QT) attention FLOPs.For accelerator peak throughput P, the theoretical token throughput is R = P (6N + 12LHQT).Source: B Compute Usage and Environmental Impact · p. 66

MFU comparison

• 57.8% analytically computed hardware FLOPs utilization was achieved by PaLM 540B, compared with 29.7% for Megatron–Turing NLG 530B without attention.PaLM’s utilization was 45.7% without rematerialization costs and 46.2% including self-attention; Megatron–Turing NLG reached 30.2% including attention.Source: B Compute Usage and Environmental Impact · p. 66

Compute usage

• PaLM’s compute-usage accounting covers training PaLM 8B and 540B to 780 billion tokens and PaLM 62B to 795B tokens, with and without rematerialization FLOPs.The 540B model uses rematerialization, recomputing 75% of non-attention forward FLOPs and all attention forward FLOPs during the backward pass; development and other inefficiencies are excluded.Source: B Compute Usage and Environmental Impact · p. 66, Table 22 · p. 66

Environmental impact

• 271.43 tCO2e were emitted while training PaLM 540B, versus estimated emissions of 552 tCO2e for GPT-3 and 180 tCO2e for one passenger-jet round trip.Training used 6144 TPU v4 chips for 1200 hours and 3072 chips for 336 hours in a datacenter operating on 89% carbon-free energy.Source: B Compute Usage and Environmental Impact · p. 66
• All energy use and emissions from PaLM training and the paper’s experiments were compensated with renewable energy sources.The reported carbon-intensity values use 2021 annual grid-level estimates and may change pending annual review.Source: B Compute Usage and Environmental Impact · p. 66

Descriptive Statistics

• 32% of data points contain people’s names, compared with 1.75% containing addresses, 1.5% phone numbers, and 0.7% emails, although these are not necessarily PII.For example, 10% of phone numbers are 1-800 or 1-888 numbers, and 11% of names come from the books corpus.Source: C Dataset Analysis · p. 67

Topic Modeling

• Hierarchical topic modeling identifies online image galleries, how-to and expert content, computer-game references, photographic and digital arts, and visual art and design among the dataset’s top categories.The hierarchy provides both a high-level overview and lower-level details of specific sub-classifications; its full structure appears in Figure 25.Source: C Dataset Analysis · p. 67, Figure 25 · p. 67

Association analysis

• The association analysis compares word and topic distributions linked to gender pronouns, religions, and racial or ethnic identities.Religion and race or ethnicity annotations use a post-hoc knowledge-graph method and TF-IDF co-occurrence tables, with identity selections intended to match Brown et al. (2020), while acknowledging that the lists do not capture identity complexity or fluidity.Source: C Dataset Analysis · pp. 67–68, Table 25 · p. 69, Table 26 · p. 69

Gender pronouns

• 41% of data points contain they/them pronouns, compared with 30% containing he/him and 14% containing she/her pronouns.The analysis also examines occupations and other TF-IDF tokens co-occurring with each pronoun set, reported in Tables 23 and 24.Source: C Dataset Analysis · p. 68, Table 23 · p. 68, Table 24 · p. 68

D Datasheet

• PaLM’s training data combines filtered webpages, books, Wikipedia, and public-domain social-media conversations, following Du et al. (2021) and Adiwardana et al. (2020).The data was collected from publicly available sources.Source: D Datasheet · pp. 70–71
• Dataset modifications adjusted component proportions, added deduplicated license-filtered GitHub code, and inserted conversation date markers for date-conditioned generation.The proportion adjustments aimed to reduce repeated examples and risks of unstable training or overfitting; date conditioning helps avoid outdated facts.Source: D Datasheet · p. 70
• The training mixture covers 124 languages, with English comprising approximately 78% of training tokens, and includes multilingual Wikipedia and conversation data.Language proportions are reported in Figure 29.Source: D Datasheet · p. 70
• The dataset likely contains offensive, insulting, or threatening material because such content is prevalent on the web and potentially in older books.The model should not be used for unacceptable language-model applications such as generating toxic speech.Source: D Datasheet · p. 71
• Preprocessing removes webpage boilerplate and HTML markup with proprietary software and extracts conversations using a special-purpose algorithm.The datasheet presents these details in Table 27, following Gebru et al. (2021).Source: D Datasheet · pp. 69, 71

E Model Card

• PaLM takes text as input and generates text as output, with primary applications in language-model, NLP, fairness, safety, and limitation research.Within Google, it is also used for open-ended text and code generation, including reasoning and code synthesis and understanding.Source: E Model Card · p. 74
• The largest PaLM model has 540 billion dense parameters, alongside 8-billion- and 62-billion-parameter models trained from random initialization.The model card presents these specifications following Mitchell et al. (2019).Source: E Model Card · p. 74
• PaLM is evaluated on English NLP, BIG-bench, reasoning, code, multilingual, translation, bias, and toxicity tasks, including benchmarks by BIG-bench collaboration, Rudinger et al. (2018), and Gehman et al. (2020).The model card notes safety risks in harmful or deceitful language-generation settings, and recommends downstream rather than pretrained-level mitigation.Source: E Model Card · pp. 74–75

F Training for longer

• 67% more training FLOPs substantially improve PaLM 62B across English NLP, MMLU, BIG-bench, GSM8K, MBPP, and HumanEval, but do not close the gap to PaLM 540B.Longer training also significantly reduces the 1-shot gap between PaLM 62B and 540B on nearly all of 29 English NLP benchmarks.Source: F Training for longer · p. 76, Table 32 · p. 77, Figure 27 · p. 77
• Refreshed data outperforms repeated subcorpora for continued PaLM 62B training, motivating a 1.325-trillion-token mixture with adjusted multilingual, code, book, web, and newly added arXiv and math data.The refreshed mixture uses 40% social media, 27% filtered multilingual webpages, 13% books, 10% GitHub, 4% Wikipedia, 1% News, and 5% new arXiv and math webpages.Source: F Training for longer · p. 76, Table 33 · p. 78

G.1 Reasoning

• Scaling to 540B parameters substantially reduces GSM8K errors made by PaLM 62B, correcting semantic-understanding and missing-one-step errors.Figures 28 and 29 provide examples of both error types corrected by PaLM 540B.Source: §G.1 · p. 78, Figure 28 · p. 78, Figure 29 · p. 79

G.2 Bias and Toxicity

• Bias and toxicity analyses examine PaLM 540B using gender, race, religion, and human-versus-model continuation prompts.The supplied tables report descriptive-word distributions and toxicity probabilities for these prompt categories.Source: Table 34 · p. 79, Table 35 · p. 79, Table 36 · p. 80

G.3 Memorization

• Memorization analysis presents hand-picked training examples that PaLM 540B exactly reproduced as true continuations.The examples illustrate types of text that the model memorizes.Source: Table 39 · p. 82

H.1 English NLP tasks on smaller models

• Additional English NLP results compare PaLM 8B, 62B, and 540B across 29 benchmarks, including 1-shot PaLM 62B results before and after longer training.The table reports mostly accuracy with 5-shot evaluation unless otherwise specified.Source: Table 40 · p. 83

H.2 Additional BIG-bench results

• Additional BIG-bench results report PaLM 8B, 62B, and 540B performance on 24 BIG-bench Lite tasks and preferred-metric scores for tasks where PaLM 540B exceeds or falls below average human performance.BIG-bench Lite uses 5-shot evaluation, while the top-ten task tables provide raw and normalized preferred metrics.Source: Table 41 · p. 84, Table 42 · p. 85, Table 43 · p. 85

H.3 Additional Multilingual NLG results

• Additional multilingual NLG results expand GEM evaluations with granular ROUGE-2, ROUGE-L, and BLEURT-20 scores for data-to-text and summarization datasets.ROUGE results are split into precision, recall, and f-measure.Source: Table 44 · p. 86, Table 45 · p. 86, Table 46 · p. 87