PaLM 2 Technical Report

1 Introduction

• For a given training-compute budget, PaLM 2 scales data and model size roughly 1:1, validating compute-optimal scaling at larger compute scales.This contrasts with past trends that scaled model size three times faster than dataset size.Source: §1 · p. 3
• PaLM 2 uses a multilingual, diverse pre-training mixture spanning hundreds of languages and domains, deduplication, and a tuned mixture of pre-training objectives.The report says larger models can handle more disparate non-English datasets without reducing English-language understanding performance.Source: §1 · p. 3
• PaLM 2 significantly outperforms PaLM across natural language generation, translation, and reasoning tasks, despite PaLM 2-L being smaller than the largest PaLM model.PaLM 2-L uses more training compute; the report attributes improved inference efficiency and lower serving costs to the smaller, higher-quality model.Source: §1 · p. 3
• PaLM 2 uses inference-time control tokens for toxicity and reports lower average verbatim memorization rates than PaLM, alongside multilingual toxicity classification and evaluations of potential harms and biases.For tail languages, memorization rates rise above English only when data is repeated several times across documents.Source: §1 · p. 3
• PaLM 2 performs significantly better than PaLM on advanced language proficiency exams and passes exams in every evaluated language; Figure 1 reports Japanese and French exam outcomes.The figure caption reports level A/special A for PaLM 2 versus level F for PaLM on Japanese, and level C1 for PaLM 2 on French; writing results were simulated, not official.Source: §1 · p. 3, Figure 1 · p. 4

2.1 Scaling laws

• The study trains differently sized models at four compute budgets, using FLOPs ≈6ND to set token counts and cosine learning-rate decay through each model’s final training token.The budgets are 1×1019, 1×1020, 1×1021, and 1×1022 FLOPs.Source: §2.1 · p. 7
• Smoothed validation-loss fits across four compute scales indicate that optimal data and model size should grow in equal proportions as compute increases, as in Hoffmann et al. (2022).The study’s scaling-law models and FLOPs do not represent the sizes and compute used for PaLM 2 models.Source: §2.1 · p. 7, §2 · p. 7, Figure 5 · p. 8
• Models that approximately follow the predicted optimal parameter counts achieve the lowest training loss at the evaluated FLOP points.The parameter and token settings were derived from the scaling laws for 1×1022, 1×1021, and 1×1020 FLOPs.Source: §2.1 · pp. 7–8, Table 1 · p. 8

2.2 Downstream metric evaluations

• For 1×1022 FLOPs, the downstream-optimal model size is around 9.5B, though the 16.1B model slightly outperforms it on downstream tasks.The 9.5B model has the lowest training loss and is closest to the predicted optimum; training throughput and serving latency also affect model-size decisions.Source: §2.2 · pp. 8–9

3 Training dataset

• The pre-training corpus combines web documents, books, code, mathematics, and conversational data, and is larger and more non-English-heavy than PaLM’s (Chowdhery et al., 2022).The broader language exposure is intended to help the model learn language nuances and benefit multilingual tasks such as translation and multilingual question answering.Source: §3 · p. 9
• Parallel source–target text pairs spanning hundreds of languages, with one side in English, further support multilingual understanding, generation, and translation.No languages were explicitly filtered to keep or remove; Table 21 lists the top 50 languages and their percentages in the multilingual web documents subcorpus.Source: §3 · p. 9
• Data cleaning included deduplication, sensitive-PII removal, and filtering; despite a smaller English-data proportion than PaLM, PaLM 2 improved on English evaluations, partly attributed to higher mixture quality.Source: §3 · p. 9
• For a small fraction of training data, toxicity labels from a fixed Perspective API version marked text with control tokens, enabling inference-time toxicity control without harming unrelated-task performance.The report also describes multilingual canaries for memorization evaluations and a responsible-AI analysis of the training data.Source: §3 · p. 9
• PaLM 2’s significantly longer context length supports long dialog, long-range reasoning and comprehension, and summarization without hurting generic benchmark performance.Source: §3 · p. 9

4.1 Language proficiency exams

• PaLM 2 outperforms PaLM on all language-proficiency exams and passes in every evaluated language.The exams target C2 proficiency; scores are estimates based on reading and writing, not official grades.Source: §4.1 · p. 10

4.2 Classification and question answering

• PaLM 2-L improves over PaLM on almost all English QA and classification tasks, with especially strong gains on ANLI, ReCoRD, and RACE.PaLM 2-S is competitive with PaLM 540B, while PaLM 2-M consistently outperforms it.Source: Table 2 · p. 11, §4.2 · p. 12
• All PaLM 2 variants outperform PaLM on TyDi QA; PaLM 2-L leads in no-context evaluation, with notable gains for lower-resource and non-Latin-script languages.In Gold Passage evaluation, performance differences among PaLM 2 sizes are relatively small.Source: Table 3 · p. 12, §4.2 · p. 12
• PaLM 2 also improves over PaLM on toxicity classification in English and non-English examples, with slightly reduced performance in Spanish.It performs well on disambiguated social-identity questions without a systematic pattern of bias; it also supports multilingual capabilities such as creative text generation and register, dialect, and script conversion.Source: §4.2 · pp. 12–13

4.3 Reasoning

• PaLM 2 outperforms PaLM across reasoning datasets, improves on every BIG-Bench Hard task, and achieves results competitive with GPT-4.On MATH it is competitive with Minerva; on GSM8K it outperforms Minerva and GPT-4, and on MGSM it surpasses the state of the art without self-consistency. XCOPA gains are especially strong for under-represented languages.Source: §4.3 · pp. 13, 15, Table 5 · p. 14, Table 6 · p. 14

4.4 Coding

• PaLM 2-S* outperforms PaLM-540B-Coder on all coding benchmarks despite being smaller, cheaper, and faster to serve.PaLM 2-S* is created by continuing to train PaLM 2-S on an extended, code-heavy, heavily multilingual data mixture.Source: §4.4 · pp. 15–16, Table 8 · p. 16
• PaLM 2-S* beats PaLM on all but two BabelCode languages, including 6.3× improvement over PaLM-Coder-540B on Haskell and 4.7× on Julia.Java, JavaScript, and TypeScript performance is higher than Python.Source: Figure 6 · p. 16, §4.4 · p. 16

4.5 Translation

• PaLM 2 improves translation quality over PaLM and Google Translate on WMT21, and outperforms both in every FRMT locale.For potential misgendering harms, English translations show stable performance versus PaLM, with small worst-case disaggregated improvements across 26 languages.Source: Table 9 · p. 18, Table 10 · p. 18, §4.5 · p. 18
• In zero-shot translation, PaLM 2 surpassed PaLM and Google Translate on gender agreement in Spanish, Polish, and Portuguese, but scored lower than PaLM in Telugu, Hindi, and Arabic.Source: §4.5 · p. 19

4.6 Natural language generation

• PaLM 2-L’s NLG improvements over PaLM ranged from 59.4% on XSum to 100.8% on WikiLingua, and even the smallest PaLM 2 models outperformed PaLM.The comparison used a common one-shot setup; ROUGE-2 was used for English and SentencePiece-ROUGE-2 for other languages.Source: §4.6 · p. 19
• PaLM 2 showed slight toxicity improvements on RealToxicityPrompts but slight regressions in conversational language modeling; dialog prompting reduced toxicity relative to decontextualized modeling.Across English, German, and Portuguese, toxic-response rates ranged from 0% to 3.5% in the best case and 1% to 17.9% in the worst case; toxicity varied by language and identity term.Source: §4.6 · p. 20
• On disambiguated social-identity questions adapted from BBQ, PaLM 2 achieved 91.4% accuracy, while 3% of questions produced representational harm by reinforcing social bias.The evaluation found no systematic bias pattern, but qualitative analysis highlighted hallucinations as a potential source of salient representational harms.Source: §4.6 · p. 20

4.7 Memorization

• PaLM 2 memorized less than PaLM when sequences appeared only a few times, but was more likely to memorize sequences repeated more than a few times.The authors hypothesize this may reflect deduplication making repeated n-grams rarer and present in more unique contexts.Source: §4.7 · p. 21
• Tail-language canaries required fewer repetitions for extraction, but real-data memorization showed no strong correlation with language size; highly repeated tail sequences sometimes had higher rates.Source: §4.7 · p. 22, Figure 9 · p. 23
• PaLM 2 significantly reduced average verbatim memorization versus PaLM, including for sequences repeated fewer than three times in pre-training.In the shared English-data extraction test, PaLM 2’s medium model had its highest memorization rate but still memorized significantly fewer sequences than even PaLM’s least-memorizing model.Source: §4.7 · pp. 21–22, Figure 8 · p. 21
• Memorization rates are estimates, not a full account of what an adversary could recover, and downstream attack risks may differ with safeguards and side-channel information.The authors suggest future evaluations of privacy harms and extraction attacks in dialog or summarization uses.Source: §4.7 · pp. 22–23

5 Responsible usage

• Evaluations cover pre-trained models, not necessarily product outcomes: fine-tuning, preprocessing, postprocessing, and model evolution can change performance, while bias, hallucinations, and privacy violations remain risks.The report says user-facing products should not be expected to match its reported results.Source: §5 · pp. 23–24
• Google reviews PaLM 2 applications for societal benefit, recommends mitigations, and conducts testing and monitoring; its policies exclude injury-enabling uses, norm-violating surveillance, and human-rights-contravening uses.The passage also describes proceeding with technologies that risk overall harm only when benefits substantially outweigh risks and appropriate safety constraints are incorporated.Source: §5 · p. 23

5.1 Inference-time control

• Control tokens could decrease or increase toxic-continuation probability for non-toxic prompts, with effects across input-toxicity strata despite tagging only a small fraction of pre-training tokens.The evaluation defines a toxic continuation as toxicity probability ≥0.5.Source: §5.1 · p. 24, Figure 10 · p. 25
• Toxic responses fell from 30% to 12% on the standard dataset and from 18% to 7% on the adversarial dataset with inference-time control.These results are for conversational language modeling.Source: §5.1 · p. 25, Figure 11 · p. 26
• Dialog-prompting alone reduced toxic generation more effectively than control tokens, while specialized downstream mitigations remained more effective than general-purpose inference-time controls.Adding control tokens to dialog-prompting yielded small gains only on the standard dataset.Source: §5.1 · p. 25, Figure 11 · p. 26
• The evaluation found no clear alignment tax or penalty on other results, although systematic pre-training-data ablations are challenging at scale.The authors suggest this may be because only a small fraction of pre-training data was tagged.Source: §5.1 · p. 26
• Future work could investigate pre-training interventions to strengthen steerability during downstream adaptation and address harms such as personal-information disclosure and jailbreak robustness.Such approaches may provide stronger, more flexible controllable generation.Source: §5.1 · p. 26

5.2 Recommendations for developers

• Developers should assess application-specific harms because prompts and decoding can alter outputs, and dialog-prompting toxicity gains may not generalize across harms, methods, or contexts.The report recommends reviewing responsible-development guides and tools.Source: §5.2 · p. 26

• PaLM 2 significantly outperforms PaLM while using significantly less inference compute, with gains across English and multilingual language understanding and reasoning.Source: §6 · p. 26
• The authors independently verify scaling laws at large scales: training tokens should grow at roughly the same rate as model parameters.Source: §6 · pp. 26–27
• Improved architecture, a more varied objective, and the data mixture all contribute to PaLM 2’s performance gains.Source: §6 · p. 27
• Although translation pairs were a minor part of the data mixture, they enabled PaLM 2 to match production translation services at this scale.Source: §6 · p. 27
• For a fixed training and inference budget, investing compute in a smaller model with more tokens is more beneficial than changing architecture for inference efficiency.Source: §6 · p. 27
• Further scaling model parameters and dataset size and quality, alongside architecture and objective improvements, is expected to yield continued gains in language understanding and generation.Source: §6 · p. 27

Large Model Training

• Large model training credits a broad core-contributor group, including Andrew M. Dai, David R. So, Dmitry Lepikhin, and Zachary Nado.The listed core contributors also include Jonathan H. Clark, Maxim Krikun, Melvin Johnson, Nan Du, and others.Source: Large Model Training · p. 28

Pre-training Data and Mixture Workstream

• Pre-training data and mixture work was led by Siamak Shakeri, with Andrew M. Dai and Nan Du among its core contributors.The workstream also lists contributors including Bryan Richter, Alex Polozov, and Jacob Devlin.Source: Pre-training Data and Mixture Workstream · p. 28

Architecture and Modeling Workstream

• Architecture and modeling credits co-leads Andrew M. Dai, Dmitry Lepikhin, and Yi Tay, with separate evaluation and responsible-AI workstreams.The workstream also names core contributors and contributors across architecture, evaluation, and responsible AI.Source: Architecture and Modeling Workstream · p. 28

Fine-tuning Workstream

• Fine-tuning was led by Melvin Johnson, with core contributors and contributors spanning the workstream.The listed core contributors include Abe Ittycheriah, Aditya Siddhant, Jacob Devlin, and Rohan Anil.Source: Fine-tuning Workstream · p. 29

Infrastructure Workstream

• Infrastructure credits co-leads for long-context, optimization, code-pretraining, and serving efforts, alongside core contributors and contributors.Named leads include Melvin Johnson and Siamak Shakeri for long context, Emanuel Taropa and Rohan Anil for optimization, four co-leads for code pre-training, and Yanping Huang for serving.Source: Infrastructure Workstream · p. 29

Technical Report

• The technical report credits Andrew M. Dai, Jonathan H. Clark, Kevin Robinson, and Maysam Moussalem as core contributors.Sebastian Ruder and Siamak Shakeri are also listed as core contributors, and Jacob Austin as a contributor.Source: Technical Report · p. 30

Technical Program Management and Product Management

• Technical program and product management credits Erica Moreira as lead TPM, Eric Ni as TPM, Paige Bailey as lead PM, and Lucas Gonzalez as PM.Source: Technical Program Management and… · p. 30

Cross-workstream Leadership

• The report defines lead, core contributor, and contributor roles, states that contributions are equal within each role and names are alphabetized, and thanks reviewers and responsible-AI discussants.Acknowledged colleagues also include teams supporting infrastructure and resource management.Source: Cross-workstream Leadership · p. 30

A.1 Scaling laws

• For four compute scales from 1×10^19 to 1×10^22 FLOPs, quadratic fits to final validation losses estimate isoflop minima, then a power law predicts optimal parameter and token counts at larger scales.Total parameter count estimates FLOPs, while non-embedding parameter count fits better when estimating optimal parameter counts; leave-one-out estimates provide error bars for extrapolations.Source: §A.1 · p. 42
• At 1×10^22 FLOPs, downstream evaluations compare models trained with an architecture and data mixture different from PaLM 2; roughly equal validation loss corresponds to approximately equal downstream performance.The evaluations use a 1-shot setting and accuracy unless otherwise specified.Source: §A.1 · p. 42, Table 15 · p. 43

A.2 Instruction tuning

• Fine-tuning on Flan improves instruction following and unseen-task performance, with consistent improvements across the cited evaluation benchmarks.The Flan dataset contains more than 1,800 tasks and at least 20 instruction templates per task.Source: §A.2 · p. 43, Table 16 · p. 44

A.3 Multilingual commonsense reasoning

• PaLM 2 significantly outperforms PaLM across XCOPA languages with standard prompting, including under-represented languages, and even beats PaLM with chain-of-thought prompting.The PaLM results use chain-of-thought prompting, which improves on standard prompting; XCOPA evaluation uses a 4-shot prompt.Source: Table 17 · p. 44, §A.3 · p. 45

A.4 Coding

• BabelCode coding results are reported as pass@1 for PaLM and PaLM 2, with greedy sampling.The raw pass@1 results are shown in Table 18.Source: Table 18 · p. 45, §A.4 · p. 45

A.5 Natural language generation

• Per-language natural-language-generation results are reported for both the original data and filtered data.The results are presented in Tables 19 and 20, respectively.Source: §A.5 · p. 45

B Examples of model capabilities

• The capability examples include sample outputs, most from a PaLM 2 variant fine-tuned for instruction following.Source: B Examples of model capabilities · p. 45

B.1 Multilinguality

• PaLM 2 shows multilingual joke understanding and explains translation ambiguities, including idioms, by clarifying meanings and offering more appropriate translations.Examples include jokes, puns, and haikus in Japanese, and translation examples involving German and Swahili.Source: §B.1 · pp. 45, 47, Figure 12 · p. 47, Figure 13 · p. 48
• PaLM 2 identifies dialect-specific terms and replaces them with equivalents when translating between regional varieties, including European and Argentinian Spanish and Portuguese varieties.The examples use zero-shot translation between dialects.Source: §B.1 · p. 47, Figure 14 · p. 48
• Zero-shot examples show PaLM 2 expanding abbreviations, correcting typos, rewriting formal text as colloquial chat, and transliterating text into Latin script.The instructions do not specify the target language for the correction and style-transfer tasks; transliteration examples include Hindi and Japanese.Source: §B.1 · p. 47, Figure 15 · p. 49, Figure 16 · p. 49
• PaLM 2 generates coherent Persian email text from English instructions and follows requests to transliterate it and translate it into Arabic.The report describes the generated Persian as well-written and notes that details about Shiraz are accurate.Source: §B.1 · p. 50, Figure 21 · p. 54, Figure 22 · p. 55
• PaLM 2 successfully generates text in Tajiki and converts it to Persian despite the languages being written in different scripts.Tajiki uses Cyrillic, while Persian uses a variation of the Arabic script.Source: §B.1 · p. 50, Figure 24 · p. 57

B.2 Creative generation

• Creative-generation examples span diverse forms and languages, including an Armenian-name-based children’s game, a Taiwanese speech, German dialect explanations, Persian email, Tajiki essays, and Tamil poetry.The report describes the Taiwanese speech as fluent despite Taiwanese being under-represented in pre-training data, and the German responses as audience-appropriate, fluent, and natural.Source: §B.2 · p. 50

B.3 Coding

• PaLM 2 demonstrates coding capabilities by designing a simple website and by fixing a bug with line-by-line comments in Korean.The examples are presented as samples of its coding capabilities.Source: §B.3 · p. 50, Figure 26 · p. 59, Figure 27 · p. 60
• A coding example shows PaLM 2 generating a JavaScript palindrome-checking function with line-by-line comments in Malayalam.The report also presents this as an example of coding in a multilingual setting.Source: §B.3 · pp. 50, 61, Figure 28 · p. 61

C Language proficiency exams

• For languages with publicly available materials, the study used the latest advanced language-proficiency past or practice exams: Chinese HSK, Japanese J-Test, French TCF, Spanish DELE C2, and Italian PLIDA C2.Source: C Language proficiency exams · p. 62
• All exam questions were administered zero-shot; multiple-choice and free-text responses used temperatures 0.3 and 0.6, respectively, and native-speaker contractors graded writing.Three contractors graded each writing response, and their scores were averaged; the parameters were not tuned.Source: C Language proficiency exams · p. 62
• Scores weighted reading and writing equally; pass thresholds were HSK 60%, TCF 85.7% (C2), DELE 60%, PLIDA 60%, and J-Test 90% (A), with HSK 7–9 using HSK 6 guidance.The report stresses that these were simulated exam results, not official or accredited results.Source: C Language proficiency exams · p. 62
• German practice and French writing results were excluded for significant contamination; after removals, Chinese scored 79%, Spanish 87%, while Italian and Japanese scores were unchanged.Contamination was checked using 25-gram overlaps, with hand-checking to exclude standardized instructions and book or article excerpts.Source: C Language proficiency exams · p. 62

D Responsible AI

• The Responsible AI evaluation examines multilingual toxicity classification, potential downstream harms and biases, and representation of people in pre-training data.Source: D Responsible AI · p. 63
• The findings concern pre-trained models, not necessarily fine-tuned variants or user-facing products, which may add processing and change over time.The analyses are intended to help developers assess harms in specific application contexts and prioritize safeguards early in development.Source: D Responsible AI · p. 63

Methods

• The analysis uses Know Your Data and classifiers on optimized training formats, reports relative frequencies, and generally includes only values above 5% of a document subset.The English toxicity signal is calculated only on English text.Source: §D.1 · p. 64

Discussion

• Women-related identities are underrepresented relative to men in most signals; English pronoun mentions are an outlier, with “female” at 30% and “male” at 10%.Source: §D.1 · p. 64
• “Americans” account for 42% of references in the race, religion, ethnicity, and nationality category; “Japanese people” account for 24%, “Indian people” 19%, and “Asian people” 6%.The report notes that automated systems make identity references challenging to encode and interpret.Source: §D.1 · p. 64
• Among qualifying English-data identity subsets, documents mentioning “white people” had the highest toxicity probability (0.250), followed by “transsexual” (0.248).The analysis includes identities with toxicity probability above the overall level and present in more than 0.1% of English pre-training data; sexuality-related identities also had relatively higher toxicity.Source: §D.1 · p. 64, Table 23 · p. 66

Limitations

• Automated identity measures have limitations for assumed or unmarked identities and social contexts, and the validity of analysis measures varies with signal reliability.Source: §D.1 · p. 64

D.2 Evaluation approach

• The five-step evaluation reviews core capabilities across languages and prioritizes dialog, safety classification, translation, and question answering based on downstream developer practices.The framework approximates current and anticipated system-building workflows while making tradeoffs for practical, efficient evaluation.Source: §D.2 · p. 66
• Potential harms include toxic language, stereotypes, discrimination, exclusion, and privacy risks from memorization, arising from misuse, system failures, or harmful outcomes of proper use.The evaluation also recognizes that harms can affect individuals, communities, and societies differently, with vulnerable groups facing cumulative impacts across sociocultural contexts.Source: §D.2 · pp. 66–67
• Evaluations focus on harms observable during each task, chiefly through extrinsic measures, and separately test bias proxies for relationships to downstream-use harm measures.These task-level measures are limited approximations, not exhaustive or holistic assessments.Source: §D.2 · p. 67
• The work prioritizes automated evaluations throughout model development and shares measurement-quality rubrics to support research on validity, relevance, and inclusivity.The stated aim includes improving evaluation for vulnerable groups facing cumulative harms over time.Source: §D.2 · p. 67
• The evaluation covers harms and bias in dialog, classification, translation, and question-answering uses, as well as underlying language-model capabilities and privacy risks from memorization.Source: §D.2 · p. 67

D.3 Dialog uses

• Dialog evaluation uses Glaese et al. (2022)'s prompting method with a generic “Assistant,” harm-mitigation cues, and sometimes multiple sampled responses before additional filtering.This setup approximates a common dialog-development approach and samples possible responses for a prompt.Source: §D.3 · p. 67
• The dialog setup does not encompass all deployments: systems may add fine-tuning, instruction fine-tuning, or reinforcement-learning safety mitigations.The passage notes additional fine-tuning and mitigation methods including those discussed by Thoppilan et al. (2022), Ouyang et al. (2022), Rae et al. (2021), Glaese et al. (2022), Ganguli et al. (2022), and Bai et al. (2022).Source: §D.3 · p. 67

D.3.1 Toxic language harms in dialog generation

• The evaluation prompts dialog systems with ParlAI standard explicitly toxic and adversarial subtly harmful transcripts, sampling 20 continuations per prompt.A prompt counted as toxic if a response had a Perspective API toxicity score > 0.5; the fixed classifier may miss implicit, subtle, and contextualized harms.Source: §D.3.1 · p. 68
• With 20 samples, toxic responses occur for over 30% of adversarial queries and almost 60% of standard queries; LaMDA consistently produces less toxic language harm.The figure summarizes toxic degeneration across models, datasets, and samples per query.Source: Figure 30 · p. 68, §D.3.1 · p. 68
• Dialog-prompted PaLM 2 has significantly lower toxic degeneration than its language-modeling evaluation, while PaLM 2 and PaLM show no major difference in conversational language modeling.The reduction is influenced by prompting techniques.Source: §D.3.1 · p. 68, Figure 30 · p. 68
• For a single query sample, dialog-prompted PaLM 2 performs similarly to the specialized dialog system LaMDA.Prompting mitigates toxic degeneration when measuring one sample, but is ineffective at controlling harms across multiple responses.Source: §D.3.1 · p. 68

D.3.2 Multilingual toxic language harms and representational bias

• The multilingual harm benchmark extends prior representational-bias evaluation across 16 languages and adapts it to dialog, but prompted systems still produce toxic harms and nonuniform identity-related biases.The findings motivate application-specific mitigation, disaggregated analysis, and further research into intersectional harms; biases related to gender appeared despite explicit targeting in the dialog prompt.Source: §D.3.2 · pp. 69, 71
• 1%–17.9% of responses were toxic in English, German, and Portuguese, versus 0%–3.5% in other evaluated languages, with identity-group toxicity varying across the outlier languages.Dialog prompting controlled toxicity in most languages but did not eliminate harms; the toxicity measure relies on a classifier whose reported AUC ranges from 0.85 to 0.97 across languages, and the study leaves human validation for future work.Source: §D.3.2 · p. 69, Figure 31 · p. 70
• English, German, and Portuguese showed identity-group toxicity bias, including higher rates for Black and White queries, Judaism and Islam, and differences between Male and Female queries.For most other languages, toxicity for all identity groups was below 5%.Source: §D.3.2 · p. 70, Figure 32 · p. 71

D.4 Classification uses

• The classification-use evaluation focuses on zero-shot and few-shot workflows, while leaving prompt-tuning and fine-tuning classifier workflows for future evaluation.Source: §D.4 · pp. 71–72

D.4.1 Toxicity classification with CivilComments

• PaLM 2 made significant gains over PaLM in zero-shot and 10-shot toxicity classification on English CivilComments.The evaluation used 10,000 test comments and AUC-ROC as its primary metric; the supplied passages provide no numerical gains.Source: §D.4.1 · p. 72

D.4.2 Multilingual toxicity classification with Jigsaw Multilingual

• PaLM 2 outperformed PaLM 540B in zero-shot toxicity classification across all six analyzed languages and remained better overall in 10-shot evaluation.The evaluation used English prompts and scored normalized log-likelihoods of “yes” and “no” continuations; few-shot examples were in the sample’s language.Source: §D.4.2 · p. 72

D.5 Translation uses

• Translation misgendering evaluations cover zero-shot translation from multiple-resource-level source languages into English using automated methods, and out of English using human evaluation.For translation into English, the evaluation uses inputs containing a single gendered entity to enable automated pronoun-based scoring.Source: §D.5 · p. 73, §D.5.1 · p. 73

D.5.1 Translating to English

• Zero-shot translation into English showed no major differences from PaLM 540B, with a minor improvement in disaggregated worst-case performance.The evaluation covered 26 source languages and 1,954 passages, scoring potential misgendering by scanning English translations for gendered personal pronouns; Hindi-to-English had the lowest performance.Source: §D.5.1 · pp. 73–74, Table 24 · p. 74

D.5.2 Translating from English

• Because gender marking varies across languages and automatic agreement metrics were unreliable, the study used human evaluations of zero-shot translations from a professionally translated gender-set sample spanning 13 languages.Two raters independently scored gender agreement and translation quality on 0–3 scales, and the reported scores are their averages.Source: §D.5.2 · pp. 74–75
• Zero-shot translation out of English showed higher potential misgendering harms than translation into English; PaLM 2 beat PaLM in some languages and Google Translate in Spanish, Polish, and Portuguese.PaLM 2 had improved translation quality but lower gender agreement in Italian and Japanese, and lower gender agreement than PaLM in Telugu, Hindi, and Arabic.Source: §D.5.2 · p. 75, Table 26 · p. 75, Figure 33 · p. 76
• Most languages represented in over 2% of pre-training data had average translation quality scores above 2.50, with Arabic the notable exception; more data may help but does not ensure quality.This broad relationship was observed for both PaLM and PaLM 2.Source: §D.5.2 · p. 76, Figure 34 · p. 76
• The evaluation excludes differential harms to non-binary people and contested cross-cultural pronoun perspectives (Keyes, 2018; Dev et al., 2021a; Lee, 2019), and covers only some risks without wider sociotechnical context.Evaluating gender agreement out of English remains challenging and time-intensive.Source: §D.5.2 · p. 76

D.6 Question answering uses

• The evaluation adapted BBQ for generative QA by giving PaLM 2 contexts and questions without multiple-choice options, to capture potential harms beyond fixed answer choices.BBQ was originally designed for multiple-choice QA and covers nine social-dimension categories relevant to US English-speaking contexts.Source: §D.6 · p. 77
• 91.4% accuracy on disambiguated questions coexisted with 38.2% of errors reinforcing social bias, producing this harm in 3% of all disambiguated questions.The analysis found no systematic pattern of errors always reinforcing harmful bias.Source: §D.6 · p. 78, Table 27 · p. 78
• Correct answers in disambiguated contexts aligned with social bias 2.2 percentage points more often than anti-bias, while Other responses made up 22.5% of errors.The authors suggest future analysis of these unexpected responses.Source: §D.6 · p. 78, Table 28 · p. 78
• 0.6% accuracy on ambiguous contexts accompanied responses naming a person despite insufficient information; biased responses were 15.3 percentage points more frequent there and 1.8 points more frequent in disambiguated contexts.Across identity categories, responses aligned with social bias more often than anti-bias, except for disability status.Source: §D.6 · p. 78, Table 28 · p. 78

D.7 Language modeling

• The language-modeling discussion examines representational and toxic-language harms and how they relate to downstream-use harms; prior work found this relationship may be weak (Liang et al., 2022), while bias can persist through instruction fine-tuning (Chung et al., 2022).Source: §D.7 · p. 80

D.7.1 Open-ended generation

• PaLM 2 shows minor aggregate improvements over PaLM, with slightly lower toxic degeneration in both open-ended evaluation variants.The comparison covers the small and full variants; Table 29 presents the probability of producing a toxic continuation.Source: Table 29 · p. 80, §D.7.1 · p. 80
• Across prompt-toxicity strata, continuations have lower toxicity probability than inputs and the human baseline; this pattern also holds across task variants and models.The disaggregated results include PaLM 2 (L) and PaLM 540B comparisons in Figure 36.Source: §D.7.1 · p. 80, Figure 36 · p. 81
• In the tiny training-time evaluation, mean TPC was 0.0676±0.0026 and worst-case TPC was 0.0725 across 87% of epochs; the authors recommend the small task for future evaluation.The tiny variant used 500 examples and performed slightly below the small variant; the authors call for more informative evaluation sets to support continuous evaluation and ablations.Source: Figure 37 · p. 81, §D.7.1 · p. 81

D.7.2 Conversational language modeling

• PaLM 2 (L) shows a slight degradation versus PaLM on conversational toxicity, while adversarial prompts produce fewer unsafe responses than standard prompts.The evaluation sampled one dialogue act per prompt using top-k 40 at temperature 1.0 and treated any Perspective score above 0.5 as unsafe.Source: Table 30 · p. 82, §D.7.2 · p. 82

D.8 Measurement quality rubrics

• The appendix offers measurement-quality rubrics and CrowdWorksheets, including a worksheet for Civil Comments, a crowdsourced dataset of approximately 2 million labeled comments.For translation out of English, it evaluates PaLM 2, PaLM, and Google Translate API with the gender sets described in Chung et al. (2022).Source: §D.8 · p. 82, §D.9 · p. 85, §D.9.1 · p. 85

What are the precise instructions that were provided to annotators?

• Civil Comments toxicity judgments are subjective and can vary with missing conversational context and annotators’ cultural familiarity; annotators were warned of potential psychological harm.The task involved potentially harmful content such as hate speech and slurs.Source: What are the precise instructions that… · p. 87

Selecting Annotators

• English proficiency was the only annotator eligibility criterion, demographic information was unavailable, and whether impacted communities were represented is unknown.Annotator demographics could affect recognition of identity references or judgments about statements concerning their own groups.Source: Selecting Annotators · p. 87

Platform and Infrastructure Choices

• Raters received 1.5 cents per judgment and reported an average pay satisfaction score of 3.8/5; estimated hourly earnings ranged from $0.90 to $5.40.Rates were set with platform competitiveness and local wage targets in mind.Source: Platform and Infrastructure Choices · p. 88

(1) Gender Agreement and Coherency Evaluation

• Gender agreement is rated 0–3, from all gender references incorrect (0) to all correct (3); a gender-neutral target scores 3 when the source has gender-specific words and no other gender errors.If you can’t make a judgment, answer “Can’t tell”.Source: (1) Gender Agreement and Coherency… · p. 89

(2) General Quality Evaluation

• General translation quality was rated 0–3 for preserved meaning and grammar, with task instructions iteratively refined to seek high inter-annotator agreement.The rubric ranges from no meaning preserved to perfect meaning and grammar.Source: (2) General Quality Evaluation · p. 89
• Intermediate scores were not clearly defined, gender-agreement perceptions can vary culturally, and annotators were assumed fluent in English and the target language.The task involved reading translations that may misgender people; gender agreement was intentionally not defined solely in binary terms.Source: (2) General Quality Evaluation · p. 90

Selecting Annotators

• English and non-English proficiency was the sole eligibility criterion; annotator gender identity and lived experience may affect judgments, and impacted communities were not represented.Sociodemographic information was unavailable, and no annotator subgroups were prioritized.Source: Selecting Annotators · p. 90

Platform and Infrastructure Choices

• Venga was selected for its translation-data experience; annotators could seek clarifications and discuss sample errors, and the supplier was paid an average of $42 per hour.Google did not have visibility into annotators’ actual pay, which the supplier set according to location and experience.Source: Platform and Infrastructure Choices · p. 90

Dataset Analysis and Evaluation

• Standard vendor checks, linguist resolution of disagreements, and research-team review of outlier language ratings were used, but disagreement patterns were not systematically analyzed.Task guidelines were updated and communicated when high-level misunderstandings arose.Source: Dataset Analysis and Evaluation · p. 90
• Annotations reflect social views at collection time, perceptions of toxic language may change, and there are no current plans to update the dataset.Future shifts in attitudes toward acceptable language may reduce the dataset’s validity.Source: Dataset Analysis and Evaluation · p. 91

D.10 Model Card

• The model card describes PaLM 2 as a static model trained on an offline dataset and warns against harmful use or downstream application without further harm and bias analysis.The paper does not test the model outside research settings that can affect performance.Source: §D.10 · p. 93