Scaling Instruction-Finetuned Language Models

Motivation

• Instruction finetuning builds on pretrained models’ ability to follow natural-language task descriptions while reducing reliance on few-shot exemplars.The paper frames this as progress toward generalization to unseen tasks.Source: §1 · p. 2

Scaling and reasoning

• Instruction finetuning scales well with both the number of tasks and model size, while adding nine CoT datasets prevents severe degradation on CoT evaluations and improves all evaluations.These scaling behaviors motivate further increases in task and model scale.Source: §1 · p. 2

Flan-PaLM results

• After scaling to 540B parameters, 1.8K finetuning tasks, and CoT data, Flan-PaLM achieves 75.2% on MMLU, a new state-of-the-art result.It improves one-shot TyDiQA by 14.9% absolute and arithmetic reasoning in under-represented languages by 8.1% versus PaLM, while improving responsible-AI evaluation benchmarks. It substantially outperforms PaLM on challenging open-ended generation questions.Source: §1 · p. 2

Flan-T5 results

• Flan-T5 checkpoints from 80M to 11B parameters provide strong zero-shot, few-shot, and CoT abilities, outperforming prior public T5 checkpoints and sometimes PaLM 62B.Flan-T5 11B achieves double-digit improvements over T5 11B and surpasses PaLM 62B on some challenging BIG-Bench tasks.Source: §1 · p. 2

Finetuning data

• The finetuning data comprises 473 datasets, 146 task categories, and 1,836 total tasks.These task counts characterize the scale of the paper’s finetuning mixture.Source: Figure 2 · p. 3

2.1 Finetuning Data

• Flan scales instruction finetuning to 1,836 tasks by combining the Muffin, T0-SF, NIV2, and CoT mixtures.NIV2 contributes 1,554 tasks, while Muffin and T0-SF contribute 80 and 193 tasks, respectively.Source: §2.1 · p. 3
• The CoT mixture contains nine manually annotated datasets spanning arithmetic, multi-hop reasoning, and natural language inference.The authors manually compose ten instruction templates per task.Source: §2.1 · p. 4

2.2 Finetuning procedure

• The same training procedure is applied across T5, PaLM, and U-PaLM models ranging from 80M to 540B parameters.Only learning rate, batch size, dropout, and finetuning steps vary by model; Flan-PaLM 540B finetuning uses 0.2% of pre-training compute.Source: §2.2 · p. 4

2.3 Evaluation protocol

• Evaluation focuses on held-out tasks across MMLU, BBH, TyDiQA, and MGSM, covering world knowledge, reasoning, question answering, and multilingual mathematics.MMLU contains 57 tasks, BBH contains 23 challenging tasks, TyDiQA covers eight languages, and MGSM covers ten languages.Source: §2.3 · p. 5
• The protocol combines direct and chain-of-thought prompting with few-shot exemplars and summarizes performance using a normalized average over six benchmark metrics.The six metrics are MMLU-Direct, MMLU-CoT, BBH-Direct, BBH-CoT, TyDiQA-Direct, and MGSM-CoT.Source: §2.3 · p. 5

Scaling model size and task count

• The experiments scale PaLM across 8B, 62B, and 540B parameters and add task mixtures from CoT through Muffin, T0-SF, and NIV2.The evaluation uses held-out tasks, with scaling assessed jointly over model size and finetuning-task count.Source: §3 · p. 6
• 9.4%–15.5%: Multi-task instruction finetuning improves normalized held-out benchmark performance across all three PaLM sizes compared with no finetuning.Figure 4 aggregates held-out MMLU, BBH, TyDiQA, and MGSM results; Table 3 reports individual benchmark results.Source: §3 · p. 6, Figure 4 · p. 6, Table 3 · p. 7
• Most task-count improvement comes from using up to 282 tasks, with additional tasks yielding only small gains.Possible explanations are limited task diversity or that finetuning mainly improves expression of pretrained knowledge; finetuning uses 1.4B tokens versus 780B pretraining tokens.Source: §3 · p. 6
• 15.5% vs. 9.4%: Absolute finetuning gains are larger for 8B than 540B, but relative error-rate reduction is larger for 540B: 18.4% vs. 16.6%.Increasing model scale by an order of magnitude substantially improves both finetuned and non-finetuned models.Source: §3 · p. 6
• Further model-size scaling is expected to provide substantial gains, whereas further task scaling is expected to improve performance incrementally.The scaling curves motivate continued scaling of instruction finetuning.Source: §3 · p. 6

4 Finetuning with chain-of-thought annotations

• Instruction finetuning with nine CoT datasets improves reasoning across evaluations, whereas instruction finetuning without CoT degrades reasoning ability.The section evaluates CoT prompting on held-out benchmarks and distinguishes CoT from non-CoT task performance.Source: §4 · p. 7

4.1 Finetuning on chain-of-thought improves reasoning on held-out tasks

• 75.2% on five-shot MMLU makes Flan-PaLM 540B outperform PaLM, code-davinci-002, and Chinchilla, while CoT + self-consistency reaches 83.9% on GSM8K.Flan-PaLM also achieves 69.6% on Bengali MGSM, whereas PaLM reaches 63.6% on French and 61.2% on German; it does not surpass specialized models on every task.Source: §4.1 · pp. 7–8, Table 4 · p. 8

4.2 Some chain-of-thought data is needed to maintain reasoning ability

• Jointly finetuning on CoT and non-CoT data improves held-out CoT performance over either mixture alone without compromising held-out non-CoT performance.Finetuning only on non-CoT data substantially degrades CoT performance, indicating that both prompting paradigms require corresponding finetuning examples.Source: §4.2 · pp. 8–9, Figure 5 · p. 8

4.3 Unlocking zero-shot reasoning

• Flan-PaLM models improve on 23 unseen challenging BIG-Bench tasks by generating CoT after the zero-shot prompt “let’s think step-by-step,” unlike unf​​inetuned PaLM.This setting tests whether the model can produce reasoning without few-shot exemplars, which otherwise require substantial prompt engineering.Source: §4.3 · p. 9, Figure 6 · p. 9, Figure 7 · p. 10

Model scope

• Instruction finetuning generalizes across PaLM, encoder-decoder T5, continued-pretrained cont-PaLM, and UL2-pretrained U-PaLM models.The evaluated models vary in size, architecture, and training objective, including PaLM’s decoder-only and T5’s encoder-decoder architectures.Source: §5 · p. 10

Results

• 52.4% MMLU for 3B Flan-T5-XL surpasses GPT-3 175B’s 43.9%, while instruction finetuning improves normalized average performance by a large margin across all model types.The strongest overall model combines instruction finetuning with UL2 continued pre-training, showing these methods are complementary without increasing model scale.Source: §5 · p. 10, Table 5 · p. 11

Evaluation

• Table 5 evaluates held-out MMLU, BBH, TyDiQA, and MGSM suites using few-shot prompted accuracy and an unweighted normalized average.The aggregate averages MMLU-direct, MMLU-CoT, BBH-direct, BBH-CoT, TyDiQA, and MGSM results across their tasks or languages.Source: Table 5 · p. 11

Human evaluation results

• For every zero-shot setting, Flan-PaLM was preferred by a large margin, and CoT-triggered inputs increased its preference over PaLM by around 10%.The evaluation covered creativity, contextual reasoning, complex reasoning, planning, and explanation, including 60 CoT-triggered examples.Source: §6 · pp. 11–12
• Flan-PaLM showed no regression compared with PaLM on the few-shot examples.The evaluation set included 30 few-shot inputs in addition to 160 zero-shot inputs.Source: §6 · pp. 11–12

Qualitative analysis

• PaLM exhibited zero-shot usability problems including continuing related text, repeating questions, and failing to stop generation.These behaviors suggest that next-token pretraining alone was insufficient for good zero-shot usability, possibly because pretraining did not use end-of-sequence tokens.Source: §6 · p. 12

Human evaluation results

• 79% of the time, human raters preferred Flan-PaLM over PaLM across 190 open-ended evaluation examples.The evaluation compared five sampled responses per model, selected after log-probability ranking and repetition filtering.Source: Figure 8 · p. 12, §6 · p. 12

Summary

• Across few-shot, zero-shot, CoT, and open-ended evaluations, instruction finetuning improves performance, generalizes across models, and combines with other techniques at relatively small computational cost.The discussion therefore recommends instruction finetuning for virtually all pretrained language models.Source: §7 · pp. 12, 14

Scaling curves

• Scaling model size and finetuning-task count improves performance, with task scaling retaining positive but diminishing returns and no clear erosion of instruction-finetuning gains.The scaling curves suggest both factors should continue improving performance, while the margin over models without finetuning does not appear to decrease.Source: §7 · p. 12

CoT finetuning

• Jointly finetuning on non-CoT and CoT data substantially improves held-out CoT reasoning while maintaining non-CoT performance in one model.Non-CoT-only finetuning degraded CoT performance, whereas joint finetuning preserved gains across both evaluation types.Source: §7 · p. 13

Cross-model generality

• Instruction finetuning generalizes across decoder-only and encoder-decoder models from T5-80M to PaLM-540B and combines effectively with UL2R.This combination produced Flan-U-PaLM, the strongest model trained in the work.Source: §7 · p. 13

Usability and harms

• Flan-PaLM receives substantially better human ratings than PaLM on open-ended tasks, especially complex reasoning, planning, and explanation, and outperforms it on several Responsible AI benchmarks.Instruction finetuning also addresses usability problems such as continuing the user’s input rather than responding, supporting zero-shot use.Source: §7 · p. 13

Compute efficiency

• 0.2% of PaLM 540B’s pre-training compute yields a 9.4% normalized-average improvement, while smaller instruction-finetuned models can outperform larger untuned models.Flan-PaLM 62B outperforms PaLM 540B on TyDiQA (58.7% vs. 52.9% EM), and Flan-T5 11B outperforms PaLM 62B on BBH-direct (43.7% vs. 37.5%).Source: §7 · p. 14

Research context

• The paper combines instruction-based finetuning with rationale-based prompting and finetuning across multi-task learning, prompting, reasoning, and large-language-model research.Its broader context includes instructions, prompting, multi-step reasoning, and large language models (Radford et al., 2019; Brown et al., 2020; Aghajanyan et al., 2021; Chowdhery et al., 2022; Lewkowycz et al., 2022, inter alia).Source: §8 · p. 14

Instruction finetuning

• Instruction finetuning extends prior work with aggregated mixtures plus added chain-of-thought, dialog, and code datasets, scales to 540B parameters, and mixes zero- and few-shot formats.Prior studies considered models from 3B to 137B parameters and generally focused on either zero-shot or few-shot finetuning (Wang et al., 2022c; Sanh et al., 2021; Wei et al., 2021; Zhong et al., 2021; Ye et al., 2021; Mishra et al., 2021; Min et al., 2022; Wang et al., 2022a).Source: §8 · p. 14

Reasoning via finetuning

• Finetuning on mixed datasets including CoT annotations improves performance on unseen reasoning tasks, while one checkpoint supports both CoT and non-CoT settings.Unlike prior work centered on single reasoning datasets, smaller models, or synthetic CoT datasets, this work jointly finetunes on CoT and non-CoT data; Huang et al. (2022) is the closest comparison.Source: §8 · p. 14

Compute-efficient methods for better language models

• Flan-U-PaLM combines UL2R and instruction finetuning and achieves the strongest performance for all models trained in the paper.UL2R uses additional training with a causal-LM-plus-span-corruption objective and no additional data, whereas instruction finetuning uses the described data mixtures.Source: §8 · p. 14
• Compute-efficient language-model research also improves models through better architectures, training objectives, and data without massively scaling compute.This work is part of that broader direction (Hoffmann et al., 2022; Padmakumar et al., 2022; So et al., 2021; Tay et al., 2022a; Du et al., 2022).Source: §8 · pp. 14–15

9 Conclusions

• 75.2% on five-shot MMLU, while Flan-T5 models outperform baseline T5 models by a large margin and instruction finetuning improves usability across model sizes and architectures.The authors scale Flan-PaLM to 540B parameters and 1.8K tasks, include CoT data, and publicly release Flan-T5 checkpoints.Source: §9 · p. 15

A.1 Are instruction-finetuned models better for single-task finetuning?

• Instruction finetuning improves performance on unseen tasks in few-shot prompting, but its effect on single-task finetuning was not evaluated and remains future work.Related work found benefits for massively multitask finetuning and prompt tuning, while large models should ideally retain multitask abilities.Source: §A.1 · p. 24

A.2 Does using CoT prompting for evaluation always improve performance?

• CoT benefits depend on task and scale: it does not improve MMLU directly, while BBH gains appear only for sufficiently large or selected instruction-finetuned models.Self-consistency can improve over no CoT, and non-instruction-finetuned BBH models benefit only at 62B or larger.Source: §A.2 · p. 24

A.3 Does instruction finetuning improve performance more or less for larger models?

• 15.5% for 8B versus 9.4% for 540B in normalized average-score improvement, but relative error reduction is 16.6% versus 18.4%, respectively.The comparison is confounded because the no-instruction-finetuning baseline also improves with scale.Source: §A.3 · p. 24

A.4 How many examples were used as part of the CoT mixture in finetuning?

• 74,730 examples comprise the CoT finetuning mixture across nine datasets, with ESNLI contributing the largest share at 36,170 examples.The mixture also includes AQuA, CREAK, ECQA, GSM8K, QASC, QED, Sensemaking, and StrategyQA.Source: §A.4 · p. 24

C.1 Toxicity in open-end generation; B Qualitative examples

• Instruction-finetuned models generate fewer toxic continuations at all scales and across toxic and non-toxic prompts, while Flan models have lower TPC than PaLM at all scales.Flan-PaLM scale has no clear toxicity effect, and Flan-T5-XXL behaves differently at higher prompt toxicity; qualitative examples are provided for challenging open-ended instructions.Source: B Qualitative examples · p. 25, Figure 10 · p. 25, Table 6 · p. 27

C.2 Toxicity in prompts with identity terms; C.3 Ability to classify toxic content

• Instruction finetuning reduces toxic language and identity-group disparities, but tail toxicity remains high and can preserve identity-linked bias patterns resembling PaLM.For toxicity classification, instruction-finetuned models consistently improve zero- and few-shot AUC-ROC over base models, although Perspective API reaches 97.0%.Source: §C.2 · p. 28, §C.3 · p. 29, Table 7 · p. 30

• The evaluation is limited to harms and biases discussed in Chowdhery et al. (2022) and does not cover many other potential axes.Source: §C.6 · p. 33
• Toxicity evaluations are limited by noisy and biased automated measures, aggregated-metric shortcomings, diverse annotation perspectives, English-only coverage, and a biased subset of identity terms.They also omit classifiers built with fine-tuning or other parameter-efficient methods.Source: §C.6 · p. 33
• Winogender covers only one English bias form and US occupation statistics, excluding global cultural perspectives, non-binary gender harms, and other prompts or usage contexts.Source: §C.6 · p. 33
• Translation evaluations omit differential harms involving non-binary gender identities and contested pronoun perspectives across languages and cultures.Source: §C.6 · p. 34
• Across evaluations, the analysis covers only a subset of potential risks and examines model outputs without the wider sociotechnical context of instruction-finetuned language models.Source: §C.6 · p. 34

D.1 MMLU

• MMLU uses five-shot prompts with dev exemplars and reports individual-task validation performance, except Tables 1 and 4, which use test sets.The MMLU prompts are publicly available, and STEM prompts come from Lewkowycz et al. (2022).Source: §D.1 · p. 35, Table 11 · p. 35, Table 12 · p. 36

D.2 BBH

• BBH evaluation averages performance uniformly across 27 counted subtasks using the prompts from Suzgun et al. (2022).BBH comprises challenging BIG-Bench tasks for which the referenced model exceeded average human-rater performance.Source: §D.2 · p. 41, Table 17 · p. 41, Table 18 · p. 42

D.3 TyDiQA

• TyDiQA uses one-shot prompting and reports exact-match performance averaged uniformly across eight non-English language scores.The protocol follows Chowdhery et al. (2022), and English is excluded from the average.Source: §D.3 · p. 44, Table 20 · p. 44

D.4 MGSM

• MGSM uses target-language exemplars and chains of thought, with per-language results reported in Table 21.The evaluation follows the prompts provided by Shi et al. (2022).Source: §D.4 · p. 45, Table 21 · p. 45

E Finetuning details

• Learning rate, batch size, and dropout were the most important finetuning hyperparameters, with values listed for every finetuned model.Batch size denotes the global rather than per-device size, and packing makes the effective batch size larger.Source: E Finetuning details · p. 46, Table 22 · p. 46
• Task sampling weights are based on example counts but capped within each mixture to prevent much larger datasets from dominating smaller tasks.The four mixtures are Muffin, T0-SF, NIV2, and CoT; their caps and proportions are summarized in Table 23.Source: E Finetuning details · p. 46, Table 23 · p. 46
• Mixture proportions changed from Proportion A to Proportion B after scaling and ablation experiments showed strong gains from T0-SF.Proportion A was used for scaling and ablations, while the remaining models used Proportion B.Source: E Finetuning details · p. 46, Table 23 · p. 46

F Finetuning Data Card

• 1,836 tasks combine FLAN, T0, Natural Instructions, dialog, program synthesis, and chain-of-thought sources, with MMLU tasks removed to preserve 57 held-out evaluation tasks.All data sources are publicly available.Source: F Finetuning Data Card · p. 47, Table 24 · p. 47
• The data card warns that publicly sourced datasets may propagate bias and unknown sensitive human attributes into Flan.Downstream users are advised to recognize these potential risks.Source: F Finetuning Data Card · pp. 47–48
• The data has limited maintenance: no new versions are planned, while updates are restricted to technical, bug, and error fixes handled case by case.The maintenance status is explicitly described as limited.Source: F Finetuning Data Card · pp. 47–48
• The dataset contains 60 languages across the 1,836 tasks, including lower-resource languages whose frequency is low.Many translation and localization tasks include two or more languages.Source: F Finetuning Data Card · p. 48, Figure 18 · p. 50

G Model cards

• Flan-PaLM shares PaLM’s architecture, implementation frameworks, usage, and limitations, with Flan-PaLM-specific model-card information provided in Table 25.The source model is PaLM (Chowdhery et al., 2022), and the model-card format follows Mitchell et al. (2019).Source: §G.1 · p. 51, Table 25 · p. 51

G.2 Flan-T5

• Flan-T5’s model card is presented in Table 26.The supplied passage identifies the table but does not provide additional model-card contents.Source: §G.2 · p. 51, Table 26 · p. 52

H Ethical Considerations

• Instruction finetuning improves many zero-shot and few-shot capabilities, but developers should assess application-specific risks and harms before deployment.Prompting, sampling, and additional mitigations such as safety filters may introduce biases or harms not evaluable from the finetuned model alone; prior considerations in Chowdhery et al. (2022) also apply.Source: H Ethical Considerations · p. 51

I Open-Ended Evaluation Details

• Open-ended evaluators ranked shuffled, anonymous model responses without knowing their sources.Responses appeared in randomized order on each row to prevent model identification.Source: I Open-Ended Evaluation Details · p. 53
• Evaluators judged answers by correctness and informativeness, using web search when needed and ranking the most useful response highest.Usefulness required at least some correctness and minimal informativeness about the prompt.Source: I Open-Ended Evaluation Details · p. 53
• Ties were allowed sparingly for indistinguishable responses, while nonsensical, irrelevant, confusing, or nonresponsive answers received an “F” label.Concise answers could outrank longer ones when at least equally correct and informative.Source: I Open-Ended Evaluation Details · p. 53

J Author Contributions

• The paper assigns experiments, training infrastructure, evaluation design, and dataset construction across named contributors.These responsibilities include experimentation, infrastructure, evaluation, and building datasets.Source: J Author Contributions · p. 54
• Other contributors handled framing and writing, responsible-AI evaluations, open sourcing, and research advising.The contributions statement separately identifies these roles and their named participants.Source: J Author Contributions · p. 54

K Versioning

• Version 5 added text-davinci-003 results to Appendix D and corrected misreported PaLM-540B MMLU-CoT results; version 4 added the responsible-AI appendix.These changes span the v4-to-v5 and v3-to-v4 revisions.Source: K Versioning · p. 54
• Earlier revisions added an FAQ and contributions statement, fixed typos, incorporated feedback, and updated the MMLU table with Hypermind and Metaculus forecasts.These changes document the v2-to-v3 and v1-to-v2 updates.Source: K Versioning · p. 54