PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, Jason Wei

arXiv:2210.11416v5cs.LGcs.CL

TL;DR

Instruction finetuning must scale across tasks and model sizes while preserving reasoning ability. This paper develops Flan by scaling task and model coverage and adding chain-of-thought data, yielding broad gains across models, prompting setups, and benchmarks.

  • Problem

    The paper examines whether instruction finetuning scales with task and model size and whether chain-of-thought data preserves reasoning performance.

  • Method

    Flan instruction-finetunes models on diverse instruction-template data, varying task count, model size, architecture, training objective, and chain-of-thought inclusion.

  • Results

    Instruction-finetuned models improve across few-shot, zero-shot, and chain-of-thought evaluations; Flan-PaLM reaches 75.2% on five-shot MMLU.

  • Takeaways & Limitations

    Scaling tasks and models improves performance, while joint non-chain-of-thought and chain-of-thought finetuning supports strong performance across evaluation types.

  • Takeaways & Limitations

    Flan-PaLM does not outperform certain specialized models, including code-davinci-002 on BBH-algo and TyDiQA-specific ByT5 on one-shot TyDiQA.

Abstract

from arXiv · show

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models.

1 Introduction

The paper advances instruction finetuning by scaling task and model size and adding chain-of-thought data, finding broad gains in performance, reasoning, and usability. Flan-PaLM and Flan-T5 demonstrate these gains across benchmarks, including 75.2% on five-shot MMLU and improvements over larger or prior public models.

  • Instruction-finetuning advances: Instruction finetuning scales well with both the number of tasks and model size, motivating further scaling in both dimensions.The paper explicitly studies these two scaling axes and reports positive scaling behavior.
  • Flan-PaLM: Flan-PaLM combines a 540B-parameter model, 1.8K finetuning tasks, and chain-of-thought data to outperform PaLM and set new state-of-the-art results.Its reasoning improvements support chain-of-thought and self-consistency prompting.
  • Flan-PaLM: 75.2% on Massive Multi-task Language Understanding is achieved by Flan-PaLM using chain-of-thought and self-consistency.This is a five-shot MMLU result.
  • Flan-T5: Flan-T5 checkpoints spanning 80M to 11B parameters show strong zero-shot, few-shot, and chain-of-thought abilities, outperforming prior public T5 checkpoints.Flan-T5 11B delivers double-digit improvements over T5 11B and surpasses PaLM 62B on some challenging BIG-Bench tasks.
  • Instruction-finetuning advances: The finetuning data comprises 473 datasets, 146 task categories, and 1,836 total tasks.These data dimensions define the task scale used in the paper.

2 Flan Finetuning

Flan is an instruction-finetuning procedure applied across diverse data mixtures, templates, model families, and scales. The study evaluates it on held-out world-knowledge, reasoning, and multilingual benchmarks using direct and chain-of-thought prompting where applicable.

  • Flan finetuning: Flan instruction-finetunes language models on diverse data sources and instruction-template types, producing models such as Flan-PaLM.The procedure works across several model sizes and architectures.
  • Task mixtures: 1,836 finetuning tasks are formed by combining the Muffin, T0-SF, NIV2, and CoT mixtures.The mixtures build on prior work and scale the number of instruction-finetuning tasks.
  • Chain-of-thought finetuning: Nine datasets with human-written chain-of-thought annotations form a reasoning mixture for testing generalization to unseen reasoning tasks.The datasets include arithmetic reasoning and multi-hop reasoning tasks.
  • Model scaling: Instruction finetuning spans T5, PaLM, and U-PaLM families, ranging from Flan-T5-small with 80M parameters to PaLM and U-PaLM with 540B parameters.The same training procedure is used for each model, with adjustments to selected hyperparameters.
  • Evaluation: Evaluation targets held-out tasks across world knowledge, reasoning, and multilingual benchmarks, using direct prompting and chain-of-thought prompting for MMLU and BBH.TyDiQA is evaluated only with direct-prompting exact-match score.

3 Scaling to 540B parameters and 1.8K tasks

Scaling both model size and the number of instruction-finetuning tasks improves held-out benchmark performance, with gains of 9.4%–15.5% over no finetuning. Most task-scaling benefits arise by 282 tasks, while further model scaling remains substantially beneficial.

  • Scaling effects: 9.4% to 15.5%: Multi-task instruction finetuning improves held-out benchmark performance across all three PaLM model sizes compared with no finetuning.The experiments use 8B, 62B, and 540B PaLM models.
  • Scaling effects: 282 tasks: Most of the performance improvement from increasing the finetuning-task count occurs by this point, with only small gains afterward.The passage offers limited task-diversity or knowledge-expression as possible explanations for the smaller gains beyond 282 tasks.
  • Scaling effects: 15.5% for 8B vs. 9.4% for 540B: Absolute instruction-finetuning gains are larger for the smaller model, complicating comparisons of relative benefit across scales.Increasing model scale by an order of magnitude improves both finetuned and non-finetuned models substantially.
  • Scaling effects: Further model-size scaling is expected to yield substantial gains, whereas expanding the finetuning-task set is expected to improve performance incrementally.The scaling curves motivate continuing to scale instruction finetuning.

4 Finetuning with chain-of-thought annotations

Including chain-of-thought (CoT) data in instruction finetuning improves reasoning across held-out benchmarks, enables zero-shot CoT on unseen tasks, and works best when combined with non-CoT data. However, Flan-PaLM does not surpass specialized models on every evaluation.

  • CoT finetuning results: Including nine CoT datasets improves reasoning ability, with Flan-PaLM outperforming PaLM on four held-out CoT evaluation benchmarks.The evaluations include MMLU, BBH, and MGSM, with BBH tasks stratified into NLP and algorithmic categories.
  • Limitations: 14.9%: Flan-PaLM outperforms PaLM by this amount on one-shot TyDiQA, but remains below ByT5 finetuned on the TyDiQA training set.Flan-PaLM also does not outperform code-davinci-002 on BBH-algo, even with CoT + SC.
  • CoT and non-CoT data: Jointly finetuning on CoT and non-CoT data improves both held-out CoT and non-CoT evaluations compared with finetuning on either type alone.Finetuning only on non-CoT data substantially degrades CoT performance, showing that some CoT examples are critical for retaining reasoning abilities.
  • Zero-shot CoT: Flan-PaLM performs zero-shot CoT reasoning on 23 unseen BBH tasks when prompted with “let’s think step-by-step,” whereas unf​​inetuned PaLM does not generate useful CoT for solving them.This capability arises from instruction finetuning on CoT data and does not require few-shot CoT exemplars.

5 Putting it all together

The paper demonstrates that instruction finetuning generalizes across model sizes, architectures, and training objectives. Across model types, it substantially improves normalized average benchmark performance beyond continued pre-training.

  • Generality of instruction finetuning: Instruction finetuning is applied across models with different sizes, architectures, and training objectives, including encoder-decoder T5 and decoder-only PaLM models.The study also includes cont-PaLM, an extended version of PaLM 62B.
  • Evaluation results: Instruction finetuning improves normalized average performance by a large margin for all evaluated model types.T5 baselines were LM-adapted using 100B additional C4 tokens with a standard language-modeling objective.
  • Evaluation results: The evaluation aggregates few-shot prompted accuracy across MMLU, BBH, TyDiQA, and MGSM benchmark suites.The suites contain 57, 23, 8, and 10 tasks or languages, respectively, and the aggregate uses normalized averages across direct, chain-of-thought, and multilingual results.

6 Usability evaluation of open-ended generation

A manual evaluation of challenging open-ended requests found that instruction finetuning substantially improved human-rated response preferences. Across 190 examples, Flan-PaLM was preferred over PaLM 79% of the time, with further gains under chain-of-thought prompting and no few-shot regression.

  • Chain-of-thought prompting: Around 10%, the rater preference for Flan-PaLM over PaLM further increased when inputs used a CoT trigger phrase.This result was reported for the zero-shot evaluation settings.
  • Few-shot usability: No regression compared to PaLM was observed in the few-shot setting.The human evaluation tested both zero-shot and few-shot prompting conditions.

7 Discussion

The discussion concludes that instruction finetuning improves performance and usability across prompting and evaluation settings, generalizes across model architectures, sizes, and pre-training objectives, and complements other techniques. These benefits require relatively small computational cost compared with pre-training, motivating instruction finetuning for virtually all pretrained language models.

  • Scaling curves for instruction finetuning: Scaling both model size and the number of finetuning tasks improves instruction-finetuning performance.The paper presents scaling curves for both components.
  • CoT finetuning is critical for reasoning abilities: Joint finetuning on non-CoT and CoT data substantially improves CoT performance while maintaining non-CoT performance.Finetuning only on non-CoT tasks degraded performance on CoT tasks.
  • Instruction finetuning generalizes across models: Instruction finetuning generalizes across decoder-only and encoder-decoder architectures, model sizes from T5-80M to PaLM-540B, and multiple pre-training objectives.The evaluated objectives include causal LM, span corruption, and prefix LM + span corruption.
  • Instruction finetuning is relatively compute-efficient: 0.2% of PaLM 540B’s pre-training compute is required for instruction finetuning, while performance improves across evaluations.Instruction finetuning is presented as compute-efficient because it can leverage existing checkpoints without changing inference cost.
  • Summary: Instruction finetuning improves performance across few-shot, zero-shot, CoT, and open-ended generation evaluations, generalizes across models, and combines well with UL2R.The authors therefore recommend instruction finetuning for virtually all pretrained language models.

8 Related Work

This work sits at the intersection of instruction finetuning, rationale-based prompting, multi-step reasoning, and large language models. It extends related efforts through aggregated instruction mixtures, chain-of-thought finetuning, and compute-efficient approaches to improving language models.

  • Research context: The paper combines instruction-based finetuning with rationale-based prompting and finetuning across multi-task learning, prompting, reasoning, and large language models.It positions the work at the intersection of these broad research areas.
  • Instruction finetuning: Instruction finetuning extends prior work by using aggregated mixtures from earlier studies together with chain-of-thought data.The passage describes this as an extension in finetuning data.
  • Reasoning via finetuning: Finetuning large language models on mixtures containing chain-of-thought annotations improves performance on unseen reasoning tasks.Prior work primarily used single reasoning datasets or substantially smaller models.
  • Compute-efficient methods: The work contributes to efforts that improve language models without massively scaling compute, alongside better architectures, training objectives, and data.The passage contrasts these approaches with the substantial computational resources required to scale language models.

9 Conclusions … C.5 Translation misgendering

Instruction finetuning improves performance and usability across model sizes, architectures, tasks, and benchmarks, while also reducing several measured harms. However, effects remain mixed for some evaluations, including translation misgendering, and important limitations persist.

  • 9 Conclusions: Instruction finetuning scales to 540B parameters and 1.8K tasks, with CoT data improving performance across evaluated CoT tasks.Performance improves with larger model size and more finetuning tasks, unlike prior methods that degraded CoT performance.
  • 9 Conclusions: 75.2% on five-shot MMLU marks Flan-PaLM’s state-of-the-art performance, while released Flan-T5 models outperform baseline T5 models by a large margin.Instruction finetuning also enables zero-shot reasoning without prompt engineering or few-shot exemplars and works across model sizes, architectures, and pre-training objectives.
  • A.1 Are instruction-finetuned models better for single-task finetuning?: Instruction-finetuned models perform better on unseen tasks in few-shot prompting, but this paper does not evaluate their single-task finetuning ability.Related work reports benefits for downstream finetuning and prompt tuning, while the authors leave evaluation on their specific models for future work.
  • A.2 Does using CoT prompting for evaluation always improve performance?: CoT benefits depend on task and scale: it helps multi-step reasoning, does not improve MMLU direct prompting, and benefits non-instruction-tuned BBH models mainly at 62B or larger.Self-consistency can improve performance over no CoT even when direct CoT does not.
  • A.3 Does instruction finetuning improve performance more or less for larger models?: Instruction finetuning improves performance across model sizes and architectures, although whether its effect increases with scale is confounded by stronger no-finetuning baselines.The paper compares normalized average scores across four datasets for 8B and 540B models.
  • A.4 How many examples were used as part of the CoT mixture in finetuning?: 74,730 examples comprise the CoT finetuning mixture across nine datasets, including 36,170 ESNLI and 7,470 GSM8K examples.The mixture also includes AQuA, CREAK, ECQA, QASC, QED, Sensemaking, and StrategyQA.
  • C Representational bias and toxicity; C.1 Toxicity in open-end generation; C.2 Toxicity in prompts with identity terms; C.3 Ability to classify toxic content; B Qualitative examples: Instruction finetuning lowers toxic-generation probability at all scales and improves zero-shot and few-shot toxicity classification, but tail toxicity remains consistently high.Identity-group analyses report reduced toxicity and bias, while the qualitative section provides challenging open-ended response examples.
  • C.4 Gender and Occupation Bias; C.5 Translation misgendering: Winogender performance improves across PaLM models, especially zero-shot, and Flan-PaLM exceeds PaLM after article-ignoring scoring, but translation misgendering effects are mixed.Flan-PaLM approaches human baseline performance even at 8B and zero-shot; translation models perform worse when English should contain “she” rather than “he.”

C.6 Limitations

The evaluations cover only a subset of potential harms and biases, with important limitations in measurement, language and identity coverage, task framing, and sociotechnical context. Toxicity and gender-bias assessments also face noisy or aggregated metrics, contested annotation perspectives, and unexamined evaluation methods.

  • The evaluations cover only a subset of potential harms and biases and remain limited to evaluations discussed in Chowdhery et al. (2022).
  • Automated toxicity measures contain noise and bias, omit diverse perspectives, and cover only a biased subset of identity terms in English.
  • Toxicity classification evaluation is limited by aggregated metrics, diverse annotation perspectives, and the absence of fine-tuning or parameter-efficient classifiers.
  • Winogender examines one English bias form using US occupation statistics, omitting global cultural perspectives, non-binary gender harms, and alternative task framings.
  • Translation evaluations omit harms involving non-binary gender identities and contested cross-cultural pronoun perspectives, and assess outputs without wider sociotechnical context.

D Full experimental results … E Finetuning details

The paper reports benchmark-specific evaluation protocols for MMLU, BBH, TyDiQA, and MGSM, alongside finetuning choices for hyperparameters, task sampling, and mixture balancing.

  • D.1 MMLU: MMLU five-shot evaluation uses the dev set for exemplars and reports individual-task performance on the validation set, except Tables 1 and 4.The prompts are publicly available, and STEM prompts follow Lewkowycz et al. (2022).
  • D.2 BBH: BBH evaluation averages subtasks unweighted across the 23 cited tasks, using the prompts from Suzgun et al. (2022).Two BBH tasks have three subtasks, which are counted individually.
  • D.3 TyDiQA: TyDiQA uses one-shot prompting, evaluates exact match, and averages eight per-language scores without English.The protocol follows Chowdhery et al. (2022).
  • D.4 MGSM: MGSM uses target-language exemplars and chains of thought, with the prompts provided by Shi et al. (2022).For example, Chinese evaluation uses Chinese exemplars and chains of thought.
  • E Finetuning details: Learning rate, batch size, and dropout were the most important finetuning hyperparameters, with batch size reported globally rather than per device.Packing makes the effective batch size larger than the reported global batch size.
  • E Finetuning details: Task sampling uses example counts as mixture weights but applies maximum caps so much larger datasets cannot dominate smaller tasks.WMT translation datasets can contain millions of examples, compared with 9k for BoolQ.
  • E Finetuning details: Mixture balancing prevents any underlying task from being repeated more than once, and later models use updated Proportion B values after strong T0-SF gains.Scaling and ablation experiments used Proportion A; the remaining experiments used Proportion B.

F Finetuning Data Card … H Ethical Considerations

The finetuning suite combines 1,836 publicly available tasks while excluding MMLU tasks from Natural Instructions for held-out evaluation. The model cards inherit existing PaLM considerations, and ethical guidance emphasizes application-specific risk assessment and mitigation.

  • F Finetuning Data Card: 1,836 tasks comprise the final finetuning mixture from FLAN, T0, Natural Instructions, dialog, program synthesis, and chain-of-thought reasoning sources.All data sources are publicly available, and MMLU tasks are removed from Natural Instructions to preserve 57 held-out evaluation tasks.
  • F Finetuning Data Card: The data card warns that publicly available sources may contain bias and unknown sensitive human attributes.Downstream users are advised to recognize these potential risks.
  • F Finetuning Data Card: The data has limited maintenance: updates address only technical bugs and errors, with no new versions planned.Error handling is considered case by case.
  • F Finetuning Data Card: The dataset covers 60 languages, including many lower-resource languages whose frequency is low.Most tasks use one language, while translation and localization tasks often include two or more.
  • G.1 Flan-PaLM: Flan-PaLM shares PaLM’s architecture, implementation frameworks, usage, and limitations, with model-card details specific to Flan-PaLM provided separately.The paper presents these details in Table 25.
  • G.2 Flan-T5: The paper presents the Flan-T5 model card in Table 26.This section documents Flan-T5 separately from the Flan-PaLM model card.
  • H Ethical Considerations: Instruction finetuning improves many zero-shot and few-shot capabilities but requires developers to assess broad and application-specific risks.Prompting, sampling, and additional mitigations such as safety filters can introduce new or different risks.

I Open-Ended Evaluation Details · J Author Contributions · K Versioning

The paper specifies a blinded human evaluation for ranking model responses by correctness and informativeness, while documenting author responsibilities and revisions across versions. The evaluation also permits ties sparingly, flags failed responses, and favors concise answers when they are equally correct and informative.

  • I Open-Ended Evaluation Details: Annotators ranked shuffled, anonymous language-model responses without knowing which model produced each response.The prompts required various forms of reasoning, and responses were shuffled on each row.
  • I Open-Ended Evaluation Details: Responses were ranked by usefulness, defined through correctness and informativeness, with web searches permitted when assessing correctness.Annotators were instructed to use their best judgment and identify the best answer to each input prompt.
  • I Open-Ended Evaluation Details: Annotators could assign equal ranks only when responses were equally correct and informative and no clear winner existed.The instructions emphasized using ties sparingly.
  • I Open-Ended Evaluation Details: Nonsensical, irrelevant, highly ungrammatical, confusing, or nonresponsive answers received an “F” fail label instead of a rank.The evaluation instructions also noted that longer answers were not necessarily better than succinct, coherent responses.
  • J Author Contributions: Author contributions covered experiments, training infrastructure, evaluation design, dataset construction, framing and writing, responsible AI evaluations, open sourcing, and research advising.The listed contributors differed by responsibility across these areas.
  • J Author Contributions: Responsible AI evaluations, open sourcing, and research advising were separately attributed to designated contributors.These responsibilities were listed as distinct contribution categories.
  • K Versioning: Version updates added results and sections, corrected reported results, updated forecasts, fixed typos, and incorporated feedback across v2–v5.Changes included text-davinci-003 results, responsible AI and FAQ sections, contributions, corrected PaLM-540B MMLU-CoT results, and revised MMLU forecasts.
Loading 2210.11416v5…