PaperlayerView exact revision on arXiv

Source-linked AI summary

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, Ves Stoyanov

arXiv:2212.12017v3cs.CL

TL;DR

Instruction-tuning improves generalization, but the performance trade-offs of tuning decisions remain limitedly understood. This paper develops OPT-IML Bench to characterize these trade-offs and trains OPT-IML models, which improve over OPT across four benchmarks at 30B and 175B scales.

  • Problem

    The performance trade-offs of instruction-tuning decisions across task diversity, sampling, demonstrations, specialized datasets, and objectives remain insufficiently understood.

    Source: §1 · p. 2

  • Method

    The paper consolidates 1,991 NLP tasks from eight meta-datasets into OPT-IML Bench and evaluates generalization across held-out categories, unseen tasks, and held-out instances.

    Source: §1 · pp. 1–2

  • Results

    OPT-IML significantly improves over OPT at 30B and 175B scales across PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG, while remaining competitive with benchmark-specific models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework establishes trade-offs and best practices for scaling instruction-tuning across task collections, distributions, instruction styles, specialized datasets, and demonstrations.

    Source: §6 · p. 20

    Trade-offs studied at 30B may not show the same trends at larger scales, and different category assignments may change the preferred instruction-tuning decisions.

    Source: §6.2 · p. 22

Abstract

from arXiv · show

Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework.

1. Introduction

The paper introduces OPT-IML Bench, a large-scale framework for studying instruction meta-learning across 1,991 tasks and three levels of generalization, and uses its insights to train OPT-IML models. OPT-IML improves substantially over OPT at 30B and 175B scales across four instruction-tuning benchmarks while remaining competitive with individually tuned models.

  • The paper characterizes how task diversity and distribution, prompt formatting, fine-tuning objectives, demonstrations, and specialized reasoning and dialogue datasets affect instruction-tuning tradeoffs.Experiments with OPT 30B examine dataset and benchmark sampling, scaling with tasks and categories, demonstration incorporation, reasoning chains, and dialogue data.Source: §1 · pp. 1–3
  • The study uses its generalization findings to train OPT-IML 30B and 175B as instruction-tuned versions of OPT and releases both models and the OPT-IML Bench framework.The framework is intended to support comprehensive evaluation and future large-scale instruction-tuning research.Source: §1 · pp. 1, 3
  • OPT-IML Bench consolidates 1,991 instruction-based NLP tasks from eight meta-datasets into more than 100 categories with multiple prompts per task.Its evaluation framework covers fully held-out categories, unseen tasks in seen categories, and held-out instances from seen tasks, measuring cross-task and supervised multi-task generalization.Source: §1 · p. 2, Figure 1 · p. 2
  • OPT-IML significantly improves over OPT at both 30B and 175B scales across PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG, while remaining competitive with individually tuned models.The models are evaluated in both zero- and few-shot settings; they nevertheless underperform PaLM- and T5-based systems on MMLU and Big-Bench Hard.Source: §1 · p. 3

2. Scaling up Multi-task Benchmarks

The authors construct OPT-IML Bench by aggregating eight instruction-tuning task collections, consolidating diverse tasks into a common prompt and category framework, and splitting them to measure three levels of generalization.

2. Scaling up Multi-task Benchmarks

  • OPT-IML Bench aggregates eight existing task collections to support massive instruction fine-tuning and evaluation across diverse task categories, instruction types, and prompting setups.The collections include Super-NaturalInstructions, FLAN, T0, PromptSource, ExMix, T5, CrossFit, Structured Knowledge Grounding, Dialogue, and Chain-of-thought Reasoning.Source: §2 · p. 3, §2.1 · p. 3

2.2 Benchmark Consolidation

  • The benchmark casts all tasks into a bipartite format with instruction and output segments, manually adding instance-level instructions where source benchmarks lacked natural-language instructions.Dataset-level instructions define an overall task, whereas instance-level templates are instantiated for individual examples.Source: §2.2 · p. 4
  • Tasks are organized into a unified single-level taxonomy with over 100 categories, primarily following Super-NaturalInstructions and manually coarsening incompatible benchmark-specific clusters.This categorization supports studying generalization across and within task categories.Source: §2.2 · p. 4

2.3 Creating Benchmark Splits

  • The split design evaluates generalization to fully held-out categories, new datasets in partially held-out seen categories, and new examples from seen tasks.Thirty-five evaluation tasks spanning nine categories are reserved for validation of instruction-tuning strategies.Source: §2.3 · p. 5
  • Train and evaluation tasks are de-duplicated by excluding shared data sources and manually reviewing roughly 14,000 task pairs whose evaluation examples overlap training sequences by more than 1%.Pairs sharing broad resources such as Wikipedia are retained when their output labels are otherwise unrelated.Source: §2.3 · p. 5

2.4 Task Prompt Construction

  • Prompt construction places few-shot demonstrations according to whether benchmarks use task-level or instance-level instructions, while manually removing templates that alter the original task semantics.Zero-shot examples use the bipartite instruction scheme, and randomly sampled delimiters mitigate overfitting.Source: §2.4 · p. 6

3. Instruction Fine-tuning

OPT-IML instruction-tunes OPT at 30B and 175B scales using target-only next-token loss over mixed benchmark datasets. The procedure packs examples into 2048-token sequences with document-level attention masking and applies scale-specific distributed-training hyperparameters.

3. Instruction Fine-tuning

  • Instruction fine-tuning applies the OPT-IML Bench to OPT models at 30B and 175B parameters, extending OPT’s next-word-prediction pretraining setup.OPT is an open-source decoder-only transformer family spanning 125M to 175B parameters.Source: §3 · p. 6

3.1 Fine-tuning Objective

  • The objective predicts label tokens only, conditioning on task instructions, inputs, and previously generated target tokens.Source tokens comprise instructions and inputs, while target tokens provide the loss terms.Source: §3.1 · p. 6
  • Training minimizes this target-token loss across all OPT-IML Bench datasets by mixing examples according to dataset sizes and benchmark proportions.Source: §3.1 · p. 6

3.2 Packing and Document Attention

  • Examples are packed into 2048-token sequences separated by <eos> tokens to maximize sequence-length utilization and computational efficiency.Without additional masking, packed examples could attend to tokens from earlier examples in the same sequence.Source: §3.2 · p. 6
  • Document attention masking restricts each packed example to its own tokens, replacing the causal triangular mask with a block-triangular mask that improves stability and performance.Source: §3.2 · p. 7

3.3 Fine-tuning Hyperparameters

  • Fine-tuning uses distributed training on 64 A100 GPUs for 30B models and 128 for 175B models, with Adam, warmup-decay scheduling, dropout, gradient clipping, and dynamic loss scaling.Models see approximately 2 billion fine-tuning tokens, equal to 0.6% of OPT’s pretraining budget; fine-tuning parameters and times are summarized in Table 3.Source: §3.3 · p. 7, Table 3 · p. 7

4. What Matters for Instruction Fine-tuning?

The experiments identify how dataset mixing, task and category diversity, auxiliary data, and demonstrations affect fully held-out, partially supervised, and fully supervised generalization. These findings motivate the final OPT-IML training choices, including controlled benchmark proportions, all tasks and clusters, 5% pre-training data, and 1% reasoning data, while excluding dialogue and MetaICL.

4.1 Experimental Setup

  • Across three generalization levels, the study evaluates dataset composition, task diversity, auxiliary pre-training/reasoning/dialogue data, and demonstration-based fine-tuning on OPT 30B.Models are tuned for 4000 steps and evaluated in 0-shot and 5-shot settings using aggregated task-category performance.Source: §4.1 · pp. 7–8

4.2 Effects of varying task mixing-rate maximum

  • EPS improves instruction-tuning over no cap, but values below 4096 vary minimally; EPS=4096 is selected for the final OPT-IML models.EPS=512 caps 97% of datasets, whereas EPS=8192 caps 16%; EPS=128 closely matches EPS=4096.Source: §4.2 · p. 8, Table 4 · p. 9

4.3 Effects of varying benchmark proportions

  • Benchmark proportions create complementary gains and tradeoffs: FLAN reaches 88.8/83.6% on fully held-out tasks through increased PromptSource and Crossfit proportions, while PromptSource reaches 79.7/83.5% with 25% PromptSource.Removing Crossfit, Exmix, T5, and Unified-SKG produces the worst model; the selected final proportion is “4/2/20/25/45/2/2”.Source: §4.3 · pp. 9–10, Table 5 · p. 10

4.4 Effects of Scaling Tasks or Categories

  • Increasing training tasks most improves fully held-out and partially supervised performance while leaving fully supervised performance unchanged, whereas increasing clusters improves zero-shot but yields mixed or declining few-shot results.The final models therefore use all tasks and clusters.Source: Figure 2 · p. 10, §4.4 · p. 11, Figure 3 · p. 11

4.5 Effects of Pre-training during Instruction-Tuning

  • Adding pre-training data improves fully held-out and partially supervised generalization through 10%, after which performance deteriorates; 5% is selected for the final models.More pre-training data improves Rouge-L F1 but lowers accuracy, partly because it changes the generation-versus-classification mixture.Source: §4.5 · pp. 11–12, Figure 4 · p. 12

4.6 Effects of Adding Reasoning Datasets

  • Adding reasoning data raises held-out reasoning Rouge-L from 12.2% to 31.6%, with the largest overall gains at 1%; OPT-IML therefore uses 1% reasoning data.Other held-out categories also improve, while higher proportions reduce gains on several categories but continue helping Summarization.Source: §4.6 · p. 13, Figure 5 · p. 13

4.7 Effects of Adding Dialogue Datasets

  • Adding only 0.5% dialogue data lowers 0-shot performance, leaves 5-shot performance unchanged overall, and causes output-format failures and severe degradation on toxicity detection, so it is excluded.The model particularly struggles with tasks requiring special decision words rather than simple “yes” or “no” outputs.Source: §4.7 · pp. 13–14, Table 7 · p. 14

4.8 Effects of Meta-Training for In-Context Learning

  • MetaICL generally worsens generation performance despite benefiting scoring-based classification, while suffix loss is better than regular MetaICL; severe output degeneration leads to its exclusion.Failures include degraded stereotype and word-analogy performance, shortened reasoning answers, and overfitting to demonstration separators; random separators may mitigate these issues.Source: §4.8 · pp. 14–15, Table 8 · p. 15, Figure 6 · p. 15

5. OPT-IML Models

OPT-IML models, created by scaling instruction tuning to OPT 30B and 175B with selected data and training choices, outperform OPT across evaluated benchmarks and remain competitive with benchmark-specific instruction-tuned models. Results show strong generalization across held-out tasks, prompt formats, model scales, and structured generation settings, with some task-specific regressions.

5. OPT-IML Models

  • OPT-IML outperforms OPT on all four multi-task benchmarks and is competitive with benchmark-specific instruction-tuned models in both zero- and few-shot settings.The comparisons cover PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG, alongside OPT evaluation tasks.Source: §5 · p. 16, §5.2 · p. 17, §5.3 · p. 19

5.1 OPT Evaluations

  • OPT-IML improves zero-shot accuracy over OPT by approximately 6-7% at both 30B and 175B, with stronger 32-shot gains at 30B than 175B.Improvements are especially large on RTE, WSC, BoolQ, ARC, CB, and WiC, but not on StoryCloze, PIQA, Winograd, or Winogrande.Source: §5.1 · p. 16, Table 9 · p. 16

5.2 Evaluations on PromptSource

  • OPT-IML 175B matches T0-original-task’s zero-shot performance and significantly outperforms it at 5-shot, while OPT-IML 30B outperforms baseline OPT 175B on average.Instruction tuning improves robustness to diverse prompts; WinoGender accuracy improves, whereas OPT-IML 175B deteriorates on Crows Pairs.Source: §5.2 · pp. 17–18, Table 10 · p. 17

5.3 Evaluations on FLAN

  • OPT-IML improves every evaluated FLAN task, exceeds untuned OPT by more than 20% on average at both scales, and OPT-IML 175B improves over FLAN-137B by 3% zero-shot.OPT-IML 30B also outperforms OPT 175B by 20% zero-shot and 12% five-shot, while matching FLAN-137B competitively at five shots.Source: Table 11 · p. 18, §5.3 · p. 19

5.4 Evaluations on Super-NaturalInstructions

  • OPT-IML outperforms OPT across all three Super-NaturalInstructions clusters and settings; OPT-IML 175B beats Tk-Instruct 11B zero-shot, but Tk-Instruct wins at 2- and 5-shot.Tk-Instruct’s score drops from 65.3 to 58.4 from 2-shot to 5-shot, whereas OPT-IML maintains uniform performance.Source: §5.4 · pp. 19–20, Table 12 · p. 19

5.5 Evaluations on UnifiedSKG

  • OPT-IML retains nearly 85 Rouge-L F1 on Spider, gains most strongly zero-shot on DART with 30B surpassing OPT 175B, but deteriorates on MultiWoZ at both scales.Spider is fully supervised, DART is held-out, and MultiWoZ is a held-out dialogue-state-tracking task.Source: §5.5 · p. 20, Table 13 · p. 20

6. Discussion and Limitations

The authors evaluate OPT-IML-Max on challenging external benchmarks, finding consistent gains over untuned OPT but performance gaps versus several stronger instruction-tuned models. They also identify methodological, scalability, categorization, and responsible-use limitations.

6.1 Evaluations on MMLU, BBH and RAFT

  • Performance gaps are difficult to explain because compared models differ in pre-training tokens and data, architecture, and fine-tuning algorithms, including RLHF for some davinci models.Meta-ICL was excluded from the final models despite yielding 2–3% improvements on MMLU and BBH because it was not holistically better.Source: §6.1 · p. 21
  • OPT-IML-Max outperforms untuned OPT at all scales on MMLU, BBH, and RAFT except 1.3B on BBH, but trails several stronger instruction-tuned models.It is competitive with FLAN-T5 11B on RAFT while lagging FLAN-T5, FLAN-PaLM, and *-davinci-* models on MMLU and BBH.Source: Table 14 · p. 21, §6.1 · p. 21

6.2 Limitations

  • The independently studied tuning variables may interact, their 30B trends may not transfer to larger scales, and different category selections or assignments may change the preferred settings.Category assignment is subjective because tasks requiring different skills can share an underlying format, such as toxicity detection and textual entailment.Source: §6.2 · pp. 21–22

6.3 Responsible AI

  • Although OPT-IML models outperform baseline OPT broadly, they remain susceptible to factual errors, toxic language, and stereotypes, requiring responsible best practices.The models are released to support instruction-tuning research and improve access to large instruction-tuned causal language models over 100B parameters.Source: §6.3 · p. 22

7. Related Work

The paper situates instruction fine-tuning at the intersection of instruction tuning, prompting and meta-training, reasoning, multitask learning, and continuous adaptation. It distinguishes its focus on massive multitask instruction tuning and systematic tradeoff analysis from prior approaches.

Instruction Tuning

  • Instruction tuning aligns language models with natural instructions; this work consolidates publicly available instructed datasets to study downstream tradeoffs across tuning decisions.It follows supervised fine-tuning on academic benchmarks rather than human-feedback instruction tuning, contrasting its goals with Chung et al. (2022a).Source: §7 · p. 22

Prompting and Meta-Training

  • The paper combines benchmark prompt variants with meta-training on demonstrations from many tasks to induce robustness across prompting languages and setups.This builds on prompting and in-context-learning methods, including meta-tuning approaches designed to improve adaptation from few examples.Source: §7 · pp. 22–23

Multi-task Learning

  • Unlike conventional multitask learning, instruction-based fine-tuning unifies tasks through instructions and shares all model weights to generalize to entirely unseen tasks.Traditional multitask learning primarily improves training tasks or transfers to new domains through related-task signals.Source: §7 · p. 23

Continuous Learning

  • The paper studies massive simultaneous adaptation by fine-tuning on 2000 tasks, whereas continuous-learning work revisits prior instructions or examples to prevent forgetting.Other approaches add new capabilities through arithmetic task-vector combinations or soft prompts without changing base-model parameters; continual adaptation remains future work.Source: §7 · p. 23

8. Conclusions

The paper contributes a 2,000-task instruction-tuning benchmark with evaluation splits for three generalization settings, derives tuning best practices, and releases OPT-IML 30B and 175B models that outperform OPT across five benchmarks.

Conclusions

  • OPT-IML 30B and 175B strongly outperform OPT on five evaluation benchmarks and remain competitive with recent models tuned on individual benchmarks.These models use the best settings identified through the paper’s experiments.Source: §8 · p. 23
  • The authors curate a 2,000-task benchmark from eight dataset collections, with splits measuring fully supervised performance, unseen tasks in seen categories, and tasks from held-out categories.The benchmark’s tasks are annotated into task categories to support these generalization evaluations.Source: §8 · p. 23
  • The evaluation suite establishes tradeoffs and best practices for task and category sampling, task demonstrations, and specialized reasoning and dialogue datasets.These findings guide the settings used to train the released OPT-IML models.Source: §8 · p. 23

Appendix A. Benchmark Preparation Details

This appendix describes how tasks from the paper’s benchmarks were downloaded and pre-processed.

  • The appendix details the downloading and pre-processing of tasks from the benchmarks used in the paper.Source: Appendix A · p. 32

A.1 Data Curation

The benchmark data were collected primarily from official releases, with source-specific handling for unavailable ExMix data, task instantiations, template validity, and overlap removal. The curation also included dedicated reasoning datasets and retained selected non-overlapping T5 datasets.

  • Data were downloaded from official benchmark releases, except ExMix, whose official data are not open-sourced.Source: §A.1 · p. 32
  • PromptSource and FLAN retained templates representing the original tasks, while PromptSource validation and test templates had to apply to all examples except for turk.Source: §A.1 · p. 32
  • ExMix tasks without overlap with other benchmarks were collected from task-specific repositories and dataset sources.Source: §A.1 · p. 32
  • After removing overlaps, the T5 collection retained seven datasets, including translation and CNN/DailyMail, which was kept for its high quality despite appearing in other benchmarks.Source: §A.1 · p. 32
  • The reasoning benchmark comprised 14 datasets spanning mathematical, commonsense, explanatory, and question-answering tasks.Source: §A.1 · p. 33

A.2 Details of Dialogue Datasets

The experiments in §4.7 use six dialogue datasets spanning web-grounded, encyclopedic, blended-skill, conversational, multi-session, and open-domain settings.

Dataset selection

  • §4.7 evaluates six dialogue datasets: Wizard of Internet, Wizard of Wikipedia, Blended Skill Talk, ConvAI2, Multi-Session Chat, and Light+ Wild.These datasets are a subset of those used by Shuster et al. (2022).Source: §A.2 · p. 33

A.3 Details on Validation Tasks · Appendix B. Additional Experimental Results

The validation set used in Section 4 comprises multiple tasks and categories, with Table 15 documenting their benchmarks, generalization levels, and evaluation metrics. The supplied passages provide no additional experimental results for Appendix B.

A.3 Details on Validation Tasks

  • Section 4 evaluates a validation set spanning several tasks and categories.These validation tasks are documented with category, benchmark, generalization level, and evaluation metric information.Source: §A.3 · p. 33
  • Table 15 lists the full validation-task details, excluding some sub-tasks such as those within MMLU.The table covers validation tasks used in the Section 4 experimental studies.Source: Table 15 · p. 33

B.1 Results at Additional Model Scales · B.2 Additional Results on Factors Affecting Instruction-Tuning

The appendix extends OPT-IML evaluation to the 1.3B scale and supplies additional analyses of task scaling, clustering, pre-training data, reasoning data, and MetaICL inference. These results use clearly specified task-level generalization settings and 0-shot/5-shot or 0-shot/32-shot reporting conventions.

B.1 Results at Additional Model Scales

  • OPT-IML 1.3B was trained by fine-tuning OPT 1.3B with the same settings used for OPT-IML 30B.Source: §B.1 · p. 33
  • Table 16 extends OPT’s 14-task zero- and few-shot comparison with results at the 1.3B scale, reporting 0-shot/32-shot accuracies.ARC results distinguish Easy and Challenge subsets.Source: §B.1 · p. 34, Table 16 · p. 34

B.2 Additional Results on Factors Affecting Instruction-Tuning

  • Table 17 reports the full effect of scaling training-task counts for OPT-IML 30B across all three generalization levels after 2000 training steps.Results are aggregated by task category in 0-shot/5-shot format.Source: Table 17 · p. 34
  • Table 18 evaluates how scaling the number of task clusters affects all three generalization levels for OPT-IML 30B after 2000 training steps.Results are aggregated by task category in 0-shot/5-shot format.Source: Table 18 · p. 35
  • Table 19 studies the impact of pre-training-data proportions on all three generalization levels for OPT-IML 30B after 4000 training steps.Results are aggregated by task category in 0-shot/5-shot format.Source: Table 19 · p. 35
  • Table 20 reports the effect of fine-tuning with reasoning data at each generalization level for OPT-IML 30B after 4000 training steps.Results are aggregated by task category in 0-shot/5-shot format.Source: Table 20 · p. 35
  • With a double-newline example separator at inference, all MetaICL models outperform the baseline model, although the 5-shot summarization-inclusive baseline is not comparable with other tables.Most tasks report Rouge-L, MMLU reports accuracy, and Cause Effect results average accuracy with Rouge-L.Source: §B.2 · p. 34, Table 21 · p. 35

Appendix C. Examples of Prompts from All Benchmarks

This appendix illustrates the prompt formats used for fine-tuning across all included benchmarks, distinguishing prompt text from optimized output. The examples span diverse tasks, including question answering, summarization, semantic parsing, classification, style transfer, entity linking, and reasoning.

Prompt examples

  • The appendix presents prompt examples from all benchmarks, with black text denoting prompts and green text denoting outputs optimized in the loss function.This color convention is applied throughout the examples.Source: Appendix C · p. 36

Super-NaturalInstructions

  • Super-NaturalInstructions examples include question answering, program execution, and question generation tasks, with zero-shot examples documented in Figures 7–12 across these three benchmark categories.Source: Figure 7 · p. 36, Figure 8 · p. 36, Figure 9 · p. 36

FLAN and PromptSource

  • FLAN and PromptSource examples include question answering, summarization, paraphrasing, semantic parsing, text scoring, and intent classification.The displayed prompts include passage-based answers, document synthesis, and structured parsing tasks.Source: Figure 17 · p. 38, Figure 18 · p. 39, Figure 19 · p. 39

UnifiedSKG, CrossFit, and ExMix

  • UnifiedSKG, CrossFit, and ExMix examples extend the prompt formats to semantic parsing, question answering, formal-language-to-text, linguistic probing, information extraction, entity linking, and style transfer.These examples include SQL-related tasks, Wikipedia entity linking, Shakespearean rewriting, and knowledge-intensive tasks.Source: Figure 31 · p. 45, Figure 32 · p. 45, Figure 37 · p. 47

Additional benchmarks

  • Additional examples show toxic-language detection, summarization, and step-by-step reasoning prompts from ExMix, T5, and the Reasoning benchmark.The reasoning examples explicitly instruct the model to answer by reasoning step by step.Source: Figure 60 · p. 54, Figure 61 · p. 54, Figure 62 · p. 55

Related papers

Loading 2212.12017v3…