PaperlayerView exact revision on arXiv

Source-linked AI summary

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, Ves Stoyanov

arXiv:2212.12017v3cs.CL

TL;DR

The paper addresses limited understanding of instruction-tuning trade-offs by building a large benchmark and evaluation framework spanning multiple generalization levels. Using insights from OPT-IML Bench, it trains OPT-IML at 30B and 175B scales, achieving improvements over OPT across four instruction-tuning benchmarks and competitive performance with benchmark-specific models.

  • Problem

    The performance trade-offs of instruction-tuning decisions, including task diversity, sampling, demonstrations, specialized datasets, and objectives, remain insufficiently understood.

  • Method

    The authors consolidate 1,991 NLP tasks from eight meta-datasets into OPT-IML Bench and evaluate three levels of generalization across categorized tasks.

  • Results

    OPT-IML significantly improves over OPT at 30B and 175B scales on PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG, while remaining competitive with models individually tuned on those benchmarks.

  • Takeaways & Limitations

    OPT-IML Bench provides a framework for characterizing instruction-tuning trade-offs and evaluating fully supervised, unseen-task, and held-out-category generalization.

  • Takeaways & Limitations

    Trade-offs studied with 30B instruction tuning may not show the same trends at larger scales, and different category choices or assignments may change the preferred tuning decisions.

Abstract

from arXiv · show

Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework.

1. Introduction

The paper studies how instruction-tuning choices affect generalization and develops OPT-IML Bench to evaluate these choices across multiple levels of task generalization. Insights from OPT-30B experiments guide OPT-IML models at 30B and 175B scales.

  • Instruction-tuning improves large language models’ zero- and few-shot performance, but its effectiveness depends on objectives, task diversity, sampling, demonstrations, and specialized datasets.
  • OPT-IML Bench consolidates 1,991 NLP tasks from eight meta-datasets into categorized tasks with multiple prompts.
  • The evaluation framework measures generalization to fully held-out categories, unseen tasks in seen categories, and held-out instances from seen tasks.
  • Experiments with OPT-30B characterize dataset sampling, task and category scaling, demonstrations, reasoning datasets, and dialogue datasets.
  • OPT-IML 30B and 175B significantly outperform OPT across PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG, while remaining competitive with benchmark-specific instruction-tuned models.

2. Scaling up Multi-task Benchmarks

The benchmark aggregates diverse instruction-tuning resources into categorized tasks, standardizes their prompt representation, and creates leakage-controlled splits for three generalization settings.

  • 2. Scaling up Multi-task Benchmarks: OPT-IML Bench aggregates eight task collections into a benchmark for massive instruction fine-tuning and evaluation across diverse categories, instruction types, and prompting setups.
  • 2. Scaling up Multi-task Benchmarks: Tasks and datasets are treated interchangeably, each task can use multiple prompt templates, and each benchmark contains tasks assigned to one category.
  • 2.1 Task Curation: The collection includes instruction-tuning, prompt-crowdsourcing, transfer, structured-knowledge, dialogue, and chain-of-thought reasoning sources, with 14 reasoning datasets.
  • 2.1 Task Curation: Because source benchmarks overlap substantially, the curation removes duplicate CrossFit, ExMix, and T5 tasks and caps examples at 100k per task, or 30k for FLAN.
  • 2.2 Benchmark Consolidation: All collected tasks are cast into instruction-and-output prompts, while manually written instructions supplement benchmarks lacking natural-language instructions.
  • 2.2 Benchmark Consolidation: Tasks are grouped using a manually unified taxonomy based primarily on Super-NaturalInstructions’ 76 categories, enabling cross- and within-category evaluation.
  • 2.3 Train, validation and test splits: Splits hold out entire categories, tasks within seen categories, or instances from seen tasks, while task de-duplication checks 13-gram overlap to limit leakage.
  • 2.2 Benchmark Consolidation: Zero-shot prompts use the bipartite instruction scheme, and few-shot prompts insert demonstrations between task descriptions and target examples; templates altering task semantics are removed.

3. Instruction Fine-tuning

OPT-IML fine-tunes OPT with target-only next-token loss on instruction-and-input sources, using mixed datasets and packed sequences with document attention masking. The implementation scales across 30B and 175B models.

  • 3. Instruction Fine-tuning: OPT-IML fine-tunes OPT, an open-source decoder-only transformer family spanning 125M to 175B parameters.
  • 3.1 Fine-tuning Objective: The objective predicts each target token conditioned on the source sequence and previously seen target tokens.
  • 3.1 Fine-tuning Objective: Task instructions and inputs are source tokens, while label tokens are target tokens whose prediction contributes to the loss.
  • 3.1 Fine-tuning Objective: Examples from different datasets are mixed according to dataset sizes and benchmark proportions.
  • 3.2 Packing and Document Attention: Multiple source-target examples are packed into 2048-token sequences separated by <eos> tokens for computational efficiency.
  • 3.2 Packing and Document Attention: Document attention masking restricts each example to its own tokens and changes the causal mask from triangular to block triangular.
  • 3.3 Fine-tuning Hyperparameters: 30B models use 64 40GB A100s, whereas 175B models use 128 40GB A100s, with Fully Sharded Data Parallel and Megatron-LM tensor parallelism.
  • 3.3 Fine-tuning Hyperparameters: Table 3 reports fine-tuning parameters, training times, and token counts for all OPT-IML models.

4. What Matters for Instruction Fine-tuning?

The experiments evaluate how task composition, scale, auxiliary data, and demonstration-based objectives affect instruction-tuning generalization. Results show benefits from diverse benchmarks and more tasks, but trade-offs from benchmark proportions, pre-training data, dialogue, and MetaICL settings.

  • 4.1 Experimental Setup: The setup varies dataset composition, task number and diversity, auxiliary pre-training, reasoning and dialogue data, and demonstration-based fine-tuning.Models are evaluated after 4000 steps across 0-shot and 5-shot settings and three generalization levels.
  • 4.3 Effects of varying benchmark proportions: Increasing FLAN’s proportion improves fully held-out and partially held-out FLAN performance, while benchmark-specific gains differ across SuperNatInst and PromptSource.PromptSource benefits most on fully held-out clusters, whereas its other generalization levels remain relatively unchanged.
  • 4.3 Effects of varying benchmark proportions: 88.8/83.6% is the highest accuracy on fully held-out FLAN, achieved by increasing PromptSource and Crossfit rather than maximizing FLAN itself.The best fully held-out PromptSource generation performance is 79.7/83.5% with 25% PromptSource, while optimized proportions can reduce reasoning performance.
  • 4.3 Effects of varying benchmark proportions: Removing Crossfit, Exmix, T5, and Unified-SKG produces the worst model, supporting diverse benchmark mixtures for instruction-tuning.The authors select “4/2/20/25/45/2/2” for the final OPT-IML models based on average benchmark performance.
  • 4.5 Effects of Pre-training during Instruction-Tuning: Adding pre-training data improves fully held-out and partially supervised performance up to 10%, after which performance deteriorates.More pre-training data raises Rouge-L F1 but lowers accuracy; the final models use 5% pre-training data.
  • 4.7 Effects of Adding Dialogue Datasets: Adding 0.5% dialogue data lowers 0-shot performance and damages tasks requiring tightly formatted outputs, so the final models exclude dialogue data.The degradation includes 5-shot toxicity detection and reflects weakened conformity to special decision-word formats.
  • 4.8 Effects of Meta-Training for In-Context Learning: MetaICL generally worsens 0-shot and 5-shot performance, with severe degradation on formatted generation and reasoning tasks; suffix loss performs better than regular MetaICL.The models may overfit to demonstration separators and generate short answers despite reasoning chains.

5. OPT-IML Models

OPT-IML models are instruction-tuned OPT models built from the study’s selected settings and evaluated across multiple benchmarks, task types, and shot settings. They generally improve over OPT and are competitive with benchmark-specific instruction-tuned models, though performance varies by task and format.

  • Evaluation setup: OPT-IML is evaluated in zero- and few-shot settings on OPT tasks and four multi-task benchmarks, with direct comparisons to benchmark-specific instruction-tuned models.The comparisons include FLAN-137B, T0pp 11B, Tk-Instruct 11B, and UnifiedSKG models or baselines.
  • OPT evaluation: Approximately 6-7% higher 0-shot accuracy is reported than OPT at both 30B and 175B, while 32-shot gains are stronger at 30B than at 175B.Improvements are task-dependent: gains appear on RTE, WSC, BoolQ, ARC, CB, and WiC, but not on StoryCloze, PIQA, Winograd, and Winogrande.
  • PromptSource evaluation: OPT-IML 175B matches T0-original-task’s zero-shot performance and significantly exceeds it in 5-shot performance.Both models were not trained on demonstrations, while the passage attributes the few-shot difference partly to causal versus encoder-decoder model behavior.
  • PromptSource evaluation: OPT-IML 30B outperforms baseline OPT 175B on average, indicating that instruction tuning can make smaller models more competitive with larger untuned models.This comparison is reported in the PromptSource evaluation discussion.
  • UnifiedSKG evaluation: UnifiedSKG results preserve Rouge-L F1 near 85 on Spider, improve zero-shot DART performance, and show deterioration on MultiWoZ after instruction tuning.The DART gain includes OPT-IML 30B outperforming OPT 175B, while MultiWoZ declines at both model scales.

6. Discussion and Limitations

The discussion presents OPT-IML as a broadly effective instruction-tuning approach while identifying limits in experimental interpretation and risks associated with large language models. The authors also report weaker results than several instruction-tuned systems on some external benchmarks.

  • Discussion: OPT-IML improves over untuned models on zero- and few-shot evaluations after scaling instruction-tuning data across eight task collections and three generalization levels.The framework studies task diversity, task and instruction-style distributions, reasoning and dialogue datasets, and demonstration-based fine-tuning.
  • External evaluations: OPT-IML-Max outperforms untuned counterparts across MMLU, RAFT, and BBH at nearly all tested scales, except the 1.3B model on BBH.It is competitive with FLAN-T5 11B on RAFT but lags FLAN-T5, FLAN-PaLM, and instruction-tuned GPT-3 models on MMLU and BBH.
  • External evaluations: The paper cannot definitively attribute improvements to scaling the instruction-tuning benchmark because models differ in pre-training tokens, data composition, held-out clusters, and model sizes.The passage contrasts OPT’s 180B pre-training tokens with 1T for T5 and 800B for FLAN-PaLM.
  • Limitations: Tradeoffs measured independently on OPT 30B may change when tuning variables interact, when models scale, or when different task-category assignments are used.The authors note that category assignments are partly subjective and that different skills can be expressed through the same format.
  • Responsible AI: OPT-IML remains susceptible to risks involving factual correctness, toxic language generation, and stereotype enforcement, motivating responsible-use practices.These risks are stated as applicable despite the models’ reported evaluation improvements.

7. Related Work

The paper situates instruction-based fine-tuning at the intersection of multitask learning, prompting, meta-training, and specialized reasoning approaches. It uses diverse prompts and demonstrations to study instruction-tuning settings that support robustness across tasks and prompting setups.

  • Prompting and Meta-Training: Prompting modifies task inputs and outputs, while the paper studies instruction-based fine-tuning using prompt variants and demonstrations from many tasks.
  • Learning to Reason: Reasoning research uses rationales or scratchpads to address challenging commonsense and arithmetic reasoning tasks.
  • Multi-task Learning: Instruction-based fine-tuning unifies tasks through instructions and shares model weights to improve generalization to unseen tasks.This contrasts with multitask learning work focused primarily on training tasks or new domains.
  • Continuous Learning: Continuous-learning methods revisit prior instructions or examples to adapt language models while preventing catastrophic forgetting.

8. Conclusions

The paper contributes a large instruction-tuning benchmark and evaluation suite, uses it to study tuning tradeoffs, and releases instruction-tuned OPT models. These models strongly outperform OPT and remain competitive with models tuned on individual benchmarks.

  • 8. Conclusions: The benchmark contains about 2,000 NLP tasks from eight collections, with splits evaluating fully supervised, unseen-task, and held-out-category generalization.
  • 8. Conclusions: The evaluation suite establishes tradeoffs and best practices for task sampling, demonstrations, and specialized reasoning and dialogue datasets.
  • 8. Conclusions: OPT-IML 30B and 175B strongly outperform OPT on five evaluation benchmarks and compete with recently instruction-tuned models.

Appendix A. Benchmark Preparation Details

The appendix describes how the paper obtains and preprocesses tasks from the benchmark collections used for instruction-tuning and evaluation.

  • Appendix A. Benchmark Preparation Details: The benchmark-preparation appendix provides details about downloading and preprocessing tasks from the paper’s source benchmarks.

A.1 Data Curation

The data-curation process downloads benchmark data from official releases or author-provided sources, retaining selected datasets and templates after overlap and applicability checks.

  • A.1 Data Curation: Most benchmarks are downloaded from official data releases, while ExMix uses alternative sources because its official data is not open-sourced.
  • A.1 Data Curation: Super-NaturalInstructions data is downloaded from its AllenAI repository.
  • A.1 Data Curation: PromptSource retains original-task templates and excludes validation or test templates that do not apply to all examples, except for turk.
  • A.1 Data Curation: CrossFit data is downloaded from its repository at the specified commit.
  • A.1 Data Curation: FLAN uses all task-example instantiations while retaining original-task templates and excluding templates that invert the task.
  • A.1 Data Curation: ExMix contributes selected non-overlapping tasks from several repositories, including COGS, StylePTB, and Parsing to FunQL.
  • A.1 Data Curation: After removing overlap with other benchmarks, the T5 collection retains seven datasets, including translation and CNN/DailyMail.
  • A.1 Data Curation: UnifiedSKG examples are downloaded from the Google Drive link provided by its authors.

A.2 Details of Dialogue Datasets

The dialogue experiments use six datasets spanning internet and Wikipedia wizards, blended conversation, ConvAI2, multi-session chat, and wild dialogue.

  • Six dialogue datasets are included in the experiments: Wizard of Internet, Wizard of Wikipedia, Blended Skill Talk, ConvAI2, Multi-Session Chat, and Light+ Wild.These datasets form a subset of those used by Shuster et al. (2022).

A.3 Details on Validation Tasks

The validation set contains multiple tasks and categories, with a companion table documenting their benchmarks, generalization levels, and evaluation metrics.

  • The validation set comprises several tasks and categories used in the experimental studies.
  • Table 15 lists each validation task’s category, benchmark, generalization level, and evaluation metric.Some listed tasks contain subtasks, such as MMLU, that are omitted from the table.

B.1 Results at Additional Model Scales

The appendix extends the evaluation and ablation results across additional model scales and reports how task, cluster, pre-training-data, reasoning-data, and MetaICL settings affect performance.

  • Results at Additional Model Scales: OPT-IML 1.3B was trained by fine-tuning OPT 1.3B with the same settings used for OPT-IML 30B.
  • Results at Additional Model Scales: Table 16 extends the 14-task OPT comparison with results at the 1.3B scale in 0-shot/32-shot format.
  • Additional Experimental Results: Tables 17–20 provide full ablation results for scaling training tasks, scaling task clusters, adding pre-training data, and adding reasoning data.These results are aggregated by task category and generalization level, using 0-shot/5-shot reporting.
  • Additional Experimental Results: Table 21 repeats the MetaICL experiments with “\n\n” as the inference example separator, under which all MetaICL models outperform the baseline.The table notes that its 5-shot baseline is not comparable with other experiment tables because summarization tasks are included.

Appendix C. Examples of Prompts from All Benchmarks

The appendix illustrates the prompts used for instruction-tuning and zero-shot evaluation across the paper’s benchmarks, covering diverse task categories and output formats.

  • Prompt Presentation: Prompt examples distinguish instruction text from optimized outputs, with black text representing prompts and green text representing outputs.
  • Super-NaturalInstructions: Super-NaturalInstructions examples span question answering, program execution, question generation, sentiment analysis, and text matching tasks.
  • FLAN: FLAN examples cover question answering, summarization, paraphrasing, sentiment analysis, and data-to-text tasks.
  • PromptSource: PromptSource examples include question answering, summarization, semantic parsing, text scoring, and intent classification.The multi_news presentation truncates part of the input and substitutes “[...continued]”.
  • UnifiedSKG: UnifiedSKG examples include semantic parsing, question answering, formal-language-to-text, and data-to-text tasks.
  • CrossFit and ExMix: Additional examples cover linguistic probing, information extraction, entity linking, style transfer, semantic parsing, and knowledge-intensive language tasks.These examples come from CrossFit and ExMix benchmarks.
Loading 2212.12017v3…