PaperlayerView exact revision on arXiv

Source-linked AI summary

Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, Daniel Khashabi

arXiv:2204.07705v3cs.CLcs.AI

TL;DR

The paper addresses limited evidence about how NLP models generalize to unseen tasks from instructions and the lack of broad public benchmarks for studying this question. It introduces SUP-NATINST and Tk-INSTRUCT, then finds strong cross-task performance, including a 9.9 ROUGE-L advantage over much larger InstructGPT on 119 unseen English tasks.

  • Problem

    Limited public supervised data and inaccessible large models leave instruction-based generalization to unseen NLP tasks insufficiently studied.

  • Method

    The paper constructs SUP-NATINST, a benchmark of 1,616 instructed NLP tasks, and meta-trains Tk-INSTRUCT to follow declarative in-context instructions.

  • Results

    9.9 ROUGE-L points separate 11B-parameter Tk-INSTRUCT from 175B-parameter InstructGPT on 119 unseen English tasks.

  • Takeaways & Limitations

    Scaling analyses identify training-task diversity and model size as important factors for generalization to unseen tasks.

  • Takeaways & Limitations

    The benchmark is biased toward English and short outputs, underrepresenting the long tail of tasks.

Abstract

from arXiv · show

How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones. Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.

1 Introduction

The paper introduces SUP-NATINST, a large, diverse public benchmark for studying instruction-based generalization to unseen NLP tasks, and trains Tk-INSTRUCT on it. Tk-INSTRUCT outperforms much larger instruction-following models, while analyses identify task diversity and model size as important factors.

  • Limited public data and inaccessible large models make the role of supervised instruction data in generalization difficult to study.
  • Tk-INSTRUCT is a model trained on diverse instructed tasks to perform new tasks from declarative in-context instructions.
  • 1,616 tasks across 76 task types and 55 languages form the SUP-NATINST benchmark, with natural-language instructions and demonstrations.
  • 9.9 ROUGE-L points separate 11B-parameter Tk-INSTRUCT from 175B-parameter InstructGPT on 119 unseen English tasks.
  • 13.3 points separate multilingual mTk-INSTRUCT from InstructGPT on 35 non-English tasks, while human evaluation rates Tk-INSTRUCT at least as well as ground truth on 77% of instances.
  • Scaling analyses find that training-task diversity and model size are important for generalization to unseen tasks.

2 Related Work

Prior work studies instruction-based generalization and multi-task learning across NLP tasks, but available benchmarks differ in scale, task diversity, instruction style, and accessibility. SUP-NATINST extends this landscape while complementing other collections.

  • Recent NLP research targets generalization across tasks using either few-shot examples or natural-language task definitions.
  • SUP-NATINST extends NATINST with 26× more tasks and a greater variety of task types.
  • PROMPTSOURCE uses relatively concise task definitions, whereas SUP-NATINST aims for complete definitions averaging 56 tokens rather than 24.
  • BIG-BENCH provides 204 tasks emphasizing challenging tests of current-model behaviors, with little overlap with SUP-NATINST.
  • Unlike traditional multi-task learning evaluated on observed tasks, this work emphasizes cross-task generalization to unseen tasks.

3 SUPER-NATURALINSTRUCTIONS

SUP-NATINST is a community-built meta-dataset of diverse NLP tasks organized under a uniform instruction and instance schema. Its collection and quality-control procedures support systematic cross-task evaluation.

  • SUP-NATINST is a meta-dataset containing diverse NLP tasks and plain-language instructions describing them.
  • Instruction schema: Each instruction combines a natural-language definition with positive and negative input-output examples and explanations.
  • Task instances: Each task instance contains textual input and acceptable textual outputs, with no task exceeding 6.5K instances.
  • Benchmark collection: Tasks were contributed by NLP practitioners through a public GitHub-based community effort using existing, intermediate-annotation, and synthetic sources.
  • Quality control: Automatic tests, expert peer review, and iterative review were used to control task quality.
  • Diversity of tasks: Task-type, language, and domain categorizations support different generalization analyses, while the empirical studies focus on task types.
  • Statistics: The benchmark contains 1,616 tasks and 5M instances, with instructions averaging 2.8 positive examples, 2.4 negative examples, and 56.6 words.

4 Tk-INSTRUCT: Learning to Follow Instructions at Scale

Tk-INSTRUCT is meta-trained to solve NLP tasks from natural-language instructions, using SUP-NATINST to scale multi-task training across diverse tasks. Evaluation formalizes unseen-task generalization as producing outputs from task instructions and inputs.

  • For task t, a model maps an input x and natural-language instruction I_t to an output y.
  • Unseen-task evaluation excludes a task’s instances from training, leaving its definition and demonstration examples as the inference-time learning signal.
  • Tk-INSTRUCT is meta-trained on SUP-NATINST to solve tasks given in-context instructions.
  • SUP-NATINST enables multi-task meta-training at a larger scale because it contains a large variety of tasks.

5 Benchmarking Cross-Task Generalization with SUP-NATINST

SUP-NATINST benchmarks generalization to unseen tasks by separating evaluation tasks from supervision tasks across English and cross-lingual tracks. The benchmark uses ROUGE-L and human evaluation alongside diverse baselines and upper-bound estimates.

  • Evaluation Split of Unseen Tasks: 154 manually selected tasks form the unseen-task evaluation split, covering 12 categories and word-, sentence-, and document-level classification and generation.The remaining tasks provide supervision, with 119 English and 35 non-English evaluation tasks.
  • Divided Tracks for English and X-lingual Tasks: 119 English and 35 non-English tasks define separate tracks for English cross-task and cross-lingual cross-task generalization.The cross-lingual track evaluates generalization to unseen tasks in different languages.
  • Evaluation Metrics: ROUGE-L is the aggregated metric because it applies across diverse open-ended generation tasks, with ranking shown to correlate well with classification accuracy.The benchmark also includes human evaluation for language generation tasks.
  • Evaluation Split of Unseen Tasks: Training excludes tasks sourced from the same datasets as test tasks to avoid data leakage, producing 757 English and 1271 cross-lingual training tasks.Models are evaluated in an open-ended generation setting without task-specific assumptions.
  • Baselines and Existing Models: The benchmark compares heuristic baselines, pretrained language models, instruction-tuned models, and oracle models fine-tuned on labeled evaluation instances.The oracle models provide estimated upper bounds because they observe hidden evaluation-task instances.
  • Overall Benchmarking Results: Instruction-leveraging models show stronger unseen-task generalization, and the diverse-task Tk-INSTRUCT model outperforms InstructGPT and T0 by a large margin on aggregated ROUGE-L.Table 3 reports overall performance on unseen tasks in the SUP-NATINST test set.

6 Experimental Results

Instruction-tuned models generally generalize better to unseen tasks, with Tk-INSTRUCT leading across evaluation task categories. However, a sizable gap remains between instruction-based generalization and supervised training, while human evaluation supports the automatic results.

  • Overall Results: Instruction-tuned models generally outperform untuned language models and heuristic baselines on unseen tasks.The comparison includes Tk-INSTRUCT versus T5-LM and InstructGPT versus GPT-3.
  • Overall Results: Tk-INSTRUCT consistently outperforms other generalization-based methods across all evaluation task types, while remaining below supervised training.Figure 4 presents performance by evaluation task type.
  • Overall Results: T0 is only slightly better than T5-LM, possibly because its training prompts differ substantially from the instructions used here.The explanation is presented as a suspicion rather than an established cause.
  • Overall Results: Tk-INSTRUCT and mTk-INSTRUCT generalize best to unseen English and non-English tasks across all evaluation categories.The authors note that overlap between InstructGPT’s unavailable training data and the evaluation tasks is unclear.
  • Overall Results: A sizable gap remains between instruction-based models and supervised training, leaving room for improvement.This limitation is reported despite the strong performance of current models.
  • Human Evaluation: Human evaluation measures how often predictions are rated at least as good as ground-truth labels, complementing automatic evaluation for generation tasks.The theoretical upper bound of this metric is 100%.

7 Further Analysis

The analysis examines how training-task diversity, instances per task, model size, and instructional elements affect generalization to unseen tasks. Task diversity and model size improve performance, whereas additional instances yield limited gains and instruction encoding affects transfer.

  • Scaling Trends of Generalization: Generalization performance grows log-linearly as the number of training tasks increases, with the trend holding across 757 diverse training tasks.
  • Scaling Trends of Generalization: Performance saturates with only 64 instances per task, while more instances increase training time and risk overfitting to training tasks.
  • Scaling Trends of Generalization: 48.0 ROUGE-L from T5-large trained on 757 tasks is comparable to 48.4 ROUGE-L from T5-3B trained on 128 tasks.
  • Scaling Trends of Generalization: Increasing model size consistently improves performance log-linearly with parameter size.
  • Instructing with Different Elements: Task definitions consistently improve generalization, and combining them with positive demonstrations yields further improvement, while additional demonstrations provide negligible gains.
  • Instructing with Different Elements: Negative examples help slightly, whereas explanations decrease performance when the model is not large enough.
  • Instructing with Different Elements: Definition-only and example-only models fail to generalize across those encoding types, while models trained with both are robust across encoding variations.

8 Conclusion

The paper introduces a diverse benchmark of NLP tasks and instructions, trains Tk-INSTRUCT on it, and evaluates generalization to unseen tasks. It provides analysis of factors affecting this generalization and releases resources intended to support future work on more general-purpose models.

  • The benchmark contains a diverse set of NLP tasks and their instructions for training or evaluating models on unseen tasks.
  • Tk-INSTRUCT is trained on the benchmark and demonstrates substantial capability on unseen tasks.
  • The paper analyzes important factors governing generalization across tasks and aims to facilitate future research on more general-purpose models.

9 Limitations

The paper identifies limitations in benchmark coverage, automatic evaluation, and computational reach. These constraints motivate broader task distributions, stronger generation metrics, and models beyond the accessible compute budget.

  • The benchmark is biased toward English, short outputs, and certain task distributions, under-representing long-tail tasks.The authors leave these distributional imbalances for future work and note extensions to vision or speech.
  • ROUGE-L aligns with human evaluation overall but may mismeasure rewriting and error-correction quality.Copying the input can produce a high ROUGE-L score for some such tasks.
  • The experiments cover only models accessible to the authors, excluding larger models because of computational-budget limitations.

Supplemental Material

The supplemental material describes crowdsourced quality checks for instructions and model outputs, alongside the simplified instruction schema used for data collection.

  • Instruction quality assessment: Crowdworkers reviewed collected instructions using task definitions and positive or negative examples presented in a feedback template.Workers were predominantly from English-speaking countries and had substantial prior HIT experience with high approval ratings.
  • Generation quality assessment: Human evaluation compared model predictions with benchmark ground-truth outputs, allowing preferences, ties, and equal-or-better judgments.The resulting score averages instance-level credit over evaluated examples.
  • Evaluation scope: Human evaluation was restricted to English tasks because non-English-speaking crowdworkers were difficult to recruit.
  • Generation quality assessment: The human-evaluation metric reached 77% of testing instances where model responses were rated at least as good as ground truth.
  • Instruction schema: The instruction schema simplifies NATINST by merging several fields into DEFINITION and dropping TITLE and PROMPT.

D Model Implementation Details

Tk-INSTRUCT encodes each instruction with its input and fine-tunes encoder-decoder T5 models, evaluating them with ROUGE-L alongside classification accuracy and human assessment.

  • Model configuration: The main T5-11B experiments use the T5.1.1.xxl checkpoint, while smaller analyses use Hugging Face T5 checkpoints on GPUs.
  • Training setup: Fine-tuning uses two epochs, a 16-example batch size, learning rate 1e-5, maximum input length 1024, and maximum output length 128.Training uses eight 48GB A100 GPUs with DeepSpeed and bfloat16 precision.
  • Instruction encoding: Tk-INSTRUCT maps an instruction and input into a textual sequence that an encoder-decoder model uses to predict the output.Instruction elements are converted to text and prepended to the input.
  • Evaluation: ROUGE-L is the aggregated automatic metric for the diverse open-ended tasks, with human evaluation also conducted.The paper reports that ROUGE-L rankings correlate well with classification accuracy.
  • Evaluation coverage: The task collection spans diverse types, languages, and domains but remains skewed toward certain tasks and languages.

H Performance Improvement per Evaluation Task

Per-task analyses compare Tk-INSTRUCT models with heuristic baselines across English and cross-lingual evaluation tasks. The figures also document benchmark composition and evaluation categories.

  • Performance improvement: Tk-INSTRUCT shows notable generalization on most evaluation tasks compared with the best of two heuristic baselines.Figure 11 covers English tasks, while Figure 12 covers cross-lingual tasks.
  • Benchmark composition: Figure 10 summarizes task distributions by task type, language, and domain using logarithmic y-axes.
  • Evaluation categories: The evaluation organizes tasks into 12 categories with category-specific Exact Matching or ROUGE-L metrics.Table 5 lists the tasks included in each category.
  • Cross-lingual track: Figure 12 reports per-task improvement for mTk-INSTRUCT on 35 cross-lingual evaluation tasks.
Loading 2204.07705v3…