PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Task Generalization via Natural Language Crowdsourcing Instructions

Swaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh Hajishirzi

arXiv:2104.08773v4cs.CLcs.AIcs.CVcs.LG

TL;DR

The paper addresses whether models can learn new NLP tasks from human-readable instructions, rather than relying only on task-specific supervision. It constructs NATURAL INSTRUCTIONS and evaluates instruction-based models across seen and unseen tasks, finding a 19% cross-task generalization gain while observing substantial remaining headroom.

  • Problem

    Cross-task generalization remains under-explored: models trained on individual tasks often struggle to perform unseen tasks, despite humans’ ability to follow instructions across problems.

  • Method

    The paper constructs NATURAL INSTRUCTIONS from crowdsourcing instructions and task instances, maps instructions to a unified schema, and trains models on seen tasks for evaluation on unseen tasks.

  • Results

    19% gain in cross-task generalization is achieved by BART using instructions compared with a model not using instructions.

  • Takeaways & Limitations

    Models benefit from natural-language instructions for generalizing to new tasks, and broader task diversity in training is associated with better unseen-task generalization.

  • Takeaways & Limitations

    The benchmark excludes unseen tasks without relatively reliable automatic evaluation, and model generalization remains substantially below an estimated upper bound.

Abstract

from arXiv · show

Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.

1 Introduction

The paper studies whether models can generalize to unseen NLP tasks by understanding natural-language crowdsourcing instructions. It introduces NATURAL INSTRUCTIONS and finds that instructions improve cross-task generalization, while substantial headroom remains.

  • Research goal: Cross-task generalization asks whether models trained on seen tasks can perform unseen tasks from their instructions without task-specific labeled data.This differs from instance-level generalization, which evaluates on tasks observed during training.
  • Dataset: NATURAL INSTRUCTIONS contains 61 distinct NLP tasks and 193k instances compiled from crowdsourcing instructions for existing datasets.The instructions are mapped to a unified schema covering elements such as definitions, constraints, and positive and negative examples.
  • Dataset: The dataset includes minimal stand-alone subtasks, including intermediate activities from QASC alongside typical downstream NLP tasks.This design supports systematic study of how instruction elements relate to cross-task generalization.
  • Results: 19% gain in cross-task generalization is achieved by BART using instructions compared with a model not using instructions.The experiments indicate that language models learn to leverage natural-language instructions when generalizing to new tasks.
  • Results: Generalization to unseen tasks improves as models observe more tasks during training, but remains substantially below an estimated upper bound.The upward trajectory suggests potential from increasing task diversity, while the gap indicates considerable room for improvement.
  • Contributions: The paper contributes a benchmark and models for studying instruction-based cross-task generalization, emphasizing both its benefits and unresolved difficulty.The authors describe this as the first work to show improved cross-task generalization from instructions.

2 Related Works

The paper distinguishes instruction-based cross-task generalization from related approaches that focus on following short descriptions, task-specific prompts, or seen-task multitask performance. Its instructions and evaluation target unseen tasks explicitly.

  • Instruction learning: The paper differs from prior instruction-following work by constructing a dataset of crowdsourcing instructions and task instances and evaluating on unseen tasks.Training uses seen tasks, while evaluation measures generalization to the remaining unseen tasks.
  • Instruction learning: Compared with Weller et al., its instructions are longer, more complex, and more natural because they were used to collect datasets through crowdsourcing.The comparison concerns the data and instruction construction strategy.
  • Instruction construction: Unlike concurrent PromptSource and FLAN, the paper uses natural instructions written by NLP researchers before crowdworkers created dataset instances.These instructions contain complete task definitions, including constraints and negative examples.
  • Prompt engineering: Prompt-engineering approaches typically seek higher performance on a particular task, whereas this meta-dataset enables measurement of generalization to unseen tasks.The paper also contrasts detailed dataset-collection instructions with often extremely short prompts that may omit complex task definitions.
  • Multitask learning: The work extends conventional multitask learning by adding natural-language instructions to bridge from tasks seen during training to tasks not seen during training.Most conventional setups evaluate on instances belonging to observed tasks.

3 Defining Cross-Task Generalization

The paper formalizes cross-task generalization as predicting outputs from both an input and a task’s natural-language instruction, without task-specific training data. This contrasts with task-specific models that learn from labeled instances of the same task.

  • Formal setup: Each task consists of input/output instances and is described by natural-language instructions.The formal setup represents a task through its instances and instruction description.
  • Task-specific models: Task-specific models learn a mapping from input x to output y using labeled training instances from a task and are evaluated on test instances of that task.This is called instance-level generalization.
  • Cross-task models: Cross-task models predict y from the pair (I_t, x), where I_t is the task instruction and x is the input.The model learns a mapping that incorporates the task description at inference time.
  • Cross-task models: Cross-task models use no task-specific training data for the mapping and are trained on seen tasks before being evaluated on an unseen task.This setup is called task-level generalization.

4 NATURAL INSTRUCTIONS

NATURAL INSTRUCTIONS organizes crowdsourcing-derived NLP tasks into a unified instruction schema and pairs them with task instances. The resulting dataset contains 61 tasks and 193k instances, with curated instructions and extracted subtasks supporting systematic study.

  • 4.2.1 Collecting Data: NATURAL INSTRUCTIONS contains task instructions and instances collected from existing crowdsourced NLP datasets.The dataset focuses on textual instructions and excludes tasks involving visual or auditory steps.
  • 4.1 Instruction Schema: The unified instruction schema represents diverse task descriptions using fields including TITLE, PROMPT, DEFINITION, constraints, cautions, and examples.The schema also includes reasons and suggestions associated with positive or negative examples.
  • 4.1 Instruction Schema: THINGS TO AVOID defines undesirable annotations, while POSITIVE and NEGATIVE EXAMPLES illustrate acceptable and unacceptable inputs and outputs.REASON explains example labels, and SUGGESTION describes how negative examples could be changed into positive ones.
  • 4.2.1 Collecting Data: Crowdsourcing workflows are split into minimal, standalone subtasks so their instructions can be mapped consistently into the schema.For example, Quoref separates question generation and answer generation into distinct subtasks.
  • 4.2.1 Collecting Data: 61 tasks are organized into 6 semantic categories, with distinct task variants even within the same category.For example, question-generation tasks from different datasets are treated as separate variants.
  • 4.2.2 Mapping Raw Instructions to Schema: Curators manually map raw crowdsourcing instructions into schema fields, verify modifications, and edit writing issues and repetitions.They also add negative examples, reasons, and suggestions when those elements are missing.
  • 4.2.3 NATURAL INSTRUCTIONS Statistics: The dataset includes 61 tasks and 193k instances, with instructions averaging 4.9 positive and 2.2 negative examples.DEFINITION is usually the longest instruction element at 65.5 tokens, while TITLE is the shortest at 8.3 tokens.

5 Problem Setup and Models

The paper evaluates whether models can generalize from seen tasks to unseen tasks by following natural-language instructions. It defines several task-level splits and encodes instructions with inputs for BART and GPT-3 comparisons.

  • Task splits: Random splitting evaluates 12 unseen tasks and trains on 49 remaining tasks, with two evaluation tasks selected from each category.Tasks without relatively reliable automatic evaluation are excluded from the unseen-task set.
  • Task splits: Leave-one-category, leave-one-dataset, and leave-one-task settings impose progressively restrictive separations between training and evaluation tasks.The settings test generalization to an unseen category, dataset, or individual task.
  • Instruction encoding: The model maps an instruction and input instance into text, then an encoder-decoder predicts the output.Instruction elements are serialized and appended before the input instance.
  • Instruction encoding: Experiments compare encodings ranging from PROMPT and selected elements to FULL INSTRUCTION to assess which instruction components affect cross-task generalization.The compared elements include definitions, positive examples, things to avoid, and emphasis.
  • Models: The study fine-tunes 140m-parameter BART and evaluates GPT-3 as a 175B-parameter few-shot comparison model.GPT-3 is used as-is because its parameters cannot be fine-tuned in this setup.
  • Models: Table 4 reports cross-task generalization using ROUGE-L and compares instruction-based fine-tuned BART with GPT-3 across splits.The caption states that BART with instructions improves performance and exceeds GPT-3 despite being over 1k times smaller.

6 Experiments

The experiments evaluate cross-task generalization under multiple task splits, instruction encodings, task categories, and model settings. Instructions improve generalization, but performance remains well below task-specific estimates.

  • Evaluation setup: ROUGE-L evaluates generated outputs across random, category-held-out, dataset-held-out, and task-held-out splits, with GPT3 serving as a no-fine-tuning comparison.BART is fine-tuned on seen tasks, whereas GPT3 uses no fine-tuning.
  • Generalization across task splits: +19% gain under the random split shows that FULL INSTRUCTIONS improve BART’s cross-task generalization over NO INSTRUCTIONS.The instruction-using BART also achieves 8% higher performance than GPT3 under the random split despite being over 1k times smaller.
  • Generalization across task splits: Leave-one-category evaluation is more difficult because training and evaluation tasks are forced to be more dissimilar.All settings evaluate on unseen tasks, but leave-one-category removes every task from the target category during training.
  • Instruction encoding: Encoding more instruction elements generally outperforms using only PROMPT or POSITIVE EXAMPLES, with benefits varying by task category.POSITIVE EXAMPLES help question generation most, while DEFINITION is more helpful for classification; verification shows little improvement.
  • Number of seen tasks: Instruction-encoding models improve as they observe more seen tasks, whereas NO-INSTRUCTION models show no tangible value from observing more tasks.The trend suggests that increasing task diversity may strengthen instruction-following systems.

7 Conclusion

The paper introduces NATURAL INSTRUCTIONS to study whether models can generalize to new tasks by understanding crowdsourcing instructions. It reports improved cross-task generalization from instructions while identifying substantial room for improvement.

  • Conclusion: NATURAL INSTRUCTIONS is built from existing crowdsourced datasets to support systematic evaluation of instruction-based generalization to new tasks.The paper presents this as the first work to show improved cross-task generalization from instructions.
  • Conclusion: Models benefit from encoding crowdsourcing instructions, but the proposed task still has substantial room for improvement.The authors frame this open gap as motivation for stronger models that generalize across a wider range of tasks.

Supplemental Material

The supplemental material describes how tasks are decomposed and represented in NATURAL INSTRUCTIONS, and compares its broader, naturally collected instructions with PromptSource.

  • Task construction: MC-TACO divides five categories into paired question-generation and answer-generation subtasks.Each category contributes two subtasks, one for generating questions and one for generating answers.
  • Task construction: The six analyzed datasets vary in the number of steps used during data creation, with QASC and MC-TACO having relatively more steps.DROP, Quoref, CosmosQA, and Winogrande have relatively fewer steps in comparison.
  • Dataset documentation: The dataset’s task collection includes a detailed task inventory and a documented curation process with estimated effort by step.Parsing instances is identified as the main bottleneck in the curation process.
  • Instruction schema: NATURAL INSTRUCTIONS includes detailed task fields such as definitions, examples, and cautions, illustrated through examples from question generation and answer generation.The supplemental examples are drawn from MC-TACO and Winogrande tasks.
  • Comparison with PromptSource: Compared with PromptSource, NATURAL INSTRUCTIONS contains a broader variety of naturally collected tasks, including intermediate data-creation steps and tasks such as question typing.Its instructions were written before crowdworkers created the dataset instances, rather than retroactively from existing instances.

A.2 Analysis of Crowdsourcing Templates

The analysis of six crowdsourcing templates finds substantial variation in process structure, instruction size, examples, and explanatory guidance across datasets.

  • Dataset coverage: The analysis covers crowdsourcing templates from CosmosQA, DROP, MC-TACO, QASC, Quoref, and Winogrande.The comparison is intended to identify similarities and differences before collecting additional templates.
  • Instruction size and structure: Instruction size varies substantially across datasets: QASC uses very large step instructions, while Winogrande uses very small ones.The major mode of instruction also differs across datasets, including common instructions, step-specific instructions, and examples.
  • Examples: QASC has no negative examples, Winogrande has equal positive and negative examples, and DROP contains relatively more examples overall.These patterns show that example composition differs considerably across the source datasets.
  • Explanatory guidance: Quoref uniquely provides reasons for all positive and negative examples and suggestions with its negative-example reasons.Other datasets vary in whether explanations accompany positive or negative examples.
  • Subtask decomposition: MC-TACO illustrates how one data-creation task can be divided into multiple subtasks across categories and output types.Its five categories each contribute question-generation and answer-generation subtasks.

A.3 Qualitative Analysis

The qualitative analysis shows that crowdsourcing instructions vary in writing style, information content, and detail across tasks, while model inputs can encode instructions and examples in several ways.

  • Writing Style: Instructions for datasets with a common objective can differ substantially in writing style.DROP uses an AI-related warning, whereas Quoref emphasizes requiring paragraph understanding.
  • Information: Instruction information can transfer across related datasets even when it is absent from their own instructions.CosmosQA asks workers to skip inappropriate or offensive passages, a consideration also relevant to Quoref and DROP.
  • Hardness: Harder crowdsourcing tasks benefit from additional tips and detailed guidance.For question generation, QASC provides an easy and detailed creation procedure, while CosmosQA and Quoref provide different levels of additional information.
  • Instruction Encodings: Full instruction content can include task definitions, constraints, cautions, and example inputs, outputs, and reasons.Example encodings may alternate positive and negative examples or include only positive examples, subject to the input limit.
  • Instruction Encodings: The study compares instruction encodings ranging from no instructions and prompts to full instructions containing definitions, cautions, and positive and negative examples.The encodings map an instruction and input instance to plain text for language-model processing.

C Analysis on Baseline Results

The baseline analysis compares GPT-3 on raw crowdsourcing instructions with the paper’s structured encoding. GPT-3 performs better with the structured encoding, while raw instructions are much longer and more verbose.

  • Experimental Setup: The analysis compares performance on raw instructions with performance using the paper’s structured instruction encoding.It evaluates raw instructions and compares them with the FULL INSTRUCTION - NEG EXAMPLES encoding.
  • Results: GPT-3 achieves higher ROUGE-L performance with the paper’s encoding than with raw crowdsourcing instructions.The comparison is reported in Table 13, whose metric is ROUGE-L.
  • Results: Raw instructions average 2.5k tokens, compared with 950 tokens for the paper’s encoding.The authors suggest that concise instructions are more effective for computers than verbose raw instructions.
Loading 2104.08773v4…