Cross-Task Generalization via Natural Language Crowdsourcing Instructions

1 Introduction

• The evaluation trains on seen tasks and tests whether models can perform unseen tasks from instructions alone, without task-specific labeled data.Unlike instance-level generalization, this setup provides instructions as additional input and evaluates on tasks not observed during training.Source: §1 · pp. 1–2
• Generalization to unseen tasks improves as models observe more tasks in training, while a sizable gap remains from estimated upper bounds.The upward trajectory suggests potential from scaling the diversity of tasks in a meta-dataset of instructions.Source: Figure 2 · p. 2, §1 · p. 2
• NATURAL INSTRUCTIONS contains 61 NLP tasks and 193k instances, including intermediate subtasks, with task descriptions mapped to a unified schema.The schema covers definitions, constraints, and positive and negative examples; tasks are minimal stand-alone steps for crowdworkers.Source: §1 · p. 2
• 19% gain in cross-task generalization was achieved by BART compared with a model not using instructions.The introduction reports this as an example of improved generalization to new tasks.Source: §1 · p. 2
• The work also investigates the importance of different instruction elements for model performance.This is listed among the study’s contributions alongside instruction-based generalization and benchmark headroom.Source: §1 · p. 2

Learning from instructions

• Unlike prior instruction-following studies and short task-description datasets, this work builds a natural-instruction dataset and evaluates generalization from seen to unseen tasks.For PromptSource and FLAN, instructions are collected retroactively from existing instances; here, NLP researchers provide them before data collection, including complete task definitions, and the approaches are described as complementary.Source: §2 · pp. 2–3

Prompt engineering

• Unlike short, task-specific prompt-engineering approaches, this meta-dataset uses detailed dataset-collection instructions to measure generalization to unseen tasks.The section notes that prompt-engineering approaches often omit complete definitions of complex tasks and make assumptions that are non-trivial to generalize.Source: §2 · p. 3

Beyond standard multi-task learning

• Conventional multitask learning generally evaluates on tasks seen during training; natural language instructions extend this setup to unseen tasks.In the conventional setup, evaluation instances come from tasks whose labeled instances were observed during training.Source: §2 · p. 3

3 Defining Cross-Task Generalization

• Each task consists of input-output instances and is described by natural-language instructions.Source: §3 · p. 3
• Task-specific models learn an input-to-output mapping from labeled instances and are evaluated on test instances of the same or a similar task.The section calls this instance-level generalization.Source: §3 · p. 3
• Cross-task models use an instruction and input to produce an output, without task-specific training data, and are evaluated on following instructions for unseen tasks.The setup asks whether models trained to follow instructions on seen tasks can generalize to unseen ones.Source: §3 · p. 3

4.1 Instruction Schema

• The unified schema includes TITLE, PROMPT, DEFINITION, and THINGS TO AVOID to capture a task, connect inputs to instructions, specify requirements, and define unacceptable responses.TITLE describes the task and skill; PROMPT is a command often placed before the input.Source: §4.1 · p. 4
• The schema also represents EMPHASIS AND CAUTION, positive and negative examples, REASON explanations, and SUGGESTION edits that could turn negative examples positive.Negative examples illustrate outputs to avoid, while positive examples help explain the task.Source: §4.1 · p. 4

4.2.1 Collecting Data

• The dataset draws on crowdsourcing templates and raw data from existing NLP benchmarks, focusing on textual instructions and excluding datasets requiring visual or auditory steps.The authors also excluded tasks involving model-in-the-loop.Source: §4.2.1 · p. 4, §4.2.2 · p. 5
• Crowdsourcing instructions are split into minimal, standalone subtasks, such as separating Quoref question generation from answer generation.QASC and MCTACO templates contained seven and nineteen steps, respectively.Source: §4.2.1 · pp. 4–5

4.2.2 Mapping Raw Instructions to Schema

• Authors manually mapped crowdsourcing content into schema fields, incorporated highlighted material into EMPHASIS AND CAUTION, and verified modifications with a second author.They also edited instructions to fix writing issues and remove repetition.Source: §4.2.2 · p. 5
• To improve coverage, they added negative examples so tasks had at least two where possible, plus missing reasons or suggestions for examples.The raw instructions varied in how many examples they supplied and often lacked negative examples, reasons, or suggestions.Source: §4.2.2 · p. 5
• Because subtasks often represent intermediate crowdworker steps, the authors parsed annotations into input-output instances, then iterated on dataset-author feedback; human responses to 240 instances exceeded 96% accuracy.An annotator verified the resulting data in consultation with dataset authors, averaging three feedback iterations.Source: §4.2.2 · p. 5

4.2.3 NATURAL INSTRUCTIONS Statistics

• 61 tasks and 193k instances span six semantic categories; instructions average 4.9 positive and 2.2 negative examples, with DEFINITIONS longest at 65.5 tokens and TITLE shortest at 8.3.The categories group tasks for analysis, but tasks within a category remain distinct.Source: §4.2.1 · p. 5, §4.2.3 · p. 6

5.1 Task Splits and Generalizations Types

• Random splits hold out two tasks per category, yielding 12 evaluation tasks and 49 training tasks; tasks without reliable automatic evaluation are excluded from evaluation.Source: §5.1 · p. 6
• Leave-one-category, leave-one-dataset, and leave-one-task splits test generalization to a category, a dataset, or one task after training on the others.The leave-one-dataset split excludes tasks from the same source dataset to prevent leakage across them.Source: §5.1 · p. 6

5.2 Models

• The experiments use fine-tuned 140m-parameter BART and few-shot 175B-parameter GPT3, which is evaluated as-is without fine-tuning.GPT3 is described as approximately 1.2k times larger than BART.Source: §5.2 · pp. 6–7
• The model encodes the instruction and input as text, appends textualized instruction elements before the input, and predicts the output.Instruction encodings range from individual components and combinations to FULL INSTRUCTION.Source: §5.2 · p. 6
• Fine-tuned BART performs better with instructions and outperforms GPT3 despite being over 1k times smaller.Table 4 reports ROUGE-L results across the cross-task splits.Source: Table 4 · p. 7

6.1 Generalization Under Various Task Splits

• Under the random split, FULL INSTRUCTIONS yields +19% over BART without instructions, and instruction-using BART scores 8% higher than GPT3.The paper also reports that instruction benefits hold across the evaluated task splits, including leave-one-category-out generalization.Source: §6.1 · p. 7

6.2 Generalization Under Instruction Encoding and Task Categories

• Encoding more instruction elements generally improves results, but their benefits differ across task categories: positive examples help QG most, while classification and verification show limitations.Positive examples offer little help for classification, where DEFINITION may be more useful; the authors hypothesize that verification’s limited improvement reflects the tasks’ distinctness from other dataset tasks.Source: §6.2 · pp. 7–8

6.3 Generalization vs. Number of Seen Tasks

• With instruction encoding, generalization improves as the number of seen tasks increases; without instructions, observing more tasks brings no tangible value.The authors suggest that scaling the dataset to include more tasks may strengthen instruction-following systems.Source: §6.3 · p. 8

6.4 Analyses

• The authors characterize negative instructions as surprisingly difficult for models to learn from when evaluating the effect of excluding negative examples from FULL INSTRUCTION encoding.Source: Table 6 · p. 8
• Task-specific BART models average 66%, versus 32% for the best generalization model, indicating substantial room to improve instruction-based generalization.The task-specific models are trained separately on each task’s annotated training data and provide a loose upper-bound estimate.Source: §6.4 · p. 8
• BART offers more control than GPT3 but can still generate nonsensical or vague questions, copy or alter input facts incorrectly, or produce yes/no questions.These are examples of erroneous BART predictions in the error analysis.Source: §6.4 · p. 9
• Human annotators’ judgments broadly align with model results for instruction elements, excluding negative examples; they consider DEFINITION and THINGS TO AVOID useful for classification and minimal text modification, respectively.This aligns with the reported high score for PROMPT + DEFINITION in the classification category.Source: §6.4 · p. 9, Table 7 · p. 9

• NATURAL INSTRUCTIONS, built from existing crowdsourced datasets, enables building models that generalize to new tasks and systematically evaluating them.Source: §7 · p. 9
• The authors report this is the first work to show that instructions improve cross-task generalization.They qualify this as being to the best of their knowledge.Source: §7 · p. 9
• The proposed task leaves substantial room for improvement, which the authors believe can draw more attention to building stronger models that generalize across a wider range of tasks.Source: §7 · p. 9

A.1 Division of Crowdsourcing Instructions into Minimal Tasks

• In MC-TACO, each of five categories contributes two subtasks: one for question generation and one for answer generation.Figure 9 is identified as an example of this task division.Source: §A.1 · p. 12
• QASC and MC-TACO have relatively more data-creation steps than DROP, Quoref, CosmosQA, and Winogrande.Figure 6 illustrates variation in the number of subtasks across six datasets.Source: §A.1 · p. 12, Figure 6 · p. 12

A.4 Data Curation Effort

• Parsing instances in Step 8 is the main bottleneck in NATURAL INSTRUCTIONS’ data-curation process.Table 9 presents the curation steps and their estimated times, but no specific times are supplied here.Source: §A.4 · p. 13, Table 9 · p. 13
• The supplement presents a detailed task inventory and examples of four NATURAL INSTRUCTIONS tasks that follow the schema in Figure 4.The inventory is in Table 10, and the examples are in Figure 11.Source: §A.4 · p. 13, Figure 11 · p. 14, Table 10 · p. 15

A.5 Qualitative Comparison to PromptSource

• NATURAL INSTRUCTIONS covers intermediate data-creation tasks beyond PromptSource’s common NLP tasks, including fact chaining, question typing, and inappropriate-content detection.The examples are tasks 38, 27, and 22, respectively, listed in Table 10.Source: Table 10 · p. 15, §A.5 · p. 16
• NATURAL INSTRUCTIONS’ task definitions are elaborate and provide the complete task definition for crowdworkers.An example defines an “absolute timepoint” and explains it with the example of going to school usually happening during the day.Source: §A.5 · p. 16
• The structured instruction format allows researchers to ablate definitions and positive or negative examples and quantify their contributions.The passage points to the empirical analysis in §6.Source: §A.5 · p. 16
• GPT3 performs better with NATURAL INSTRUCTIONS than with PromptSource instructions, despite the former being lengthier.The passage attributes the comparison to Table 11 but supplies no performance values.Source: Table 11 · p. 16, §A.5 · p. 16

Size of the instructions

• Instruction size and task explanation vary across datasets: QASC emphasizes step instructions, Winogrande and DROP common instructions, Quoref examples and step instructions, and CosmosQA examples.MC-TACO relies on examples to explain its task; Winogrande and QASC rely mostly on common and step instructions, respectively.Source: §A.2 · p. 12, Figure 8 · p. 12

The number of positive/negative examples

• Only Winogrande has equal numbers of positive and negative examples; QASC has no negative examples, while DROP has relatively more examples than other datasets.Figure 7 illustrates variation in positive and negative example counts across the datasets.Source: §A.2 · p. 12, Figure 7 · p. 12

Presence of reasons/suggestions in examples

• All datasets except QASC include positive and negative examples; Quoref alone gives reasons for every example and suggestions alongside reasons for negative examples.Explanations accompany negative examples, while explanations for positive examples vary across datasets.Source: §A.2 · pp. 12–13

A.3 Qualitative Analysis

• Instructions differ substantially in writing style even across datasets with a common objective, as illustrated by DROP and Quoref.DROP’s instruction describes an AI that may give the same answer, while Quoref’s emphasizes avoiding questions answerable without understanding the passage.Source: §A.3 · p. 13
• Some instruction information could apply to related datasets but is absent there; CosmosQA asks users to skip offensive passages, unlike Quoref and DROP.All three datasets involve reading comprehension, and the passage-skipping guidance is described as relevant to Quoref and DROP.Source: §A.3 · p. 13
• Tips for hard tasks can help with data creation: QASC gives an easy, detailed question-generation approach, while CosmosQA describes question-quality attributes that may help Quoref.The figure contrasts Quoref’s single-line instruction with CosmosQA’s detailed instruction and QASC’s examples alongside instructions.Source: §A.3 · p. 13, Figure 10 · p. 13

B Building Baselines for NATURAL INSTRUCTIONS

• The baseline section compares input representations ranging from no instructions to prompt, prompt-plus-definition, and full-instruction encodings.These alternatives encode an instruction and input instance as plain text in different ways.Source: §B.1 · p. 17

B.1 Encoding of the instructions

• The PROMPT + DEFINITION encoding places both the prompt and task definition before the input, making it more informative and complex than PROMPT.PROMPT appends the prompt message before the input.Source: §B.1 · p. 17
• FULL INSTRUCTIONS includes the definition, things to avoid, emphasis or caution, and positive-example material.The encoding can include as many alternating positive and negative examples as possible before reaching the input limit.Source: §B.1 · p. 17
• The POSITIVE EXAMPLES encoding uses positive subtask examples without a task description, followed by the input instance.The paper notes that example-only encodings have been used in prior studies.Source: §B.1 · p. 17

C.1 Comparison to Raw Instructions

• The analysis focuses on whether breaking tasks into sub-tasks and mapping them to the proposed schema improves their instruction encoding.The comparison operationalizes this question using raw instructions and FULL INSTRUCTION - NEG EXAMPLES encoding.Source: §C.1 · p. 18
• The comparison uses raw instructions for the first sub-task of four datasets and follows the setup of Efrat and Levy (2020).The authors compare these with their FULL INSTRUCTION - NEG EXAMPLES encoding to assess the value of splitting tasks into sub-tasks and mapping them to the proposed schema.Source: §C.1 · p. 18
• Weak language-model performance on raw instructions aligns with Efrat and Levy (2020)’s finding that language models perform poorly.The paper says the prior finding and its own raw-instruction results are consistent.Source: §C.1 · p. 18
• GPT3 performs better with FULL INSTRUCTION - NEG EXAMPLES encoding than with raw crowdsourcing instructions on ROUGE-L.The comparison is reported in Table 13; the supplied passages do not provide its numeric scores.Source: Table 13 · p. 18, §C.1 · p. 18
• Raw instructions average 2.5k tokens, compared with 950 tokens for the paper’s encoding.The authors suggest verbose raw instructions may partly explain the performance difference.Source: §C.1 · p. 18
• The authors suggest repetition can help human understanding, while concise instructions seem more effective for computers.This is presented as a possible explanation, not as a separately tested result.Source: §C.1 · p. 18