Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning

• The study annotates eight definition-content categories and ablates each category to test its contribution to performance.The analysis uses human annotation and model retraining with category occurrences removed.Source: §1 · p. 1
• Removing label information causes the lowest performance, while input descriptions generally contribute little and larger models use them more.Natural-language label definitions clarify task-specific meanings of common verbalizers; verbalizers alone help with a new label space.Source: §1 · p. 1
• The findings suggest that costly human-written definitions may not always help more than basic task metadata.Source: §1 · p. 2
• Syntax-guided Task Definition Compression removes around 60% of tokens while improving T5-XL performance by about 3 points on a held-out set.The method uses model predictions and requires only black-box access; models may infer the full label space from a partial one.Source: §1 · p. 2
• The paper proposes a JSON-like input-action-output format and meta-tuning before instruction learning to help models use task definitions.Source: §1 · p. 2
• The two proposed instruction strategies improve Rouge-L by 4.2 on BART-Large, 4.0 on T5-Large, and 2.1 on T5-XL when combined.Source: §1 · p. 2

Instruction Learning

• In this paper, task instructions contain a high-level task definition and demonstration examples, though they may include other content.The task definition describes the task's input and output.Source: §2 · p. 2
• Models are fine-tuned on upstream tasks, then infer on an unseen test task using its instruction, which is prepended to the input.Source: §2 · p. 2

Benchmark

• The English NIv2 benchmark contains 757 training tasks and 119 unseen test tasks; reported results use Rouge-L and average three random seeds.The test tasks are divided into 12 categories, with an additional classification-versus-generation split; the study sets aside 76 training tasks for validation.Source: §2 · pp. 2–3

3 Ablation Analysis of Annotated Task Definitions

• Authors categorized task-definition text into eight hierarchical input, action, and output content types, with 0.91 Fleiss kappa across 150 initial annotations.The ablation models were retrained after content removal, and results were averaged over three random seeds.Source: §3 · p. 3, §3.1 · p. 3

3.1 Ablation Analysis

• Removing all label information (Label List and Label Definition) drops performance to its lowest level, similar to having no task definition; removing direct input descriptions has nearly no negative impact.Additional input/output details comprise 44% of task-definition tokens and become slightly more influential as model size grows.Source: §3.1 · pp. 4–5

3.2 The Role of Label Information

• Label List removal chiefly hurts unseen-verbalizer tasks, whereas Label Definition removal hurts both groups, especially tasks with seen verbalizers.The paper hypothesizes that definitions help distinguish the semantics of identical label names across tasks.Source: Table 3 · p. 5, §3.2 · p. 5

3.1 Ablation Analysis

• The Metadata baseline achieved performance similar to full definitions using categorical task information in a JSON-like template.For classification tasks, its metadata included the label space; generation tasks used “generate free text.”Source: §3.1 · pp. 4–5

Method

• STDC iteratively removes syntactically plausible definition phrases when their removal does not decrease performance.It traverses constituency parse trees top-down and stops after attempting leaf-node removals.Source: §4 · p. 5

Results

• 2.8 Rouge-L points gained by T5-XL while keeping 41% of definition length; all three tested models improved after removing approximately half or more of the tokens.Coverage averaged around 90%, indicating the increase affected a large majority of samples.Source: Table 4 · p. 6, §4 · p. 6

Compression Ratio Distribution

• STDC compresses generation-task definitions more than classification-task definitions, despite roughly similar original lengths except for Code to Text.The authors offer longer classification definitions or less accurate model interpretation of generation definitions as possible explanations.Source: §4 · p. 6, Figure 2 · p. 6

Information Kept by Type

• Compressed definitions retain 66% of Output content and 80% of Label Definitions, versus about 33% of Input content and 47% of Additional input details.This pattern supports the finding that output descriptions are more essential than input descriptions.Source: §4 · p. 6, Figure 3 · p. 6

Qualitative Case Study

• Compressed definitions can omit some label verbalizers and even Action root verbs while maintaining or improving model performance.Only 52% of Action Content remains in compressed definitions.Source: §4 · pp. 6–7, Figure 4 · p. 7

5 Improving Model Understanding of Task Definitions

• The two strategies are to organize definitions into input/action/output triplets and add meta-tuning before instruction learning.The meta-tuning stage is intended to adapt models to the writing style of task definitions.Source: §5 · p. 7
• Triplet definitions were extracted from NIv2 using human annotation and automated processing, with parsing and coreference corrections.The resulting triplets also condense the information in task definitions.Source: §5 · p. 7
• Meta-tuning trains the model to produce a tagged triplet entry from two demonstration examples, then initializes the instruction-learning model with the meta-tuned parameters.The same examples are used in meta-tuning and instruction learning to avoid adding extra data.Source: §5 · p. 7

5.1 Experiments

• Around 4 Rouge-L points of improvement were achieved for BART-Large and T5-Large, while triplets improved T5-XL by 1.5 points.For the smaller models, around 3.1 points came from structuring definitions into triplets; the authors suggest larger models may extract key information more effectively from unstructured definitions.Source: Table 5 · p. 8, §5.1 · p. 8

Instruction Learning

• Instruction-learning studies fine-tune pretrained language models to follow language instructions and generalize to new tasks (Sanh et al., 2022; Wei et al., 2022a; Ouyang et al., 2022; Wang et al., 2022b; Chung et al., 2022; OpenAI, 2023; Taori et al., 2023).Source: §6 · p. 8

Benchmarks of Instruction Learning

• NIv2 provides an English collection of around 800 crowd-sourced tasks, while other benchmarks include Ye et al. (2021)'s 160+ task collection and PromptSource (Bach et al., 2022).T0 (Sanh et al., 2022) is trained on PromptSource, and NIv2 test tasks are also used to evaluate models trained on other data.Source: §6 · p. 8

Prompt Engineering

• Prompt engineering searches for better prompts, while this work compresses task definitions and forms them around structured triplets (Schick and Schütze, 2021b,a; Gao et al., 2021; Mishra et al., 2021; Shin et al., 2020; Prasad et al., 2022; Deng et al., 2022).The paper describes searching for better demonstration examples as complementary.Source: §6 · p. 8

Prompt Analysis

• Unlike prompt-analysis work on short prompts or in-context examples, this study examines task definitions in zero-shot settings and reports that task metadata can be comparable to human-written definitions.It also reports that Label Definitions can matter when familiar labels take different meanings at test time (Min et al., 2022; Kung and Peng, 2023; Zhao et al., 2021).Source: §6 · pp. 8–9

More powerful instruction learning models

• The analysis may apply to stronger models such as Alpaca, but some conclusions might change, and the authors leave that question for future work.They also describe explaining how models leverage instructions for generalization as an open question.Source: §7 · p. 9

Automatically created training data for instruction learning

• Automatically generated definitions need further quality improvement, and combining the proposed format with automatic generation is suggested as a future direction.The authors also note that content-type boundaries may be vaguer in generated definitions and raise safety concerns about distilling LLMs to generate instruction-tuning data (Gudibande et al., 2023).Source: §7 · p. 9

From task instructions to instructions for open-ended generation

• The analysis mainly applies to definitions for typical NLP tasks, especially classification, while open-ended generation instructions remain a topic for later work.Source: §7 · p. 9

8 Conclusion

• Label information is critical for model performance, whereas input descriptions and additional constraints are not important.The paper also reports that different types of definition content have widely varying impacts.Source: §8 · p. 9
• Task definitions can be extensively compressed without relying on the full natural-language version.Source: §8 · p. 9
• A JSON-formatted template can provide structured task information, even when that information is basic task metadata.Source: §8 · p. 9

9 Limitations

• The analysis covers English-language tasks, so its conclusions may not extend to task definitions in other languages.The authors attribute the English-only scope to the annotators’ common language.Source: §9 · p. 9
• The study rewrites existing definitions into triplets but leaves writing triplets from scratch for future evaluation.The authors describe their approach as a starting point for demonstrating structured, condensed definitions.Source: §9 · pp. 9–10
• The largest evaluated model is a 3B-parameter T5, leaving larger-model behavior and possible emergent abilities outside the study’s scope.Source: §9 · p. 10
• The effect of removing classification label information on generation-task performance is counter-intuitive and requires further exploration.Source: §9 · p. 10

A.1 Validation Task set

• The authors create and fix their own validation split across experiments because NIv2 provides no official split.Source: §A.1 · p. 13
• The validation tasks were selected to have roughly equal numbers of classification and generation tasks.Source: §A.1 · p. 13
• Table 6 reports the task types represented in the validation set and the number of tasks in each category.Source: Table 6 · p. 13

A.2 Model Training

• Instruction-learning runs used two epochs, constant learning rates, model-specific batch sizes, and input and output limits of 1024 and 128 tokens.The reported setup used A100 GPUs with 40G memory and DeepSpeed.Source: §A.2 · p. 13

B Annotation Procedure Details

• Three authors annotated the 876 NIv2 tasks, with the remaining tasks divided equally after a pilot and agreement assessment.Two annotators were native English speakers; one was a graduate student in the United States.Source: B Annotation Procedure Details · p. 13, §B.1 · p. 13

B.1 Overview of the Annotation Procedure

• The annotation procedure refined candidate categories on 50 training tasks, measured agreement on a held-out 150, then split the remaining work among annotators and corrected obvious errors.Source: §B.1 · p. 13

B.2 A Hierarchy of Content Types in Definitions

• The content hierarchy distinguishes input, action, and output descriptions, with further subdivisions for details, constraints, labels, and input mentions within action phrases.Label List and Label Definition distinguish label verbalizations from descriptions of label-space semantics.Source: §B.2 · p. 13, Table 7 · p. 14

B.3 Inter-Annotator Agreement Level

• Fleiss’ kappa is used to measure agreement among the three annotators for each content category, with the authors reporting consistency across categories.Source: §B.3 · p. 14

B.4 Pre-process and Post-process of The Annotations

• Annotations were generally sentence-level, with pattern-based splitting for input phrases and syntactic parsing plus another human pass to identify input mentions and output content in action sentences.Source: §B.4 · p. 14

D Examples of Triplet

• The appendix points to triplet examples in Table 9 and reports grid search over learning rate, batch size, and epochs after meta-tuning.The post-meta-tuning learning rate uses a constant schedule with 300 warm-up steps.Source: D Examples of Triplet · p. 14, E Hyperparameters Tuning for… · p. 14, Table 8 · p. 14

C Compression Algorithm

• STDC tests removing parse-tree nodes and keeps a removal when model performance is at least as high as with the full definition, returning the compressed definition.The listed algorithm takes a model, task examples, a full definition, and its constituency tree as input.Source: D Examples of Triplet · p. 14