PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruction Induction: From Few Examples to Natural Language Task Descriptions

Or Honovich, Uri Shaham, Samuel R. Bowman, Omer Levy

arXiv:2205.10782v1cs.CL

TL;DR

The paper investigates whether language models can explicitly infer and describe unseen tasks from a few demonstrations. It introduces a zero-shot instruction-induction benchmark and execution-based evaluation, finding that instruction generation emerges mainly in the largest instruction-tuned model, which reaches 65.7% of human performance. The results motivate natural-language task descriptions as an interpretable learning paradigm, while ambiguity may remain harder for complex task descriptions.

  • Problem

    Although language models can infer unseen tasks from a few labeled examples through in-context learning, whether they can explicitly describe those tasks in natural language remains unexplored.

  • Method

    The paper introduces a 24-task instruction-induction challenge in which models generate natural-language instructions from demonstrations and evaluates them using execution accuracy.

  • Results

    InstructGPT achieves 65.7% of human performance on execution accuracy, while GPT-3 reaches 9.8%; instruction induction emerges mainly in the largest instruction-following model.

  • Takeaways & Limitations

    Instruction induction may serve as a learning paradigm that searches for interpretable task descriptions in the natural-language hypothesis space.

  • Takeaways & Limitations

    Ambiguity may pose a greater challenge for complex task descriptions, and the authors hypothesize that more than five demonstrations could help.

Abstract

from arXiv · show

Large language models are able to perform a task by conditioning on a few input-output demonstrations - a paradigm known as in-context learning. We show that language models can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples. To explore this ability, we introduce the instruction induction challenge, compile a dataset consisting of 24 tasks, and define a novel evaluation metric based on executing the generated instruction. We discover that, to a large extent, the ability to generate instructions does indeed emerge when using a model that is both large enough and aligned to follow instructions; InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance. This surprising result suggests that instruction induction might be a viable learning paradigm in and of itself, where instead of fitting a set of latent continuous parameters to the data, one searches for the best description in the natural language hypothesis space.

1 Introduction

The paper asks whether language models can explicitly describe tasks from a few demonstrations, rather than only infer them implicitly for execution. It introduces instruction induction and finds strong performance only for a large, instruction-tuned model, motivating natural-language task descriptions as a possible learning paradigm.

  • Motivation and approach: Instruction induction prompts a language model to generate a natural-language instruction describing the relationship between a few input-output demonstrations.The challenge is performed without fine-tuning or labeled instruction-induction data.
  • Evaluation: The benchmark covers 24 tasks spanning morphosyntax, style transfer, and sentiment analysis, evaluated with human-reference similarity and execution accuracy.Execution accuracy tests whether a language model can perform the task zero-shot using the generated instruction.
  • Results: 44.4 BERTScore and 43.6 execution accuracy were achieved by InstructGPT, versus human scores of 60.0 and 66.4, respectively.These figures summarize the best-performing model against human-written instructions.
  • Results: Instruction-generation ability was limited to InstructGPT, while smaller InstructGPT models and the original 175B-parameter GPT-3 performed dramatically worse.The finding links successful instruction induction to both sufficient model scale and alignment for instruction following.
  • Implications: Instruction induction may provide an interpretable learning paradigm that searches for task descriptions in natural language rather than fitting only continuous parameters.The paper presents this as a proof-of-concept and suggests possible benefits for spurious-correlation and overfitting issues.

2 Instruction Induction

Instruction induction changes the goal of in-context learning from executing a latent task to expressing its input-output relation as a single natural-language instruction. The paper elicits this behavior zero-shot with a human-designed challenge prompt.

  • 2 Instruction Induction: Given n input-output demonstrations, instruction induction generates one natural-language instruction whose execution should produce each demonstrated output.The task resembles in-context learning, but the desired output is a description of the relation rather than a prediction for a new input.
  • 2 Instruction Induction: The model performs instruction induction zero-shot, without fine-tuning on labeled instruction-induction examples.The experiments simply prompt models to explain a small set of demonstrations.
  • 2 Instruction Induction: Prior work primarily assessed latent-task inference through held-out task execution, whereas instruction induction requires describing the underlying task in natural language.This distinction makes explicit task description the target capability.

3 Data

The benchmark contains 24 varied language tasks with demonstration-based induction, held-out execution evaluation, human reference instructions, and verification studies. The authors also identify ambiguity as a theoretical challenge that five demonstrations usually reduce.

  • 3 Data: The dataset includes 24 tasks ranging from spelling to sentence similarity and causality detection, selected to vary in difficulty and language-understanding demands.The authors describe task formats, annotation and verification procedures, and a theoretical limitation.
  • 3 Data: Each example uses five randomly sampled input-output demonstrations from an induce set, while a separate execute set is held out for execution-accuracy evaluation.The study creates 100 instruction-induction examples per task.
  • 3.2 Annotating Reference Instructions: The benchmark collects 10 manually verified gold-reference instructions per task from college-graduate English-speaking annotators.Annotators receive the same five demonstrations and prompt format used for models.
  • 3.3 Verification: The authors verify task feasibility through model in-context learning and a human study using five demonstrations.The model test repeats 100 times per task, while the human study assesses whether people can describe the latent tasks.
  • 3.3 Verification: At least four of five annotators produced correct task descriptions for every task except Larger Animal.These control annotations also establish human baselines for reference-based evaluation and execution accuracy.
  • 3.4 Ambiguity: Ambiguity can permit multiple valid task interpretations, although five demonstrations typically resolve it for the benchmark’s tasks.The authors hypothesize that more than five demonstrations could mitigate ambiguity for more complex descriptions.

4 Evaluating Generated Instructions

The paper evaluates generated instructions both by similarity to human references and by whether an execution model can perform the task from them. Execution accuracy averages task-specific scores on held-out examples, complementing reference-based BERTScore.

  • 4 Evaluating Generated Instructions: The evaluation exploits the fact that induced instructions are executable, allowing correctness to be measured directly through prompted task performance.
  • 4.1 Reference-Based Evaluation: BERTScore compares generated instructions with human gold annotations, using the maximum BERTScore-F1 across verified references to accommodate wording variation.
  • 4.2 Execution Accuracy: Execution accuracy tests whether a language model prompted with an instruction and input produces the correct task output.For example, an instruction for pluralization is tested on an input such as “cat” and the expected output “cats.”
  • 4.2 Execution Accuracy: Execution accuracy is averaged over task-specific scores on 100 held-out execute examples for each task.The predefined language model executes each generated instruction on the held-out inputs.
  • 4.2 Execution Accuracy: BERTScore uses DeBERTa-xl-MNLI in version 0.3.11 of the metric implementation.
  • 4.2 Execution Accuracy: Figure 2 compares each task’s average BERTScore for model-generated instructions with manually authored control-group instructions, using human gold annotations as references.
  • 4.2 Execution Accuracy: Figure 3 compares per-task execution accuracy for generated and human-written instructions, alongside a Human baseline and a Gold ceiling based on verified annotations.
  • 4.2 Execution Accuracy: Table 2 reports average BERTScore and execution accuracy across tasks, with generated-instruction execution evaluated by InstructGPT and human performance based on control-group instructions.

5 Results

Results show that InstructGPT can induce instructions across many tasks, whereas GPT-3 and smaller models perform substantially worse. The strongest instruction-induction ability appears only in the largest instruction-tuned model, though it remains below human performance.

  • 5 Results: InstructGPT and GPT-3 are compared using eight GPT-3 versions, including four instruction-tuned models and four originally published models.The largest InstructGPT and original 175B-parameter GPT-3 are the default representatives of their respective families.
  • 5.1 Comparing to Gold Annotations: InstructGPT reaches at least 75% of human BERTScore performance on 13 of 24 tasks, while GPT-3 remains far from human performance across tasks.
  • 5.1 Comparing to Gold Annotations: InstructGPT’s average BERTScore is 15.6 points below human performance, compared with a 45.4-point gap for GPT-3.
  • 5.2 Execution Accuracy: Execution accuracy uses the largest InstructGPT as both the instruction-induction model and execution model, which may bias results toward its generations.
  • 5 Results: Table 3 presents examples of instructions generated by GPT-3 and InstructGPT for five tasks.
  • 5.2 Execution Accuracy: InstructGPT achieves at least 75% of human execution accuracy on 12 of 24 tasks, whereas GPT-3 scores below 10% on 20 tasks.
  • 5.2 Execution Accuracy: 65.7% of human performance is reached by InstructGPT on average execution accuracy, versus 9.8% for GPT-3.
  • 5.2 Execution Accuracy: The largest InstructGPT model is the only model showing a substantial emergence of instruction generation, while smaller instruction-tuned and original GPT-3 models remain weak.

6 Analysis

The analysis reveals that instruction induction can succeed on some tasks but often produces incomplete or misleading descriptions, with execution exposing failures that text similarity can obscure.

  • InstructGPT achieved high or near-human execution accuracy on First Letter, Sentence Similarity, and Pluralization.Its pluralization instruction was incomplete, but the execution model often supplied the correct plural form anyway.
  • 98% of InstructGPT’s Passivization instructions reduced the task to reversing subject and object order, omitting required surface-form changes.For example, “The scientist supported the authors” is not the correct passive form, “The scientist was supported by the authors”.
  • GPT-3’s unrelated phrase “The friend wrote the following output:” achieved higher Passivization execution accuracy than InstructGPT’s descriptions.This phrase appeared in 24% of GPT-3’s predictions and apparently prompted the execution model to rephrase inputs in passive form.
  • In Antonyms, 60% of InstructGPT predictions said “Reverse the input” and 11% said “Reverse the word”, which the execution model interpreted literally.The model reversed letters rather than the words’ meanings.
  • GPT-3 showed no instruction-induction ability, although generic imperative outputs inflated BERTScore while receiving very low execution-accuracy scores.Execution accuracy is therefore more robust than reference overlap to empty instructions that merely match the expected format.

7 Related Work

Prior work studies learning from demonstrations by executing latent tasks or learning from explicit instructions, while this paper focuses on explicitly describing inferred tasks in natural language.

  • In-Context Learning: In-context learning studies task execution from a few demonstrations without fine-tuning or gradient updates.Related work addresses efficiency, stability, meta-training, prompt design, reframing, and retrieval.
  • The Instruction Paradigm: The instruction paradigm studies learning new tasks from natural-language instructions, including benchmarks and instruction tuning for zero-shot learning.This literature contrasts with inducing an instruction from demonstrations.
  • Intermediate Reasoning Steps: Scratchpads and chain-of-thought prompting enrich demonstrations with intermediate reasoning steps to improve performance on complex or reasoning-intensive tasks.Subsequent work also uses self-consistency ensembles for such tasks.

8 Discussion

The paper shows that large language models can infer and describe tasks from a few demonstrations, while suggesting instruction induction as a potentially interpretable learning paradigm.

  • Large language models can infer new tasks from a handful of demonstrations and describe them in natural language.Evidence comes from a diverse set of language tasks.
  • Instruction induction abilities were limited to a single state-of-the-art model, which approached human performance on about half the tasks.The result is presented as evidence that the capability exists but remains limited across models.
  • The proposed learning paradigm searches for a natural-language instruction describing data instead of converting the dataset into continuous parameters.The authors associate this representation with human interpretability and possible relief from overfitting and spurious correlations.

A Dataset Details

The appendix documents the datasets used for each task and identifies a shared collection of common English nouns used by some datasets.

  • The appendix details each task’s dataset and notes that some datasets rely on common English nouns.The common English nouns are described as CEN in a separate appendix section.

A.1 Tasks

The task suite spans linguistic transformations, semantic judgments, arithmetic operations, and list or character manipulations, using varied datasets and evaluation criteria.

  • The suite includes word-level tasks such as synonyms, antonyms, rhymes, translations, and selecting the larger animal.
  • Evaluation uses task-specific criteria, including containment checks, unigram-overlap F1, and exact string match where applicable.
  • Membership and starting-with tasks map lists or sentences to items satisfying a specified category or initial-letter criterion.
  • Sentence-level tasks cover cause selection, common-concept identification, formality, sentiment, similarity, passivization, and negation.
  • Symbolic tasks require summing or comparing numbers, converting numbers to words, extracting letters, or listing letters.

B Data Verification

Data verification compares models’ in-context task inference with humans’ ability to write correct instructions. Most tasks can be inferred in context, and all but one are accurately described by at least four of five human annotators.

  • Most tasks can be inferred in context by the evaluated models.
  • All tasks but one were accurately described by at least 4 out of 5 human annotators.
Loading 2205.10782v1…