PaperlayerView exact revision on arXiv

Source-linked AI summary

Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor

Or Honovich, Thomas Scialom, Omer Levy, Timo Schick

arXiv:2212.09689v1cs.CLcs.AIcs.LG

TL;DR

Instruction tuning needs broad instruction data, but existing collection methods rely on benchmarks, user prompts, or substantial human annotation. Unnatural Instructions automatically generates and paraphrases examples from 15 seeds, producing a large dataset that performs strongly across benchmarks. The findings support model-generated data as a cost-effective alternative to human-curated expansion, while noting challenges in generating correct outputs.

  • Problem

    Existing instruction data comes from limited academic benchmarks or costly user and human annotation processes, motivating diverse task data without human labor.

  • Method

    A pretrained language model generates structured instruction-input-output examples from three-example seeds, then produces alternative instruction formulations.

  • Results

    Models trained on Unnatural Instructions outperform T0++ and Tk-Instruct across several benchmarks and perform competitively with a matched Super-Natural Instructions baseline.

  • Takeaways & Limitations

    Fully automated language-model data generation is a viable, cost-effective alternative to human-curated data and can provide enhanced instruction diversity and task creativity.

  • Takeaways & Limitations

    Weaker models can generate useful instructions but may struggle to produce the corresponding outputs, making output annotation a remaining challenge.

Abstract

from arXiv · show

Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions. These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions. In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor. We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth. This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs. Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training on open-source manually-curated datasets, surpassing the performance of models such as T0++ and Tk-Instruct across various benchmarks. These results demonstrate the potential of model-generated data as a cost-effective alternative to crowdsourcing for dataset expansion and diversification.

1 Introduction

Unnatural Instructions addresses whether language models can generate large, diverse instruction datasets without human labor. The resulting data supports strong instruction-tuning performance across multiple benchmarks.

  • Existing instruction datasets derive from academic benchmarks or user prompts, while the instruction paradigm can describe broader text-based tasks.
  • The authors collect 64,000 instruction-input-output triplets using only 15 seed examples across five three-example prompts.The process prompts a pretrained language model to generate a fourth example from three demonstrations.
  • Paraphrasing each instruction expands the dataset to approximately 240,000 examples while preserving input arguments and outputs.
  • Despite noise, Unnatural Instructions outperforms T0++ and Tk-Instruct across several benchmarks, including Super-Natural Instructions, BIG-bench Hard, and LMentry.
  • An 18-point gain on BIG-bench Hard and a 16-point gain on LMentry appears when comparing against the controlled baseline setup.
  • The work presents automatic dataset expansion with language models as a faster, cheaper alternative to human labor that can produce creative and diverse data.

2 Data Collection

The dataset is generated automatically from a small set of structured demonstrations, then expanded with natural-language paraphrases. Structured fields and decoding choices separate creative task generation from output generation aimed at correctness.

  • Unnatural Instructions contains 240,670 instruction-input-output examples covering a wide variety of natural-language tasks.
  • 2.1 Core Dataset Generation: The core dataset starts from 15 manually constructed examples and uses a pretrained language model to generate structured instruction, input, constraint, and output fields.Five seeds contain three demonstrations each; the core collection phase uses structured formatting and filtering heuristics.
  • 2.1 Core Dataset Generation: Stochastic decoding generates example inputs to promote creativity, while deterministic decoding generates outputs to prioritize accuracy.
  • 2.1 Core Dataset Generation: Automatic filters remove malformed generations, prompt duplicates, and examples sharing the same instruction and input argument.
  • 2.2 Template Expansion: The expansion stage prompts a language model to produce two alternative free-form formulations for each structured instruction while preserving its content.An {INPUT} placeholder marks where the input is inserted.
  • 2.2 Template Expansion: More than 97.5% of instructions have two valid and distinct alternative formulations, although some outputs are duplicated or lack the required placeholder.
  • 2.2 Template Expansion: Cross-referencing paraphrases with shared instructions and different inputs expands the collection to 240,670 examples without additional cost.

3 Data Analysis

Unnatural Instructions combines creative task coverage with substantial noise and greater input diversity than Super-Natural Instructions. Manual analysis found 117 task types and 56.5% fully correct examples.

  • Creativity: The analysis presents generated instructions as evidence of creativity, including tasks beyond canonical sentiment analysis, question answering, and summarization.The examples include highly specific tasks such as detecting a recipe from ingredients.
  • Correctness: 56.5% of the 200 manually analyzed examples were fully correct across instructions, inputs, and outputs.Correctness required logical instructions, matching input arguments, and correct outputs.
  • Correctness: The dataset includes both valid and noisy instruction-input-output triples, with some incorrect examples remaining informative.One example had a format mismatch but an output that correctly answered the clue.
  • Diversity: 117 distinct tasks appeared among 200 manually analyzed examples, including canonical NLP tasks and highly specific tasks.Eighty-five task types appeared only once.
  • Diversity: Unnatural Instructions inputs tend to be less similar to one another than Super-Natural Instructions inputs.The comparison used 10,000 sampled example pairs from each dataset and BERTScore similarity.

4 Experimental Setup

The experiments fine-tune T5-LM and compare automatically generated instruction data with manually annotated and existing instruction-tuned baselines. Evaluation covers four benchmarks in zero-shot settings using benchmark-specific decoding and metrics.

  • Training: T5-LM models are fine-tuned with batch size 16 for 3 epochs, using a 1,000-example validation set for model selection.Core-dataset training uses the Super-Natural Instructions template.
  • Baselines: The study compares Unnatural Instructions with manually annotated instruction data and models including T0++, Tk-Instruct, and FLAN-T5.The direct comparison baseline uses T5-LM trained on 64,000 Super-Natural Instructions examples.
  • Evaluation: Four benchmarks measure capabilities in zero-shot evaluation unless instructions explicitly provide demonstrations.The benchmarks include Super-Natural Instructions, T0: Zero-Shot, BIG-bench Hard, and LMentry.
  • Evaluation: Super-Natural Instructions uses greedy decoding and Rouge-L, while T0: Zero-Shot uses rank classification and accuracy.T0: Zero-Shot contains six held-out tasks with free-form task descriptions.
  • Evaluation: BIG-bench Hard is evaluated in two formats using greedy decoding and exact match against references.The formats are BIG-bench’s original format and a manually reformulated question-answering format without few-shot demonstrations.
  • Evaluation: LMentry tests basic language abilities using greedy decoding, high-accuracy regular expressions, and an LMentry score combining accuracy with robustness.The score combines accuracy with multiple robustness aspects.
  • Results: T5-LM on Unnatural Instructions performs better than several strong baselines and remains competitive with the direct comparison baseline across the reported setups.Table 4 reports results across the four considered benchmarks.

5 Results

Unnatural Instructions performs competitively with manually curated instruction data across benchmarks, while scaling and template expansion improve performance and cost efficiency.

  • Main Results: Unnatural Instructions achieves stronger or equal performance than Super-Natural Instructions on every dataset except Super-Natural Instructions itself in a direct comparison.The broader baseline comparison also reports that T0++ and Tk-Instruct are generally outperformed, with BIG-bench Hard (Orig) as an exception for T0++.
  • Main Results: Unnatural Instructions is outperformed by FLAN-T5, whose training data is larger by several orders of magnitude.
  • Performance with Template Expansion: Adding instruction paraphrases boosts performance on T0: Zero-Shot (+3.3), Big-bench: Hard in its original format (+12.1), and LMentry (+8.7).The reported gains are attributed partly to increased format diversity and are retained when dataset size is controlled.
  • Performance Scaling by Dataset Size: 250 to 64,000 examples are used to evaluate scaling for both Super-Natural Instructions and Unnatural Instructions.The experiments test whether generated examples become repetitive and yield diminishing downstream returns.
  • Performance Scaling by Dataset Size: Log-linear scaling laws appear for the core, full, and Super-Natural Instructions datasets, indicating that more generated examples continue adding valuable training signal.
  • Performance Scaling by Cost: Unnatural Instructions is clearly more cost-efficient than manually curated data, including on the Super-Natural Instructions test set when annotation cost is held constant.The comparison assumes $0.02 or $0.01 per generated example and a lower-bound human annotation cost of $0.50.

6 Data Collection Ablations

Ablations examine the generation model, meta-prompt, demonstrations, constraints, and pipeline structure, showing that prompt and model choices affect data quality.

  • Ablation Setup: Meta-prompts, demonstration constraints, and separate input-output generation are evaluated using 11B T5-LM models trained on 2,000 examples.The ablations evaluate Super-Natural Instructions validation performance, averaged across three random seeds, with the relevant results reported in Tables 5–9.
  • Generative Model: Replacing the instruction-tuned generator with vanilla GPT-3 reduces generated-input quality by 4.5 points but still produces informative and diverse instructions.
  • Generative Model: Generating outputs with vanilla GPT-3 requires instruction tuning because outputs often begin correctly before degenerating into repetitions or tangents.The authors leave post-processing heuristics for this failure mode to future work.
  • Meta-Prompts: Enumeration meta-prompts elicit more informative examples than minimalistic or verbose prompts.The verbose prompt may perform worse because its final command interrupts the pattern and mismatches pretraining patterns.
  • Demonstration Seeds: The generation pipeline is largely robust to different demonstration seeds, except seed 4, which produces substantially less constrained instructions.Seed 4 has 1,376 of 2,000 examples without constraints, compared with 28 to 880 for the other sets.
  • Pipeline Structure: One-step triplet generation scores 1.7 points lower than the default two-step process.

7 Related Work

Prior instruction-tuning and automatic-data-generation work establishes the surrounding methods, while Unnatural Instructions extends automatic generation to a general-purpose, task-diverse dataset.

  • Instruction Tuning: Instruction tuning lets pretrained language models generalize to unseen tasks from natural-language instructions in a zero-shot setting.
  • Instruction Tuning: Existing instruction benchmarks reformulate crowdsourced NLP datasets into a uniform instruction-input-output format.
  • Instruction Tuning: User-generated prompts provide a different instruction distribution but require live applications and substantial human annotation investment.
  • Automatic Data Generation: Automatic data-generation methods commonly augment existing labeled datasets or combine models with human annotators.
  • Automatic Data Generation: Unnatural Instructions is presented as the first general-purpose, large-scale dataset automatically generated without labeled data and emphasizing task diversity.

8 Conclusion

The conclusion presents Unnatural Instructions as an automatically generated alternative to manually annotated datasets, with evidence of cost-effectiveness, diversity, and creative task generation. It also identifies output generation as a challenge for weaker models.

  • Unnatural Instructions is an automatically generated dataset of natural language instructions with corresponding inputs and outputs.
  • Models trained on Unnatural Instructions can outperform models trained on manually annotated datasets across several benchmarks.
  • The dataset is described as cost-effective and more diverse and creative in its instructions and tasks than crowd workers may provide.
  • Weaker models can generate useful instructions but may struggle to produce the corresponding outputs.
  • The authors view model-based general-purpose data generation as a direction for future research.

A Fine-Tuning Hyperparameters

The fine-tuning experiments use shared training settings, including a bounded training schedule, validation-based checkpoint selection, and fixed optimization and sequence-length parameters.

  • The same hyperparameters are used for the T5-LM fine-tuning experiments.
  • Models are trained for up to max(3 epochs, 3000 steps), with the final model selected using Rouge-L on a validation set.Validation is performed every 100 steps.
  • The experiments use batch size 16, maximum learning rate 1 · 10^-5, 10% warm-up, and weight decay 0.01.
  • Inputs are truncated at 1,024 tokens and outputs at 128 tokens.

B Evaluation Details

Evaluation uses official scripts for several benchmarks and normalized exact-match scoring for BIG-bench Hard, with one punctuation-only task handled separately.

  • Official evaluation scripts are used for Super-Natural Instructions, T0: Zero-Shot, and LMEntry.
  • BIG-bench Hard outputs are lowercased, stripped of punctuation, and trimmed for whitespace before exact-match scoring.
  • The dyck_languages task is exempt from punctuation removal because its target outputs consist entirely of punctuation characters.

C Data Generation Prompts

The data-generation prompts span diverse natural-language tasks, including reference understanding, commonsense plausibility, ordering, affordances, sentiment, topic classification, and constrained question answering.

  • In-Context Demonstrations: Reference-understanding prompts ask questions linking pronouns or other mentions to the entities they refer to.
  • In-Context Demonstrations: Commonsense prompts classify everyday events or general statements as True or False according to plausibility and commonsense.
  • In-Context Demonstrations: Ordering prompts determine whether process steps are coherent or return a corrected order for a five-sentence story.
  • In-Context Demonstrations: Other prompts cover object affordances, review sentiment, and classification into ten question topics.
  • In-Context Demonstrations: Constrained examples include science multiple-choice answers, arithmetic solutions, sentence simplification, review conversion, and binary passage classification.
Loading 2212.09689v1…