Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor

• Existing instruction datasets derive from academic benchmarks or user prompts, while the instruction paradigm can describe broader text-based tasks.Source: §1 · p. 1
• The authors collect 64,000 instruction-input-output triplets using only 15 seed examples across five three-example prompts.The process prompts a pretrained language model to generate a fourth example from three demonstrations.Source: §1 · pp. 1–2
• Paraphrasing each instruction expands the dataset to approximately 240,000 examples while preserving input arguments and outputs.Source: §1 · p. 2
• Despite noise, Unnatural Instructions outperforms T0++ and Tk-Instruct across several benchmarks, including Super-Natural Instructions, BIG-bench Hard, and LMentry.Source: §1 · p. 2
• An 18-point gain on BIG-bench Hard and a 16-point gain on LMentry appears when comparing against the controlled baseline setup.Source: §1 · p. 2
• The work presents automatic dataset expansion with language models as a faster, cheaper alternative to human labor that can produce creative and diverse data.Source: §1 · p. 2

• Unnatural Instructions contains 240,670 instruction-input-output examples covering a wide variety of natural-language tasks.Source: §2 · p. 2

2.1 Core Dataset Generation

• The core dataset starts from 15 manually constructed examples and uses a pretrained language model to generate structured instruction, input, constraint, and output fields.Five seeds contain three demonstrations each; the core collection phase uses structured formatting and filtering heuristics.Source: §2 · p. 2, §2.1 · pp. 2–4
• Stochastic decoding generates example inputs to promote creativity, while deterministic decoding generates outputs to prioritize accuracy.Source: §2.1 · pp. 2, 4
• Automatic filters remove malformed generations, prompt duplicates, and examples sharing the same instruction and input argument.Source: §2.1 · p. 4

2.2 Template Expansion

• The expansion stage prompts a language model to produce two alternative free-form formulations for each structured instruction while preserving its content.An {INPUT} placeholder marks where the input is inserted.Source: §2.2 · p. 4
• More than 97.5% of instructions have two valid and distinct alternative formulations, although some outputs are duplicated or lack the required placeholder.Source: §2.2 · p. 4
• Cross-referencing paraphrases with shared instructions and different inputs expands the collection to 240,670 examples without additional cost.Source: §2.2 · p. 4

Creativity

• The analysis presents generated instructions as evidence of creativity, including tasks beyond canonical sentiment analysis, question answering, and summarization.The examples include highly specific tasks such as detecting a recipe from ingredients.Source: §3 · pp. 5–6, Table 1 · p. 5

Correctness

• 56.5% of the 200 manually analyzed examples were fully correct across instructions, inputs, and outputs.Correctness required logical instructions, matching input arguments, and correct outputs.Source: §3 · p. 5
• The dataset includes both valid and noisy instruction-input-output triples, with some incorrect examples remaining informative.One example had a format mismatch but an output that correctly answered the clue.Source: §3 · p. 5, Table 2 · p. 6

Diversity

• 117 distinct tasks appeared among 200 manually analyzed examples, including canonical NLP tasks and highly specific tasks.Eighty-five task types appeared only once.Source: §3 · p. 6
• Unnatural Instructions inputs tend to be less similar to one another than Super-Natural Instructions inputs.The comparison used 10,000 sampled example pairs from each dataset and BERTScore similarity.Source: §3 · p. 6, Figure 5 · p. 6

Training

• T5-LM models are fine-tuned with batch size 16 for 3 epochs, using a 1,000-example validation set for model selection.Core-dataset training uses the Super-Natural Instructions template.Source: §4.1 · p. 7

Baselines

• The study compares Unnatural Instructions with manually annotated instruction data and models including T0++, Tk-Instruct, and FLAN-T5.The direct comparison baseline uses T5-LM trained on 64,000 Super-Natural Instructions examples.Source: §4.2 · p. 7

Evaluation

• Four benchmarks measure capabilities in zero-shot evaluation unless instructions explicitly provide demonstrations.The benchmarks include Super-Natural Instructions, T0: Zero-Shot, BIG-bench Hard, and LMentry.Source: §4.3 · p. 7
• Super-Natural Instructions uses greedy decoding and Rouge-L, while T0: Zero-Shot uses rank classification and accuracy.T0: Zero-Shot contains six held-out tasks with free-form task descriptions.Source: §4.3 · p. 7
• BIG-bench Hard is evaluated in two formats using greedy decoding and exact match against references.The formats are BIG-bench’s original format and a manually reformulated question-answering format without few-shot demonstrations.Source: §4.3 · p. 7
• LMentry tests basic language abilities using greedy decoding, high-accuracy regular expressions, and an LMentry score combining accuracy with robustness.The score combines accuracy with multiple robustness aspects.Source: §4.3 · pp. 7–8

Results

• T5-LM on Unnatural Instructions performs better than several strong baselines and remains competitive with the direct comparison baseline across the reported setups.Table 4 reports results across the four considered benchmarks.Source: Table 4 · p. 8

Main Results

• Unnatural Instructions achieves stronger or equal performance than Super-Natural Instructions on every dataset except Super-Natural Instructions itself in a direct comparison.The broader baseline comparison also reports that T0++ and Tk-Instruct are generally outperformed, with BIG-bench Hard (Orig) as an exception for T0++.Source: §5 · p. 8
• Unnatural Instructions is outperformed by FLAN-T5, whose training data is larger by several orders of magnitude.Source: §5 · p. 8

Performance with Template Expansion

• Adding instruction paraphrases boosts performance on T0: Zero-Shot (+3.3), Big-bench: Hard in its original format (+12.1), and LMentry (+8.7).The reported gains are attributed partly to increased format diversity and are retained when dataset size is controlled.Source: §5.1 · p. 8

Performance Scaling by Dataset Size

• 250 to 64,000 examples are used to evaluate scaling for both Super-Natural Instructions and Unnatural Instructions.The experiments test whether generated examples become repetitive and yield diminishing downstream returns.Source: §5.2 · p. 8
• Log-linear scaling laws appear for the core, full, and Super-Natural Instructions datasets, indicating that more generated examples continue adding valuable training signal.Source: §5.2 · p. 9

Performance Scaling by Cost

• Unnatural Instructions is clearly more cost-efficient than manually curated data, including on the Super-Natural Instructions test set when annotation cost is held constant.The comparison assumes $0.02 or $0.01 per generated example and a lower-bound human annotation cost of $0.50.Source: §5.3 · p. 9

Ablation Setup

• Meta-prompts, demonstration constraints, and separate input-output generation are evaluated using 11B T5-LM models trained on 2,000 examples.The ablations evaluate Super-Natural Instructions validation performance, averaged across three random seeds, with the relevant results reported in Tables 5–9.Source: §6 · p. 10, Table 5 · p. 10, Table 6 · p. 10

Generative Model

• Replacing the instruction-tuned generator with vanilla GPT-3 reduces generated-input quality by 4.5 points but still produces informative and diverse instructions.Source: §6.1 · p. 10
• Generating outputs with vanilla GPT-3 requires instruction tuning because outputs often begin correctly before degenerating into repetitions or tangents.The authors leave post-processing heuristics for this failure mode to future work.Source: §6.1 · p. 10

Meta-Prompts

• Enumeration meta-prompts elicit more informative examples than minimalistic or verbose prompts.The verbose prompt may perform worse because its final command interrupts the pattern and mismatches pretraining patterns.Source: §6.2 · p. 10

Demonstration Seeds

• The generation pipeline is largely robust to different demonstration seeds, except seed 4, which produces substantially less constrained instructions.Seed 4 has 1,376 of 2,000 examples without constraints, compared with 28 to 880 for the other sets.Source: §6.3 · p. 11

Pipeline Structure

• One-step triplet generation scores 1.7 points lower than the default two-step process.Source: §6.5 · p. 11

Instruction Tuning

• Instruction tuning lets pretrained language models generalize to unseen tasks from natural-language instructions in a zero-shot setting.Source: §7 · p. 11
• Existing instruction benchmarks reformulate crowdsourced NLP datasets into a uniform instruction-input-output format.Source: §7 · p. 11
• User-generated prompts provide a different instruction distribution but require live applications and substantial human annotation investment.Source: §7 · p. 12

Automatic Data Generation

• Automatic data-generation methods commonly augment existing labeled datasets or combine models with human annotators.Source: §7 · p. 12
• Unnatural Instructions is presented as the first general-purpose, large-scale dataset automatically generated without labeled data and emphasizing task diversity.Source: §7 · p. 12

• Unnatural Instructions is an automatically generated dataset of natural language instructions with corresponding inputs and outputs.Source: §8 · p. 12
• Models trained on Unnatural Instructions can outperform models trained on manually annotated datasets across several benchmarks.Source: §8 · p. 12
• The dataset is described as cost-effective and more diverse and creative in its instructions and tasks than crowd workers may provide.Source: §8 · p. 12
• Weaker models can generate useful instructions but may struggle to produce the corresponding outputs.Source: §8 · p. 12
• The authors view model-based general-purpose data generation as a direction for future research.Source: §8 · p. 12

• The same hyperparameters are used for the T5-LM fine-tuning experiments.Source: A Fine-Tuning Hyperparameters · p. 15
• Models are trained for up to max(3 epochs, 3000 steps), with the final model selected using Rouge-L on a validation set.Validation is performed every 100 steps.Source: A Fine-Tuning Hyperparameters · p. 15
• The experiments use batch size 16, maximum learning rate 1 · 10^-5, 10% warm-up, and weight decay 0.01.Source: A Fine-Tuning Hyperparameters · p. 15
• Inputs are truncated at 1,024 tokens and outputs at 128 tokens.Source: A Fine-Tuning Hyperparameters · p. 15

• Official evaluation scripts are used for Super-Natural Instructions, T0: Zero-Shot, and LMEntry.Source: B Evaluation Details · p. 15
• BIG-bench Hard outputs are lowercased, stripped of punctuation, and trimmed for whitespace before exact-match scoring.Source: B Evaluation Details · p. 15
• The dyck_languages task is exempt from punctuation removal because its target outputs consist entirely of punctuation characters.Source: B Evaluation Details · p. 15

In-Context Demonstrations

• Reference-understanding prompts ask questions linking pronouns or other mentions to the entities they refer to.Source: C Data Generation Prompts · p. 16
• Commonsense prompts classify everyday events or general statements as True or False according to plausibility and commonsense.Source: C Data Generation Prompts · p. 16
• Ordering prompts determine whether process steps are coherent or return a corrected order for a five-sentence story.Source: C Data Generation Prompts · p. 16
• Other prompts cover object affordances, review sentiment, and classification into ten question topics.Source: C Data Generation Prompts · pp. 16–17
• Constrained examples include science multiple-choice answers, arithmetic solutions, sentence simplification, review conversion, and binary passage classification.Source: C Data Generation Prompts · p. 17