PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi

arXiv:2212.10560v2cs.CLcs.AI

TL;DR

Human-written instruction data is costly and limited in diversity, constraining progress toward more generalizable instruction-following models. Self-Instruct bootstraps instruction data from a pretrained model’s own generations, and finetuning GPT3 on the resulting data improves Super-NaturalInstructions performance by 33% while closely matching InstructGPT-001 in human evaluation.

  • Problem

    Human-written instruction data is costly and often lacks diversity, limiting coverage of tasks needed for more generalizable instruction-following models.

  • Method

    Self-Instruct iteratively uses a pretrained language model to generate and filter instructions plus input-output instances, then finetunes the model on the synthetic data.

  • Results

    33.1% absolute improvement on SUPERNI over GPT3 was observed, while human evaluation on novel tasks showed performance closely matching InstructGPT-001 and exceeding public instruction datasets.

  • Takeaways & Limitations

    Self-Instruct offers an almost annotation-free route to instruction tuning and releases a large synthetic dataset for future instruction-following research.

  • Takeaways & Limitations

    The iterative approach may amplify problematic social biases, including stereotypes or slurs, and may produce imbalanced labels reflecting pretrained model biases.

Abstract

from arXiv · show

Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning. Our code and data are available at https://github.com/yizhongw/self-instruct.

1 Introduction

SELF-INSTRUCT addresses the limited diversity and costly collection of human-written instruction data by bootstrapping instruction-tuning signals from a pretrained language model using a small manually written seed set. Applied to vanilla GPT3, it generates a large, diverse synthetic dataset for finetuning and evaluation.

  • Motivation: Human-written instruction data is costly and often lacks diversity, limiting coverage of task types and instruction formulations.The paper motivates alternative approaches for supervising instruction tuning to improve model quality and coverage.
  • Method: SELF-INSTRUCT iteratively bootstraps instruction-tuning data from a pretrained LM, beginning with a limited seed set of manually written tasks.The model generates new instructions and corresponding instances, which are filtered and added back to the task repository.
  • Empirical evaluation: 52k instructions, paired with about 82K instance inputs and target outputs, are generated by applying SELF-INSTRUCT to vanilla GPT3.The resulting data covers creative tasks that deviate from typical NLP-task distributions and overlap only modestly with seed tasks.
  • Empirical evaluation: The authors finetune GPT3 on the generated data to build GPT3SELF-INST and evaluate it against other models on typical SUPERNI tasks and manually written novel tasks.The work also releases the synthetic dataset and manually written novel tasks for future instruction-following research.

2 Method

SELF-INSTRUCT bootstraps instruction data from a pretrained language model, filters low-quality generations, and uses the resulting synthetic examples for instruction tuning. Its four-stage pipeline generates tasks, identifies classification tasks, creates instances with task-specific strategies, and removes invalid or insufficiently diverse data.

  • Pipeline: SELF-INSTRUCT generates tasks from seed instructions, classifies each task, creates input-output instances, and filters low-quality data before tuning.The pipeline comprises four steps: instruction generation, classification-task identification, instance generation, and filtering.
  • Instruction Generation: The task pool begins with 175 human-written tasks, while each generation step samples 8 instructions, including 6 human-written and 2 model-generated examples.The model-generated examples are included to promote diversity during bootstrapping.
  • Classification Task Identification: For classification-task identification, the language model uses few-shot prompting with 12 classification and 19 non-classification seed instructions.Tasks with a small, limited output-label space are treated as classification tasks.
  • Instance Generation: Instance generation requires interpreting the task, determining needed input fields, and producing outputs; classification tasks additionally use output-first generation to condition inputs on class labels.The output-first approach addresses input-label bias, such as generating mostly grammatical inputs for grammar-error detection.
  • Filtering and Postprocessing: Filtering enforces diversity with a ROUGE-L similarity threshold below 0.7 and removes unsupported instructions, duplicate or conflicting instances, and heuristic invalid generations.Excluded instructions include those containing keywords such as image, picture, or graph; invalid examples include overly long instructions and repetitive outputs.
  • Instruction Tuning: The generated instruction and input are concatenated into prompts, and the original language model is fine-tuned to produce the output using supervised training and multiple formatting templates.Templates vary prefixes, output markers, and line breaks to improve robustness to different formats.

3 SELF-INSTRUCT Data from GPT3

Applying SELF-INSTRUCT to GPT3 produced over 52K filtered instructions and more than 82K corresponding instances. The generated instructions were diverse, included many novel formulations relative to the 175 seeds, and were mostly meaningful in expert review.

  • Generated data overview: Over 52K instructions and more than 82K corresponding instances were generated after filtering with GPT3.The case study used GPT3’s largest “davinci” engine through the OpenAI API.
  • Instruction diversity: 26,559 out of 52,445 instructions contained a simple verb-noun structure, while others used more complex clauses.The structure was extracted using the Berkeley Neural Parser by identifying the root-nearest verb and its first direct noun object.
  • Instruction diversity: The top 20 root verbs and their four most common direct noun objects accounted for 14% of the instruction set, indicating diverse intents and formats.Many instructions lacked the analyzed verb-noun structure, so the plotted patterns covered only a minority of the data.
  • Instruction novelty: Generated instructions showed limited ROUGE-L overlap with some of the 175 seed instructions, demonstrating the production of novel instructions.Diversity also appeared in the lengths of instructions, instance inputs, and instance outputs.
  • Data quality: An expert review of 200 randomly sampled instructions and one instance per instruction found that most generated instructions were meaningful.The review assessed correctness of each instruction, instance input, and instance output.

4 Experimental Results

The experiments compare SELF-INSTRUCT with vanilla, publicly instruction-tuned, and InstructGPT models on zero-shot SUPERNI tasks and diverse user-oriented instructions. SELF-INSTRUCT substantially improves GPT3, nearly matches InstructGPT001, and remains complementary to labeled instruction data.

  • Compared models: The study compares vanilla T5-LM and GPT3 baselines, public T0 and T𝑘-INSTRUCT models, and instruction-tuned GPT3 models including InstructGPT.The public T5-derived instruction-tuned models are evaluated using their largest 11B-parameter versions.
  • SUPERNI evaluation: SELF-INSTRUCT boosts GPT3 performance on unseen SUPERNI tasks by +33.1% and nearly matches InstructGPT001.The evaluation covers 119 tasks with 100 instances each in a zero-shot setting without in-context demonstrations.
  • SUPERNI evaluation: GPT3SELF-INST outperforms T0 and GPT3 finetuned on T0 data, while nearly matching InstructGPT001 trained with private user data and human-annotated labels.Vanilla GPT3 generally produces irrelevant, repetitive text and fails to stop generation.
  • SUPERNI evaluation: SELF-INSTRUCT provides additional gains when combined with SUPERNI training data, demonstrating complementary value beyond labeled instruction data.Models trained directly on SUPERNI perform better on its evaluation set, which the authors attribute to matching instruction style and formatting.
  • User-oriented evaluation: On 252 diverse user-oriented instructions judged by human experts, GPT3SELF-INST outperforms GPT3 variants trained on publicly available instruction datasets and nearly matches InstructGPT001.The evaluation uses four response-quality ratings and reports inter-rater agreement of κ=0.57.

5 Related Work

Prior instruction-following work relies on human-annotated data and often targets existing NLP tasks, whereas SELF-INSTRUCT uses pretrained language models to generate diverse, previously undefined tasks from scratch. It also differs from related generation and distillation methods in being task-agnostic and self-distilling instruction tasks.

  • Instruction-following LMs: Instruction-tuned language models benefit from larger and more diverse human-annotated instructional datasets, but dependence on such data creates a bottleneck for generalizable models.SELF-INSTRUCT addresses this limitation by employing pretrained language models to create diverse instruction data.
  • Language models for data generation and augmentation: Unlike task-specific data generation or augmentation, SELF-INSTRUCT bootstraps new task definitions that may not have been defined by NLP practitioners.The approach is not restricted to a particular task such as QA or NLI.
  • Language models for data generation and augmentation: Compared with Unnatural Instructions, SELF-INSTRUCT uses different seed tasks and a different generating model, producing a different distribution of generated tasks.Unnatural Instructions uses SUPERNI tasks as seeds and InstructGPT002 to generate the data.
  • Instruction generation: Unlike prior instruction-generation methods that infer instructions from a few examples, SELF-INSTRUCT is task-agnostic and generates new tasks with instructions and instances from scratch.This distinction concerns the scope of task creation rather than merely generating instructions for an existing task.
  • Knowledge distillation: SELF-INSTRUCT resembles knowledge distillation but transfers a model’s knowledge to itself in the form of instruction tasks rather than from a larger model to a smaller one.The distilled content consists of task-defining instructions and example instances.

6 Conclusion

SELF-INSTRUCT improves language models’ instruction-following ability by having them generate their own instruction data. Applied to vanilla GPT3, the method produces a large dataset and yields a substantial SUPERNI improvement.

  • 6 Conclusion: SELF-INSTRUCT improves language models’ instruction-following ability through their own generation of instruction data.The method is presented as a way to bootstrap instruction data from the model itself.
  • 6 Conclusion: 52K instructions are automatically constructed for diverse tasks when experimenting with vanilla GPT3.The resulting dataset is described as large-scale and task-diverse.
  • 6 Conclusion: 33% absolute improvement on SUPERNI follows from finetuning GPT3 on the SELF-INSTRUCT dataset versus the original GPT3.This comparison is reported for vanilla GPT3 finetuned on the automatically constructed data.
  • 6 Conclusion: Expert-written instructions for novel tasks are curated for further human evaluation.The passage introduces this evaluation set after reporting the SUPERNI result.

7 Broader Impact

SELF-INSTRUCT may improve transparency into how widely used instruction-tuned models are constructed and why they exhibit impressive capabilities, especially because industrial models and their datasets remain inaccessible behind APIs.

  • Transparency: SELF-INSTRUCT may bring more transparency to the hidden construction of widely used instruction-tuned models such as InstructGPT and ChatGPT.The passage frames this as a broader impact beyond the paper’s immediate focus.
  • Transparency: Because industrial models remain behind API walls and their datasets are unreleased, their construction is poorly understood.The passage directly links restricted access to limited understanding of these models.
  • Academic responsibility: The passage argues that academia bears responsibility for understanding why these models demonstrate impressive capabilities.This responsibility follows from limited transparency into industrial models’ development.

8 Limitations

The authors identify three limitations of SELF-INSTRUCT: inherited weaknesses in low-frequency language contexts, possible dependence on large models, and amplification of problematic social biases. These limitations may constrain gains, accessibility, and label balance, motivating further study.

  • Tail phenomena: SELF-INSTRUCT may yield minimal gains in low-frequency language contexts because it inherits language-model limitations and performs best on frequent language uses.The authors describe these frequent uses as the head of the language-use distribution and identify tail phenomena as a serious challenge for language models.
  • Dependence on large models: SELF-INSTRUCT might work best for larger models because it depends on inductive biases extracted from language models.If confirmed, this dependence could create access barriers for researchers without large computing resources.
  • Dependence on large models: Instruction-tuning gains are higher for larger models, so this limitation also affects approaches using human annotation.The authors note that human-annotation-based instruction tuning suffers from a similar limitation.
  • Reinforcing LM biases: The iterative algorithm may amplify problematic social biases and has difficulty producing balanced labels that reflect models’ prior biases.Examples include stereotypes or slurs concerning gender and race.

Supplemental Material … A.4 Prompting Templates for Data Generation

The supplemental material details how Self-Instruct is implemented, from author-written seed tasks and GPT3 querying to finetuning and prompting templates for generating instructions and instances.

  • A.1 Writing the Seed Tasks: Seed-task diversity encourages broader, higher-quality generation by demonstrating both varied tasks and correct solution formats.For example, coding seeds increase the likelihood and quality of generated coding-related tasks.
  • A.1 Writing the Seed Tasks: Seed tasks were designed by the authors and labmates for diverse LLM uses, comprising 25 classification and 150 non-classification tasks.They were written without explicit reference to existing datasets or specific testing tasks.
  • A.2 Querying the GPT3 API: GPT3 querying uses purpose-specific hyperparameters selected to work well with the “davinci” engine and instruction-tuned GPT3 variants.The hyperparameters are listed in Table 4.
  • A.2 Querying the GPT3 API: $600 was the approximate cost of generating the entire dataset, with OpenAI charging $0.02 per 1000 tokens for “davinci” completions as of December 2022.The stated pricing applies to completion requests to the “davinci” engine.
  • A.3 Finetuning GPT3: GPT3SELF-INST and baselines are finetuned from the 175B-parameter GPT3 “davinci” model through OpenAI’s finetuning API.The API’s internal finetuning details, such as updated parameters and optimizer, are unavailable.
  • A.3 Finetuning GPT3: Two training epochs, default API hyperparameters, and prompt_loss_weight=0 are used, while finetuning GPT3SELF-INST on all generated data costs $338.Two epochs are intended to avoid overfitting the training tasks, and finetuning charges depend on training-file tokens.
  • A.4 Prompting Templates for Data Generation: SELF-INSTRUCT uses four prompting templates for instruction generation, task-type classification, and non-classification or classification instance generation.The templates are presented in Tables 5–8.
  • A.4 Prompting Templates for Data Generation: Instruction generation samples 8 existing instructions for in-context demonstration, while instance generation uses input-first for non-classification and output-first for classification tasks.The instruction prompt stops at generation termination, its length limit, or “Task 16” tokens; input-first generates input before output, whereas output-first generates the class label before input.

B Human Evaluation Details for Following the User-oriented Instructions · C Task and Instance Examples from the Generated Instruction Data

The appendix details a blinded, independently rated human evaluation of model responses, reports moderate-to-high evaluator agreement, and illustrates both model predictions and generated-task validity. Example tables show that GPT3SELF-INST can provide extensive reasoning even when its final answer is incorrect, while generated tasks span varied formats and include both valid and error-containing cases.

  • B.1 Human Evaluation Setup: Two instruction authors independently rated all responses using coordinated standards and a four-level A–D quality scale.Evaluators reviewed the instruction, input, target, and anonymized model response, with predictions shown in random order.
  • B.1 Human Evaluation Setup: Predictions were presented in random order with model identities anonymized during evaluation.Evaluators selected A/B/C/D after reading the instruction and input and consulting the target.
  • B.2 Human Evaluation Agreement: 0.58 Cohen’s κ indicated moderate agreement for categorical A–D ratings.The four-level rating was treated as a categorical variable when computing Cohen’s κ.
  • B.2 Human Evaluation Agreement: 0.75 Cohen’s κ indicated substantial agreement when responses were classified as acceptable versus unacceptable.Acceptable responses were grouped as A or B, while unacceptable responses were grouped as C or D.
  • B.2 Human Evaluation Agreement: 0.81 Spearman correlation indicated high correlation between the two evaluators.The reported coefficient was ρ = 0.81.
  • B.3 Example Predictions from GPT3SELF-INST: Table 9 presents user-oriented tasks, GPT3SELF-INST responses, and annotator ratings on a four-level scale from A to D.A denotes valid and satisfying responses, whereas D denotes irrelevant or invalid responses.
  • B.3 Example Predictions from GPT3SELF-INST: Even responses rated C can show extensive task-solving steps despite producing an incorrect final output.The examples illustrate that detailed intermediate reasoning does not guarantee a correct final answer.
  • C Task and Instance Examples from the Generated Instruction Data: Generated-task examples include broad formats, text types, and expertise, with valid tasks correct on more than half of generations and invalid tasks still offering instructional signals.Table 10 shows representative valid tasks, while Table 11 identifies problematic fields in representative invalid tasks.
Loading 2212.10560v2…