PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi

arXiv:2212.10560v2cs.CLcs.AI

TL;DR

Instruction tuning relies on costly human-written data with limited task diversity. Self-Instruct generates and filters instruction examples from a pretrained language model, then fine-tunes that model on them. It yields a 33% absolute improvement over GPT3 on SUPERNI and outperforms public-dataset tuning on novel tasks, approaching InstructGPT001 in human evaluation.

  • Problem

    Instruction tuning depends on human-annotated data that is costly to collect and often limited in task diversity.

    Source: §1 · p. 1, §5 · p. 8

  • Method

    Self-Instruct generates instructions and input-output examples with a pretrained language model, filters the generated data, and uses it to instruction-tune that model.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    33% absolute improvement over GPT3 on SUPERNI accompanied novel-task human-evaluation performance above public-dataset tuning and close to InstructGPT001.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Pretrained language models can acquire instruction-following capabilities with minimal human-labeled data.

    Source: §1 · p. 2

    Self-Instruct risks reinforcing model biases, with observed difficulty producing balanced labels and concern about amplifying problematic social biases.

    Source: §8 · p. 10

Abstract

from arXiv · show

Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning. Our code and data are available at https://github.com/yizhongw/self-instruct.

1 Introduction

SELF-INSTRUCT bootstraps instruction-tuning data from a language model’s own generations, reducing reliance on costly, limited-diversity human-written instructions. Finetuning GPT3 on this data substantially improves instruction following and approaches InstructGPT001.

1 Introduction

  • Human-written instruction collection is costly and biased toward popular NLP tasks, motivating alternative supervision that covers more tasks and ways of describing them.Improving instruction-tuned models requires broader instruction quality and coverage.Source: §1 · pp. 1–2
  • 175 manually written seed tasks initiate SELF-INSTRUCT’s iterative generation of new instructions and input-output instances, with heuristic filtering before tasks reenter the pool.The accumulated data is then used to instruction-tune the same language model that generated it.Source: §1 · p. 2, Figure 2 · p. 2
  • About 52K instructions and 82K input-output instances are generated using vanilla GPT3 (Brown et al., 2020).Generated tasks depart from typical NLP task distributions and overlap little with the seeds. The authors release the synthetic instructions and manually written novel evaluation tasks.Source: §1 · p. 2
  • +33.1% improvement over original GPT3 is reported for GPT3SELF-INST on SUPERNI, nearly matching InstructGPT001.InstructGPT comparisons use text-davinci-001, chosen for its comparable supervised-finetuning setup.Source: §1 · pp. 1–2
  • 5% behind InstructGPT001, GPT3SELF-INST outperforms models trained on other public instruction datasets in human evaluation on newly created instructions.These instructions test novel uses of instruction-following models beyond typical NLP tasks.Source: §1 · p. 2

2 Method

SELF-INSTRUCT uses a pretrained language model to bootstrap instruction data, filter unsuitable generations, and finetune itself to follow instructions. Its generation pipeline distinguishes classification tasks so it can choose between input-first and output-first instance generation.

2.1 Defining Instruction Data

  • Instruction data pairs each natural-language task instruction with one or more input-output instances, allowing empty inputs when the instruction already specifies the complete task.The model is expected to produce each instance’s output given its instruction and input; allowing empty inputs accommodates the flexible boundary between instructions and inputs.Source: §2.1 · p. 3

2.2 Automatic Instruction Data Generation

  • 175 human-written seed tasks initialize SELF-INSTRUCT, which bootstraps instructions, identifies classification tasks, generates and filters instances, then finetunes the original language model.Each seed task contains one instruction and one instance. Each instruction-generation prompt samples eight instructions: six human-written and two previously generated.Source: §2.2 · p. 3, §2 · p. 3
  • 12 classification and 19 non-classification seed instructions provide few-shot examples for identifying whether generated instructions define classification tasks.Classification tasks are defined as having a small, limited output label space; this identification determines which instance-generation approach is used.Source: §2.2 · p. 3
  • Classification instances use output-first generation, producing labels before conditioning inputs on each label; non-classification instances use input-first generation followed by corresponding outputs.Cross-task instruction-input-output demonstrations guide instance generation, but input-first generation can bias classification inputs toward one label. Fixed seed demonstrations limit generation to a small number of instances per task in one round.Source: §2.2 · p. 3
  • 0.7 is the ROUGE-L similarity cutoff: a new instruction enters the pool only if its similarity to every existing instruction is lower.Filtering also excludes instructions containing keywords associated with unsupported content, such as images. Instance filters remove exact duplicates, identical inputs with conflicting outputs, and heuristic failures such as outputs repeating inputs.Source: §2.2 · pp. 3–4

2.3 Finetuning the LM to Follow Instructions

  • Supervised finetuning trains the original model to generate instance outputs from concatenated instructions and inputs, using varied prompt templates for format robustness.Templates vary instruction and input prefixes, the presence of an output prefix, and the number of intervening line breaks.Source: §2.3 · p. 4

3 SELF-INSTRUCT Data from GPT3

SELF-INSTRUCT applied to GPT3’s “davinci” engine produces over 52K filtered instructions spanning diverse intents, formats, and lengths. Expert review finds mostly meaningful instructions, although their associated instances contain more noise.

3.1 Statistics

  • Over 52K instructions and more than 82K corresponding instances remain after filtering the SELF-INSTRUCT data generated with GPT3.Generation used GPT3’s largest language model, the “davinci” engine, accessed through the OpenAI API.Source: Table 1 · p. 4, §3.1 · p. 4, §3 · p. 4

3.2 Diversity

  • 14% of generated instructions are covered by the 20 most common root verbs and their four leading noun objects, illustrating diverse intents and textual formats.The analysis extracted the verb closest to the parse root and its first direct noun object, finding this structure in 26,559 of 52,445 instructions. Other instructions typically contain more complex clauses or questions.Source: §3.2 · p. 4, Figure 3 · p. 5
  • ROUGE-L comparisons against 175 seed instructions indicate new instructions with little seed overlap; instruction, input, and output lengths also vary.Each generated instruction was compared with its most similar seed instruction using its highest ROUGE-L overlap.Source: §3.2 · p. 4, Figure 4 · p. 5

3.3 Quality

  • Most generated instructions are meaningful, but their instances contain more noise; erroneous generations often retain correct formatting or partial correctness that may aid training.An expert annotator, also an author, assessed instruction, input, and output correctness for one randomly selected instance from each of 200 randomly sampled instructions.Source: §3.3 · p. 4, Table 2 · p. 5

4 Experimental Results

Self-generated instruction tuning substantially improves GPT3 on unseen SUPERNI tasks and novel user-oriented instructions, approaching InstructGPT001. Increasing data size yields diminishing returns, while regenerating training outputs with InstructGPT003 improves performance further.

4.1 GPT3SELF-INST: finetuning GPT3 on its own instruction data

  • GPT3SELF-INST trains GPT3’s davinci engine to generate outputs from instructions and inputs, substantially improving instruction following over vanilla GPT3.Finetuning uses OpenAI’s API with default hyperparameters except a prompt loss weight of 0 and training for 2 epochs.Source: §4.1 · p. 5, §4.3 · p. 6

4.2 Baselines

  • Baselines include pretrained models, public instruction-tuned models, InstructGPT (Ouyang et al., 2022), and GPT3 finetuned on comparable public instruction datasets.Pretrained baselines are T5-LM (Lester et al., 2021; Raffel et al., 2020) and GPT3 (Brown et al., 2020); public baselines are 11B-parameter T0 and T𝑘-INSTRUCT, proposed in Sanh et al. (2022) and Wang et al. (2022), respectively. GPT3 public-data baselines use 50K instances covering all instructions from PROMPTSOURCE or SUPERNI; benchmark comparisons exclude InstructGPT002/003 because their newer user data may contain the SUPERNI test set, while human evaluation includes all three engines.Source: §4.2 · pp. 5–6

4.3 Experiment 1: Zero-Shot Generalization on SUPERNI benchmark

  • +33.1% improvement over vanilla GPT3 on unseen SUPERNI tasks lets GPT3SELF-INST nearly match InstructGPT001 and outperform T0 and GPT3 tuned on T0 data.Evaluation uses 119 tasks with 100 instances each from SUPERNI, prompting with task definitions without demonstrations. GPT3 variants use deterministic generation: temperature 0, no nucleus sampling, and no specific stop sequences.Source: Table 3 · p. 6, §4.3 · p. 6
  • SELF-INSTRUCT adds gains when combined with SUPERNI training data, although models trained on SUPERNI still perform better on its evaluation set.The authors attribute the advantage of SUPERNI-trained models to similar instruction styles and formatting, positioning generated instructions as complementary training data.Source: §4.3 · p. 6

4.4 Experiment 2: Generalization to User-oriented Instructions on Novel Tasks

  • 5% behind InstructGPT001 when minor-imperfection responses count as valid, GPT3SELF-INST substantially outperforms GPT3 tuned on T0 or SUPERNI on user-oriented tasks.Instruction authors judged responses to 252 newly written, diverse user-oriented instructions using four quality levels, with inter-rater agreement κ=0.57. All instruction-tuned models outperform vanilla GPT3, and InstructGPT002/003 demonstrate strong instruction-following ability.Source: §4.4 · pp. 6–7, Figure 6 · p. 7

4.5 Effect of Data Size and Quality

  • 16K generated instructions nearly saturate performance on user-oriented tasks, whereas SUPERNI gains plateau after only hundreds of instructions.The analysis finetunes GPT3 on subsampled instruction sets and applies the same human evaluation to 252 user-oriented instructions. The authors suggest that differences between generated tasks and SUPERNI motivate combining instruction datasets.Source: §4.5 · p. 8
  • 10% improvement over training on original generated data results from finetuning GPT3 on outputs regenerated by InstructGPT003.InstructGPT003 regenerates every output using the existing instruction and input, effectively distilling its capabilities into GPT3. The authors suggest improving generated supervision through stronger models or human experts.Source: §4.5 · p. 8

5 Related Work

SELF-INSTRUCT generates new tasks with a vanilla language model, addressing limitations of human-annotated instruction data and task-specific data generation. The related-work discussion distinguishes its task-agnostic approach from instruction generation, self-training, and conventional knowledge distillation.

Instruction-following LMs

  • SELF-INSTRUCT addresses human-annotation dependence and limited task coverage by generating diverse instruction data with pretrained language models.Instruction-data size and diversity correlate with unseen-task generalization (Chung et al., 2022). InstructGPT (Ouyang et al., 2022) pursues similar generality, but its private data and limited transparency motivate a large, public dataset.Source: §5 · p. 8

Language models for data generation and augmentation; Bootstrapping with limited resources

  • SELF-INSTRUCT bootstraps potentially new task definitions rather than generating data for a fixed task or improving specialized inference procedures.NPPrompt (Zhao et al., 2022) reduces manual verbalizer dependence without finetuning; STAR bootstraps reasoning; Self-Correction (Welleck et al., 2023) trains a separate corrector.Source: §5 · pp. 8–9

Language models for data generation and augmentation; Knowledge distillation

  • Unnatural Instructions (Honovich et al., 2022a) uses SUPERNI seeds and InstructGPT002, whereas SELF-INSTRUCT relies on a vanilla language model.SUPERNI seeds yield a different task distribution, and generation pipelines and templates also differ. Viewed as distillation, SELF-INSTRUCT transfers knowledge into the same model through task-defining instructions and examples, rather than the conventional larger-to-smaller transfer.Source: §5 · p. 9

Instruction generation; Model self-training

Multi-modal instruction-following

6 Conclusion

SELF-INSTRUCT improves instruction-following by training language models on their own generated instruction data. In GPT3 experiments, it improves SUPERNI performance and outperforms public instruction datasets in human evaluation on novel tasks.

6 Conclusion

  • 33% absolute improvement on SUPERNI over original GPT3 results from finetuning GPT3 on SELF-INSTRUCT’s generated instruction dataset.Source: §6 · p. 9
  • Human evaluation on expert-written novel tasks shows SELF-INSTRUCT-tuned GPT3 substantially outperforms tuning with existing public instruction datasets and performs closely to InstructGPT001.Source: §6 · p. 9
  • 52K instructions covering diverse tasks are automatically constructed using vanilla GPT3 for the SELF-INSTRUCT finetuning dataset.The method uses a language model’s own generated instruction data to improve its instruction-following ability.Source: §6 · p. 9

7 Broader Impact

Self-Instruct may help clarify how instruction-tuned models achieve their capabilities, while its synthetic dataset offers a step toward higher-quality instruction data. Its central idea has also been adopted in follow-up work (Taori et al., 2023; Xu et al., 2023; Sun et al., 2023, i.a.).

7 Broader Impact

  • Self-Instruct may improve transparency around instruction-tuned models such as InstructGPT and ChatGPT, whose unreleased datasets limit understanding of their construction and capabilities.The authors argue that academia must investigate these models’ success and pursue better, more open alternatives.Source: §7 · p. 10
  • The authors emphasize diverse instruction data and position their large synthetic dataset as a first step toward higher-quality data for better instruction-following models.Source: §7 · p. 10
  • Self-Instruct’s central idea has been adopted in follow-up work pursuing these goals (Taori et al., 2023; Xu et al., 2023; Sun et al., 2023, i.a.).Source: §7 · p. 10

8 Limitations

SELF-INSTRUCT may struggle with rare instructions, work best with larger models, and amplify problematic social biases.

Tail phenomena

Dependence on large models

  • SELF-INSTRUCT may work best with larger models, potentially creating access barriers for researchers with limited computing resources.Future studies should examine gains across model sizes and other parameters. Human-annotated instruction tuning likewise yields larger gains for larger models (Wei et al., 2022).Source: §8 · p. 10

Reinforcing LM biases

  • SELF-INSTRUCT’s iterative generation may amplify problematic social biases, while observed difficulty producing balanced labels reflects models’ existing biases.Potential harms include reinforcing stereotypes or slurs about gender and race.Source: §8 · p. 10

Supplemental Material … A.4 Prompting Templates for Data Generation

Self-Instruct bootstraps generation from author-written seed tasks, queries GPT3 with purpose-specific settings, and finetunes using largely default API settings. Its prompting templates cover instruction generation, task classification, and input-first or output-first instance generation.

A.1 Writing the Seed Tasks

  • 25 classification tasks and 150 non-classification tasks form the seed pool, written by the authors and UWNLP labmates without referencing existing datasets or specific tests.Classification tasks have limited output label spaces. Diverse seeds encourage diverse, higher-quality generated tasks while demonstrating correct solutions.Source: §A.1 · p. 14
  • 0.21 for SUPERNI and 0.34 for user-oriented instructions are average ROUGE-L similarities between seed instructions and their closest instructions in each test set.Exactly one seed instruction also appears verbatim in the user-oriented test set: “answer the following question”, but the associated questions differ.Source: Figure 8 · p. 14, §A.1 · p. 14

A.2 Querying the GPT3 API

  • Around $600 was spent generating the entire dataset using purpose-specific GPT3 API hyperparameters.The settings worked well for “davinci” and instruction-tuned GPT3 variants. As of December 2022, “davinci” completion requests cost $0.02 per 1000 tokens.Source: §A.2 · p. 14

A.3 Finetuning GPT3

  • $338 was the cost of finetuning GPT3SELF-INST on the entire generated dataset through OpenAI’s API.GPT3SELF-INST and some baselines start from 175B-parameter GPT3 (“davinci”); with API training internals undisclosed, default hyperparameters support comparability. The authors set prompt_loss_weight to 0 because it worked better and train every experiment for two epochs to avoid overfitting.Source: §A.3 · pp. 14–15

A.4 Prompting Templates for Data Generation

  • Non-classification instances use input-first prompting, generating an input before its corresponding output, or generating output directly when no additional input is required.Instruction-generation prompts randomly sample eight existing instructions as demonstrations and stop at model termination, the length limit, or “Task 16”. A separate template identifies whether an instruction describes a classification task.Source: §A.4 · p. 15, Table 5 · p. 15, Table 6 · p. 16
  • Classification instances use output-first prompting: the model generates a class label before generating the corresponding input.Source: Table 8 · p. 18

B Human Evaluation Details for Following the User-oriented Instructions · C Task and Instance Examples from the Generated Instruction Data

Two instruction-author evaluators independently rated model responses, showing moderate agreement on four-level ratings and substantial agreement on response acceptability. Examples illustrate detailed but incorrect model responses and both valid and erroneous generated instruction data.

B.1 Human Evaluation Setup

  • 252 user-oriented instructions were independently rated by two instruction-author evaluators using coordinated four-level standards and anonymized model responses presented in random order.Evaluators saw instructions, inputs, reference targets, and responses in a Google Sheet. Reported performance uses one evaluator’s ratings; the other evaluator’s results show the same trends.Source: §B.1 · p. 19, Figure 9 · p. 19

B.2 Human Evaluation Agreement

  • 0.58 Cohen’s κ for four-level ratings and 0.75 for acceptable-versus-unacceptable responses indicate moderate and substantial inter-rater agreement, respectively.Acceptable responses combine ratings A and B; unacceptable responses combine C and D.Source: §B.2 · p. 19
  • 0.81 Spearman correlation coefficient ρ indicates high correlation between the two evaluators when ratings are treated as ordinal variables, A>B>C>D.This complements Cohen’s κ by accounting for the ordering of the rating levels.Source: §B.2 · pp. 19–20

B.3 Example Predictions from GPT3SELF-INST

  • GPT3SELF-INST examples demonstrate extensive problem-solving steps even when level-C responses have incorrect final outputs.Table 9 pairs selected user-oriented tasks and model predictions with annotator ratings, ranging from A for valid, satisfying responses to D for irrelevant or invalid responses.Source: §B.3 · p. 20, Table 9 · p. 21

C Task and Instance Examples from the Generated Instruction Data

  • More than half of all generated tasks are correct, while representative valid GPT3 tasks span diverse formats, text types, and underlying expertise.Table 10 presents representative valid generated tasks.Source: Table 10 · p. 22
  • Erroneous GPT3-generated tasks can still provide useful supervision for instruction following.Table 11 presents representative invalid tasks and identifies their problematic fields in a validity column.Source: Table 11 · p. 23

Related papers

Loading 2212.10560v2…