Instruction Tuning with GPT-4

Baolin Peng; Chunyuan Li; Pengcheng He; Michel Galley; Jianfeng Gao

Source-linked AI summary

Instruction Tuning with GPT-4

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, Jianfeng Gao

arXiv:2304.03277v1Apr 6, 2023cs.CLcs.AI

TL;DR

Whether GPT-4 could serve as a teacher for self-instruct tuning remained untested. The paper uses GPT-4 to generate instruction-following data for LLaMA tuning, which often performs better than Alpaca and untuned LLaMA.

Problem
Whether GPT-4 could serve as a teacher for self-instruct tuning had not yet been explored.
Source: §1 · p. 1
Method
The authors use GPT-4 as a self-instruct teacher to generate English and Chinese tuning data, plus feedback and comparison data for evaluation and reward modeling.
Source: §1 · p. 1, §2 · p. 2
Results
54.12% of human-evaluation votes favored GPT-4-tuned LLaMA for helpfulness, versus 19.74% for GPT-3-tuned LLaMA, while GPT-4 tuning often outperformed Alpaca and untuned LLaMA.
Source: §4.2 · p. 5, §4.3 · p. 7
Takeaways & Limitations
The released GPT-4-generated data and finetuned checkpoints provide resources for developing open-source general-purpose LLMs.
Source: §6 · p. 8
The study remains preliminary at 52K GPT-4 examples and a 7B LLaMA base, with larger-scale data and models left for future work.
Source: §6 · p. 8

Abstract

from arXiv · show

Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed. In this paper, we present the first attempt to use GPT-4 to generate instruction-following data for LLM finetuning. Our early experiments on instruction-tuned LLaMA models show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks to the instruction-following data generated by previous state-of-the-art models. We also collect feedback and comparison data from GPT-4 to enable a comprehensive evaluation and reward model training. We make our data generated using GPT-4 as well as our codebase publicly available.

1 INTRODUCTION

The paper proposes using GPT-4 as a teacher for self-instruct tuning, building on instruction tuning’s potential to improve LLM generalization. It introduces GPT-4-generated data and models for training and evaluation.

1 INTRODUCTION

The paper proposes the first use of GPT-4 as a teacher for self-instruct tuning to advance LLM instruction tuning.The introduction frames this approach against prior instruction-tuning methods that improve zero- and few-shot generalization.Source: §1 · p. 1
The authors release GPT-4-generated 52K English and Chinese instruction-following data and feedback ratings for outputs from three instruction-tuned models.They also report developing instruction-tuned LLaMA models and reward models from the generated data.Source: §1 · p. 1
Evaluation on test samples with unseen instructions uses human judgments on three alignment criteria, GPT-4 feedback, and ROUGE-L.The supplied passage ends mid-description of the ROUGE-L evaluation scope.Source: §1 · p. 1

1 PROMPT DICT{

The section gives GPT-4 instruction-tuning prompt formats for tasks with and without additional input context, and reports that GPT-4-generated data was effective for instruction-tuning.

1 PROMPT DICT{

With input, the prompt presents an instruction and optional context, then requests a response that appropriately completes the request.Its template labels the fields “Instruction,” “Input,” and “Response.”Source: §1 · p. 2
Without input, the prompt describes a task and requests a response that appropriately completes the request.This version uses an instruction field followed by a response field, without an input field.Source: §1 · p. 2
The empirical study validates GPT-4-generated data for LLM instruction-tuning and suggests practical tips for building general-purpose instruction-following agents.The passage does not specify quantitative results here.Source: §1 · p. 2

2 DATASET

The dataset reuses Alpaca’s 52K instructions and replaces GPT-3.5 answers with GPT-4 generations. It includes English and Chinese instruction-following data, model-comparison ratings, and answers to Unnatural Instructions prompts.

Data Collection

GPT-4 provides one English answer for each of Alpaca’s 52K instructions, while Chinese data uses ChatGPT-translated instructions answered by GPT-4.The authors leave building their own instruction set through an iterative GPT-4 and self-instruct process to future work.Source: §2 · p. 2
GPT-4 rates its own responses from 1 to 10 and compares responses from GPT-4, GPT-3.5, and OPT-IML to create reward-model training data.OPT-IML is identified as Iyer et al. (2022).Source: §2 · p. 2
GPT-4 answers the core 68K instruction-input-output triplets from Unnatural Instructions to quantify the gap from instruction-tuned models at scale.The cited dataset is Honovich et al. (2022).Source: §2 · p. 2

Data Statistics

GPT-4 tends to produce longer sequences than GPT-3.5, while Alpaca’s GPT-3.5 outputs have a longer tail, possibly due to iterative deduplication absent from the one-time GPT-4 generation.The paper reports more favorable alignment performance for the GPT-4 data despite its simple one-time generation process.Source: §2 · pp. 2–3, Figure 1 · p. 3

3 INSTRUCTION-TUNING LANGUAGE MODELS

The section trains English- and Chinese-instruction-tuned LLaMA models on GPT-4 data, then uses GPT-4 response comparisons to develop and evaluate a reward model.

3.1 SELF-INSTRUCT TUNING

Both LLaMA-GPT4 and LLaMA-GPT4-CN use supervised finetuning from the LLaMA 7B checkpoint, each with 52K GPT-4-generated instructions in its respective language.The English data distribution is shown in Figure 1.Source: §3.1 · p. 4
Following Taori et al. (2023)’s training schedule, the models test GPT-4 data quality and cross-language generalization in instruction tuning.Source: §3.1 · p. 4

3.2 REWARD MODELS

RLHF aligns LLM behavior with human preferences, and reward modeling formulates this as predicting a scalar reward for a prompt and response.Source: §3.2 · p. 4
Human comparison-labeling costs kept Alpaca, Vicuna, and Dolly from RLHF, while GPT-4’s response-judging ability motivated generating comparison data (Peng et al., 2023; Bai et al., 2022; Kim et al., 2023).Source: §3.2 · p. 4
An OPT 1.3B reward model rates responses, while GPT-4 assigns each response a score from 1 to 10 for prompts with K responses.Source: §3.2 · p. 4
For each prompt, score-ordered response pairs train the reward model to prefer the higher-scored response through a sigmoid-based objective; their distribution appears in Figure 2.Source: §3.2 · p. 4

4 EXPERIMENTAL RESULTS

The experiments assess GPT-4-instruction-tuned LLaMA on unseen tasks using established instruction benchmarks, human HHH judgments, and GPT-4-based comparisons.

4.1 BENCHMARKS

The evaluation targets unseen instructions using three benchmarks: user-oriented applications, challenging questions, and text-davinci-002-generated unnatural instructions.The sets comprise 252 user-oriented instructions, 80 challenging questions, and 68,478 unnatural-instruction samples.Source: §4.1 · p. 4

4.2 HUMAN EVALUATION WITH ALIGNMENT CRITERIA

Helpfulness votes favored GPT-4 tuning (54.12%) over GPT-3 tuning (19.74%); honesty and harmlessness votes mostly tied, with GPT-3 slightly superior.GPT-4-tuned LLaMA performed similarly to GPT-4 across the three HHH criteria.Source: §4.2 · pp. 5–6, Figure 3 · p. 5

4.3 COMPARISONS WITH SOTA USING AUTOMATIC EVALUATION

GPT-4 evaluations of five reward-model-ranked response groups followed the reward model’s ordering, supporting the value of feedback data and reward-model effectiveness.The ranked groups were compared with baseline decoding that generated one response per question.Source: §4.3 · pp. 6–7, Figure 4 · p. 6
GPT-4-tuned 7B LLaMA outperforms 13B Alpaca and LLaMA, though it remains behind commercial GPT-4.The comparison uses GPT-4 evaluation against ChatGPT and GPT-4.Source: Figure 4 · p. 6, §4.3 · p. 7
GPT-4’s translated English responses performed better than its generated Chinese responses, while relative scores were consistent across opponents and languages.The Chinese comparisons included translated English responses and responses generated in Chinese.Source: §4.3 · p. 7, Figure 5 · p. 7
On unnatural instructions, Alpaca led in average ROUGE-L, but LLaMA-GPT4 and GPT-4 improved as ground-truth response length increased and exceeded Alpaca beyond length 4.For short responses, LLaMA-GPT4 and GPT-4 often added chat-like words beyond the simple ground-truth answer, lowering ROUGE-L.Source: §4.3 · pp. 7–8, Figure 6 · p. 8

5 RELATED WORK

Related work develops instruction-tuning pipelines across data, foundation models, and evaluation, alongside open-source efforts to build aligned general-purpose text and multimodal assistants.