PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruction Tuning with GPT-4

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, Jianfeng Gao

arXiv:2304.03277v1cs.CLcs.AI

TL;DR

Instruction-tuning research needs effective machine-generated data and scalable evaluation for improving LLM instruction following. This paper uses GPT-4 to generate instruction-following and comparison data, fine-tunes LLaMA, and trains reward models. The resulting GPT-4-tuned models outperform relevant smaller baselines on evaluated unseen tasks, while remaining behind large commercial chatbots.

  • Problem

    Instruction-tuning requires effective instruction-following data and scalable comparison data for evaluating and aligning LLMs.

  • Method

    The paper uses GPT-4 to generate 52K English and Chinese instruction-following examples, comparison data, and answers for unseen-task evaluation, then fine-tunes LLaMA and trains a reward model.

  • Results

    Instruction tuning of LLaMA with GPT-4 often achieves higher performance than tuning with text-davinci-003 or no tuning, though it remains behind large commercial chatbots such as GPT-4.

  • Takeaways & Limitations

    The released GPT-4-generated data and models provide a promising direction for developing open-source instruction-following LLMs.

  • Takeaways & Limitations

    The study uses 52K GPT-4 examples and a 7B base LLaMA model, and the reward model is used only during decoding rather than for further LLM training.

Abstract

from arXiv · show

Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed. In this paper, we present the first attempt to use GPT-4 to generate instruction-following data for LLM finetuning. Our early experiments on instruction-tuned LLaMA models show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks to the instruction-following data generated by previous state-of-the-art models. We also collect feedback and comparison data from GPT-4 to enable a comprehensive evaluation and reward model training. We make our data generated using GPT-4 as well as our codebase publicly available.

1 INTRODUCTION

The paper proposes using GPT-4 as a teacher for self-instruct tuning and releases GPT-4-generated instruction-following and feedback data for training and evaluation.

  • The released resources include 52K English and Chinese instruction-following instances and GPT-4-generated feedback data.
  • The evaluation uses human alignment judgments, GPT-4 feedback, and ROUGE-L on unseen test samples.
  • The release is preliminary, with plans to expand the dataset and fine-tune larger models.

1 PROMPT DICT{

The prompt template frames each task as an instruction paired with optional input context and requests an appropriately completing response.

  • The prompt asks the model to write a response that appropriately completes the request.
  • With input, the template separates the instruction, contextual input, and response fields.
  • Without input, the template omits the input field and presents only the instruction and response fields.
  • The study reports that GPT-4-generated instruction-following data demonstrates more favorable alignment performance.

2 DATASET

The dataset reuses 52K instructions and generates English, Chinese, comparison, and unnatural-instruction answers with GPT-4, while comparing response statistics against GPT-3.5.

  • GPT-4 generates one English answer for each of 52K instructions reused from the Alpaca dataset.
  • ChatGPT translates the instructions into Chinese, after which GPT-4 generates Chinese answers for cross-language study.
  • GPT-4 rates its own responses and compares responses from GPT-4, GPT-3.5, and OPT-IML to create reward-model comparison data.
  • GPT-4 answers are generated for the 68K-triplet Unnatural Instructions core dataset to quantify the gap with instruction-tuned models.
  • The analysis extracts root verb–direct-object noun pairs, compares frequent pairs, and compares sequence-length distributions across GPT-4 and GPT-3.5 outputs.
  • GPT-4 tends to generate longer sequences, while Alpaca's GPT-3.5 outputs have a longer tail attributed to iterative deduplication.

3 INSTRUCTION-TUNING LANGUAGE MODELS

The paper fine-tunes LLaMA with GPT-4-generated data and constructs a reward model from GPT-4 response comparisons to evaluate instruction-following quality.

  • 3.1 SELF-INSTRUCT TUNING: LLaMA-GPT4 is trained on 52K English GPT-4 instruction-following examples, while LLaMA-GPT4-CN uses 52K Chinese examples.
  • 3.1 SELF-INSTRUCT TUNING: These models study GPT-4 data quality and cross-language generalization when instruction tuning is performed in one language.
  • 3.2 REWARD MODELS: Reward modeling predicts a scalar reward for a prompt and response, typically requiring comparisons between two responses to the same prompt.
  • 3.2 REWARD MODELS: GPT-4 assigns scores from 1 to 10 to K responses for each prompt, producing ordered response pairs for reward-model training.
  • 3.2 REWARD MODELS: The reward model is trained to score the higher-rated response above the lower-rated response using a sigmoid objective.
  • 3.2 REWARD MODELS: The comparison-data distribution is reported in Figure 2.

4 EXPERIMENTAL RESULTS

The experiments evaluate GPT-4-instruction-tuned LLaMA models on unseen instructions using human alignment judgments, GPT-4 comparisons, and ROUGE-L. GPT-4-generated data yields strong performance, including results comparable to GPT-4 on unseen instructional tasks, while remaining below large commercial chatbots in some comparisons.

  • Human evaluation: 54.12% of helpfulness votes favored GPT-4-generated-data tuning, compared with 19.74% for GPT-3-generated-data tuning.For honesty and harmlessness, ties received the largest share, with GPT-3 slightly superior.
  • Human evaluation: GPT-4-instruction-tuned LLaMA performed similarly to GPT-4 across helpfulness, honesty, and harmlessness on unseen instructional tasks.The authors conclude that learning from GPT-4-generated data can produce performance very comparable to the original GPT-4.
  • Automatic evaluation: The reward-model ranking agreed with ChatGPT and GPT-4 evaluations, supporting the value of GPT-4 feedback data and the reward model.Five responses per question were ranked into top-1 through top-5 groups and compared with one-response baseline decoding.
  • Automatic evaluation: 7B LLaMA-GPT4 outperformed 13B Alpaca and LLaMA, but remained behind large commercial chatbots such as GPT-4.The comparison used GPT-4 automatic evaluations against strong opponent models including ChatGPT and GPT-4.
  • Automatic evaluation: On unnatural instructions, LLaMA-GPT4 followed GPT-4’s behavior closely and exceeded GPT-4 in ROUGE-L when ground-truth responses were longer than four tokens.Alpaca had the highest average ROUGE-L overall, while LLaMA-GPT4 and GPT-4 improved as ground-truth response length increased.

5 RELATED WORK

Instruction tuning builds on open-source language models, prompts, datasets, and benchmarks to improve general-purpose instruction following and alignment with human values.

  • Instruction-tuning research improves instruction-following data, foundation language models, and evaluation benchmarks as interconnected development factors.
  • Existing instruction-tuning pipelines combine resources such as FLAN, P3, Super-NaturalInstructions, UnifiedSKG, and OPT-IML Bench.
  • Open-source efforts aim to develop general-purpose text-based assistants aligned with human values.
  • Representative open-source foundation models include BLOOM, GPT-J, GPT-NEO, OPT, and LLaMA.

6 CONCLUSIONS

The paper demonstrates GPT-4-based instruction tuning and releases generated data and LLaMA checkpoints, while identifying scale and reinforcement learning as directions for continued work.

  • The paper demonstrates the effectiveness of instruction tuning using GPT-4 and releases 52K English and Chinese instruction-following instances.
  • Released resources include GPT-4-generated instruction-following data and model checkpoints finetuned from LLaMA.
  • The 52K GPT-4 dataset and 7B LLaMA base model define current scale limits for the work.
  • Future work includes collecting more GPT-4 data, combining it with ShareGPT data, and training larger LLaMA models.
  • Because the reward model is used only during decoding, reinforcement learning with machine-generated feedback is identified as a promising direction.

A.1 HUMAN EVALUATION

Human evaluation uses Amazon Mechanical Turk to compare model responses under the HHH criteria of helpfulness, honesty, and harmlessness, with results presented through a five-option form.

  • Amazon Mechanical Turk evaluates model-generated responses using the HHH alignment criteria.
  • The evaluation asks participants to choose between two responses based on helpfulness, honesty, and harmlessness.
  • The evaluation form provides five options, merging the first two and last two options for easier analysis.
Loading 2304.03277v1…