PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruction Tuning for Large Language Models: A Survey

Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, Guoyin Wang

arXiv:2308.10792v10cs.CLcs.AIcs.LG

TL;DR

LLM next-word training does not directly match users’ need for helpful, safe instruction following. This survey systematically reviews instruction tuning and its datasets, models, applications, analyses, and criticisms, concluding that SFT supports broad adaptation but remains relevant alongside preference-optimization methods and faces important limitations.

  • Problem

    LLMs are trained for contextual word prediction, while users want helpful and safe instruction following.

  • Method

    The paper systematically surveys SFT methodology, dataset construction, model training, applications across modalities and domains, efficiency, evaluation, analyses, and criticisms.

  • Results

    Instruction tuning supports controllable adaptation across language, multimodal, domain, and low-resource settings, with 6% of downstream training data achieving SOTA performance in multi-task learning.

  • Takeaways & Limitations

    SFT remains practically relevant because it can rapidly adapt models and commonly provides the initial instruction-following model for RLHF and DPO.

  • Takeaways & Limitations

    RLHF and DPO still commonly depend on SFT-trained initial models, while each has documented limitations involving complexity, data quality, distribution shift, or generalization.

Abstract

from arXiv · show

This paper surveys research works in the quickly advancing field of instruction tuning (IT), which can also be referred to as supervised fine-tuning (SFT)\footnote{In this paper, unless specified otherwise, supervised fine-tuning (SFT) and instruction tuning (IT) are used interchangeably.}, a crucial technique to enhance the capabilities and controllability of large language models (LLMs). Instruction tuning refers to the process of further training LLMs on a dataset consisting of \textsc{(instruction, output)} pairs in a supervised fashion, which bridges the gap between the next-word prediction objective of LLMs and the users' objective of having LLMs adhere to human instructions. In this work, we make a systematic review of the literature, including the general methodology of SFT, the construction of SFT datasets, the training of SFT models, and applications to different modalities, domains and application, along with analysis on aspects that influence the outcome of SFT (e.g., generation of instruction outputs, size of the instruction dataset, etc). We also review the potential pitfalls of SFT along with criticism against it, along with efforts pointing out current deficiencies of existing strategies and suggest some avenues for fruitful research. Project Page: github.com/xiaoya-li/Instruction-Tuning-Survey

1 Introduction

Instruction tuning addresses the mismatch between LLM training objectives and users’ need for helpful, safe instruction following. The survey organizes methodology, datasets, models, applications, efficiency, evaluation, and comparisons with reinforcement-learning methods, while reviewing benefits and open challenges.

  • LLMs are typically trained for contextual word prediction, whereas users want them to follow instructions helpfully and safely.
  • Instruction tuning, also called supervised fine-tuning, further trains LLMs on instruction-output pairs to address this objective mismatch.
  • SFT can improve controllability and predictability by constraining outputs toward desired response characteristics or domain knowledge.
  • SFT is computationally efficient and can adapt LLMs to specific domains without extensive retraining or architectural changes.
  • Key challenges include limited instruction-data quantity, diversity, and creativity, possible concentration of gains on supported tasks, and criticism of surface-level learning.Improving instruction adherence and handling unanticipated responses remain open problems.
  • The survey covers instruction-tuning methodology, datasets, models, multimodal and domain applications, efficiency, evaluation, criticism, and reinforcement-learning methods.Its reinforcement-learning discussion includes RLHF, DPO, and GRPO.

2 Methodology

Instruction tuning constructs instruction datasets and fine-tunes pretrained models to predict desired outputs from instructions and inputs. The methodology combines data integration, LLM-generated outputs, conversational self-play, and sequential output-token prediction.

  • Each instruction-dataset instance contains an instruction, optional contextual input, and an anticipated output.
  • Instruction datasets can be built by transforming annotated text-label pairs into instruction-output pairs with templates.Flan and P3 are examples of datasets constructed through data integration.
  • LLMs such as GPT-3.5-Turbo or GPT-4 can generate desired outputs for instructions instead of relying on manual output collection.
  • For multi-turn conversational datasets, LLMs can self-play user and assistant roles to generate conversational messages.
  • With a collected SFT dataset, a pretrained model is fine-tuned by predicting each output token sequentially given the instruction and input.
  • The general instruction-tuning pipeline comprises instruction-dataset construction followed by instruction tuning.

3 Datasets

Instruction-tuning datasets are organized into human-crafted, distillation-based, and self-improvement-based resources, with additional datasets targeting multilingual tasks and complex reasoning. They vary substantially in scale, language coverage, supervision format, and synthetic-data construction process.

  • Dataset taxonomy: Instruction-tuning datasets are categorized as human-crafted data, synthetic data via distillation, and synthetic data via self-improvement, with reasoning datasets often combining these strategies.The survey separately reviews reasoning datasets designed for multi-step reasoning and describes them as using one or more of the three construction strategies.
  • Human-crafted Data: Human-crafted datasets rely on manual annotation or internet sources and are generally smaller, including Natural Instructions, P3, xP3, Flan 2021, and LIMA.Natural Instructions contains 193K instances from 61 tasks; P3 integrates 170 datasets and 2,052 prompts; xP3 covers 16 tasks in 46 languages; Flan 2021 transforms 62 benchmarks; LIMA has 1K training pairs.
  • Human-crafted Data: Natural Instructions represents tasks through instructions and input-output instances, while P3 uses inputs, answer choices, and targets produced from collaboratively authored prompts.Natural Instructions unifies instances across 61 NLP datasets, whereas P3 maps conventional NLP examples into natural-language prompt triplets.
  • Human-crafted Data: Super-Natural Instructions contains 1,616 tasks and 5M instances spanning 76 task types and 55 languages, with definitions, positive examples, and negative examples.Its source data combines public NLP datasets, crowdsourced intermediate annotations, and synthetic tasks transformed from symbolic tasks.
  • Synthetic Data via Self-improvement: Self-improvement generates synthetic data from the model itself, but requires a robust foundational LLM because weak backbones may constrain learning and amplify biases or errors.Reported examples include 502K synthetic instances for instruction back-translation and SPIN, which improves models without additional human data or stronger-model feedback.
  • Reasoning Datasets: Reasoning datasets add structured supervision such as step-level correctness labels or long chains of thought, exemplified by PRM800K and O1-Journey.PRM800K contains 800K annotated steps from 75K solutions to 12K MATH problems, while O1-Journey contains 677 instances, including 327 training instances.

4 Instruction Tuned LLMs

Instruction-tuned models are initialized from pretrained LLMs and fine-tuned on instruction datasets, with evaluations generally showing improved task performance and alignment-related behavior.

  • 4 Instruction Tuned LLMs: The section reviews representative instruction-tuned LLMs, including InstructGPT, BLOOMZ, FLAN-T5, and Alpaca.
  • 4.1 InstructonGPT: InstructGPT applies supervised fine-tuning, reward-model training, and policy optimization to GPT-3 using human instructions and preference data.
  • 4.1 InstructonGPT: InstructGPT improves over GPT-3 by 10% on TruthfulQA truthfulness, 7% on RealToxicityPrompts toxicity, and multiple human-evaluation dimensions.
  • 4.2 BLOOMZ: BLOOMZ is fine-tuned on xP3, which combines English and multilingual instruction datasets spanning 46 languages.
  • 4.2 BLOOMZ: BLOOMZ exceeds BLOOM by +10.4% on coreference resolution, 20.5% on sentence completion, and 9.8% on natural language inference in zero-shot evaluation.
  • 4.3 Flan-T5: FLAN-T5 is fine-tuned on FLAN, a unified collection of 62 datasets covering 12 NLP tasks.
  • 4.3 Flan-T5: FLAN-T5 fine-tuning costs 0.2% of T5 pretraining resources, approximately 128 TPU v4 chips for 37 hours.
  • 4.3 Flan-T5 / 4.4 Alpaca: FLAN-T5 outperforms T5 and achieves comparable few-shot results to larger models including PaLM, while Alpaca matches InstructGPT in human evaluation.Alpaca records 90 wins versus 89 for InstructGPT on the self-instruct dataset.

4.5 Vicuna

Vicuna is produced by fine-tuning LLaMA on filtered user-shared ChatGPT conversations, with expanded context handling and GPT-4-based evaluation.

  • 4.5 Vicuna: Vicuna fine-tunes LLaMA 13B on 70K filtered conversation records collected from ShareGPT.com.
  • 4.5 Vicuna: The overview table lists instruction-tuned models by parameter count, base model, self-built dataset status, dataset name, and training-set size.
  • 4.5 Vicuna: The table includes models ranging from InstructGPT and BLOOMZ to Alpaca, Vicuna, WizardLM, and other instruction-tuned systems.
  • 4.5 Vicuna: Additional listed systems include OPT-IML, Dolly 2.0, Falcon-Instruct, Guanaco, Minotaur, Nous-Hermes, TÜLU, and YuLan-Chat.
  • 4.5 Vicuna: The table also lists MOSS, Airoboros, UltraLM, and an overview of LLMs tuned on instruction-tuning datasets.
  • 4.5 Vicuna: Vicuna training uses a modified loss for multi-turn conversations, extends context length from 512 to 2048, and applies memory-saving techniques.
  • 4.5 Vicuna: Vicuna is evaluated on eight chatbot question categories, with GPT-4 rating helpfulness, relevance, accuracy, and detail.
  • 4.5 Vicuna: Vicuna outperforms Alpaca and LLaMA on 90% of test questions and matches or exceeds ChatGPT ratings on 45%.

4.6 GPT-4-LLM

GPT-4-LLM combines supervised fine-tuning on GPT-4-generated responses with PPO guided by a GPT-4-rated reward model, outperforming smaller and larger comparison models.

  • 4.6 GPT-4-LLM: GPT-4-LLM fine-tunes LLaMA 7B on Alpaca instructions paired with GPT-4 responses, then optimizes the model using PPO.
  • 4.6 GPT-4-LLM: GPT-4-LLM 7B outperforms Alpaca 7B, Alpaca 13B, and LLaMA 13B in the reported evaluations.
  • 4.6 GPT-4-LLM: Human evaluation improvements over Alpaca are 11.7 for helpfulness, 20.9 for honesty, and 28.6 for harmlessness.

4.7 Claude

Claude, LIMA, WizardLM, and ChatGLM2 illustrate instruction-tuning strategies using preference optimization, small datasets, evolved instructions, and bilingual data, with broad reported gains.

  • 4.7 Claude: Claude uses GPT-4-generated instruction responses for supervised fine-tuning, followed by PPO based on a reward model trained from rated model responses.
  • 4.7 Claude: Claude improves over GPT-3 by 7% on RealToxicityPrompts toxicity and by +10% to +20% on several human-evaluation dimensions.
  • 4.8 WizardLM: WizardLM evaluates Evol-Instruct and Vicuna test sets, outperforming Alpaca by +6.2% and +5.3% and Vicuna by +5.8 and +1.7%.
  • 4.9 ChatGLM2: ChatGLM2 fine-tunes GLM on a bilingual English-Chinese instruction dataset and expands maximum context length from 1024 to 32K.
  • 4.9 ChatGLM2: ChatGLM2 outperforms GLM on MMLU, C-Eval, GSM8K, and BBH, with gains of +3.1, +5.0, +8.6, and +2.2, respectively.
  • 4.10 LIMA: LIMA is based on the superficial alignment hypothesis, which separates pretrained knowledge acquisition from alignment to user-preferred response formalizations.
  • 4.10 LIMA: LIMA outperforms InstructGPT and Alpaca by 20% and 36% in automatic evaluation, while underperforming Claude and GPT-4.

4.11 Others

The survey describes instruction-tuned models spanning bilingual dialogue, general instruction following, and mixed-task training, with reported gains over their backbones and several open or closed-model comparisons.

  • OPT-IML fine-tunes OPT-175B on over 1,500 NLP tasks from eight public benchmarks and outperforms OPT across all benchmarks.
  • TÜLU reaches 83% of ChatGPT’s performance and 68% of GPT-4’s performance on average after mixed-dataset instruction tuning.
  • YuLan-Chat uses 250,000 Chinese-English instruction pairs, matches ChatGLM-6B comparably, and outperforms Vicuna-13B on English BBH3K.
  • Airoboros significantly outperforms LLaMA-13B on all benchmarks and approaches models fine-tuned for specific benchmarks.
  • UltraLM outperforms Dolly and achieves winning rates up to 98%, exceeding Vicuna and WizardLM by 9% and 28%, respectively.

5 Multi-modality Instruction Tuning

Multimodal instruction tuning extends supervised instruction-response learning across images, text, video, audio, and 3D point clouds. The surveyed datasets and models combine diverse task coverage with generated or curated multimodal supervision.

  • 5.1 Multi-modality Datasets: MUL-TIINSTRUCT unifies 62 multimodal tasks across 10 categories from 21 datasets, with five expert-written instructions per task.
  • 5.1 Multi-modality Datasets: 227k image-question pairs over 149k medical images support PMC-VQA, and MedVInT reaches 81.6% on VQA-RAD and 88.0% on SLAKE.
  • 5.1 Multi-modality Datasets: LAMM provides 186K language-image and 10K language-point-cloud instruction-response pairs generated from public datasets using GPT-API and self-instruction.
  • 5.1 Multi-modality Datasets: Vision-Flan contains 1,664,261 human-annotated instances spanning 200+ vision-language tasks derived from 101 computer-vision datasets.
  • 5.1 Multi-modality Datasets: ALLaVA and ShareGPT4V scale visual supervision through GPT-4V-produced captions, answers, and descriptive image-text pairs.
  • 5.2 Multi-modality Instruction Tuning Models: InstructPix2Pix trains a conditional diffusion model on more than 450K text-editing instructions paired with images before and after editing.
  • 5.2 Multi-modality Instruction Tuning Models: LLaVA connects CLIP’s visual encoder with LLaMA’s decoder, uses 158K generated instruction-following samples, and achieves 92.53% with GPT-4 on Science QA.
  • 5.2 Multi-modality Instruction Tuning Models: Video-LLaMA integrates visual and auditory branches, while Otter and MultiModal-GPT fine-tune OpenFlamingo-based systems for broad multimodal instruction following.

6 Domain-specific Instruction Tuning

The survey reviews instruction tuning across dialogue, language understanding, writing, medicine, arithmetic, and code. Across these domains, task-specific instruction formats and datasets are associated with reported gains in transfer, assistance, or benchmark performance.

  • 6.1 Dialogue: InstructDial converts 59 dialogue datasets into 48 text-to-text tasks and adds instruction selection and adherence meta-tasks.
  • 6.2 Intent Classification and Slot Tagging: LINGUIST uses structured five-block instructions and improves novel-intent classification while surpassing a translation baseline across six crosslingual languages.
  • 6.3 Information Extraction: InstructUIE unifies 32 information-extraction datasets in text-to-text form and outperforms state-of-the-art systems and GPT-3.5 in zero-shot settings.
  • 6.4 Aspect-based Sentiment Analysis: Instruction templates decompose aspect-based sentiment analysis into five question-answering tasks covering aspects, categories, opinions, and sentiment.
  • 6.5 Writing: Writing-Alpaca-7B improves LLaMA’s performance on all writing tasks and outperforms larger off-the-shelf language models.
  • 6.5 Writing: CoEdIT trains FLAN-T5 on approximately 82K text-editing pairs and achieves state-of-the-art results on several editing tasks.
  • 6.6 Medical: Medical instruction tuning improves domain performance in Radiology-GPT and ChatGLM-Med, while ChatDoctor incorporates retrieval prompts for external medical knowledge.
  • 6.7 Arithmetic / 6.8 Code: Goat reaches state-of-the-art arithmetic performance, while WizardCoder surpasses open-source code models and larger closed models on HumanEval and HumanEval+.

7 Efficient Tuning Techniques

Efficient tuning techniques reduce the parameters or memory required to adapt large language models. The surveyed approaches add modules, restrict updates to low-dimensional subspaces, quantize weights, or fuse optimization operations.

  • Efficient fine-tuning methods include addition-based, specification-based, and reparameterization-based approaches that optimize only a small parameter fraction.
  • LoRA represents weight changes as low-rank updates and trains matrices A and B instead of directly optimizing the full pretrained weight matrix.
  • For GPT-3, LoRA reduces trainable parameters by 10,000x and memory usage by 3x versus full fine-tuning.
  • HINT uses hypernetworks to generate parameter-efficient adapter and prefix modules from natural-language instructions and few-shot examples.
  • QLORA freezes a 4-bit quantized base model and learns a small set of 16-bit low-rank adapter weights without degradation versus full 16-bit fine-tuning.
  • LOMO enables full-parameter fine-tuning under limited resources by fusing gradient computation and parameter updates to avoid storing full gradient tensors.
  • Delta-tuning restricts optimization to a low-dimensional manifold, interpreting tuned parameters as controllers of downstream model behavior.

8 Evaluation, Analysis and Criticism

The survey reviews evaluation frameworks for instruction-tuned models and summarizes evidence about data efficiency, dataset quality, and superficial alignment. It also identifies resource and coverage limitations affecting interpretation of SFT results.

  • Evaluation Benchmarks: Close-ended benchmarks assess core capabilities across language understanding, mathematics, reasoning, coding, and instruction following.The reviewed evaluations include MMLU, MATH, GSM8K, BBH, HumanEval, and IFEval.
  • Evaluation Benchmarks: HELM advocates evaluating language models across broad task, scenario, and metric coverage to improve transparency about capabilities, risks, and limitations.
  • Open-ended Evaluation: WildBench addresses limited task composition and skill coverage by using 1,024 test instructions drawn from real-world user interactions.
  • Open-ended Evaluation: Pearson correlations of 0.98 for WB-Reward and 0.95 for WB-Score indicate strong alignment with human evaluations.
  • Low-resource Instruction Tuning: 25% of downstream training data in single-task learning and 6% in multi-task learning enabled SFT models to achieve or exceed state-of-the-art performance on 119 SuperNI tasks.
  • Limitations: Further investigation with larger language models and datasets is needed because resource limitations prevented experiments on models such as T5-11B.
  • Dataset Quality and Size: LIMA matched several models using 1,000 carefully curated demonstrations, while dataset studies found no single best SFT dataset across all tasks.
  • Criticism of SFT: Imitation models improve strongly on tasks supported by imitation data but may fail to improve or decline on unsupported tasks, while gains can reflect stylistic mimicry.

9 The Role of Instruction Fine-tuning

Instruction fine-tuning remains central to alignment because it establishes instruction-following behavior, while RLHF, DPO, and prompting offer complementary trade-offs. Evidence also raises concerns that SFT gains can reflect superficial alignment rather than task understanding.

  • 9 The Role of Instruction Fine-tuning: SFT trains models on prompt-response examples to align outputs with user instructions and intended goals.
  • 9.1.1 Reinforcement Learning from Human Feedback (RLHF): RLHF combines SFT, reward-model training, and policy optimization to target complex user preferences, but requires substantial data, tuning, and computational effort.
  • 9.1.1 Reinforcement Learning from Human Feedback (RLHF): RLHF can be time-consuming and brittle, introduce performance drops on some tasks, and fail to generalize when its reward model is mis-specified.
  • 9.1.2 Direct Preference Optimization (DPO): DPO directly fine-tunes on preferred and dispreferred responses, avoiding a separate reward model and reinforcement-learning stage.
  • 9.1.2 Direct Preference Optimization (DPO): DPO is reported to be simpler, more stable, and computationally lightweight than PPO-based RLHF, while matching or exceeding RLHF on some tasks.
  • 9.1.2 Direct Preference Optimization (DPO): DPO remains dependent on preference-data quality and typically benefits from an SFT-trained initial model, especially under distribution shift.
  • 9.1.3 Prompt Engineering (In-Context Learning): Prompt engineering requires no weight updates or retraining, enabling rapid alignment when parameter fine-tuning is infeasible.
  • 9.1.3 Prompt Engineering (In-Context Learning): Prompt-based alignment is limited by context length, brittle rephrasing, jailbreaks, and reliance on existing capabilities rather than new knowledge or deep reasoning.

10 Conclusion

The paper surveys instruction tuning across methods, datasets, models, modalities, domains, and applications. It also analyzes benefits and pitfalls to identify deficiencies and motivate further research.

  • The survey systematically reviews recent instruction-tuning research, including methodology, dataset construction, and model training.
  • It covers applications across different modalities, domains, and tasks, alongside analyses of factors affecting SFT outcomes.
  • The paper reviews instruction tuning’s advantages and potential pitfalls to motivate work addressing deficiencies in current SFT models.

A Datasets

The survey’s dataset overview organizes instruction-tuning resources by scale, language, construction method, and openness. The collection spans human-crafted, distilled, self-improved, conversational, mathematical, and coding datasets.

  • The dataset collection records instance counts, languages, construction approaches, and open-source availability.
  • Human-crafted Data: Human-crafted datasets include UnifiedQA, Natural Instructions, Super-Natural Instructions, P3, xP3, and Flan 2021.
  • Human-crafted Data: The collection includes human-crafted conversational datasets such as InstructGPT, Dolly, LIMA, ChatGPT, OpenAssistant, and ShareGPT.
  • Synthetic Data via Distillation: Synthetic distillation resources include Unnatural Instructions, InstructWild, Evol-Instruct, Alpaca, GPT-4-LLM, Baize, UltraChat, Guanaco, and Orca.
  • Code Data: The overview also lists code datasets such as WizardCoder, Magicoder, WaveCoder, Phi-1, and Phi-1.5.
  • Synthetic Data via Self-improvement: Self-improvement and reasoning resources include Self-Instruct, Instruction Backtranslation, SPIN, PRM800K, and O1-Journey.
  • Table 7 presents an overview of instruction-tuning datasets.
Loading 2308.10792v10…