Source-linked AI summary
Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning
Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Heng Huang, Jiuxiang Gu, Tianyi Zhou
TL;DR
Low-quality instruction-tuning data can undermine model outputs. Reflection-tuning uses an oracle LLM to improve both instructions and responses in existing data, and the authors report that models trained on recycled data outperform those trained on traditional datasets, with GPT-4 the Figure 1 exception.
Problem
Low-quality or skewed instruction-tuning data can produce unreliable model outputs and impair model efficacy.
Source: §1 · p. 1Method
Reflection-tuning uses an oracle LLM to assess and improve instruction-response data, reflecting on both instructions and responses.
Source: §1 · pp. 1–2, §3.2 · p. 3Results
Models trained with recycled data outperform models trained with traditional datasets across reported evaluations, with GPT-4 the Figure 1 exception.
Source: §5.1 · p. 5, §7 · p. 7Takeaways & Limitations
The method is described as needing no supplementary model training and adapting to diverse instruction-response pair architectures.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recent advancements in Large Language Models (LLMs) have expanded the horizons of natural language understanding and generation. Notably, the output control and alignment with the input of LLMs can be refined through instruction tuning. However, as highlighted in several studies, low-quality data in the training set are usually detrimental to instruction tuning, resulting in inconsistent or even misleading LLM outputs. We propose a novel method, termed "reflection-tuning," which addresses the problem by self-improvement and judging capabilities of LLMs. This approach utilizes an oracle LLM to recycle the original training data by introspecting and enhancing the quality of instructions and responses in the data. Extensive experiments on widely used evaluation benchmarks show that LLMs trained with our recycled data outperform those trained with existing datasets in various benchmarks.
1 Introduction
Instruction tuning is vulnerable to low-quality training data, so the paper introduces reflection-tuning to improve existing datasets and reports stronger benchmark performance from models trained on recycled data.
1 Introduction
- Low-quality or skewed training data can undermine instruction-tuned models, while prior work reports performance gains from using selected high-quality subsets.The cited discussion links data quality to fluency, pertinence, and precision, and notes that even a small fraction of skewed prompts can impair model efficacy.Source: §1 · p. 1
- Reflection-tuning uses an oracle LLM to assess and improve existing instruction-tuning data, and modifies both instructions and responses.The authors describe it as requiring no supplementary model training and adapting to diverse instruction-response pair architectures.Source: §1 · pp. 1–2
- Recycled WizardLM 7B achieved the highest win rate among open-source 7B models on Alpaca-Eval; recycled Alpaca and WizardLM reached 88.75% and 81.25% win rates, respectively, on the Vicuna test set.The Vicuna comparison used the same number of training data and model size.Source: §1 · p. 2
2 Related Work
The related work situates reflection-tuning among instruction tuning, dataset-quality efforts, and LLM self-alignment, emphasizing its posthoc application to existing instruction-response data.
Instruction Tuning of LLMs
High-quality data generation
- Reflection-tuning is presented as a posthoc tool that can further improve instruction-response data from different sources, complementing dataset curation and response distillation.The authors characterize it as orthogonal to prior data-generation work.Source: §2 · p. 2
3 Methodology
The method formalizes instruction tuning and then uses an oracle model to reflect on instructions and responses, producing recycled pairs for training.
3.1 Preliminaries
- Instruction tuning maps a raw instruction to a response and trains the model with a loss over the response given the instruction.The preliminaries define the pre-trained model, oracle, text segments, and initial dataset; the supplied loss passage is truncated.Source: §3.1 · pp. 2–3
3.2 Reflection-Tuning
- Reflection-tuning has two phases and uses criterion-specific critical responses from an oracle to guide generation of improved instruction-response pairs.The criteria are intended to address a potential lack of diversity in LLM judgments.Source: §3.2 · p. 3
3.2.1 Reflection on Instruction
- Instruction reflection assesses the original instruction-response pair against criteria including topic complexity, response detail, required knowledge, ambiguity, and reasoning demands, then generates a revised pair.The original response is included in the oracle prompt, and the critical responses guide generation.Source: §3.2.1 · p. 3
3.2.2 Reflection on Response
- Response reflection further revises the response for the modified instruction using criteria of helpfulness, relevance, accuracy, and detail; the resulting pair becomes recycled training data.The base model is trained on the recycled dataset to produce a Recycled Model.Source: §3.2.2 · p. 3
4 Experimental Setup
The experiments use Alpaca and WizardLM data, train models under stated implementation settings, evaluate pairwise preferences, and report results on automatic benchmarks.
4.1 Base Datasets
- Experiments use Alpaca’s 52,002 instruction-following samples and focus on the 70,000-sample WizardLM-7B subset from a 250,000-sample dataset.Alpaca is described as self-instruct data generated with text-davinci-003; WizardLM uses Evol-Instruct and ChatGPT refinement.Source: §4.1 · pp. 3–4
4.2 Implementation Details
- Training uses Llama2-7B with Adam at a 2 × 10^-5 learning rate, batch size 128, three epochs, maximum length 2048, and warmup rate 0.03.The prompt and code base come from Vicuna, while training arguments follow Alpaca and WizardLM protocols.Source: §4.2 · p. 4
4.3.1 Pari-wise comparison
- Pairwise evaluation uses GPT-4 or ChatGPT to score outputs from 1 to 10, with responses shown in two orders to mitigate positional bias.A win requires superiority in both orders or one order with parity in the other; ties and losses are defined across the two orderings.Source: §4.3.1 · p. 4
4.4 Benchmarks
- The Huggingface Open LLM Leaderboard covers ARC, HellaSwag, MMLU, and TruthfulQA, while AlpacaEval compares instruction-following outputs against Davinci003 responses.AlpacaEval uses automatic annotation and the section reports strong correlation between its rankings and human-annotator rankings.Source: §4.4 · pp. 4–5
5 Experimental Results
Across pairwise and leaderboard evaluations, reflection-tuned models generally outperform comparison models, with stated exceptions and qualifications. Their average scores across four representative benchmarks are better than related models and comparable to Llama-2-7B-chat.
5.1 Pair-wise Comparison
- On the Vicuna pairwise evaluation, recycled models outperform the compared models except GPT-4.SelFee used more training data but did not enhance instructions or use granular self-enhancement criteria; the recycled models also surpass several counterparts using extra RLHF.Source: Figure 1 · p. 5, §5.1 · p. 5
5.2 Alpaca Eval Leaderboard
- On AlpacaEval, the recycled models outperform all extant 7B open-source counterparts except Xwin-LM, and surpass some larger models.The passage notes that Xwin-LM's training data are unknown and that it uses extra RLHF.Source: §5.2 · p. 5, Table 1 · p. 6
5.3 Open LLM Leaderboard
- Across four representative Huggingface Open LLM benchmarks, recycled models achieve better average performance and results comparable to Llama-2-7B-chat.Llama-2-7B-chat is described as elaborately fine-tuned with extra RLHF.Source: Table 2 · p. 6, §5.3 · p. 6
6 Discussion
The reported analyses show dataset-dependent shifts in instruction length, alongside longer responses, improved coherence, and rising IFD scores after recycling. With 52k recycled Alpaca examples, the 13B model also reaches the stated AlpacaEval win rate and Open LLM average score.
6.1 Statistic Analysis
- After recycling, Alpaca instructions grow longer while WizardLM instructions become shorter; responses are longer, coherence improves, and IFD scores rise.The table compares instruction-response data before and after recycling and defines the reported length, perplexity, coherence, and IFD metrics.Source: §6.1 · p. 6, Table 3 · p. 7
6.2 Performances on 13B Models
- With 52k recycled Alpaca examples, Recycled Alpaca 13B reaches an 83.42% AlpacaEval win rate and a 58.93% average score on the Huggingface Open LLM leaderboard.The authors characterize the data amount as small compared with other models.Source: §6.2 · p. 7
7 Conclusion
The conclusion presents reflection-tuning as an LLM-based way to improve instruction-tuning datasets by recycling both instructions and responses. It reports that models trained on the resulting data consistently outperform models trained on traditional datasets.
7 Conclusion
- Reflection-tuning uses LLM self-improvement to improve and recycle instruction-tuning data, reflecting on both instructions and responses.The conclusion calls this a reflect-and-recycle mechanism applied comprehensively to both dimensions.Source: §7 · p. 7
- Models trained with reflection-tuning consistently outperform models trained with traditional datasets, according to the reported experiments.Source: §7 · p. 7
- The conclusion frames high-quality data recycling as a path toward more reliable, consistent, and high-performing LLMs.Source: §7 · p. 7
Related papers
This paper cites
- Training language models to follow instructions with human feedback2022
- LLaMA: Open and Efficient Foundation Language Models2023
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models2023
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge2018
- Constitutional AI: Harmlessness from AI Feedback2022
- TruthfulQA: Measuring How Models Mimic Human Falsehoods2021