PaperlayerView exact revision on arXiv

Source-linked AI summary

From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, Jing Xiao

arXiv:2308.12032v5cs.CL

TL;DR

Instruction tuning faces a quality-versus-quantity challenge in choosing effective data. The paper proposes self-guided cherry-data selection using the model-specific IFD metric, and reports that approximately 5% selected data beats the official Alpaca model while approximately 10% beats the reimplemented WizardLM model. The main limitation is the inconvenience of training the pre-experienced model.

  • Problem

    Instruction tuning has emphasized large datasets, while prior findings motivate examining whether data quality can matter more than sheer volume.

  • Method

    A self-guided pipeline briefly trains a model, uses IFD to evaluate instruction difficulty from conditioned and unconditioned response losses, and retrains on selected cherry data.

  • Results

    Approximately 5% selected data outperformed the official Alpaca model, and approximately 10% selected data outperformed the reimplemented WizardLM model.

  • Takeaways & Limitations

    The findings support model-specific autonomous data selection as a way to emphasize instruction-data quality and reduce the data used for instruction tuning.

  • Takeaways & Limitations

    The pre-experienced phase is inconvenient to train, making direct real-world use difficult despite promising direct IFD calculation on base LLaMA2 models.

Abstract

from arXiv · show

In the realm of Large Language Models (LLMs), the balance between instruction data quality and quantity is a focal point. Recognizing this, we introduce a self-guided methodology for LLMs to autonomously discern and select cherry samples from open-source datasets, effectively minimizing manual curation and potential cost for instruction tuning an LLM. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability. Through the application of IFD, cherry samples can be pinpointed, leading to a marked uptick in model training efficiency. Empirical validations on datasets like Alpaca and WizardLM underpin our findings; with a mere $10\%$ of original data input, our strategy showcases improved results. This synthesis of self-guided cherry-picking and the IFD metric signifies a transformative leap in the instruction tuning of LLMs, promising both efficiency and resource-conscious advancements. Codes, data, and models are available: https://github.com/tianyi-lab/Cherry_LLM

1 Introduction

The paper argues that instruction-tuning data quality can matter more than quantity and introduces a self-guided method for selecting model-relevant “cherry data” from open-source datasets. Its IFD metric supports autonomous selection, with reported gains using substantially reduced data on Alpaca and WizardLM.

  • Motivation: The paper frames instruction tuning as a quality-versus-quantity problem, building on evidence that effective tuning need not rely on sheer data volume.It describes instruction tuning as using explicit guidelines to produce desired outputs and contrasts dataset accumulation with data quality.
  • Approach: The method autonomously identifies impactful “cherry data” from extensive open-source instruction datasets.The approach relies on a model learning from a small initial subset before evaluating and selecting training samples.
  • Approach: The self-guided process uses brief experience, experience-based evaluation, and retraining from selected samples.The supplied introduction describes familiarizing the model with a small subset, applying IFD, and then using selected data for final training.
  • Results: Approximately 5% selected data outperformed the official Alpaca model, while approximately 10% outperformed the reimplemented WizardLM model.These dataset-specific comparisons are reported as experimental validation of the selection strategy.
  • Approach: The Instruction-Following Difficulty (IFD) score measures how much an instruction helps generate its corresponding response, revealing model-specific sample difficulty.The score compares response loss with and without instructional context; higher IFD indicates less instructional help and greater difficulty.
  • Implications: The work presents model-specific difficulty measurement as a possible benefit for future instruction-data generation.The authors also report improved outcomes with only 10% of the original data input in validation on Alpaca and WizardLM.

2 Methodology

The methodology gives an LLM brief experience, evaluates each sample’s instruction-following difficulty with IFD, and retrains on selected cherry data. IFD compares conditioned and direct answer losses to separate instruction-following difficulty from answer-string difficulty.

  • Overview: The pipeline has three phases: brief experience, experience-based evaluation, and retraining on selected cherry samples.A pre-experienced model first gains basic instruction-following ability, then scores target samples before the final model is trained.
  • Learning from Brief Experience: Brief experience uses K-Means clustering to diversify a small pre-experienced subset before one training epoch.The procedure generates 100 instruction-embedding clusters and samples 10 instances from each cluster.
  • Evaluating Based on Experience: The conditioned answer score measures average cross-entropy loss when predicting the answer given the instruction.It evaluates the model’s ability to generate an appropriate response based on the instruction and correct answer.
  • Evaluating Based on Experience: The direct answer score measures the model’s ability to generate the answer without contextual guidance from its instruction.A higher score can reflect inherent difficulty or intricacy in the answer string itself.
  • Evaluating Based on Experience: IFD is a model-specific ratio comparing conditioned and direct answer scores to estimate how much the instruction helps response generation.Higher IFD indicates weaker alignment to the instruction and greater instruction difficulty, while partially reducing the influence of intrinsic answer-generation difficulty.
  • Evaluating Based on Experience: Samples with IFD above 1 are treated as potentially misaligned because the instruction provides no useful predictive context for the response.The threshold reflects cases where conditioned answer loss exceeds direct answer loss.

3 Experimental Setup

Experiments use Alpaca and WizardLM datasets with model configurations aligned to their original training implementations. Evaluation combines established leaderboards, automated model judging, and a human comparison study.

  • Training Datasets: Alpaca contains 52,002 instruction-following samples, while WizardLM uses Evol-Instruct and ChatGPT-assisted reformulation to improve instruction-data quality.The datasets provide contrasting sources and construction approaches for instruction tuning experiments.
  • Training Configuration: Experiments use the Alpaca codebase for LLaMA-7B and the Vicuna codebase for LLaMA2-7B and LLaMA2-13B models.Training configurations align with the original Alpaca and WizardLM setups.
  • Automated Evaluation: Automated comparisons use GPT4 and ChatGPT judges, with responses rated from 1 to 10 and order-swapped to address positional bias.A model is counted as winning only when it does not lose in both response orderings.
  • Benchmarks: Evaluation includes the Huggingface Open LLM Leaderboard and AlpacaEval Leaderboard.The Open LLM Leaderboard covers ARC, HellaSwag, MMLU, and TruthfulQA through a unified evaluation framework.
  • Human Evaluation: Human evaluation compares model responses on 100 randomly sampled instructions using majority votes among three participants.Each comparison records Win, Tie, or Loss outcomes.

4 Experimental Results

Experiments show that self-guided selection can outperform full-data baselines using substantially smaller subsets, across datasets, benchmarks, judges, selection strategies, and model scales. The results also examine the roles of IFD scoring and pre-experience data.

  • 4.1 Main Results: Approximately 5% of Alpaca data beats the full-data official Alpaca model, while approximately 10% of WizardLM data beats the reimplemented full-data WizardLM model.The primary pairwise comparisons use GPT4 as judge across specific test sets.
  • 4.1 Main Results: With merely 10% of selectively chosen data, models exceed full-dataset models consistently across both Alpaca and WizardLM.Winning scores are calculated across five test sets as (Num(Win)−Num(Lose))/Num(All) +1.
  • 4.1 Main Results: Cherry Alpaca using 5% data outperforms official Alpaca on both leaderboards, while Cherry WizardLM using 10% has close performance to reimplemented WizardLM.The evaluated benchmarks are the Huggingface Open LLM Leaderboard and AlpacaEval Leaderboard.
  • 4.1 Main Results: Human evaluation records 49 wins, 25 ties, and 26 losses for Cherry Alpaca versus Alpaca, and 37 wins, 32 ties, and 31 losses for Cherry WizardLM versus WizardLM.Each comparison contains 100 evaluations.
  • 4.2 Ablation Studies: Random and diversity selection underperform or remain comparable to baselines, whereas higher IFD selection outperforms low IFD, high conditioned-answer, random, and diversity alternatives.Diversity-only filtering is described as insufficient, and low IFD produces the least performance among the methods compared.
  • 4.3 Pre-Experience Studies: Pre-experience matters more than its sampling strategy: random, diversity, and difficulty-based pre-experience all surpass Alpaca and are comparable, while 100 samples are insufficient for basic instruction-following ability.Without a pre-experienced model, cherry models perform worst, although 10% data still outperforms Alpaca.
  • 4.4 Results on LLaMA2 Models: On LLaMA2-7B and LLaMA2-13B, cherry models trained with much less data outperform models trained with the original full data.IFD scores are calculated from the corresponding LLaMA2 pretrained models using Vicuna prompts.

5 Cherry Data Characteristics

Cherry data is not uniformly distributed in instruction-embedding space: high-IFD samples cluster around more creative and complex tasks, whereas low-IFD samples cluster around simpler rule-following tasks.

  • 5 Cherry Data Characteristics: Top-5% and least-5% IFD samples form distinct distributions in t-SNE visualizations of Alpaca instruction embeddings.Red denotes top-5% IFD samples and blue denotes least-5% samples.
  • 5 Cherry Data Characteristics: Clusters dominated by low IFD contain rudimentary editing tasks, while high-IFD clusters contain storytelling and phenomenon-explanation tasks.The paper characterizes high-IFD tasks as deeper and more intricate.
  • 5 Cherry Data Characteristics: Verb-noun analysis compares the ten most frequent instruction patterns in the top and least 5% IFD groups.The analysis uses the Berkeley Neural Parser on Alpaca instructions.
  • 5 Cherry Data Characteristics: High-IFD instructions mainly involve creative or complex actions such as writing stories, generating lists, and explaining concepts, unlike lower-creativity rule-following instructions.The contrast reflects variation in the thinking and creativity demanded by tasks.

6 Related Work

Related work spans handcrafted and teacher-generated instruction collections, coreset selection, and quality-based filtering. This paper addresses the under-explored problem of finding high-quality instruction data without relying primarily on human curation or external fully trained models.

  • Instruction Data: Earlier instruction-tuning collections were typically handcrafted or task-related, while later work generated instructions using teacher models and open-source datasets.The related work cites GPT-3-based generation and instruction datasets released after Meta LLaMA.
  • Coreset Selection: Coreset selection identifies representative subsets to accelerate learning and has been studied in SVMs, K-means, logistic regression, and neural-network training.Prior neural-network work examines data-point utility dynamics during training.
  • Quality-Based Filtering: Although instruction-tuning consensus emphasizes quality, non-human methods for finding high-quality data remain under-explored.The paper frames this as a central motivation for its approach.
  • Quality-Based Filtering: Instruction Mining trains numerous models for statistical selection, whereas ALPAGASUS uses ChatGPT scoring and may neglect the base model’s intrinsic abilities.The paper positions its approach as addressing reliance on external models.
  • Related Metrics: IFD is related to PMI because both assess relationships between questions and answers, despite using different methodologies.PMI has been used to assess response-question alignment in generative language models.

7 Conclusion

The paper concludes that LLMs can use their innate capabilities to select instruction-tuning data suited to the model. Its self-guided approach centers on IFD and supports quality-focused, potentially cost-effective training.

  • 7 Conclusion: The self-guided approach enables LLMs to discern and cherry-pick pertinent instruction-tuning samples.The conclusion presents this as harnessing the models’ innate capabilities.
  • 7 Conclusion: The Instruction-Following Difficulty score gauges differences between autonomous model outputs and expected responses.The conclusion identifies IFD as the central metric of the methodology.
  • 7 Conclusion: The findings emphasize data quality over quantity and the potential for cost-effective LLM training.These conclusions are stated within the paper’s instruction-tuning experiments.

Limitation

The method’s main limitation is the inconvenience of training a pre-experienced model, creating a trade-off between research performance and real-world efficiency.

  • Training the pre-experienced model is the method’s main limitation.
  • Using pre-experienced models is considered more reasonable and performs better for research purposes.
  • Directly calculating IFD scores on base LLaMA2 models is more efficient while still promising effective data selection.

B Ablation on Data Selection Mechanism

The cherry models generally match or exceed their comparison models across instruction-task subcategories, with weaker results concentrated in several data-intensive or difficult categories.

  • Evaluation measures: Human evaluation and Open LLM Leaderboard ablations report win-tie-lose counts from the perspective of the cherry model.
  • Evaluation setup: The evaluation compares cherry models trained on 5% Alpaca data and 10% WizardLM data with corresponding Alpaca and WizardLM models across test-set subcategories.
  • Alpaca results: The Alpaca cherry model performs better or comparably on most Vicuna and WizardLM subcategories, but underperforms in Math and Coding.
  • WizardLM results: The WizardLM cherry model performs better or comparably on most subcategories, but underperforms in Math, Code, Complex Format, and Counterfactual.
  • Evaluation measures: Tables 6 and 7 present GPT4-judged comparisons on Vicuna and WizardLM test-set subcategories.

D Results with Official WizardLM

Using 40% of WizardLM data, the cherry model achieves performance comparable to official WizardLM despite shorter maximum input length and an uncensored-versus-censored comparison setting.

  • Using 40% of WizardLM data, the model reaches performance comparable to official WizardLM despite an inherent maximum-length disadvantage.The cherry model uses a 1024-token maximum, whereas official WizardLM uses 2048 tokens.
  • Figure 7 compares models trained on cherry data with official WizardLM trained on full data using GPT4 as judge.
  • Table 8 compares performance on the Huggingface Open LLM Leaderboard and AlpacaEval Leaderboard.

E Cherry Data General Characteristics

Higher IFD samples exhibit greater instruction difficulty across scope, complexity, depth, and knowledge required, while lower-IFD samples score higher on clarity and simplicity.

  • Evaluation setup: The analysis samples 100 instances from the top and least 5% IFD groups and scores them with ChatGPT on six instruction qualities.The evaluated aspects are Scope, Complexity, Clarity, Depth, Simplicity, and Knowledge Required.
  • Aspect definitions: Scope concerns task breadth, Complexity concerns multiple steps or concepts, and Clarity concerns straightforward, unambiguous wording.
  • Aspect definitions: Depth concerns thorough task details, Simplicity concerns accessible wording, and Knowledge Required concerns necessary foundational context.
  • Findings: Higher-IFD samples score higher on Scope, Complexity, Depth, and Knowledge Required, the aspects reflecting instruction difficulty.
  • Findings: Lower-IFD samples score higher on Clarity and Simplicity than higher-IFD samples.

F Cherry Example Analysis

The examples illustrate how IFD distinguishes valuable difficult samples from easy, short, or insufficiently instruction-sensitive responses. Across comparisons, cherry models generally outperform baselines and alternative selection strategies with small data fractions, although the optimal fraction remains unresolved.

  • Positive Examples: High DA and CA scores identify valuable samples that remain difficult despite instructional context.The first positive example treats high intrinsic generation difficulty and limited instructional help as valuable for learning.
  • Negative Examples: 0.914 IFD accompanies the coronavirus example, where a relatively short response produces a large score despite familiar content.The passage explains that short responses can yield relatively high DA scores and consequently large IFD scores.
  • Positive Examples: 0.091 IFD marks the frustration example, where the instruction substantially lowers the response loss.Its DA score is 6.593 and CA score is 0.601.
  • Negative Examples: 0.053 IFD marks the quote example, whose low conditioned and direct answer scores indicate that the response is already easy to generate.The example is a quoted passage from Pride and Prejudice with DA 0.497 and CA 0.026.
  • Comparative Results: At 5% of the data, cherry models outperform official Alpaca, while they surpass reimplemented WizardLM from 10% and match official WizardLM at 40%.The comparisons are reported across test sets using ChatGPT as the judge; the authors also state that the optimal data fraction remains under-explored.
  • Comparative Results: From 5% to 15% of the data, cherry models consistently outperform randomly selected, low-IFD, and high-conditioned-answer-score models.These comparisons use ChatGPT as the judge across different test sets.
Loading 2308.12032v5…