PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruction Mining: Instruction Data Selection for Tuning Large Language Models

Yihan Cao, Yanbin Kang, Chi Wang, Lichao Sun

arXiv:2307.06290v3cs.CLcs.AIcs.LG

TL;DR

Selecting high-quality instruction data remains difficult, and human filtering can be costly. INSTRUCTMINING estimates data quality with natural-language indicators and searches for a suitable subset. Using 2,532 examples, it achieves state-of-the-art performance on LLM-as-a-Judge and Huggingface OpenLLM benchmarks.

  • Problem

    No standardized guideline for selecting high-quality instruction datasets is established, and human expert filtering can be time-consuming and expensive.

    Source: §1 · p. 1

  • Method

    INSTRUCTMINING uses natural-language indicators to estimate instruction-data quality and applies BLENDSEARCH to search for a suitable subset.

    Source: §1 · p. 1, §6 · p. 9

  • Results

    2.5% of the dataset (2,532 of 100,000 examples) achieves state-of-the-art performance on LLM-AS-A-JUDGE and Huggingface OPENLLM benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that using less training data can reduce training time and cost while achieving state-of-the-art benchmark performance.

    Source: §1 · p. 2

    The quality assumption follows the superficial alignment hypothesis, and valid inference-loss evaluation requires a selected, unbiased, high-quality instruction-following evaluation set.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Large language models (LLMs) are initially pretrained for broad capabilities and then finetuned with instruction-following datasets to improve their performance in interacting with humans. Despite advances in finetuning, a standardized guideline for selecting high-quality datasets to optimize this process remains elusive. In this paper, we first propose InstructMining, an innovative method designed for automatically selecting premium instruction-following data for finetuning LLMs. Specifically, InstructMining utilizes natural language indicators as a measure of data quality, applying them to evaluate unseen datasets. During experimentation, we discover that double descent phenomenon exists in large language model finetuning. Based on this observation, we further leverage BlendSearch to help find the best subset among the entire dataset (i.e., 2,532 out of 100,000). Experiment results show that InstructMining-7B achieves state-of-the-art performance on two of the most popular benchmarks: LLM-as-a-judge and Huggingface OpenLLM leaderboard.

1 Introduction

The paper addresses costly human selection of high-quality instruction data by introducing automatic evaluation and subset selection. Its experiments identify double descent during finetuning, and the selected data achieves strong benchmark results using a small subset.

1 Introduction

  • Human experts must filter examples in existing high-quality data selection, a process described as time-consuming and expensive.LIMA showed that a small amount of carefully selected instruction data can improve performance through instruction tuning.Source: §1 · p. 1
  • INSTRUCTMINING uses natural-language indicators to predict finetuned-model inference loss, estimating dataset quality without finetuning for each evaluation.The method combines data assessment with automatic selection of a suitable subset.Source: §1 · pp. 1–2
  • As subset size grows, finetuning shows double descent, so the best subset size is not necessarily the largest or the top-K highest-quality examples.The authors use BLENDSEARCH to search for a suitable subset size.Source: §1 · p. 2, Figure 1 · p. 2
  • Using 2,532 of 100,000 examples, INSTRUCTMINING achieves state-of-the-art performance on LLM-AS-A-JUDGE and Huggingface OPENLLM benchmarks.This is 2.5% of the complete dataset.Source: §1 · p. 2
  • INSTRUCTMINING improves LLAMA-2-7B by 4.93 on the Huggingface OPENLLM benchmark.Its finetuned models generate equivalent or superior responses in 64.67% of instances compared with VICUNA-7B-v1.5.Source: §1 · p. 2

2 Methodology

INSTRUCTMINING estimates instruction-data quality from the inference loss of a finetuned model, then uses natural-language indicators to avoid repeated finetuning for evaluation. Because performance varies non-monotonically with data size, the method also searches for a suitable subset size.

2 Methodology

  • The methodology has two parts: estimating instruction-data quality and searching for a data threshold; it aligns quality with finetuned-model inference loss.The proposed selector uses a natural-language-indicator rule and BLENDSEARCH.Source: §2 · pp. 2–3

2.1 What is instruction quality?

  • Instruction quality is treated as a dataset’s ability to steer a model toward generating responses in a particular manner, following the superficial alignment hypothesis.The inference-loss measure requires a selected evaluation set of unbiased, high-quality instruction-following samples.Source: §2.1 · p. 3

2.2 How to estimate instruction quality?

  • Natural-language indicators are used to approximate finetuned-model inference loss, with a multivariate linear rule whose unknown parameters are estimated using Least Squares.The indicator values are computed for each dataset, and the fit minimizes squared residuals between observed and predicted values.Source: §2.2 · pp. 3–4

2.3 Instruction Data Selector

  • Finetuning performance can first improve, then worsen, and later improve as data size increases, making the best data size a search problem.BLENDSEARCH combines Bayesian optimization with local search; it samples dataset size logarithmically and minimizes evaluation-set loss.Source: §2.3 · p. 4

3 Experiment Settings

Experiments are designed to fit the INSTRUCTMINING rule and evaluate it across finetuning settings. The authors construct mixed-quality datasets, test on unseen datasets, and use consistent LLaMA-2-7B finetuning settings.

3 Experiment Settings

  • Experiments aim both to estimate the INSTRUCTMINING rule’s parameters and to evaluate its performance across varied finetuning scenarios.The section describes the design, datasets, and finetuning settings for these two goals.Source: §3 · p. 4

3.1 Empirical Experiment Design

  • The rule-fitting experiment combines commonly used datasets in random proportions to create finetuning datasets with varied indicator values.The resulting datasets are finetuned and evaluated, and their quality-indicator relationships are analyzed through regression.Source: §3.1 · pp. 4–5

3.2 Datasets

  • Rule fitting uses ALPACA, OPEN-ASSISTANT, STACKEXCHANGE, and WIKIHOW; evaluation tests selection on unseen OPENORCA and DOLLY-15K datasets.SELF-INSTRUCT supplies the rule-fitting test set; MT-BENCH is an unseen evaluation set using only its first turn.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 Finetuning Settings

  • All finetuning experiments use LLAMA-2-7B, three epochs, and a per-step batch size of 128.Estimation-phase datasets each contain 1,000 examples; experiments use eight A100 GPUs.Source: §3.3 · p. 5

4 Experimental Results

The experiments fit a quality rule from natural-language indicators, then use it to select instruction data and compare the resulting models with existing systems. The reported results include GPT-4 preference judgments, OpenLLM performance, training-time observations, and an ablation identifying Rew as an important indicator.

4.1 INSTRUCTMINING Parameter Fit

  • The fitted quality rule identifies reward score and unieval as the most significant indicators; its prose interpretation says Und is negatively correlated with data quality and the other three indicators positively correlated.The rule was estimated from regression results on sampled subsets.Source: §4.1 · p. 6

4.2 Quality-Guided Instruction Selection

  • BlendSearch searched data sizes from 512 to 10,000 to address performance declines and recoveries as training data increased.Figure 2 presents the search procedure from the perspective of steps, with loss calculated on MT-BENCH.Source: §4.2 · p. 6, Figure 2 · p. 6
  • Selecting the top 1,000 OPENORCA examples took around two hours, followed by training a better LLM within 15 minutes, compared with up to 30 hours on 8 GPUs for full-data finetuning.The authors report the same efficiency pattern for DOLLY.Source: §4.2 · p. 6
  • In Figure 3, the model produced better-or-equal results than VICUNA-1.5-7B in 64.67% of cases; GPT-4 assessment from different perspectives is described separately.Source: §4.2 · p. 7, Figure 3 · p. 7
  • INSTRUCTMINING-finetuned models achieved performance similar to STABLEBELUGA-7B on the OPENLLM benchmark.The authors report around two hours of indicator inference and ten hours of finetuning for a comparably strong model.Source: §4.2 · p. 7, Table 4 · p. 8

4.3 Ablation Study

  • Removing Rew from the rule increased SELF-INSTRUCT inference loss by 0.03 and MT-BENCH inference loss by 0.051, making Rew the most important tested indicator.Ablations selected 1,000 examples and finetuned LLAMA-2-7B for three epochs.Source: §4.3 · p. 8, Table 5 · p. 8

5 Analysis

The analysis reports non-monotonic performance as finetuning data grows and compares quality-selected data with random selection. It also reports that the INSTRUCTMINING rule applies across different base models, model sizes, and parameter-efficient finetuning.

5.1 Double Descent in Generative Language Models

  • Performance improves, worsens after the data size reaches around 10,000 as loss rises, then improves again, a double-descent-like pattern also observed in vanilla language-model training.Figure 4 evaluates loss on SELF-INSTRUCT and MT-BENCH, OPENLLM scores, and MMLU scores.Source: Figure 4 · p. 8, §5.1 · p. 9
  • As training data grows, quality-selected and random-data performance curves intersect and their distance shrinks, so data quality matters less beyond a certain size.The authors report that quality measures can improve performance initially, but become less important as data size grows.Source: §5.1 · p. 9

5.2 Robustness

  • The INSTRUCTMINING rule was applicable with LLAMA-1-7B, a 13B model, and LoRA parameter-efficient finetuning.Table 6 presents the robustness test results.Source: Table 6 · p. 9, §5.2 · p. 9

6 Conclusion

The conclusion presents INSTRUCTMINING as a high-quality example selection method, reports evidence for its rule’s validity and scalability, and connects double descent to a BlendSearch search for the best subset. The accompanying sections describe the search pipeline and prior instruction-data work.

6 Conclusion

  • INSTRUCTMINING combines quality-rule estimation with BLENDSEARCH to search for a high-quality subset after observing double descent in language-model finetuning.The rule is estimated from curated experiments, then applied to score and rank examples before the search.Source: §6 · p. 9, Figure 5 · p. 13

A Search Procedure

  • The search procedure fuses and samples candidate datasets, estimates a linear quality rule, ranks examples, and uses FLAML to run BLENDSEARCH.Rule estimation evaluates finetuned models on a shared evaluation set and calculates indicator values for each dataset.Source: Figure 5 · p. 13

B Related Work

  • Prior work explores instruction tuning, dataset size, and quality, while other instruction-data studies consider diversity, quality assessment, and learning-algorithm use.Some work reports that smaller volumes of high-quality data can still produce effective models.Source: B Related Work · p. 13

C Justification of Loss as a Suitable Indicator

This section evaluates inference loss as a data-quality indicator, considering its representativeness and computational cost. It notes a potential risk of overfitting to the golden evaluation set but reports that the finetuned model did not do so.

D Indicator Descriptive Analysis

This section describes the distributions of natural-language indicators across sampled subsets and reports that the indicators used in the regression follow a normal distribution.

E Empirical Test of Instruction Quality Evaluation Hypothesis

The empirical test finds that GPT-4-labeled datasets tend to yield lower evaluation loss, and larger finetuned models also yield lower loss. The experiments use 1,000-example finetuning sets to test loss as an indicator of model capability and data quality.

F Other Emergent Phenomena

The section reports correlations between data-quality indicators and evaluation loss, then presents regression analysis and case studies of selected examples and model responses.

F Other Emergent Phenomena

  • Higher perplexity corresponds to increased inference loss, reward score is positively correlated with data quality, and reward models are described as reflecting human preferences.The paper also links elevated perplexity to examples infrequently encountered during pretraining.Source: F Other Emergent Phenomena · pp. 15–16
  • As output length increases, loss tends to increase and data quality tends to decrease; LLAMA-2-7B has a 4096-token maximum sequence length, so long instances may be truncated during preprocessing.The paper suggests this limit may account for the observed association.Source: F Other Emergent Phenomena · pp. 15–16
  • Figure 7 plots four indicators against actual evaluation loss with univariate linear fits; understandability is negatively correlated with data quality, possibly because translation-related examples are uncommon in the evaluation set.Unieval-understandability is described as representing sentence complexity; increasing complexity is associated with decreasing data quality.Source: F Other Emergent Phenomena · p. 16, Figure 7 · p. 16

G More Details on Parameter Fitting

  • Regression analysis identifies Rew, Nat, and Und as the most prominent indicators; Und is negatively associated with data quality, while the other three indicators are positively associated.Table 10 reports OLS regression parameter estimates and describes p values and R2 measures.Source: G More Details on Parameter Fitting · p. 16, Table 10 · p. 16

H Case Study

  • The case studies cover the INSTRUCTMINING rule and the finetuned model.The section introduces examples of selected data and model responses.Source: H Case Study · p. 17

H.1 Rule Selected Examples

  • Three rule-selected examples show an assistant completing a task with good interaction, completing it without suitable human interaction, and failing to complete it.These examples are presented in Tables 11–13.Source: Table 11 · p. 17, Table 12 · p. 17, §H.1 · p. 17

H.2 Language Model Response Examples

I Random Sampling Experiment Results

The section collects results from 129 randomly selected subsets and presents them in three tables.

I Random Sampling Experiment Results

Loading 2307.06290v3…