PaperlayerView exact revision on arXiv

Source-linked AI summary

MoDS: Model-oriented Data Selection for Instruction Tuning

Qianlong Du, Chengqing Zong, Jiajun Zhang

arXiv:2311.15653v1cs.CL

TL;DR

How to select appropriate instruction data for a given LLM remains an open problem despite evidence that a small number of high-quality examples can suffice. MoDS filters for quality, selects a broad-coverage seed set, then uses the initial model’s weak responses to identify necessary additions. The LLM fine-tuned with 4,000 instruction data selected by MoDS achieved better performance than the LLM fine-tuned with the full original dataset (214k).

  • Problem

    How to select appropriate instruction data for a given LLM remains an open problem despite evidence that a small number of high-quality examples can suffice.

    Source: §3.1 · p. 3

  • Method

    MoDS filters for quality, selects a broad-coverage seed set, then uses the initial model’s weak responses to identify necessary additions.

    Source: §1 · p. 2, §3.2.1 · p. 3, repeat · p. 5

  • Results

    The LLM fine-tuned with 4,000 instruction data selected by MoDS achieved better performance than the LLM fine-tuned with the full original dataset (214k).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that MoDS can select a small, high-quality, broad-coverage, high-necessity subset that outperforms fine-tuning on the full original dataset.

    Source: §1 · p. 2

Abstract

from arXiv · show

Instruction tuning has become the de facto method to equip large language models (LLMs) with the ability of following user instructions. Usually, hundreds of thousands or millions of instruction-following pairs are employed to fine-tune the foundation LLMs. Recently, some studies show that a small number of high-quality instruction data is enough. However, how to select appropriate instruction data for a given LLM is still an open problem. To address this problem, in this paper we present a model-oriented data selection (MoDS) approach, which selects instruction data based on a new criteria considering three aspects: quality, coverage and necessity. First, our approach utilizes a quality evaluation model to filter out the high-quality subset from the original instruction dataset, and then designs an algorithm to further select from the high-quality subset a seed instruction dataset with good coverage. The seed dataset is applied to fine-tune the foundation LLM to obtain an initial instruction-following LLM. Finally, we develop a necessity evaluation model to find out the instruction data which are performed badly in the initial instruction-following LLM and consider them necessary instructions to further improve the LLMs. In this way, we can get a small high-quality, broad-coverage and high-necessity subset from the original instruction datasets. Experimental results show that, the model fine-tuned with 4,000 instruction pairs selected by our approach could perform better than the model fine-tuned with the full original dataset which includes 214k instruction data.

1 Introduction

Instruction tuning can require large datasets, but evidence that a small number of high-quality instructions can suffice has made data selection an open problem. MoDS selects for quality, coverage, and model-specific necessity, and the paper reports better performance with 4,000 selected pairs than with the full 214k-pair dataset.

1 Introduction

  • Earlier approaches focused on constructing large, diverse, high-quality datasets, while more recent work examined selecting smaller subsets.The paper notes that some filtering methods rely heavily on extra LLMs or focus mainly on instruction quality.Source: §1 · p. 1
  • MoDS filters for quality, selects a broad-coverage seed set, then identifies instructions the initial model handles poorly.The seed and augmented data are combined for fine-tuning.Source: §1 · p. 2
  • MoDS adds model-specific necessity to instruction selection alongside quality and coverage.The authors define necessity by whether instructions address the ability gap of the particular LLM.Source: §1 · p. 2
  • Experimental results show that an LLM fine-tuned with 4,000 instruction data selected by the authors’ approach achieved better performance than an LLM fine-tuned with the full original dataset (214k).The authors say this indicates their approach selects valuable instruction data with high-quality, broad-coverage, and high-necessity.Source: §1 · p. 2

2 Related Work

Prior work explored manual curation, quality scoring, and self-guided instruction selection. The paper describes limitations involving curation cost, reliance on external models, or incomplete attention to data quality and coverage.

2 Related Work

  • Manual selection of carefully crafted instructions was described as time-consuming and costly.Zhou et al. (2023) reported strong results with a small number of such instructions.Source: §2 · p. 2
  • Cao et al. (2023) used a linear quality rule and indicators, but the paper says they lacked full-dataset comparisons and used a complex approach.Source: §2 · p. 2
  • Chen et al. (2023) used ChatGPT to score instructions and selected 9k Alpaca examples, while relying heavily on an external LLM.Their model was reported to surpass the official Alpaca model trained on the complete dataset.Source: §2 · p. 2
  • Li et al. (2023a) introduced the IFD metric for self-guided selection, but used one answer per instruction and gave limited attention to quality and coverage.Source: §2 · pp. 2–3

3 Methodology

The methodology defines valuable instructions by quality, coverage, and necessity, then uses quality filtering, diversity-based seed selection, and model-specific augmentation before fine-tuning.

3.1 Which instructions are the valuable data for a given LLM

  • Quality concerns both instructions and responses; coverage concerns the diversity of instruction types; necessity reflects an instruction’s importance and uniqueness for a specific LLM.The same instruction may be necessary for one model but not another, depending on its response ability.Source: §3.1 · p. 3

3.2 Instruction Data Selection

  • The selection pipeline consists of quality evaluation, diverse seed selection, and augmented data selection.The three modules operationalize quality, coverage, and necessity.Source: §3.2 · p. 3

3.2.1 Quality Evaluation

  • A DeBERTa-based OpenAssistant reward model scores each instruction-input-output triplet, and examples above threshold α form the high-quality subset.The paper uses the model’s reward-scoring capability for quality evaluation.Source: §3.2.1 · pp. 3–5

repeat

  • At each k-center greedy step, the algorithm adds the data point farthest from the current selected set until the budget is reached.Source: §3.2.2 · p. 5, repeat · p. 5

3.2.2 Diverse Data Selection for Seed Instrucitons

  • K-center greedy selection iteratively adds the point farthest from existing centers, seeking to minimize the maximum distance to a selected center.The method uses BERT sentence embeddings to compute distances and form a diverse seed dataset.Source: repeat · p. 5

3.2.3 Augmented Data Selection

  • After fine-tuning an initial LLM on the seed data, the method scores its responses to high-quality instructions and selects low-scoring cases for augmentation.It reuses the reward model for necessity scoring, applies threshold β, then uses k-center greedy selection on the low-scoring instructions.Source: §3.2.3 · p. 5

3.2.4 Fine-tuning with Selected Instruction Data

  • The seed and augmented datasets are merged to fine-tune the raw pre-trained LLM; the paper uses LLaMA 2.Source: §3.2.4 · p. 5

4 Experiments

Experiments compare MoDS with full-dataset training and alternative selection baselines across instruction-following test sets. The results and ablations examine the roles of coverage-oriented selection and augmented data.

4.1.1 Training set

  • The experiments use Alpaca's 52,002 triplets and a 214,526-triplet mixture dataset, with the latter combining six instruction sources and covering multiple domains.Evaluation uses five human-curated test sets covering math, coding, writing, knowledge, computers, and other domains.Source: §4.1.1 · p. 6, §4.1.2 · p. 6

4.2 Detatils of Training and Testing

  • Fine-tuning uses LLaMA 2 7B for three epochs, while ChatGPT and GPT-4 score response relevance and accuracy on a 1–10 scale.Paired-model prompts are evaluated in both orders to address positional bias.Source: §4.2 · p. 6

4.3 Results and Analysis

  • MoDS(2000) outperforms Self-guided on all five test sets, while MoDS(1000) is comparable on four and better on Sinstruct; both use fewer examples.Table 1 reports comparisons using GPT-4 as the judge.Source: §4.3 · p. 7, Table 1 · p. 7
  • 4,000 MoDS-selected instructions outperform training on the full 214K-example Mixture Dataset; on Alpaca, 1,000 selected instructions also outperform full-dataset training.The Mixture comparison uses 1,000 seed and 3,000 augmented instructions.Source: §4.3 · pp. 6–7, Figure 3 · p. 6, Figure 4 · p. 7

4.4 Ablation Study

  • With 1,000 selected Mixture examples, K-center greedy outperforms random sampling across the test sets.Both subsets are selected from the high-quality dataset before fine-tuning.Source: Figure 5 · p. 7, §4.4 · p. 8
  • MoDS outperforms a model trained on 4,000 seed instructions alone, supporting the reported value of augmented data for addressing LLM capacity gaps.The comparison uses 1,000 seed and 3,000 augmented instructions for MoDS.Source: Figure 6 · p. 8, §4.4 · p. 8
  • Winning scores improve as augmented-data counts rise, but show no significant further improvement once the augmented set reaches 3,000 examples.The comparisons are judged by ChatGPT; scores above 1.0 indicate better performance than the full Mixture Dataset model.Source: Figure 7 · p. 8, §4.4 · p. 8

5 Conclustion

MoDS selects valuable instructions for a target foundation LLM by considering data quality, coverage, and necessity tied to that model’s abilities. It combines a diverse seed set with instructions that address the fine-tuned model’s capability gaps.

5 Conclustion

  • MoDS selects instruction data for a target foundation LLM using quality, coverage, and model-specific necessity.The approach evaluates instruction, input, and output triplets, then filters for high quality.Source: §5 · p. 8
  • K-center greedy selection forms a diverse, broad-coverage seed dataset from the high-quality subset.The seed fine-tunes the foundation LLM before its performance is evaluated on high-quality instructions to identify augmented data.Source: §5 · p. 9
  • Merging the seed and augmented instructions yields a high-quality, broad-coverage, high-necessity dataset for fine-tuning the optimized LLM.Source: §5 · p. 9

Related papers

Loading 2311.15653v1…