PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning

Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, Chang Zhou

arXiv:2311.08182v1cs.CLcs.LG

TL;DR

Large instruction-tuning datasets carry computational and annotation costs, while compact-subset selection has often relied on human or advanced-LLM supervision. DIVERSEEVOL iteratively selects diverse training data using the model’s own current representations; with less than 8% of the original data, its models match or surpass full-data baselines.

  • Problem

    Large instruction-tuning datasets impose computational and annotation costs, and prior compact-subset selection methods require human or advanced-LLM supervision.

    Source: §1 · p. 1

  • Method

    DIVERSEEVOL iteratively adds diverse data selected using the model’s current embedding space, without external oversight.

    Source: §1 · p. 1

  • Results

    With less than 8% of the original data, DIVERSEEVOL models match or surpass full-data baselines across three datasets and benchmarks.

    Source: §1 · p. 1, §5 · p. 8

  • Takeaways & Limitations

    The paper reports that dataset diversity correlates with improved performance and iterative evolving sampling outperforms one-shot sampling.

    Source: §1 · p. 2

    K-Center distance computations may strain GPU memory as datasets grow, and GPT-4-judge evaluations may retain model biases.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Enhancing the instruction-following ability of Large Language Models (LLMs) primarily demands substantial instruction-tuning datasets. However, the sheer volume of these imposes a considerable computational burden and annotation cost. To investigate a label-efficient instruction tuning method that allows the model itself to actively sample subsets that are equally or even more effective, we introduce a self-evolving mechanism DiverseEvol. In this process, a model iteratively augments its training subset to refine its own performance, without requiring any intervention from humans or more advanced LLMs. The key to our data sampling technique lies in the enhancement of diversity in the chosen subsets, as the model selects new data points most distinct from any existing ones according to its current embedding space. Extensive experiments across three datasets and benchmarks demonstrate the effectiveness of DiverseEvol. Our models, trained on less than 8% of the original dataset, maintain or improve performance compared with finetuning on full data. We also provide empirical evidence to analyze the importance of diversity in instruction data and the iterative scheme as opposed to one-time sampling. Our code is publicly available at https://github.com/OFA-Sys/DiverseEvol.git.

1 Introduction

DIVERSEEVOL addresses the cost and supervision demands of selecting compact instruction-tuning datasets by letting a model iteratively select diverse data for its own training. Across three datasets, models trained on less than 8% of the data match or outperform full-data baselines; the paper also reports benefits from diversity and iterative sampling.

1 Introduction

  • Large instruction-tuning datasets impose computational burdens, and their curation requires substantial annotation labor.Prior data-distillation approaches also require human or advanced-LLM supervision to select subsets.Source: §1 · p. 1
  • DIVERSEEVOL iteratively uses the model’s current embedding space to select additional training examples, refining the model without external oversight.It uses a K-Center strategy to select points far from existing labeled data.Source: §1 · p. 1
  • With less than 8% of the original data, DIVERSEEVOL models match or outperform full-data baselines across benchmarks and three instruction-tuning datasets.The datasets include human-annotated and machine-generated data.Source: §1 · p. 1
  • Dataset diversity, measured with Vendi Score, correlates with improved performance, while iterative evolving sampling outperforms one-shot sampling.The paper presents diversity and iterative sampling as central findings.Source: §1 · p. 2
  • The paper’s contributions include efficient self-evolving sampling, an analysis of dataset diversity, and evidence favoring iterative over static sampling.These contributions are presented as a three-part summary.Source: §1 · p. 2

2 Related Works

Prior work improves instruction-tuning efficiency through smaller curated datasets and external labeling or filtering. DIVERSEEVOL instead uses self-supervision and examines sampling strategies that prioritize uncertainty or embedding-space diversity.

2 Related Works

  • Instruction datasets have been built from human annotations, parent-LLM distillation, and real-world interactions, while efficiency work explores smaller datasets and advanced-LLM labeling.Examples include manually curated data and ChatGPT-based rating and filtration.Source: §2 · p. 2
  • DIVERSEEVOL conducts diverse sampling with only its own supervision, unlike methods requiring humans or more advanced LLMs.Source: §2 · p. 2
  • Sampling strategies include uncertainty-based selection and embedding-space diversity methods; the paper reports diversity-based sampling reduces instruction data without harming performance.Examples include entropy, least-confidence, K-Center, and Cluster-Margin sampling.Source: §2 · p. 2

3 DIVERSEEVOL

DIVERSEEVOL progressively expands an instruction-data training pool, using each model iteration to guide the next selection. Its K-Center selection favors candidates distant from existing training points in the current model’s embedding space.

3.1 Iterative Instruction Data Selection

  • At each iteration, a selection function chooses new instruction-response pairs for an expanded pool that trains the next model.The procedure starts from a randomized pool; each step selects St using the current model Mt and pool Pt, then trains Mt+1 on Pt+1.Source: §3.1 · pp. 2–3

3 DIVERSEEVOL

  • The workflow repeatedly trains a model, embeds datapoints in its representation space, and adds selected points from the remaining source data to the next training pool.Figure 1 depicts the process beginning with P0 and Q0, then adding S0 to form P1.Source: §3 · p. 2, Figure 1 · p. 3

3.2 Selection Algorithm: K-Center-Sampling

  • K-Center selection seeks representative, diverse additions by maximizing each candidate’s distance to its nearest existing training point.Distances are computed from average-pooled hidden states produced by the currently trained model, so selection reflects its current embedding space.Source: §3.2 · pp. 3–4

4 Experiments

Across three instruction-tuning datasets, DIVERSEEVOL matches or exceeds full-data baselines with far fewer samples. Analyses find that selected-data diversity and iterative sampling distinguish the approach from one-shot selection.

4.1 Experimental Setup

  • Experiments use Databricks-Dolly and two Self-Instruct datasets, evaluating on Vicuna-, Koala-, and Wizardlm-Bench with GPT-4 as judge and RS/WTR versus ChatGPT.The datasets include human-annotated and machine-generated sources.Source: §4.1 · pp. 4–5
  • Comparisons include full-source-data LLaMA-7B baselines and random, least-confidence, margin, and K-Center sampling; training uses 100 initial samples and adds 100 per iteration for 10 iterations.Unless stated otherwise, models are finetuned for three epochs with batch size 128 and learning rate 2 × 10−5.Source: §4.1 · pp. 4–5

4.2 Main Results

  • DIVERSEEVOL matches full-data baselines on Dolly with 700–1,100 samples (<8%) and tops most SelfInstruct metrics with ≤1,100 (≤2%); on Dolly it matches by 600 (4%).The table compares RS, WTR, and optimal data sizes; Figure 2 traces performance as training samples increase.Source: §4.2 · pp. 5–6, Table 1 · p. 5, Figure 2 · p. 6

4.3 Analyses

  • K-Center selection raises pool diversity early above the original dataset and other samplers, as measured by Vendi-Score across iterations.Figure 3 plots Vendi-Score against increasing data size and includes the full-data diversity line.Source: §4.3 · p. 7, Figure 3 · p. 7
  • At fixed training volumes of 300, 700, and 1,100, higher Vendi-Score generally accompanies higher RS across benchmarks, with only marginal Wizardlm-Bench deviations.The controlled-diversity analysis uses Databricks-Dolly and targets Vendi-Score ranges [3, 4], [5, 6], and [9, 10].Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • With the same K-Center selector and initial 100 samples, iterative expansion consistently outperforms one-time selection at matched data volumes.The authors attribute the difference to iterative feedback from each trained model guiding subsequent selections.Source: Table 3 · p. 7, §4.3 · p. 8

5 Conclusion

DIVERSEEVOL is a self-evolving approach to efficient instruction tuning that iteratively selects diverse subsets without external supervision. The authors report that it matches or surpasses strong baselines with less than 8% of the original data.

5 Conclusion

  • Under 8% of the original data, DIVERSEEVOL matches or surpasses strong baselines in performance.The conclusion also identifies larger instruction datasets as a direction for future work.Source: §5 · p. 8
  • DIVERSEEVOL iteratively selects diverse subsets using K-Center sampling, without external supervision.The method progressively improves itself through this selection scheme.Source: §5 · p. 8
  • Future work could apply DIVERSEEVOL to larger instruction datasets for potentially more refined results.Source: §5 · p. 8
  • The authors suggest that more advanced diverse-sampling algorithms may further enhance model performance.Source: §5 · p. 8

Limitations

The authors identify potential GPU-memory expense as the source dataset grows and possible bias in GPT-4-based evaluation as limitations.

Limitations

  • As the source dataset grows, computing distances between high-dimensional embeddings may impose considerable GPU-memory expense.Source: Limitations · p. 9
  • The evaluation outcomes rely heavily on GPT-4 as judge.Source: Limitations · p. 9
  • Even with temperature set to 0 and alternating answer positions across two queries, GPT-4 evaluation may retain inherent biases.Source: Limitations · p. 9

Ethics Statement

The authors report following applicable data and API usage policies and warn that finetuned models may produce toxic or unsafe content.

Ethics Statement

  • The authors state that data, pretrained models, and results were collected and processed according to their respective usage policies.Source: Ethics Statement · p. 9
  • Finetuned DIVERSEEVOL models may produce toxic or unsafe content.Source: Ethics Statement · p. 9
  • The authors call for careful verification before applying model outputs in real-world settings.Source: Ethics Statement · p. 9

A GPT4-Judge Template

The evaluation template uses GPT-4 to judge two assistant answers to a test-set instruction. It requests ratings and an explanation while instructing the judge to avoid order bias.

A GPT4-Judge Template

Related papers

Loading 2311.08182v1…