PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Data Efficiency via Curating LLM-Driven Rating Systems

Jinlong Pang, Jiaheng Wei, Ankit Parag Shah, Zhaowei Zhu, Yaxuan Wang, Chen Qian, Yang Liu, Yujia Bao, Wei Wei

arXiv:2410.10877v2cs.CLcs.AI

TL;DR

LLM-generated data-quality ratings can be inaccurate, complicating selection for instruction tuning. DS2 models score errors to curate ratings and select diverse data; using 3.3% of the original data, it outperforms full-dataset training and exceeds LIMA at the same sample size.

  • Problem

    LLM-based quality ratings can contain inaccuracies and biases, and the errors are widespread and vary across rating models.

    Source: §1 · pp. 1–2

  • Method

    DS2 uses a score transition matrix to curate LLM-generated quality ratings and combines curated scores with long-tail selection to favor quality and diversity.

    Source: §1 · p. 2

  • Results

    Using 3.3% of the original data, DS2 outperforms training on the full 300k-sample dataset and exceeds LIMA at the same sample size of 1k samples.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The conclusion is that smaller, high-quality instruction datasets can achieve superior results while avoiding performance drops associated with low-rated or redundant data.

    Source: §7 · p. 10

    DS2 relies on 2-NN clusterability, which can be violated by subtle token-level differences that change scores; the authors say averaging across clusters mitigates noise from some violations.

    Source: §3.2 · pp. 4–5

Abstract

from arXiv · show

Instruction tuning is critical for adapting large language models (LLMs) to downstream tasks, and recent studies have demonstrated that small amounts of human-curated data can outperform larger datasets, challenging traditional data scaling laws. While LLM-based data quality rating systems offer a cost-effective alternative to human annotation, they often suffer from inaccuracies and biases, even in powerful models like GPT-4. In this work, we introduce DS2, a Diversity-aware Score curation method for Data Selection. By systematically modeling error patterns through a score transition matrix, DS2 corrects LLM-based scores and promotes diversity in the selected data samples. Our approach shows that a curated subset (just 3.3% of the original dataset) outperforms full-scale datasets (300k samples) across various machine-alignment benchmarks, and matches or surpasses human-aligned datasets such as LIMA with the same sample size (1k samples). These findings challenge conventional data scaling assumptions, highlighting that redundant, low-quality samples can degrade performance and reaffirming that "more can be less."

1 INTRODUCTION

Instruction tuning has often followed data-scaling approaches, but studies of small human-curated datasets challenge that assumption and make data quality selection important. DS2 addresses inaccuracies in LLM-generated ratings through score curation and diversity-aware selection.

1 INTRODUCTION

  • Small human-annotated datasets have been reported to suffice for alignment, challenging traditional data-scaling assumptions and underscoring the importance of selecting high-quality instruction data.The introduction notes that raw LLM ratings may contain inaccuracies or model-specific biases, which can yield sub-optimal selection.Source: §1 · p. 1
  • DS2’s pipeline uses LLM ratings, k-NN-based score-transition estimation and curation, then long-tail scoring to select rare data while prioritizing quality and diversity.Source: Figure 1 · p. 2
  • Score errors are widespread and vary significantly across GPT, LLaMA, and Mistral rating models.The paper models these errors with a score transition matrix without relying on ground-truth scores.Source: §1 · p. 2
  • DS2 combines automatic score curation with diversity-aware selection, and the paper reports downstream performance improvements across different initial rating models.Its score curation uses a learned transition matrix to refine ratings, while selection promotes examples that vary from one another.Source: §1 · p. 2
  • With 3.3% of the data (10k of 300k samples), DS2 significantly outperforms fine-tuning on the full data pool and outperforms the human-curated LIMA dataset.The reported comparisons also find DS2 superior to nine baselines.Source: §1 · p. 2

2 RELATED WORK

Related work covers data-selection methods based on statistical metrics and LLM-driven assessment or generation. DS2 is distinguished from prior prompt-based rating approaches by explicitly modeling score errors for curation.

2 RELATED WORK

  • Data selection and filtering methods are broadly categorized as LLM-free or LLM-based approaches.Source: §2 · p. 3
  • LLM-free selection work uses metrics such as k-NN embedding distance, input and output length, Shapley values, and importance resampling to assess or select data.Source: §2 · p. 3
  • LLM-based approaches select or generate data using semantic analysis, fine-grained tags, quality metrics, and gradient-based measures.Examples include perplexity, confidence and reward scores, gradient matching, and influence-function scores.Source: §2 · p. 3
  • Prompt-based rating work includes Chen et al. (2023), focused on quality, and Liu et al. (2023a), emphasizing diversity; DS2 explicitly models score errors for curation.Source: §2 · p. 3

3 UNDERSTANDING THE ERROR PATTERN OF LLM SCORES

The section describes prompt-based LLM rating and models its score errors with a transition matrix estimated from neighboring samples. The matrices show that error patterns vary across rating models, while the analysis relies on a k-NN clusterability assumption that can be violated in practice.

3.1 PROMPT-BASED LLM RATING

  • Samples are rated from 0 to 5 by GPT-4o-mini, LLaMA-3.1-8B-Instruct, or Mistral-7B-Instruct-v0.3, over a pool of five instruction-tuning datasets.GPT-4o-mini scores are more spread across the median range, while the other two models concentrate heavily on score 3.Source: §3.1 · pp. 3–4

3.2 SCORE TRANSITION MATRIX

  • The transition matrix models probabilities from ground-truth scores to observed ratings, and the method estimates it without ground-truth labels using k-NN consensus patterns and a linear program.The analysis assumes nearby embeddings share a ground-truth class; third-order consensus vectors suffice to estimate the matrix and score distribution.Source: §3.2 · pp. 4–5
  • GPT’s score errors are mostly between adjacent values, while LLaMA and Mistral exhibit more errors and rating variation.Figure 3 compares the derived score transition matrices across rating models.Source: §3.2 · p. 5, Figure 3 · p. 5
  • The k-NN clusterability assumption can be violated when subtle token-level differences correspond to different correctness, but averaging over 2-NN clusters helps tolerate some violations.The scoring approach also considers rarity and informativeness, not correctness alone.Source: §3.2 · pp. 4–5

4 DS2: DIVERSITY-AWARE SCORE CURATION FOR DATA SELECTION

DS2 combines LLM ratings, transition-matrix-based score curation, and a long-tail diversity score to select data. Its curation uses k-NN agreement and a confidence probability to address score errors and imbalanced ratings, while diversity is measured from sample embeddings.

4 DS2: DIVERSITY-AWARE SCORE CURATION FOR DATA SELECTION

  • DS2 first rates samples with an LLM, then generates curated quality and long-tail diversity scores, and finally sorts by curated score followed by long-tail score.The two-stage sorting prioritizes quality while retaining diverse samples.Source: §4 · p. 5

4.1 CURATED QUALITY SCORE

  • DS2 uses k-NN score agreement to identify candidate corrections, then applies the estimated transition matrix and ground-truth score distribution to set a misrating threshold.Because the transition matrix is dataset-level, k-NN agreement supplies instance-level quality information.Source: §4.1 · p. 6
  • The confidence probability scales curation effort to mitigate imbalanced ratings, which can otherwise push many high-rated samples toward the majority score and degrade performance.The default confidence probability is 0.5.Source: §4.1 · p. 6

4.2 LONG-TAIL DIVERSITY SCORE

  • The long-tail diversity score uses average cosine similarity to nearest neighbors; a higher score indicates greater diversity and the measure is independent of the rating LLM.Samples whose neighbors are farther away receive higher diversity scores.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

DS2 outperforms nine baselines and the 300k data pool with 10k examples, and at 1k samples matches or exceeds LIMA. The experiments also compare score curation across rating models and assess benchmark performance against human-curated data.

5.1 EXPERIMENTAL SETUP

  • Experiments fine-tune LLaMA-2-7B, LLaMA-3.1-8B, and Mistral-7B-v0.3, evaluating five OpenLLM tasks with Exact Match or 1-shot F1.The baselines include random selection, completion length, perplexity, k-NN, AlpaGasus, DEITA, LESS, and full-data fine-tuning.Source: §5.1 · pp. 7–8, §5.2 · p. 8

5.2 OPENLLM LEADERBOARD EVALUATION RESULTS

  • With 10k samples, DS2 consistently beats nine baselines and significantly outperforms the 300k full data pool, using 96.7% less data.The reported comparison holds under different rating-model settings.Source: §5.2 · p. 8
  • After score curation, LLaMA and Mistral average 60.2 and 61.1, respectively, matching or exceeding GPT’s uncurated average of 60.2.The passage reports that LLaMA-3.1-8B-Instruct has more score errors than the other two rating models, alongside a performance drop without curation.Source: §5.2 · p. 8
  • Score curation consistently raises DS2’s average performance across rating models, including GPT-4o-mini, from 60.2 to 61.4.Additional results on other base models support this finding.Source: §5.2 · p. 8

5.3 HUMAN ALIGNMENT V.S. MACHINE ALIGNMENT

  • At 1k samples, DS2 matches or outperforms LIMA; at 10k, it improves further over LIMA on Vicuna-Bench and MT-bench judging.The comparison uses GPT-4o-mini as the judge and reports results in Win-Tie-Loss form.Source: §5.3 · p. 9, Figure 6 · p. 9

6 ABALTION STUDY

Across data budgets, DS2 performs best among the compared selection methods, while a matched-setting comparison also reports a 15% average improvement over AlpaGasus. The section examines data scaling and score curation alongside that comparison.

6.1 REVISITING DATA SCALING LAWS

  • Across different data budgets, the method consistently achieves the best data-selection performance versus representative baselines; redundant samples may be uninformative, unnecessary, or detrimental to model performance due to overfitting.Source: §6.1 · p. 9

6.2 EXPLORING THE IMPACT OF SCORE CURATION

  • Table 5 compares AlpaGasus and DEITA results without versus with score curation using GPT-4o-mini scores.Results are presented as without curation / with curation.Source: §6.2 · p. 9, Table 5 · p. 10

6.3 APPLES-TO-APPLES COMPARISON WITH ALPAGASUS

  • In an apples-to-apples comparison, DS2 improves average performance by 15% over AlpaGasus despite using a weaker rating model.The comparison replicates AlpaGasus settings and uses GPT-4o-mini for consistency.Source: §6.3 · p. 10, Figure 7 · p. 10

7 CONCLUSION

The conclusion presents DS2 as a way to improve instruction-tuning data efficiency by curating LLM-rated scores. It reports that a small selected dataset can outperform full-data training and match or exceed LIMA at the same sample size.

7 CONCLUSION

  • Using 3.3% of the original data, DS2 outperforms training on 300k samples and exceeds LIMA with the same 1k-sample size.The conclusion attributes the results to curated data that avoids performance drops associated with low-rated or redundant samples.Source: §7 · p. 10

ORGANIZATION OF THE APPENDIX

  • The appendix lists material on limitations, rating prompts, the DS2 algorithm, and additional training and evaluation details.It also points to further experimental results, including LLM judging and LIMA comparisons.Source: ORGANIZATION OF THE APPENDIX · p. 14
  • Additional appendix sections cover embedding and score-curation analyses, computational complexity, diversity, and wrongly rated examples.Source: ORGANIZATION OF THE APPENDIX · p. 14

A LIMITATIONS

The paper identifies limitations in its clusterability and model-scale assumptions, alongside the data pool, rating prompt, score-transition derivation, and DS2 pipeline details. It also explains how LLM ratings are rescaled and how a binary example derives transition probabilities from score consensuses.

A LIMITATIONS

  • The method assumes sample-independent scores, k-NN clusterability, and mainly 7B/8B models; these assumptions may introduce data-specific errors, and performance at larger scales remains uncertain.The authors suggest group-dependent approaches as a future direction and note that subtle textual differences can produce different scores.Source: A LIMITATIONS · p. 14

B.1 DATA POOL

  • The data pool combines five instruction-tuning datasets originating from human annotations or powerful LLM generations, with varied formats, quality, prompt lengths, and tasks.The datasets are standardized using the TULU template; WizardLM and Flan_v2 are sampled to 100K to match dataset size.Source: §B.1 · p. 14, Table 6 · p. 15

B.2 QUALITY-BASED PROMPT TEMPLATE

  • The rating prompt scores rarity, complexity, and informativeness, using an initial 1–10 scale that is compressed to 0–5 to distinguish quality differences.The authors say direct rating on a small scale can concentrate scores around 3, while rescaling merges low and high values for simpler analysis.Source: §B.2 · p. 15

C.1 WARM-UP OF DERIVING SCORE TRANSITION MATRIX: A BINARY EXAMPLE

  • The binary derivation assumes that a sample and its two nearest neighbors share a true score, then uses observed score consensuses to estimate ground-truth prevalence and rating-error rates.The resulting error rates determine the transition matrix, with score-probability information pooled across 2-NN clusters.Source: §C.1 · pp. 15–16

C.2 ALGORITHM DETAILS

  • Algorithm 1 provides the proposed DS2 data-selection pipeline details.The supplied passage identifies the algorithm but does not describe its individual steps.Source: §C.2 · p. 16

C.3 KNN CLUSTERABILITY HYPOTHESIS ANALYSIS

The section examines whether nearby text embeddings tend to share ratings, as required by the k-NN clusterability hypothesis. It reports that score curation increases low-gap clusters, while embedding choice has limited impact on error patterns and curation shifts the rating distribution.

C.3 KNN CLUSTERABILITY HYPOTHESIS ANALYSIS

  • The k-NN clusterability hypothesis assumes semantically similar embeddings tend to share ratings, but subtle token changes can alter correctness and scores.The authors argue that their prompt's broader quality criteria and granular ratings help address this challenge.Source: §C.3 · p. 16
  • Averaging consensus probabilities across 2-NN clusters can tolerate some clusterability violations, but their rate cannot be measured without ground-truth scores.The authors cite prior image-classification work reporting that methods can still outperform baselines with 20% violations.Source: §C.3 · p. 17
  • The analysis measures cluster score gaps as the mean absolute difference between a target sample's score and its nearest neighbors' scores.Table 7 groups the score-gap distribution, and Figure 8 visualizes proportions with and without curation.Source: §C.3 · p. 17, Table 7 · p. 18, Figure 8 · p. 18
  • 81.0% of GPT samples, compared with Mistral's 70.2% and LLaMA's 58.3%, fall in the 0.0–1.0 score-gap range without curation.The authors interpret this pattern as stronger alignment with k-NN clusterability for more powerful rating models.Source: §C.3 · p. 18
  • All three rating models show an increased proportion of samples in the 0.0–1.0 score-gap range after score curation.Table 8 also includes corresponding average performance on LLM Leaderboard tasks.Source: Table 8 · p. 18, §C.3 · p. 18

D EXPLORING THE IMPACT OF EMBEDDING MODELS

E.1 IMPACT OF SCORE CURATION ON DISTRIBUTION

  • Score curation decreases high-rated examples and substantially increases examples rated 3, which the authors link to the imbalanced score distribution.They say confidence probabilities are important because rare rating-5 samples are drawn toward the majority rating of 3.Source: §E.1 · p. 20, Figure 11 · p. 21
  • 60%-70% of DS2's selected 10K examples come from WizardLM, corresponding to quality differences among the five source subsets.Figure 12 depicts the subset proportions within the selected samples.Source: §E.1 · p. 20, Figure 12 · p. 21

E.2 IMPACT OF SCORE CURATION ON SCORE ERRORS

The section reports that score curation reduces incorrect transitions in the estimated rating matrices. It also gives evaluation setup details and additional OpenLLM leaderboard results on two base models.

E.2 IMPACT OF SCORE CURATION ON SCORE ERRORS

  • Score curation significantly reduces the probability of incorrect score transitions in matrices across rating models.Figure 13 compares post-curation error patterns with those without curation in Figure 3.Source: Figure 13 · p. 21, §E.2 · p. 21

F SETUP DETAILS

  • The evaluation covers MMLU, BBH, GSM, TydiQA, and TruthfulQA, with task-specific prompting and scoring setups.Training uses LoRA on 7B/8B models, with batch size 128, learning rate 1e-4, five epochs, and maximum input length 2048 tokens.Source: F SETUP DETAILS · pp. 21–22

G.1 OPENLLM LEADERBOARD EVALUATION RESULTS

  • DS2 consistently outperforms the baselines on OpenLLM leaderboard evaluations using Mistral-7B-v0.3 and LLaMA-2-7B-hf base models.The reported outcomes are in Tables 10 and 11, respectively.Source: §G.1 · p. 22

G.2 LLM JUDGE EVALUATION

Across these supplementary evaluations, score curation generally improves judge-based comparisons, while DS2’s comparison with LIMA depends on sample size and diversity-only ranking is not universally effective.

G.2 LLM JUDGE EVALUATION

  • In most cases, score curation lowers Vicuna-Bench loss rates without reducing original win rates, and DS2 outperforms almost all baselines.Source: §G.2 · p. 22, Table 13 · p. 24

G.3 EXPLORING THE CURATION IMPACT ON OTHER SCORE-AWARE METHODS

  • Table 14 compares performance with versus without score curation across score-aware methods on LLaMA-3.1-8B and Mistral-7B-v0.3.Source: §G.3 · p. 22, Table 14 · p. 25

G.5 EXPLORING THE IMPACT OF CONCATENATING HIGH-RATED EXAMPLES ACROSS RATING MODELS

  • The combined baseline's 8K score-5 examples plus 2K examples jointly rated 4 still fail to achieve strong performance.Source: Table 15 · p. 25, §G.5 · p. 25

G.4 COMPARISON WITH HIGH-QUALITY HUMAN-ANNOTATED EXAMPLES: LIMA

  • At 1k samples, DS2 performs worse than LIMA; at 10k samples, it surpasses LIMA on the 300-sample LIMA test set.The authors attribute LIMA’s advantage at 1k to the IID nature of its training and test sets.Source: §G.4 · p. 24, Figure 14 · p. 26

G.6 APPLES-TO-APPLES PERFORMANCE COMPARISON WITH ALPAGASUS

  • AlpaGasus comparisons use matched 4-bit quantization and LoRA settings, but DS2 uses GPT-4o-mini scores, weaker than the GPT-4 used by AlpaGasus.Source: §G.6 · p. 26

H COMPUTATIONAL COMPLEXITY

I EXPLORING THE IMPACT OF DIVERSITY SCORE

J MIS-RATED EXAMPLES

The appendix shows examples of ratings revised downward and compares high- and low-rated samples from five instruction-tuning datasets.

J MIS-RATED EXAMPLES

J.1 ORIGINAL HIGH-RATED EXAMPLES VS. ORIGINAL LOW-RATED EXAMPLES

Related papers

Loading 2410.10877v2…