Source-linked AI summary
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, Junxian He
TL;DR
The paper addresses the lack of a principled, automatic way to identify effective instruction-tuning data for alignment. It measures data by complexity, quality, and diversity, then uses these measurements for selection; DEITA matches or exceeds open-source alignment models with 6K SFT samples, while its DPO variant reaches 7.55 MT-Bench and 90.06% AlpacaEval.
Problem
Earlier instruction-tuning datasets often use heuristic automation or manual selection, leaving effective data characteristics and systematic low-data curation unclear.
Method
The paper studies complexity, quality, and diversity, develops measurement techniques including evolution-based scorers and embedding-based diversity filtering, and selects data using a simple score-first strategy.
Results
DEITA performs better or comparably to state-of-the-art open-source models with over 10x fewer samples; DEITA-Mistral-7B + DPO reaches 7.55 MT-Bench and 90.06% AlpacaEval.
Takeaways & Limitations
Automatic selection of complex, high-quality, and diverse data supports data-efficient alignment, and the authors release selected datasets for future research.
Takeaways & Limitations
The selection algorithm is intentionally kept simple, while more advanced selection algorithms are left for future work.
Abstract
from arXiv · showhide
Instruction tuning is a standard technique employed to align large language models to end tasks and user preferences after the initial pretraining phase. Recent research indicates the critical role of data engineering in instruction tuning -- when appropriately selected, only limited data is necessary to achieve superior performance. However, we still lack a principled understanding of what makes good instruction tuning data for alignment, and how we should select data automatically and effectively. In this work, we delve deeply into automatic data selection strategies for alignment. We start with controlled studies to measure data across three dimensions: complexity, quality, and diversity, along which we examine existing methods and introduce novel techniques for enhanced data measurement. Subsequently, we propose a simple strategy to select data samples based on the measurement. We present deita (short for Data-Efficient Instruction Tuning for Alignment), a series of models fine-tuned from LLaMA and Mistral models using data samples automatically selected with our proposed approach. Empirically, deita performs better or on par with the state-of-the-art open-source alignment models with only 6K SFT training data samples -- over 10x less than the data used in the baselines. When further trained with direct preference optimization (DPO), deita-Mistral-7B + DPO trained with 6K SFT and 10K DPO samples achieve 7.55 MT-Bench and 90.06% AlpacaEval scores. We anticipate this work to provide tools on automatic data selection, facilitating data-efficient alignment. We release our models as well as the selected datasets for future researches to effectively align models more efficiently.
1 INTRODUCTION
The paper studies how to automatically identify effective instruction-tuning data for alignment, focusing on complexity, quality, and diversity. It introduces DEITA, whose automatically selected datasets achieve competitive alignment performance with substantially fewer examples.
- Motivation: Instruction tuning can align pretrained models using relatively small datasets, but earlier construction commonly relies on heuristics or manual selection.The paper frames systematic, data-efficient curation as an unresolved problem.
- Approach: The study measures instruction-tuning data along complexity, quality, and diversity to characterize effective alignment data.It examines existing measurement baselines and introduces new metrics intended to correlate more strongly with alignment performance.
- Approach: EVOL COMPLEXITY and EVOL QUALITY generate variants with differing complexity or quality, score a seed subset with ChatGPT, and train scorers to predict those scores.The resulting scores support automatic ranking and selection of data samples.
- DEITA: DEITA models fine-tuned from LLaMA and Mistral use the proposed automatic selection technique to maximize data efficiency.The models are evaluated on MT-Bench, AlpacaEval, and the Open LLM Leaderboard.
- Results: 7.22 MT-bench and 80.78% AlpacaEval are achieved by DEITA-Mistral-7B with only 6K samples under vanilla SFT.With DPO, the model trained on 6K SFT and 10K DPO samples obtains 7.55 MT-Bench and 90.06% AlpacaEval.
2 WHAT MAKES GOOD DATA FOR ALIGNMENT?
The study frames data selection as choosing a budgeted subset whose resulting alignment performance measures whether the selection metric identifies effective examples. It evaluates complexity, quality, and diversity through controlled experiments and simple selection procedures.
- 2.1 THE DATA SELECTION PROBLEM: Data selection chooses m examples from a large instruction-tuning pool, with m serving as the data budget tied to computation.The selected subset is used for instruction tuning, and alignment performance evaluates the selection strategy.
- 2.2 EXPERIMENTAL SETUP: The controlled-study pipeline selects data by one metric, instruction-tunes a pretrained model, and evaluates its instruction-following abilities.The selection algorithm is intentionally kept simple, such as choosing examples with the largest metric scores.
- 2.3 FROM THE COMPLEXITY PERSPECTIVE – EVOL COMPLEXITY: EVOL COMPLEXITY ranks examples after iterative instruction evolution and achieves the best alignment performance across both Xsota and Xbase.The method uses evolved samples with different complexities to obtain finer-grained complexity scores.
- 2.4 FROM THE QUALITY PERSPECTIVE – EVOL QUALITY: EVOL QUALITY consistently delivers superior alignment performance, while quality metrics matter especially in pools containing many low-quality examples.Response-quality scores are obtained by ranking responses of varying quality, enabling more nuanced distinctions.
- 2.5 FROM THE DIVERSITY PERSPECTIVE – AN EMBEDDING-BASED APPROACH: Diversity-aware selection substantially outperforms random selection, and Repr Filter outperforms Instag Diversity on both data pools.Repr Filter sorts data by complexity and quality, then adds examples according to embedding-distance-based diversity criteria until the budget is reached.
3 DEITA– DATA EFFICIENT INSTRUCTION TUNING FOR ALIGNMENT
DEITA selects instruction-tuning data by combining complexity, quality, and diversity, then trains models on the resulting compact subsets. Across several backbones and benchmarks, the selected data achieves strong alignment performance with substantially reduced data budgets.
- 3.1 METHOD: DEITA combines complexity and quality into an evol score, then applies diversity-aware filtering to select the final subset.The method sorts samples by s = q * c and discards redundant examples using nearest-neighbor distance.
- 3.1 METHOD: DEITA trains LLaMA-1-13B, LLaMA-2-13B, and Mistral-7B models using selected datasets of 6K or 10K examples.The models are evaluated against state-of-the-art SFT models and random-selection baselines.
- 3.3 RESULTS: DEITA-LLaMA16K clearly outperforms other data-selection approaches in the main comparison using a LLaMA-1-13B backbone.Table 5 compares multiple selection approaches; its caption notes cost-related sampling and tagger-model evaluation details for Alpagasus.
- 3.3 RESULTS: Across three backbones, DEITA SFT models outperform almost all compared SFT-aligned models, with DEITA-Mistral-7B10K reaching 7.32 MT-Bench.DEITA-LLaMA-2 models also outperform LLaMA2-13B-Chat in the reported comparison.
- 3.3 RESULTS: DEITA-Mistral-7B10K+DPO achieves 7.55 MT-Bench and 90.06% AlpacaEval, comparable to zephyr-beta trained on 30x more data.It slightly trails Mistral-7B-Instruct-v0.2, whose alignment approach and data are not public.
- 3.3 RESULTS: The radar plot compares DEITA models and major baselines across eight MT-Bench subtasks, highlighting detailed task-level performance.The figure provides the requested multi-subtask comparison view for the reported models.
- 3.3 RESULTS: On the Open LLM Leaderboard, DEITA SFT models achieve the best average results among SFT-aligned models across different backbones.Further DPO training boosts DEITA-Mistral-7B by around 5 points on average and helps it surpass Zephyr.
- 3.3 RESULTS: DEITA consistently provides the best data-selection performance across data volumes and matches training on all 300K samples with only 3K selected examples.Performance initially rises as selected data increases but eventually declines, indicating that only a limited proportion of the pool is especially suitable for alignment.
4 CONCLUSION
The paper studies what makes good alignment data and develops automatic selection methods around complexity, quality, and diversity. DEITA achieves comparable or superior performance to open-source state-of-the-art models with far fewer training samples.
- 4 CONCLUSION: The study investigates good alignment data through controlled analyses of complexity, quality, and diversity.It proposes automatic selection methods and trains DEITA models on the selected data.
- 4 CONCLUSION: DEITA achieves superior or comparable performance to state-of-the-art open-source models using 10x fewer training samples.The authors release the selected data for more efficient model alignment.
A SETUP DETAILS
The setup controls selected-data properties for diversity analysis and specifies the computational infrastructure used to train DEITA models.
- A SETUP DETAILS: The diversity experiment constrains the product of selected-data mean complexity and quality to remain within 2 of the complete pool means.This controls complexity and quality while comparing diversity-based selection methods.
- A SETUP DETAILS: Training uses four or eight NVIDIA Tesla A100 GPUs for 7B or 13B models, respectively.DeepSpeed ZeRO-3, FlashAttention-2, Vicuna-style templates, and a maximum input length of 2048 are used.
B CASE STUDY
The case study compares Direct Scoring with the proposed Rank & Scoring method for measuring complexity and quality. Rank & Scoring better differentiates subtle variations among samples.
- Rank & Scoring distinguishes subtle differences in sample complexity and quality more effectively than Direct Scoring.Direct Scoring tends to assign similar and excessively high scores, whereas Rank & Scoring produces more differentiated scores.
C.1 ANALYSIS OF REPR FILTER
The Repr Filter is evaluated across thresholds τ from 0.8 to 0.9 and two sentence-encoding strategies. Model-based encoding is robust across thresholds, while Semantic-based performance declines as τ changes.
- The Repr Filter experiment varies thresholds τ from 0.8 to 0.9 and compares Model-based with Semantic-based sentence encoding.Model-based encoding uses representations from the model trained after selection; Semantic-based encoding uses sentence-embedding models such as E5-Large-V2.
- Model-based encoding remains robust across thresholds and maintains a substantial margin over the baseline.
- Semantic-based encoding shows a significant performance decline when τ changes.
D HUMAN EVALUATION
The human evaluation uses colleague researchers as annotators after difficulties with MTurk responses and instruction following. DEITA-LLaMA16K outperforms random selection and performs on par with Vicuna-13B-v1.3, despite using fewer training samples than Vicuna.
- Setup: Human evaluation uses the LIMA test dataset, with 100 randomly sampled examples selected for evaluation.
- Setup: Four colleague researchers serve as annotators after MTurk answers were inconsistent and instruction following was difficult.Annotators were assigned samples without knowing which model produced each response.
- Results: Both DEITA and Random Selection use 6K training samples, whereas Vicuna uses 125K samples.All models use the LLaMA1-13B backbone in the reported comparison.
- Results: DEITA-LLaMA16K shows a significant human-evaluation advantage over random selection.
- Results: DEITA-LLaMA16K performs on par with Vicuna-13B-v1.3 in human evaluation, with most responses judged ties.
E PROMPT EXAMPLES
The appendix documents prompt templates for complexity and quality measurement, enhancement, and ranking. It covers baseline prompts alongside prompts designed to modify and score instruction characteristics.
- E.1 COMPLEXITY BASELINE: Complexity baselines include Direct Scoring and Instruction Node prompt templates.
- E.2 EVOL COMPLEXITY: Complexity-enhancement prompts add constraints and deepen instructions.
- E.2 EVOL COMPLEXITY: Additional complexity prompts concretize instructions and increase reasoning steps.
- E.2 EVOL COMPLEXITY: Complexity-ranking prompts rank and score instructions with different complexity levels.
- E.3 QUALITY BASELINE: Quality baselines include Random Selection, Response Length, and Direct Scoring.
- E.4 EVOL QUALITY: Quality-enhancement prompts improve helpfulness, relevance, and depth.
- E.4 EVOL QUALITY: Other quality-enhancement prompts foster creativity and supply additional details.
- E.4 EVOL QUALITY: Quality-ranking prompts rank and score responses with different quality levels.