Less is More: Improving LLM Alignment via Preference Data Selection

1 Introduction

• Prior DPO work has focused on modified objectives and iterative learning, while the contribution here centers on preference-data characteristics and selection.Source: §1 · p. 1
• Noisy reward-model preferences can shrink model parameters, while margin-based selection can address this issue by inducing parameter inflation.Source: §1 · p. 2
• Using 10% of UltraFeedback and its variants, BeeS improves AlpacaEval 2.0 by 3% to 8% across Llama, Mistral, and Qwen models versus full-dataset DPO.The introduction also reports iterative DPO results using 25% of online data: 48.49% win rate and 54.99% length-control win rate on AlpacaEval 2.0.Source: §1 · p. 2
• BeeS aggregates external and implicit DPO reward signals, deprioritizing pairs with a low margin from any single source to mitigate noise.The introduction presents this strategy as a way to reduce computational overhead while improving performance over full-dataset training.Source: §1 · p. 2

1.1 Related Work

• Related work identifies limited exploration of data efficiency in preference learning; this work proposes criteria for informative-sample selection and extends them to iterative DPO.The related-work discussion also notes prior research on annotation costs, margin-based losses, and debate over whether hard samples help or hinder preference learning.Source: §1.1 · p. 2

2 Background

• RLHF uses preference data to train a reward model, then uses that model to guide policy optimization, often with reinforcement learning such as PPO.Source: §2 · pp. 2–3
• The RL approach can be computationally expensive, sensitive to reward misspecification, and require careful hyperparameter tuning.Source: §2 · p. 3
• DPO directly incorporates human preference data into supervised training instead of explicitly defining and optimizing a reward function.It bypasses the intermediate reinforcement-learning step and is described as more stable and computationally efficient than standard RLHF.Source: §2 · p. 3

3.1 Parameter Shrinkage and Inflation Analysis

• Selecting large-margin examples reduces the likelihood of preference reversals and can inflate parameter estimates, counteracting shrinkage.The analysis also notes that regularization or early stopping is indispensable when realized preferences are separable.Source: §3.1 · p. 4
• Noisy reward labels can shrink parameters toward zero, while selecting larger-margin data can compensate for the resulting performance degradation.The margin-maximization principle motivates estimating margins across the dataset before full-set training and filtering for high-margin examples.Source: §3.1 · p. 4

3.2 Multi-source Margin Aggregation

• Implicit and external reward margins are weakly correlated, while implicit margins from 1B and 3B models are strongly correlated.The reported patterns also hold across other datasets.Source: §3.2 · p. 5
• Online max-margin pairs selected by one reward model can remain ambiguous under another, suggesting online-generated preference data may be redundant.The ambiguity is reported even when the two reward models show similar RewardBench performance.Source: §3.2 · p. 5, Figure 2 · p. 5
• BeeS combines external and implicit margins as preference-probability signals and deprioritizes pairs with a low margin from any source.It projects margins into a shared probability space under a conditional-independence assumption, approximating each source probability with a clipped linear projection.Source: §3.2 · pp. 5–6
• One-pass DPO selects examples with the highest aggregated preference probabilities while excluding negative-margin samples; iterative DPO reapplies BeeS each iteration.For iterative DPO, only implicit reward models need training in the first iteration.Source: §3.2 · pp. 5–6

4.1 Experimental Setup

• Experiments cover TL;DR, HH, UltraFeedback, and on-policy UltraFeedback variants across Llama, Mistral, and Qwen models, using random, IFD, and reward-margin selection baselines.The evaluation uses AlpacaEval benchmarks for UltraFeedback and GPT-4 as a proxy for human judgment in three settings.Source: §4.1 · pp. 6–7

4.2 Win Rate Comparison with Baselines on Classic Preference Datasets

• Using 2–5% of selected data can surpass full-dataset DPO, and BeeS achieves optimal or near-optimal win rates across TL;DR, HH, and UltraFeedback.Baselines are weak on at least one task, with some margin-based strategies performing worse than random selection.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 AlpacaEval 2.0 Win Rate Comparison

• BeeS-selected 6,000-sample subsets achieve around 4% higher AlpacaEval 2.0 win rates than full-dataset training across four settings.BeeS subsets also show the fastest training-loss decrease and DPO-margin increase in the reported training dynamics.Source: Table 3 · p. 8, §4.3 · p. 8, Figure 3 · p. 8
• For iterative DPO, online data contains substantial redundancy, but selection remains important because the data includes ambiguous, low-margin samples.BeeS reserves 5,000 samples per iteration compared with 20,000 prompts used for on-policy preference-pair generation.Source: §4.3 · pp. 8–9, Figure 4 · p. 9
• With β set to 0.01, a BeeS-selected 3,000-sample subset achieves performance comparable to a full dataset 20 times larger.The authors report that relaxing the DPO-loss margin constraint improves data efficiency.Source: §4.3 · p. 9
• Unlike SimPO, β-DPO, and α-DPO, BeeS improves both raw and length-controlled win rates, while those variants primarily improve length-controlled win rate.Source: §4.3 · p. 9

4.4 Ablation Study

• On Mistral-7B and Qwen-2.5-7B/14B Instruct models, BeeS-selected 10% subsets outperform full-dataset DPO; Qwen-14B improves win rate by approximately 3%.Source: §4.4 · p. 9, Figure 5 · p. 10
• A 6,000-sample BeeS subset from Llama-UltraFeedback outperforms full-set training with IPO, KTO, and SLiC.Source: §4.4 · p. 9, Figure 5 · p. 10

5 Conclusion

• Using around 10% of the original training data, BeeS maintains or improves performance with consistent AlpacaEval2 improvements and extends successfully to iterative DPO.The paper presents these results as evidence for the importance of careful data curation in alignment.Source: §5 · p. 10

A Datasets and Evaluation Details

• TL;DR and HH test sets are sampled from their original testing pools, while AlpacaEval prompts serve as test sets for UltraFeedback models.Table 2 uses Text-Davinci-003 reference answers; other tables use GPT4-1106-preview answers.Source: A Datasets and Evaluation Details · p. 15
• Table 4 reports statistical information about the training and evaluation datasets used in the experiments.Source: Table 4 · p. 15
• Win-rate judgments are a key evaluation metric, using modified prompts for TL;DR and HH and original prompts for AlpacaEval and AlpacaEval 2.0.The TL;DR prompt judges summary precision and concision; the HH prompt judges helpfulness or harmlessness.Source: A Datasets and Evaluation Details · p. 15

A.1 More Implementation Details

• The Base-model SFT setup uses two epochs, a 2 × 10−5 learning rate, sample packing, and a block size of 4096.Source: §A.1 · p. 16
• Baseline P and N select samples from opposite ranking extremes, while Z randomly samples margins within metric-specific intervals after outlier removal.The Z intervals are [−τ, τ], with τ set to 0.1 for CPPL and 1.0 for External and Implicit rewards.Source: §A.1 · p. 16
• BeeS uses −2 as the lower bound for external and implicit reward margins and dynamically sets the upper bound using sample-count conditions.The upper bound criteria compare the count above U with 30 or with max margin − U.Source: §A.1 · p. 16
• Figure 6 shows abnormal responses from Llama-3-8B-Instruct and Mistral-7B-Instruct-V0.2 trained with DPO at β = 0.01.Source: Figure 6 · p. 16

B.1 Singular Margin Distribution

• Margins cluster around zero, indicating ambiguous preference labels that show slower loss decrease and margin increase than larger-margin samples.Source: §B.1 · p. 16

B.2 Joint Margin Distribution

• Figure 7 plots implicit reward-margin distributions on TL;DR, HH, and UltraFeedback using an SFT model and a DPO model trained on 2,000 randomly selected samples.Source: Figure 7 · p. 17
• Figure 8 plots external reward distributions on TL;DR, HH, and UltraFeedback using Skywork-Reward-Llama-3.1-8B-v0.2.Source: Figure 8 · p. 17
• External and implicit margins have minimal correlation across four datasets, while implicit margins from different models are highly correlated.The paper relates these patterns to BeeS’s fusion of external and implicit margins and separation of implicit margins from the target model.Source: §B.2 · p. 17

C.1 Train Loss and Margin Curves - 3B

• BeeS achieved even larger training margins than Implicit Margin-P, while P strategies reached higher margins than N and Z approaches.Both P-margin-oriented and N strategies rapidly reduced training loss; Z converged more slowly and ended with significantly higher loss.Source: §C.1 · p. 17

C.2 Resources and computation cost

• Training 8B models on UltraFeedback for two epochs took about 9 hours, while iterative-round generation and reward annotation took about 1.5 and 2 hours, respectively.Experiments used 8 A100 GPUs, with 4 GPUs allocated per SFT/DPO run and 4 GPUs for iterative generation and annotation.Source: §C.2 · p. 17
• Figure 9 plots conditional-perplexity margins, also called instruction-following-difficulty margins, on TL;DR, HH, and UltraFeedback using a Llama-3.2-3B SFT model.Figure 10 plots external and implicit reward-margin distributions across four datasets and compares implicit margins from 1B- and 3B-scale models on two datasets.Source: Figure 9 · p. 18, Figure 10 · p. 18

C.3 More Results for Ablation Study on the DPO Variants

• In a complementary study using Llama-3-8B-Instruct while maintaining all other experimental parameters, models trained on subsets selected by BeeS achieved significantly higher win rates across most evaluation scenarios.Source: §C.3 · p. 18

C.4 Hyperparameters Risks

• With β = 0.01 and two epochs, Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2 sometimes produced repeated or chaotic tokens, and their AlpacaEval 2.0 win rates fell by more than 10 points.The authors associate small β with a relaxed KL-divergence constraint that can permit excessive deviation from the initial policy.Source: §C.4 · p. 19
• During Mistral DPO training, chosen-sample log probabilities fell from -400 to -1400 and rejected-sample values from -600 to -4600; the drop was greater than for Llama.The decline coincided with the model's performance decline.Source: §C.4 · p. 19
• Reducing the learning rate from 5×10^-7 to 3×10^-7 or stopping after epoch 1 can produce a relatively smaller chosen/rejected log-probability drop.Source: §C.4 · p. 19

D Limitations.

• Evaluations focused on models up to 14B parameters, leaving BeeS's scalability and performance at 70B or beyond for future investigation.The authors say computational resources constrained the larger-scale experiments.Source: D Limitations · p. 20

• The paper reports that it discusses the work’s limitations.Source: §2 · p. 21
• The authors say implementation details are provided in the experimental section and appendix.Source: §2 · p. 22
• The authors did not repeat LLM experiments or report statistical-significance metrics, citing their cost and typical robustness.Source: §2 · p. 24