Source-linked AI summary
AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
Peng Lai, He Zhu, Zhiwen Ruan, Dongdong Zhang, Yun Chen, Peng Li, Furu Wei, Yang Liu, Guanhua Chen
TL;DR
Preference datasets can contain noisy or ambiguous pairs, while selecting only high-margin examples may produce the squeezing effect and ambiguous outputs. AlignDiff uses intrinsic positive and inverse preference signals to filter for clear polarity, then applies difficulty-aware selection through the average negative log-likelihood gap. Across LLaMA and Qwen backbones and three alignment benchmarks, it consistently outperforms strong baselines, with further gains from difficulty-based curriculum learning.
Problem
Preference-data noise and high-margin selection can limit alignment quality by retaining ambiguous signals or shrinking high-likelihood regions toward ambiguous outputs.
Method
AlignDiff uses RAD with positive and inverse implicit reward margins to filter polarity-clear pairs, then uses ANG to prioritize harder informative pairs.
Results
AlignDiff consistently outperforms strong preference-data selection baselines across LLaMA and Qwen backbones on AlpacaEval 2.0, Arena-Hard, and MT-Bench.
Takeaways & Limitations
Intrinsic model signals support preference-data filtering, and difficulty-based curriculum learning further improves performance on the filtered data.
Takeaways & Limitations
Experiments use one preference dataset and 7–8B-scale base models, leaving scalability to larger models and additional corpora open.
Abstract
from arXiv · showhide
Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective alignment. Existing datasets are frequently plagued by inherent noise and distribution shifts, which inherently limit model performance. To bridge this gap, we propose AlignDiff, a preference data filtering framework driven by intrinsic model signals. AlignDiff first identifies samples with clear preferences using both positive and inverse signals, then prioritizes the more challenging samples based on the average negative log-likelihood gap, encouraging the model to learn richer information from them. AlignDiff is evaluated on two widely used model families (LLaMA and Qwen) and three benchmarks widely adopted in the alignment community (AlpacaEval 2.0, Arena-Hard, and MT-Bench). Across all settings, it consistently outperforms seven strong baselines. We conduct comprehensive ablation studies to validate the effectiveness of AlignDiff, and further show that difficulty-based curriculum learning improves model performance.
1 Introduction
Preference-data quality is central to effective LLM alignment, but large datasets can contain noisy or ambiguous pairs and promote the squeezing effect. AlignDiff addresses this by filtering for clear preference polarity with intrinsic signals, then prioritizing harder samples for learning.
- Preference data supplies core signals for DPO and supports subsequent alignment-method improvements.
- Large-scale preference training can shrink high-likelihood regions for chosen and rejected responses, biasing models toward ambiguous outputs.
- AlignDiff uses intrinsic model signals to separate preference clarity from sample difficulty during data selection.RAD identifies reliable preference directions, while ANG prioritizes informative samples after polarity is established.
- Stage 1 uses positive and inverse implicit reward margins to identify clear preferences, while Stage 2 prioritizes more challenging pairs.
- 6.3 and 3.2 points are AlignDiff’s reported AlpacaEval 2.0 length-controlled win-rate advantages over SDPO on LLaMA-3-8B-SFT and Qwen2.5-7B-SFT, respectively.The comparison is reported with comparable response lengths; AlignDiff also achieves the strongest overall performance on Arena-Hard and MT-Bench across both backbones.
2 Background
RLHF aligns models through learned rewards and policy optimization, whereas DPO uses model-derived implicit rewards to optimize preference comparisons directly. AlignDiff builds on these signals by combining positive and inverse evidence for polarity filtering and difficulty-aware selection.
- Positive and inverse implicit reward margins provide complementary information because they do not exhibit a clear inverse correlation.
- RLHF trains a reward model on pairwise preferences and refines a policy with reinforcement learning under a KL constraint.
- DPO removes the need for an explicit reward model by defining implicit rewards from the policy and reference models.
- AlignDiff’s two stages use RAD to retain polarity-clear pairs and ANG to rank informative hard pairs while mitigating the squeezing effect.
- A larger implicit reward margin indicates clearer preference signals and is associated with simpler optimization, greater stability, and faster convergence.
3 AlignDiff
AlignDiff selects preference data in two stages: first filtering for clear polarity using positive and inverse model signals, then ranking polarity-clear pairs by relative learning difficulty. Its RAD and ANG criteria retain consistent, informative pairs while avoiding ambiguous or overly easy examples.
- AlignDiff: AlignDiff separates preference-signal clarity from sample difficulty when selecting a high-quality preference dataset.Stage 1 asks whether a pair has clear polarity; Stage 2 asks which polarity-clear pairs are most informative to learn from.
- Mining Clearer Preference Samples via Alignment Discrepancy: RAD compares positive and inverse implicit reward margins to identify clear preference polarity and detect anomalous or inconsistent pairs.The method exploits DPO’s symmetry by training or evaluating positive and inverse preference signals.
- Mining Clearer Preference Samples via Alignment Discrepancy: RAD discards ambiguous pairs, retains clear positive or inverse pairs, and can reorder responses when the model’s internal polarity conflicts with the original annotation.The label ϕ = 0 denotes ambiguity, while ϕ = −1 and ϕ = 1 denote clear inverse and positive preferences, respectively.
- Sample Difficulty-Aware Dataset Calibration: ANG measures pairwise difficulty through the average negative log-likelihood gap, favoring pairs with a hard chosen response and an easier rejected response.ANG is designed to be less sensitive to response length than perplexity and reflects relative contrast rather than absolute response quality.
- Sample Difficulty-Aware Dataset Calibration: A larger positive ANG indicates a more informative pair, so AlignDiff retains the top-K% polarity-clear pairs with the largest ANG values.Smaller or negative ANG indicates an already easy pair with limited learning signal and higher squeezing risk.
4 Experiments
Experiments evaluate AlignDiff against multiple filtering baselines on two SFT backbones and standard alignment benchmarks. Results support its two-stage design: RAD improves preference polarity selection, while ANG-based difficulty selection and the full framework improve alignment performance.
- Experimental Settings: Experiments compare 30k-sample methods under identical settings across LLaMA-3-8B-SFT and Qwen2.5-7B-SFT.The evaluation uses UltraFeedback_Binarized and AlpacaEval 2.0, Arena-Hard, and MT-Bench settings described in the paper.
- Main Results: IM-filtered data consistently outperforms EM-filtered data across all three benchmarks and both backbones.Adding EM to IM does not produce a clear gain and lowers AlpacaEval performance in the reported examples.
- Main Results: AlignDiff exceeds SDPO by 6.3 and 3.2 AlpacaEval length-controlled win-rate points on LLaMA-3-8B-SFT and Qwen2.5-7B-SFT, respectively.It also improves Arena-Hard win rates over SDPO by 1.7 and 3.4 points on the two backbones.
- Hyperparameter and Ablation Studies: RAD outperforms PIM, IIM, and EM for first-stage filtering, indicating that positive and inverse implicit margins provide richer polarity signals.The comparison trains DPO on each resulting Qwen dataset.
- Hyperparameter and Ablation Studies: Difficulty-based calibration outperforms easy and random calibration, with easy-sample training trailing difficult-sample training by 9.7%.The ablation attributes the improvement to richer learning signals from challenging samples.
- Hyperparameter and Ablation Studies: Performance improves progressively when RAD filtering, reversal, and ANG-based difficulty selection are added, demonstrating complementarity between polarity clarity and difficulty-aware selection.The full framework combines the two filtering stages rather than treating them as interchangeable alternatives.
5 Analysis
Analysis examines training dynamics, filtering comparisons, curriculum ordering, and efficiency. The findings indicate that RAD stabilizes optimization, difficult-sample calibration improves learning, easy-to-hard ordering is best among tested curricula, and AlignDiff balances efficiency with performance.
- Training Dynamics and Effectiveness: RAD makes training more stable, while RAD-only can reach lower loss yet underperform the full framework because of the squeezing effect.ANG-only mitigates squeezing but struggles to converge without polarity filtering.
- Training Dynamics and Effectiveness: RAD-based filtering outperforms PIM, IIM, and EM, while difficult-sample calibration outperforms easy and random baselines.These comparisons separate the contributions of polarity filtering and difficulty-aware calibration.
- Difficulty-based Curriculum Learning: 36.40/36.89 is the best reported LC/WR result for easy-to-hard ordering, versus 33.38/33.81 for random and 27.72/27.95 for hard-to-easy ordering.The experiment uses the final 30k samples selected by AlignDiff on Qwen2.5-8B-SFT.
- Training Dynamics and Effectiveness: The filtered-data model produces a tighter, left-shifted distribution of average negative log-likelihoods.This distributional result accompanies the paper’s analysis of filtering and training behavior.
- Computational Efficiency Comparison: AlignDiff achieves a better efficiency–performance balance than EM and SDPO, although its efficiency is lower than IM’s.Efficiency is computed as average win rate per GPU hour, and EM incurs substantial scoring cost from Qwen2.5-72B.
6 A Close Look at the Filtered High-Quality Data
AlignDiff’s filtered data preserves response-length relationships while improving preference-margin quality and removing ambiguous samples. Its selections overlap substantially with internal-signal methods but less with external-model filtering.
- Length and reward distributions: The filtered dataset closely matches the original length-gap distribution, indicating that improved data quality is not attributable to longer responses.This pattern is reported in Figure 7a and is consistent with the paragraph-level analysis.
- Length and reward distributions: Filtering removes primarily ambiguous samples under external rewards and shifts the external reward-margin distribution slightly rightward.The length-gap distribution remains nearly unchanged while the external reward margins are optimized.
- Method-level overlap: AlignDiff overlaps with IM by 67.4% and SDPO by 63.2%, reflecting their shared use of internal model signals.Despite this overlap, AlignDiff identifies additional representative and higher-quality data.
- Method-level overlap: AlignDiff overlaps with EM by 38.4%, substantially less than with IM or SDPO because EM uses external models for selection.The cited passage frames this as a method-level contrast in selection signals.
7 Conclusion
AlignDiff filters preference data using model-internal signals in two stages: it removes ambiguous or conflicting pairs, then selects informative pairs using difficulty. Experiments find consistent gains across models and benchmarks, while difficulty-based curriculum learning provides an additional benefit.
- Framework: AlignDiff is a two-stage preference-data filtering framework based entirely on intrinsic model signals.The stages target preference clarity and sample informativeness.
- Framework: Bidirectional alignment discrepancies use positive and inverse preference signals to remove ambiguous or conflicting samples.The method separates reliable preference direction from sample difficulty.
- Framework: Average negative log-likelihood gaps refine selection by identifying high-quality, informative preference pairs.This second-stage criterion is intended to prioritize samples that remain useful for model learning.
- Findings: Across multiple base LLMs and benchmarks, AlignDiff consistently outperforms strong preference-data selection baselines while maintaining competitive response lengths.The conclusion reports this as the overall experimental pattern.
- Findings: Difficulty-based curriculum learning further benefits the filtered data.The conclusion identifies curriculum ordering as an additional improvement beyond filtering.
Limitation
The study’s evidence is limited by model, dataset, and selection-signal scope. AlignDiff may require model-specific tuning, and filtering can discard subtle signals while its scalability remains untested.
- Scope and transferability: Intrinsic signals may vary with model scale and architecture, requiring corresponding hyperparameter adjustments.This constrains direct transfer of the same settings across models.
- Data coverage: Filtering inevitably discards part of the training data, potentially excluding valuable but subtle preference signals.The limitation concerns information lost through selective retention.
- Evaluation scope: Experiments use one preference dataset, UltraFeedback_Binarized, and 7–8B-scale base models because of computational constraints.The reported evaluation scope does not establish behavior on larger models or additional preference corpora.
- Evaluation scope: The scalability of AlignDiff to larger models and additional preference corpora remains an open question.The paper identifies broader-scale validation as future work.
C Additional Experimental Results
Multi-judge evaluation confirms AlignDiff’s performance across base models and metrics, while downstream tasks show similar results across filtering strategies at comparable data scales. The appendix reports these additional evaluations and comparisons.
- Multi-judge evaluation: AlignDiff achieves the best performance across both base models and all reported metrics under multi-judge evaluation.The evaluation uses multiple judges to address concerns about dependence on a single evaluator.
- Multi-judge evaluation: On Qwen2.5-7B-SFT, AlignDiff leads with 38.8% LC, 39.1% WR, and 45.3% Arena-Hard WR.The same passage reports gains over SDPO of 2.1%, 4.2%, and 1.9% absolute points.
- Downstream tasks: At comparable data scales, different filtering strategies yield similar downstream-task performance, while AlignDiff remains competitive.The downstream tasks include GPQA, Toxigen, TruthfulQA, MMLU, and Winogrande.
E.1 Evaluation Datasets
The evaluation uses AlpacaEval 2.0, Arena-Hard, and MT-Bench to assess DPO-trained models across instruction following, diverse challenging tasks, and multi-turn dialogue. AlpacaEval comparisons use GPT-4-1106-Preview, Arena-Hard uses GPT-4o-mini, and MT-Bench uses GPT-4 Turbo as judges.
- AlpacaEval 2.0: AlpacaEval 2.0 compares DPO-trained model outputs with GPT-4-1106-Preview on 805 test examples.
- Arena-Hard: Arena-Hard evaluates language-model performance across natural-language understanding, reasoning, and generation using GPT-4o-mini as judge.
- MT-Bench: MT-Bench evaluates large language models in multi-turn dialogues using FastChat and GPT-4 Turbo as the judge.
- Evaluation setup: The experiments replace AlpacaEval 2.0’s expensive official gpt4_turbo annotator with deepseek-v3-0324, which is reported as more cost-effective and stronger on several annotator metrics.The comparison includes human consistency, Spearman correlation, and Pearson correlation.
F Details of DPO Training
The baseline procedures filter preference pairs using external scores, implicit or explicit reward margins, alignment potential, response length, or perplexity-based difficulty. These methods select pairs judged clearer or higher quality under their respective signals.
- External Reward Margin: External Reward Margin scores chosen and rejected responses across helpfulness, instruction following, honesty, and truthfulness with Qwen2.5-72B-Instruct.Scores are assigned from 1 to 9 and aggregated using model probabilities.
- External Reward Margin: Samples with the largest external score differences are selected as high-quality preference examples.
- PPLGAP: PPLGAP selects pairs with the highest perplexity gaps, where larger values indicate that the chosen response is much more likely under the model than the rejected response.
- Implicit Reward Margin: Implicit Reward Margin uses SFT reference models to compute margins and selects samples with the highest values as high-quality examples.The selected pairs are intended to reflect stronger alignment between model and human preferences.
- MAP: MAP ranks preference pairs by the gap between explicit reward and implicit reward margins, treating larger gaps as greater alignment potential.The method selects the top-scoring pairs as training data.
- R.I.P.: R.I.P. retains samples with positive external reward margins and the longest rejected responses, using an external-margin threshold of 0.126.
I Calculation of GPU Hours for the Methods
The methods differ substantially in estimated GPU-hour requirements, with AlignDiff costing less than EM and SDPO but more than IM. The estimates include training, scoring, and reward-margin computations where applicable.
- Scoring setup: The external-scoring setup evaluates helpfulness, instruction following, honesty, and truthfulness on a 1-to-9 scale using a structured prompt template.
- Method costs: External Margin requires an estimated 128 GPUh for Qwen2.5-72B-Instruct scoring across four dimensions.
- Method costs: Implicit Margin costs an estimated 43 GPUh, combining forward DPO training with implicit reward-margin computation.
- Method costs: AlignDiff costs an estimated 80.5 GPUh, including forward and reverse DPO training plus reward-margin and NLL computations.
- Method costs: SDPO has the highest estimated cost at 162 GPUh because it trains six reference models and computes margins for each.