Source-linked AI summary
Preference Learning Algorithms Do Not Learn Preference Rankings
Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, Kyunghyun Cho
TL;DR
The relationship between preference-learning objectives, ranking accuracy, and win rate remains unclear. The paper analyzes RLHF and DPO theoretically and empirically, deriving idealized accuracy and examining the connection to win rate. Most tested preference-tuned models fall below 60% ranking accuracy, with observed accuracy below idealized levels and win-rate alignment dependent on closeness to the reference model.
Problem
The relationship between preference-learning objectives, ranking accuracy, and win rate is unclear, leaving open what these techniques accomplish during training.
Source: §1 · p. 2Method
Results
Most evaluated preference-tuned models achieve below 60% ranking accuracy, fall short of idealized accuracy, and track win rate only when close to the reference model.
Source: §1 · p. 2Takeaways & Limitations
Ranking accuracy and win rate trend together near the reference model but become anti-correlated after the model moves too far away.
Source: §1 · p. 2Empirical models were trained on a single dataset due to computational constraints, and the analysis describes behavior on training data rather than other distributions.
Source: §8 · p. 10
Abstract
from arXiv · showhide
Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs to produce generations that are more preferred by humans, but our understanding of their inner workings is still limited. In this work, we study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs, measured via ranking accuracy. Surprisingly, we find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets. We furthermore derive the idealized ranking accuracy that a preference-tuned LLM would achieve if it optimized the DPO or RLHF objective perfectly. We demonstrate that existing models exhibit a significant alignment gap -- i.e., a gap between the observed and idealized ranking accuracies. We attribute this discrepancy to the DPO objective, which is empirically and theoretically ill-suited to fix even mild ranking errors in the reference model, and derive a simple and efficient formula for quantifying the difficulty of learning a given preference datapoint. Finally, we demonstrate that ranking accuracy strongly correlates with the empirically popular win rate metric when the model is close to the reference model used in the objective, shedding further light on the differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO) preference learning algorithms.
1 Introduction
Preference learning is intended to rank preferred outputs above dispreferred ones, but the relationship between training objectives, ranking accuracy, and win rate remains unclear. The paper finds low observed accuracy, a gap from idealized accuracy, and difficulty correcting reference-model ranking errors.
1 Introduction
- Most evaluated open-access preference-tuned LLMs achieve below 60% ranking accuracy across validation splits from commonly used preference datasets.The authors study ranking accuracy as a lens into preference-learning algorithms, not as a measure of overall model quality.Source: §1 · p. 2
- Existing models show a significant alignment gap, achieving ranking accuracy far below the idealized accuracy under perfect optimization of DPO or RLHF.The idealized accuracy assumes ground-truth preference data and perfect optimization of the objective.Source: §1 · p. 2
- Even mild reference-model ranking errors can make DPO and its variants virtually unable to correct rankings, which are rarely flipped in practice.Reference-model likelihoods generally determine ranking accuracy, and the paper derives a formula for identifying hard-to-learn datapoints without tuning.Source: §1 · p. 2
- Ranking accuracy and win rate trend together when the model is close to its reference, but become anti-correlated after it moves too far away.This comparison adds evidence about differences between on-policy and off-policy behavior.Source: §1 · p. 2
2 Preliminaries
The preliminaries define preference data and outline the RLHF and DPO training pipeline, then distinguish ranking accuracy from other alignment evaluations. They also show that a low DPO loss guarantees correct ranking only under stringent conditions.
2.1 Learning from Human Preferences
- Aggregated preference datapoints pair a prompt with the majority-preferred and dispreferred continuations, but many datasets do not provide the underlying rater proportions.The paper notes that single-rater or already-aggregated datasets often leave α(x, y1, y2) unavailable.Source: §2.1 · p. 2
- RLHF samples continuations from a reference model, trains a reward model on preferences, then uses PPO to maximize reward while regularizing against the reference.The regularization coefficient β is intended to prevent the model from straying too far from its initialization.Source: §2.1 · p. 3
- DPO avoids PPO by reparameterizing the preference-learning objective to operate directly over policies rather than rewards.The DPO objective is defined for an aggregated preference dataset and a reference model.Source: §2.1 · p. 3
2.2 Evaluation Metrics
- Win rate compares how often a model produces a favorable completion against a baseline, while human ratings are costly and LLM judges can be biased.The paper compares this on-policy evaluation with ranking accuracy, an off-policy metric.Source: §2.2 · pp. 3–4
- Ranking accuracy measures whether the policy assigns higher likelihood to the preferred continuation, but completion lengths can affect unnormalized likelihood comparisons.The paper reports both unnormalized and length-normalized accuracy and does not advocate ranking accuracy as a measure of model quality.Source: §2.2 · p. 4
- If the reference already ranks the preferred continuation higher and the DPO loss is at most 0.6, the policy ranks it correctly; this condition is unlikely across all datapoints.The result depends on the reference model already having the correct ranking.Source: §2.2 · p. 4
3 The Alignment Gap
The section examines imperfect reference models and derives the ranking accuracy attainable under perfect DPO or RLHF optimization with ground-truth preferences. Comparing that idealized accuracy with observed models reveals a substantial alignment gap.
3.1 Existing Reference Models Rarely Have Correct Rankings
- Reference models rarely achieve high ranking accuracy on common preference datasets, and fine-tuning pretrained models on preferred completions does not significantly improve it.Synthetic Instruct GPT-J Pairwise is an exception; the authors note that preferred completions alone do not convey relative quality information.Source: §3.1 · p. 4, §3.2 · p. 4
3.2 Idealized Ranking Accuracy
- Theorem 3.1 characterizes the optimal DPO or RLHF policy under ground-truth preferences that follow the Bradley–Terry model.The resulting idealized accuracy can be computed from the reference model, β, and aggregated preference proportions.Source: §3.2 · p. 5
3.3 Measuring the Alignment Gap
- Existing models fall 19–59 percentage points short of median idealized ranking accuracy on the reported measures.Table 1 compares length-normalized and non-length-normalized accuracy for open-access models on Alpaca Farm validation data with idealized accuracies across β values.Source: §3.3 · p. 6, Table 1 · p. 6
- Even perfect optimization on ground-truth preference data can yield less than 100% idealized accuracy, with the limit depending on β and reliance on the reference model.The authors compute idealized accuracy for a range of β values to assess the gap between observed and attainable accuracy.Source: §3.3 · p. 6
4 Understanding Ranking Accuracy with DPO
The analysis finds that DPO often lowers its loss without correcting incorrect preference rankings. It explains this difficulty through the reference model’s log-ratio and characterizes datapoints for which flipping a ranking requires especially low loss.
4.1 DPO Rarely Flips Preference Rankings
- At the lowest validation loss, fewer than 10% of initially incorrectly ranked training examples have flipped to the correct ranking.DPO mostly increases reward margins on already-correct pairs; ranking accuracy improves substantially only well after overfitting begins.Source: §4.1 · p. 7, Figure 2 · p. 7
4.2 Analysis: How Easy Is It To Flip A Ranking?
- For a datapoint with reference-model log-ratio c, DPO achieves the correct ranking if and only if its loss is at most −log σ(βc).The threshold’s dependence on the reference model makes moderately incorrect reference likelihoods capable of making correction very difficult.Source: §4.2 · p. 7
- The reference model’s log-ratio determines how low DPO loss must fall to correct a ranking, and even mild reference-model errors can require very low loss.The analysis yields a simple way to identify hard-to-flip datapoints without tuning.Source: §4.2 · p. 7
- Hard-to-learn datapoints are substantially longer than easy ones, while easy datapoints generally have less ambiguous preference annotations.The authors suggest stronger reference models and iterative or on-policy DPO variants as responses to these difficulties.Source: §4.2 · p. 8
5 Ranking Accuracy and Win Rate
The section investigates how off-policy ranking accuracy relates to on-policy win rate during DPO training and under an objective that varies reference-model influence. The metrics track together near the reference model but can diverge as the model moves away.
5 Ranking Accuracy and Win Rate
- The study measures win rate on 500 responses to training prompts using the Alpaca Eval GPT-4 auto-annotator.It compares metrics during DPO training and across converged models trained with Lγ DPO at multiple γ values.Source: §5 · p. 8
- Ranking accuracy and win rate are highly correlated early in training but become anti-correlated as the model moves away from the reference model.With the attenuated objective, the metrics trend together when reference-model influence is stronger.Source: §5 · p. 9, Figure 4 · p. 9
- When the model moves farther from the reference, ranking accuracy can rise while win rate declines, even as test loss continues to fall.The authors speculate that over-optimizing reward margins may harm generative capabilities acquired during pre-training.Source: §5 · p. 9
6 Related Work
The related-work section distinguishes this paper’s ranking-accuracy analysis from prior studies of preference datasets, evaluation reliability, length effects, and preference-learning objectives.
6 Related Work
- Prior work studies preference datasets, evaluation reliability [69], response-length confounds, and taxonomies of preference-learning algorithms [2, 61, 48, 50, 63, 34].Source: §6 · p. 9
- Other analyses address why DPO lowers likelihoods of both chosen and rejected outputs and why RLHF has vanishing gradients [44].Source: §6 · p. 9
- This paper instead studies DPO and RLHF through ranking accuracy, emphasizing the role of reference-model regularization.Related objectives remove reference regularization or modify its strength, while other work studies that regularization’s role more broadly.Source: §6 · p. 9
- On-policy methods have been contrasted with more efficient off-policy methods, and iterative approaches refresh the reference model and preference data during alignment.The authors relate these approaches to their observation that win rate and ranking accuracy correlate when the model remains close to its reference.Source: §6 · pp. 9–10
7 Discussion
RLHF and DPO struggle to teach correct rankings even on training data, and the relationship between ranking accuracy and win rate depends on how far the model moves from its reference. The discussion connects poor rankings to potential safety feedback loops and notes that the theoretical results are limited to training data.
7 Discussion
- Ranking accuracy and win rate trend together early in alignment but become anti-correlated after the model moves too far from its reference.The paper also reports that win rate does not increase monotonically during training despite falling train and test loss and modest ranking-accuracy gains.Source: §7 · p. 10
- Poor ranking accuracy can create strong negative feedback loops when LLMs judge other models’ responses or improve their own abilities, and such loops are costly to mitigate.Source: §7 · p. 10
- Theoretical results describe behavior on training preference data, leaving generalization to other distributions, especially model-generated data, as future work.The authors also identify optimization dynamics and interactions with calibration metrics as topics for further study.Source: §7 · p. 10
8 Limitations
The limitations section confines the theoretical analysis to training data and notes that empirical models were trained on one dataset because of computational constraints. The appendix proves results for DPO and RLHF, characterizes idealized ranking accuracy, and gives a DPO-loss threshold for correct ranking under a fixed reference-model log-ratio.
8 Limitations
- Empirical models were trained on a single dataset due to computational constraints, and the analysis cannot rigorously support claims about other distributions or generative capabilities.The theoretical results describe behavior on training data; empirical verification across datasets remains valuable.Source: §8 · p. 10
A.1 Proof of Proposition 2.6
- If the reference model already assigns at least as much likelihood to the preferred response, DPO loss at most 0.6 guarantees the correct ranking.Source: §A.1 · p. 17
A.2 Proof of Theorem 3.1
- Under ground-truth preferences following the Bradley-Terry model, an adequately expressive policy that perfectly optimizes DPO or RLHF has a characterized optimal form.The theorem’s stated conditions include perfect DPO minimization or perfect PPO maximization for the optimal reward function.Source: §A.2 · p. 17
A.3 Proof of Corollary 3.3
- The optimum’s ranking accuracy is given as a function of the reference model, objective hyperparameter, preference dataset, and rater proportions.The passage also distinguishes idealized accuracy computed using length-normalized reference-model likelihoods.Source: §A.3 · pp. 18–19
A.4 Proof of Theorem 4.1
- For a datapoint with reference-model log-ratio c, the DPO ranking is correct if and only if its loss is at most −log σ(βc).Source: §A.4 · p. 19
A.5 Extending Theorem 4.1 to IPO
The appendix extends the DPO ranking analysis to IPO and defines ranking accuracy for prompts with more than two responses. It also describes likelihood-based evaluation, tie handling, length normalization, and the preference datasets used.
A.5 Extending Theorem 4.1 to IPO
- IPO’s regression target is the reference-model log-ratio plus a constant margin, which must overcome reference-model bias for perfect ranking accuracy.The paper notes that a per-example margin dependent on the reference-model log-ratio is an alternative.Source: §A.5 · p. 20
A.6 Generalization of the Ranking Accuracy for Preference Datasets with n > 2 Outputs
- For prompts with more than two responses, ranking accuracy is exact agreement between the aggregated social ranking and the policy’s likelihood-based ranking.The definition assumes an aggregated social ranking is already available.Source: §A.6 · p. 20
B.1 Implementation of Ranking Accuracy
- Ranking accuracy uses sequence likelihoods factorized into token likelihoods; ties count as correct when tied responses receive approximately equal likelihoods, within ϵ=0.01.The evaluation samples 1,000 examples per dataset and also reports length-normalized ranking accuracy.Source: §B.1 · pp. 20–21
B.2 Datasets
- The evaluated datasets span crowdworker preferences, synthetic pairs, community voting, and model-annotated responses, including HH-RLHF, UltraFeedback, and Stanford Human Preferences.Source: §B.2 · p. 21
B.3 Full Results
The appendix reports ranking-accuracy results across models and datasets, describes approximations used for idealized-accuracy calculations, and documents training details.
B.3 Full Results
- Full results report length-normalized and non-length-normalized ranking accuracies for 16 open-access models across six datasets, with tables covering HH-RLHF, Synthetic Instruct GPT-J Pairwise, StackExchange Preferences, UltraFeedback, SHP, and Alpaca Farm.The appendix also includes generalized accuracy for datasets with more than two outputs and results with ties included.Source: §B.3 · pp. 21–22, Table 2 · p. 22, Table 3 · p. 22
B.4 Computation of the Idealized Ranking Accuracy
- Idealized-accuracy calculations use Alpaca Farm validation because most datasets lack individual rater proportions, add ϵ = 0.001 for unanimous votes, and report results across five β values.For unknown β values, the appendix reports the minimum, median, and maximum.Source: §B.4 · p. 23
C.1 Training Details
- Training experiments used GPT2, Pythia 2.8B, and Llama 2 7B across three seeds, with hyperparameters tuned on one seed and reused for the other seeds of each model class.Llama 2 7B training was limited to one epoch due to computational constraints; GPT2 and Pythia 2.8B were trained for five epochs.Source: §C.1 · p. 24
C.2 Results
The appendix reports training trends across nine models and additional plots examining ranking accuracy and win rate.
C.2 Results
- DPO loss, reward margin, and dataset trends are reported across nine models: three seeds each of GPT2, Pythia 2.8B, and Llama 2 8B.The trends appear in Figures 5, 6, and 7.Source: §C.2 · p. 24
- Figure 5 tracks average DPO loss across four data categories defined by reference-model and current-policy ranking correctness; lines end when a category has no examples.Its dashed vertical line marks the step with the lowest validation loss.Source: Figure 5 · p. 25
D Ranking Accuracy and Win Rate
- Additional plots explore the relationship between ranking accuracy and win rate.These plots are Figures 9 and 10.Source: D Ranking Accuracy and Win Rate · p. 25
D.1 Results on the Test Set
The appendix also reports test-set results in Figures 11 and 12.
D.1 Results on the Test Set
- Test-set results are presented in Figures 11 and 12.Source: §D.1 · p. 25
E Qualitative Analysis
This section qualitatively examines how DPO handles reference-model ranking errors and how ranking accuracy relates to win rate during training. It also reports the paper’s use and release of assets and whether the work involved crowdsourcing or human-subject research.
E Qualitative Analysis
- Large positive reference-model log-ratios require the DPO loss to become very small before DPO flips an incorrect ranking.This difficulty is established by Theorem 4.1 and illustrated in Figure 3.Source: E Qualitative Analysis · p. 25
- In Pythia-2.8B examples from Anthropic HH-RLHF, hard points have large RMLR, easy points very negative RMLR, and easy-to-flip errors small positive RMLR.Hard-to-learn datapoints are substantially longer; easy examples generally have chosen responses that are unambiguously better than rejected ones.Source: E Qualitative Analysis · pp. 25–26, Table 7 · p. 30, Table 8 · p. 33
- Ranking accuracy and win rate trend together only early in DPO training, then become anti-correlated even as training and test loss continue to decline.The measurements cover training through the checkpoint with the lowest validation loss.Source: Figure 9 · p. 28
- For γ ≥1.25, ranking accuracy and win rate trend in the same direction when the influence of πRef is stronger.The passage states that higher γ implies greater influence of πRef during training.Source: Figure 10 · p. 29
2. Limitations
- The authors say they cited all open-access models and datasets used and released no new assets.The checklist marks documentation of new assets as not applicable because none were released.Source: §2 · pp. 39–40
- The authors report that their work involved neither crowdsourcing nor research with human subjects.The paper marks both the crowdsourcing and IRB questions as not applicable.Source: §2 · p. 40