A General Theoretical Paradigm to Understand Learning from Human Preferences

Mohammad Gheshlaghi Azar; Mark Rowland; Bilal Piot; Daniel Guo; Daniele Calandriello; Michal Valko; Rémi Munos

Source-linked AI summary

A General Theoretical Paradigm to Understand Learning from Human Preferences

Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos

arXiv:2310.12036v2Oct 18, 2023cs.AIcs.LGstat.ML

TL;DR

The paper addresses the theoretical gap and approximation assumptions underlying practical preference learning methods such as RLHF and DPO. It introduces ΨPO, a pairwise-preference objective that contains both methods as special cases, and studies IPO as an identity-mapping instantiation. The authors report that IPO avoids the Bradley-Terry assumption, admits an efficient theoretically justified loss, and is better suited to sampled preferences in illustrative examples.

Problem
Existing practical preference-learning methods lack theoretical foundations for their RLHF, DPO, and sampled-preference behavior, while RLHF and DPO rely on pointwise-reward modeling assumptions.
Method
The paper introduces ΨPO, a general pairwise-preference objective unifying RLHF and DPO, then derives IPO by setting Ψ to the identity mapping.
Results
IPO learns directly from preferences without reward modeling or the Bradley-Terry assumption, with an efficient sampled loss and illustrative examples showing it better suited to sampled preferences than DPO.
Takeaways & Limitations
The ΨPO framework enables analysis of RLHF and DPO pitfalls, while IPO provides a simple, theoretically justified alternative for preference learning.

Abstract

from arXiv · show

The prevalent deployment of learning from human preferences through reinforcement learning (RLHF) relies on two important approximations: the first assumes that pairwise preferences can be substituted with pointwise rewards. The second assumes that a reward model trained on these pointwise rewards can generalize from collected data to out-of-distribution data sampled by the policy. Recently, Direct Preference Optimisation (DPO) has been proposed as an approach that bypasses the second approximation and learn directly a policy from collected data without the reward modelling stage. However, this method still heavily relies on the first approximation. In this paper we try to gain a deeper theoretical understanding of these practical algorithms. In particular we derive a new general objective called $Ψ$PO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations. This new general objective allows us to perform an in-depth analysis of the behavior of RLHF and DPO (as special cases of $Ψ$PO) and to identify their potential pitfalls. We then consider another special case for $Ψ$PO by setting $Ψ$ simply to Identity, for which we can derive an efficient optimisation procedure, prove performance guarantees and demonstrate its empirical superiority to DPO on some illustrative examples.

1 Introduction

Learning from human preferences aligns generative language models with human desiderata, but practical methods lack a unified theory and rely on important approximations. The paper introduces ΨPO to represent RLHF and DPO directly through pairwise preferences and analyzes their vulnerabilities.

Motivation: Learning from human preferences collects contexts, paired generations, and human choices before learning a policy that produces preferred generations.The problem is framed as an offline contextual bandit with a reference-policy constraint, often enforced through KL regularisation to avoid model drift.
Existing approaches: RLHF trains a classifier-based reward model and then optimizes a policy to maximize that learned reward while staying close to a reference policy.This two-stage approach has been used successfully for aligning generative language models with human preferences.
Existing approaches: Practical preference-optimization methods are competitive with state-of-the-art RLHF while being simpler to implement and requiring fewer resources.
Theory gap: Existing theory mainly provides regret guarantees for standard bandits rather than addressing practical RLHF, DPO, and SLiC-HF settings.
Paper contribution: ΨPO characterizes RLHF and DPO as special cases of a pairwise-preference objective and reveals that both can be vulnerable to overfitting.The analysis links this vulnerability to substituting pairwise preferences with pointwise rewards through a Bradley-Terry model, especially for deterministic or nearly deterministic preferences.

2 Notations

The paper formulates preference learning over finite contexts and actions, with human comparisons generated from a reference policy. It defines true and empirical preference quantities used throughout the analysis.

Setting: A policy assigns each context a discrete distribution over a finite action space, while the behavior policy generates compared actions independently.
Setting: Human raters compare two generated actions and identify the preferred and dispreferred generations.
Preference quantities: True pairwise preference is the probability that a human prefers one action over another given the context.The probability averages an indicator over the humans providing ratings.
Preference quantities: Expected preference of an action over a distribution is defined by averaging its pairwise preference against an independently sampled action from that distribution.
Empirical data: Observed preferences are Bernoulli samples with means given by the corresponding true pairwise preferences and are collected in a rated-generations dataset.
Empirical data: The notation distinguishes expectations under distributions from empirical expectations computed over the finite dataset.

3 Background

RLHF learns a Bradley-Terry reward model and optimizes a KL-regularized policy, while DPO avoids explicit reward modeling. Under Bradley-Terry assumptions and an optimal reward, their global optima coincide.

RLHF: RLHF consists of learning a reward model followed by policy optimization using the learned reward.
RLHF: The reward model is a binary classifier trained with logistic regression to distinguish preferred from dispreferred actions.
RLHF: The Bradley-Terry model represents pairwise preference as a sigmoid of the difference between pointwise action rewards.
RLHF: With Bradley-Terry-conforming preferences, the learned preference estimate converges to the true preference as the dataset grows.
Policy optimization: RLHF optimizes expected reward while minimizing KL divergence between the learned policy and a reference policy.The objective samples contexts from ρ and actions from the policy; PPO or similar methods optimize it.
DPO: DPO avoids training a reward model and directly optimizes a loss over the empirical preference dataset.
RLHF–DPO relationship: Under a perfectly fitting Bradley-Terry model and an optimal learned reward, RLHF and DPO have identical global optimizers.

4 A General Objective for Preference Optimisation

The paper introduces ΨPO, a preference-based objective that balances a non-linear transformation of pairwise preferences with KL regularisation, and shows that RLHF and DPO arise as special cases under the Bradley–Terry assumption. Its analysis identifies weak regularisation and overfitting risks, motivating a modification that can remain close to the reference policy with deterministic preferences.

General objective: ΨPO maximizes a non-decreasing transformation of pairwise preference probabilities while regularising the policy toward a reference policy with KL divergence.The objective is designed as a general representation of preference optimisation and strictly generalises RLHF and DPO when the Bradley–Terry model holds.
Connections to RLHF and DPO: When Ψ(q) = log(q/(1 −q)) and the Bradley–Terry model holds, ΨPO, RLHF, and standard DPO have identical optimal policies.The equivalence follows because the transformed preference objective corresponds to the RLHF reward up to an additive constant.
Connections to RLHF and DPO: Under the Bradley–Terry assumption, the closed-form solution for DPO and RLHF follows from the analytical solution of the corresponding ΨPO objective.The paper notes that the derivation is a known result and supplies it for completeness.
Weak regularisation and overfitting: Non-linear preference transformations can incentivize increases near preference probability 1 as strongly as larger increases around 50%.The paper notes that maximising logit-preferences can therefore produce counter-intuitive effects even with transitive preferences.
Weak regularisation and overfitting: With deterministic preferences, the Bradley–Terry reward difference diverges, causing the optimal policy to assign zero probability to the dispreferred action regardless of KL regularisation.The paper concludes that KL regularisation becomes weaker as preferences become more deterministic.
Weak regularisation and overfitting: Finite-sample estimates can equal 1 even when true preference probability is 0.8, leading to zero probability for the dispreferred action and substantial overfitting in large spaces.The paper contrasts this with RLHF, where reward-function underfitting can preserve regularisation toward the reference policy; it proposes modifying ΨPO to address the issue.

5 IPO: ΨPO with identity mapping

IPO is the identity-mapping case of ΨPO, yielding a directly trainable preference objective with a unique optimum and avoiding DPO’s tendency toward greedy policies.

IPO instantiates ΨPO with the identity mapping and is designed to avoid DPO’s overfitting caused by unbounded Ψ and the absence of explicit reward modeling.The identity choice produces direct regularized optimisation of total preferences.
The IPO objective can be transformed into an offline sampled loss by converting its root-finding formulation into a single optimisation problem.The sampled loss uses preference observations represented as Bernoulli samples and is equivalent to the population loss up to a policy-independent additive constant.
5.1 Derivations and Computationally Efficient Algorithm: The loss is a positive-semidefinite quadratic in logits, so every local minimizer is global; after accounting for logit-shift invariance, the policy optimum is unique.The same uniqueness result is stated directly for policies under equal support assumptions.
5.2 Sampled Loss for IPO: IPO regresses the gap between policy and reference-policy log-likelihood ratios using observed pairwise preferences.This gives a simple preference-dataset loss without requiring a separate reward-model stage.
5.3 Fully Known Preferences: In a two-action example, IPO approaches the reference policy as τ grows and the deterministic preferred policy as τ approaches zero.DPO instead converges to the deterministic policy regardless of τ in the same deterministic-preference setting.
5.4 Sampled Preferences: On sampled-preference examples, DPO ignores the reference policy and becomes deterministic, while IPO avoids greediness under strong regularisation and decreases unobserved-action probability as τ increases.These behaviors are reported for both the total-order dataset D1 and the partially unobserved dataset D3.

6 Conclusion and Future Work

The paper presents ΨPO as a unified pairwise-preference objective and IPO as an implementable identity-mapping case that avoids reward modeling and the Bradley–Terry assumption. Illustrative experiments report DPO instabilities and conclude that IPO is better suited to sampled preferences, while larger-scale evaluation remains future work.

ΨPO unifies RLHF and DPO as special cases of a general objective expressed directly through pairwise preferences.
IPO learns directly from preferences without reward modeling or the assumption that pairwise preferences can be substituted with pointwise rewards.
The paper derives a theoretically justified, simple sampled IPO loss and uses illustrative examples to expose DPO instabilities with fully known and sampled preferences.
The authors conclude that IPO is better suited to sampled preferences, while identifying scaling experiments on more complex language-model settings as future work.

A.1 Existence and uniqueness of the regularized argmaximum

The regularized criterion has a softmax optimizer under the stated finite-set and positive-temperature definitions. Uniqueness can fail when the policy’s support exceeds the reference distribution’s support, while coinciding supports restore uniqueness.

Definitions: The criterion is formulated for a finite set S, a real-valued function f, positive τ, and discrete probability distributions δ and η.The distributions are elements of ∆S, with τ a strictly positive real number.
Existence and uniqueness: Under the stated definitions, the softmax probability δ∗ shares the argmaximum of the regularized criterion.The proof identifies −KL(δ || δ∗) and Lτ(δ) as having the same argmaximum when the remaining terms are constant and τ is positive.
Non-uniqueness: If the support of π is strictly larger than the support of µ, multiple solutions can arise because µ does not constrain the whole action space.The example uses three actions, a uniform reference policy, and a policy assigning zero probability to y3.
Non-uniqueness: Assuming that the supports of πref and µ coincide restores uniqueness of the solution.The paper states that this support condition enables recovery of uniqueness, as proven in Theorem 2.

B Additional results

The additional results establish a broad equivalence between DPO and RLHF through the Bradley-Terry reward objective. Their optimal policies coincide whenever the Bradley-Terry loss has a minimizer, even when the preference model is not Bradley-Terry.

DPO–RLHF equivalence: DPO and RLHF have identical optimal policies when the Bradley-Terry loss admits a minimizer, regardless of the preference model’s form.The equivalence therefore does not require the preference model p∗ to correspond to a Bradley-Terry model.
Objective correspondence: The Bradley-Terry reward objective evaluated at r equals the DPO objective evaluated at its corresponding policy π∗r.This correspondence links reward optimization to direct policy optimization.
Proof strategy: If a reward-optimal policy were not DPO-optimal, a policy with lower DPO loss would induce a reward with lower Bradley-Terry loss, yielding a contradiction.The construction uses r′(x, y) = τ log(π′(y|x)/πref(y|x)).
Proof strategy: Conversely, a DPO-optimal policy induces an optimal reward, and the corresponding RLHF optimizer is the same policy.The induced reward is r(x, y) = τ log(π∗(y|x)/πref(y|x)).