PaperlayerView exact revision on arXiv

Source-linked AI summary

Provably Robust DPO: Aligning Language Models with Noisy Feedback

Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan

arXiv:2403.00409v2cs.LGcs.CL

TL;DR

Noisy preferences and limited theory for practical mitigation methods complicate DPO’s reliance on preference data. The paper introduces rDPO, a flip-rate-adjusted loss, and proves a policy error bound under log-linear policies and SFT feature coverage, with noise increasing the bound by a factor of O(1/(1-2ε)).

  • Problem

    Theoretical understanding of practical methods for learning preferences under random label flips remains limited, despite noisy preference data and mitigation heuristics.

    Source: §1 · p. 2

  • Method

    rDPO adapts DPO’s binary cross-entropy loss to the flip rate, yielding an unbiased estimate of the original loss that de-biases preference noise on average.

    Source: §1 · p. 2

  • Results

    Under log-linear policies and good SFT feature coverage, the policy suboptimality bound has an additional multiplicative noise cost of O(1/(1-2ε)).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework also gives performance bounds for noise-free DPO when ε = 0, addressing a theoretical gap for practical preference-learning algorithms.

    Source: §1 · p. 2

    The stated estimation-error result does not yield vanishing error for neural policies; its guarantee relies on the log-linear setting.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Learning from preference-based feedback has recently gained traction as a promising approach to align language models with human interests. While these aligned generative models have demonstrated impressive capabilities across various tasks, their dependence on high-quality human preference data poses a bottleneck in practical applications. Specifically, noisy (incorrect and ambiguous) preference pairs in the dataset might restrict the language models from capturing human intent accurately. While practitioners have recently proposed heuristics to mitigate the effect of noisy preferences, a complete theoretical understanding of their workings remain elusive. In this work, we aim to bridge this gap by by introducing a general framework for policy optimization in the presence of random preference flips. We focus on the direct preference optimization (DPO) algorithm in particular since it assumes that preferences adhere to the Bradley-Terry-Luce (BTL) model, raising concerns about the impact of noisy data on the learned policy. We design a novel loss function, which de-bias the effect of noise on average, making a policy trained by minimizing that loss robust to the noise. Under log-linear parameterization of the policy class and assuming good feature coverage of the SFT policy, we prove that the sub-optimality gap of the proposed robust DPO (rDPO) policy compared to the optimal policy is of the order $O(\frac{1}{1-2\epsilon}\sqrt{\frac{d}{n}})$, where $\epsilon < 1/2$ is flip rate of labels, $d$ is policy parameter dimension and $n$ is size of dataset. Our experiments on IMDb sentiment generation and Anthropic's helpful-harmless dataset show that rDPO is robust to noise in preference labels compared to vanilla DPO and other heuristics proposed by practitioners.

1. Introduction

Noisy preference data can undermine preference-based policy training, and existing mitigation heuristics lack a full theoretical account. The paper develops a framework for random preference flips, introducing rDPO and analyzing its guarantees and empirical robustness.

1. Introduction

  • DPO avoids RLHF’s separate reward model and RL training, but noisy or ambiguous preferences can still degrade performance at high noise rates.The paper addresses limited theory for heuristics that mitigate noisy preference data.Source: §1 · pp. 1–2
  • rDPO adapts DPO’s binary cross-entropy loss to the flip rate, making it an unbiased estimate of the original loss and tuning gradient importance weights to noise.The framework also generalizes to reward training in RLHF and other preference optimization methods.Source: §1 · p. 2
  • Under log-linear policies and good SFT feature coverage, the analysis bounds rDPO’s policy suboptimality, with preference flips adding a multiplicative O(1/(1-2ε)) cost.Setting ε = 0 also gives performance bounds for DPO without label noise.Source: §1 · p. 2
  • rDPO is robust to noisy preference labels on IMDb sentiment generation and Anthropic’s helpful-harmless data, outperforming DPO and other baselines, including label-smoothed DPO.Across sampling temperatures, rDPO policies are consistently better than other methods.Source: §1 · p. 2

1.1. Related Work

  • Related methods use hinge, square, ranking, or BCE losses; prior policy-learning guarantees focus on bandit or RL settings rather than practical methods such as DPO.Label-noise work in supervised learning bounds excess population risk, whereas this paper targets policy estimation error.Source: §1.1 · p. 2

2. Background and Problem Setup

The setup learns a language-model policy from preference pairs sampled using an SFT policy, with preferences modeled through reward differences. It estimates a policy from possibly flipped labels and evaluates performance by reward suboptimality relative to the optimal policy.

2. Background and Problem Setup

  • Prompts are sampled from a distribution, response pairs from the SFT policy, and human labelers or an oracle express preferences modeled through latent reward differences.The link function g converts reward differences into winning probabilities; a sigmoid link gives the Bradley-Terry-Luce model.Source: §2 · p. 2
  • The optimal policy maximizes the reward objective while accounting for divergence from the SFT policy, with β controlling the balance between exploration and exploitation.As β approaches zero, probability concentrates on the highest-reward response; as β grows, the policy approaches the SFT policy.Source: §2 · p. 3
  • Policies are parameterized by θ in a d-dimensional class; linear feature functions yield log-linear policies, including a language-model form using the final-layer weights.The feature map can be formed by removing the language model’s last layer.Source: §2 · p. 3
  • DPO estimates the policy by minimizing empirical binary cross-entropy over preference pairs, although the minimizer is not strictly an MLE because pairs come from the SFT policy rather than the target policy.Directly sampling preference pairs from the target policy is challenging.Source: §2 · p. 3
  • The noise model flips true preferences with probability ε, which is assumed known to the learner and may be tuned through cross-validation in practice.The learner receives the potentially noisy preference labels.Source: §2 · p. 3
  • Policy performance is measured by the reward suboptimality gap between the optimal policy and the learned policy, with the goal of a gap shrinking toward zero as sample size grows.The learned policy is obtained from a dataset of noisy preference samples.Source: §2 · p. 3

3. Our Approach: Robust DPO

The approach corrects for random preference flips in DPO, first addressing bias in losses computed from noisy preferences and then examining how flip-aware gradient weights affect updates. Its robust loss is unbiased for the clean DPO loss, while its gradients account for whether observed preferences may have been flipped.

3. Our Approach: Robust DPO

  • Conservative DPO is a weighted sum of DPO losses on noisy preferences, but both that conservative loss and the noisy-preference BCE loss are biased relative to the DPO loss.The paper uses this shared drawback to motivate an unbiased loss.Source: §3.1 · p. 4, §3 · p. 4

3.1. An Unbiased Loss Function

  • The rDPO loss is an unbiased estimator of the DPO loss under noisy preferences and reduces to the DPO loss when preferences are clean.When the flip rate ε is zero, the trained rDPO policy coincides with the DPO policy.Source: §3.1 · p. 4
  • For the un-normalized rDPO loss, the variance is ε(1 − ε) times the squared difference between the two clean preference losses and is bounded by C/4 for ε ≤ 1/2.The bound assumes a neural policy class with bounded fθ.Source: §3.1 · p. 4

3.2. Gradients of rDPO Loss

  • rDPO gradient terms weight incorrectly ordered observed preferences by the probability of no flip and correctly ordered ones by the probability of a flip, debiasing noise on average.The weights are defined using the implicit reward difference between the two answers.Source: §3.2 · p. 4
  • With no flip, rDPO updates are more aggressive in the desired direction than DPO and cDPO; with flips below one half, its updates move correctly faster on average.Individual flipped-preference updates can be more aggressive in the wrong direction than DPO and cDPO.Source: §3.2 · p. 5

4. Theoretical Analysis

The analysis bounds rDPO’s estimation error and the learned policy’s suboptimality under noisy preferences, with guarantees depending on noise and SFT feature coverage. It also identifies settings in which the bounds do not ensure vanishing parameter error or learning becomes impossible.

4. Theoretical Analysis

  • Although the empirical rDPO loss is unbiased for the population DPO loss, the learned policy need not converge to the optimum because samples come from the SFT policy.The analysis therefore bounds parameter estimation error as a function of sample size and flip rate; neural-policy suboptimality bounds are described as an adaptation of the preceding reasoning.Source: §4 · p. 5, §4.2 · p. 7

4.1. Estimation Error

  • An invertible feature covariance matrix permits vanishing ℓ2 parameter error without regularization; otherwise, λ=O(d/n) yields vanishing semi-norm error for log-linear policies, but not neural policies.Without invertibility, the learned parameter need not converge to θ*; neural-policy error does not vanish in the stated case.Source: §4.1 · pp. 5–6
  • Preference flips increase the estimation error bound by a multiplicative factor of order 1/(1-2ε) compared with DPO under clean preferences.The bound is for rDPO with flip rate ε > 0; the factor grows as the expected number of flips increases.Source: §4.1 · p. 6
  • At β=0 or as β approaches infinity, preference probabilities make learning impossible or the loss degenerate, respectively, so β must be tuned.At β=0 both actions are equally preferred; as β→∞ one action is always preferred with probability 1.Source: §4.1 · p. 6

4.2. Performance Bounds of Learned Policy

  • The policy suboptimality bound depends on estimation error and a condition number, and assumes the SFT policy covers the feature space.The feature-coverage assumption is λmin(Σπsft) > 0; the stated theorem is for log-linear policies and bounded rewards.Source: §4.2 · p. 6
  • With invertible sample feature covariance, rDPO needs roughly 1/(1-2ε)^2 times as many samples as clean-data DPO to achieve the same suboptimality gap.The stated sample complexity for noisy data is n ≥ κd/(∆²γ²β²(1−2ε)²), compared with n ≥ κd/(∆²γ²β²) for clean data.Source: §4.2 · p. 6

5. Generalizations and Extensions

The framework extends beyond DPO with BTL preferences to other preference losses, preference models, and reward training in RLHF. The extensions include unbiased noisy-feedback losses for ranking and a performance-bound result for the probit model.

Reward training in RLHF

  • The noise correction also applies to reward-model training in RLHF, producing a robust reward model that can be used with PPO as robust PPO (rPPO).The reward-model loss is an unbiased estimate of the clean-data BCE loss.Source: §5 · p. 7

Other optimization methods

  • The loss correction extends to SLiC and IPO, whose robust counterparts are unbiased under noisy preferences and can support robust policy learning.These methods do not assume a BTL preference model and can work with general preference probabilities.Source: §5 · p. 7

Other Preference Models

  • For probit preferences, the paper derives similar performance bounds when the preference model’s link function is strongly log-concave near zero.The probit model uses the standard Gaussian CDF Φ for preference probabilities.Source: §5 · p. 7
  • For Plackett-Luce K-wise rankings, perturbing the true ranking with noise yields a robust loss that is an unbiased estimate of the clean-ranking loss.The perturbation assigns probability ε/(N−1) to another ranking, where N is the number of possible rankings.Source: §5 · p. 7

6. Experiments

The experiments evaluate rDPO on IMDb sentiment generation and Anthropic helpful-harmless dialogue, comparing it with DPO and other baselines. Under noisy preferences, the reported results favor rDPO over DPO and cDPO, with rDPO and rPPO also achieving the highest rewards in their respective families across sampling temperatures.

6. Experiments

  • The sentiment task compared rDPO with DPO, cDPO, SLiC and IPO, and compared rPPO with PPO and cPPO; dialogue comparisons included DPO and cDPO.The reported experiments use IMDb sentiment generation and Anthropic helpful-harmless data.Source: §6 · pp. 7–8
  • IMDb sentiment data used movie-review prefixes, four generated reviews and six preference pairs per prefix, retaining 12,000 triplets split into 10,000 training and 2,000 evaluation examples.Preferences were randomly flipped at ε=0.4 for the noisy comparisons.Source: §6 · p. 8
  • At ε=0.4, rDPO maintains performance across training steps while DPO degrades; rDPO also significantly outperforms DPO and cDPO in both dialogue evaluations.For IMDb, rDPO and rPPO achieve the best reward by a significant margin over peers in their families across sampling temperatures.Source: Figure 1 · p. 8, §6 · pp. 8–9, Table 3 · p. 8
  • For helpful-harmless dialogue, the true noise rate was unknown, so rDPO was tested at ε values from 0.1 to 0.4; ε=0.1 gave its best results.The study also trained DPO and cDPO with label-smoothing ε=0.1 on the same data.Source: §6 · p. 8

A.1. Proof of Lemma 3.1

The supplied proof excerpt asserts that a result is easy to see, then presents an expression.

A.1. Proof of Lemma 3.1

  • The excerpt pairs “It is easy to see that” with the expression 1 −2ε + ε(1 −ε)L(θ; s, al, aw) −ε2L(θ; s, aw, al).Source: §A.1 · p. 11

A.2. Variance of rDPO loss

The appendix defines an un-normalized rDPO loss and gives its variance in terms of the two orderings of the preference pair.

A.2. Variance of rDPO loss

  • The un-normalized rDPO loss combines the loss for the observed answer ordering with the reverse ordering, weighted by 1−ε and −ε.The displayed definition also relates this quantity to the normalized loss through a factor of 1−2ε.Source: §A.2 · p. 11
  • The variance expression contains squared losses for both answer orderings and a cross-product term, with coefficients depending on ε.The excerpt gives the combined expression as (1 −3ε + 3ε^2)L(θ; s, a_w, a_l)^2 + ε(1−ε)L(θ; s, a_l, a_w)^2 −2ε(1−ε)L(θ; s, a_w, a_l)L(θ; s, a_l, a_w).Source: §A.2 · p. 11
  • The appendix states that ε(1−ε) times the squared difference between the two ordered losses equals their squared terms minus twice their product.This identity is presented as an algebraic step in the variance derivation.Source: §A.2 · p. 11

A.3. Proof of Lemma 3.2

The appendix writes the rDPO and cDPO gradients as weighted combinations of gradients for the observed and reversed preference orderings.

A.3. Proof of Lemma 3.2

  • The rDPO gradient weights the observed-order loss gradient by 1−ε and the reversed-order gradient by −ε.The excerpt expresses the weights of the DPO gradients using terms involving ε, 1−2ε, and the sigmoid of the pairwise policy score.Source: §A.3 · pp. 11–12
  • The cDPO gradient instead adds the reversed-order gradient with weight ε to the observed-order gradient weighted by 1−ε.The corresponding weights are written as (1−ε) and ε.Source: §A.3 · p. 12
  • The excerpt relates the cDPO gradient weight to the DPO weight by subtracting ε, and also gives its relation to the rDPO weight.The displayed relation includes a correction term of −2ε(1−ε).Source: §A.3 · p. 12

A.4. Proof of Theorem 4.2

The proof rewrites preference margins relative to the SFT policy and analyzes a loss adjusted for random preference flips. It uses gradient concentration and curvature conditions to complete the argument.

A.4. Proof of Theorem 4.2

  • The margin hθ compares the policy's preference-score difference with the corresponding SFT-policy difference.The proof represents the noisy preference outcome as Bernoulli under the random-noise model before rewriting the de-biased loss.Source: §A.4 · p. 12
  • At θ*, the gradient-related variable Vθ*,i is zero-mean and 1-sub-Gaussian conditional on zi.The proof then invokes Bernstein's inequality for sub-Gaussian random variables in quadratic form.Source: §A.4 · pp. 12–13
  • The cited curvature step is supported when ε < 1/2 and the uniform derivative bound σ′(βhθ(zi)) ≥ γ holds for every θ ∈ Θ.The passages also bound |hθ(s,a,a′)| ≤ 2α0 and, for θ ∈ Θ with ∥θ∥ ≤ B, ∥∇hθ*(zi)∥ ≤ 2α1 and ∥vi∥ ≤ 2α2B.Source: §A.4 · pp. 12–13

A.5. Proof of Theorem 4.4

The proof relates policy sub-optimality to covariance of centered features and controls the empirical covariance using clean preference data. It then applies a relative condition number and a KL-to-Bregman identity in the bound.

A.5. Proof of Theorem 4.4

  • For log-linear policies, centered features are feature vectors minus their policy expectation, and their covariance is used in the sub-optimality bound.The proof defines centered gradients for general policies and identifies them with centered features in the log-linear case.Source: §A.5 · pp. 14–15
  • Clean preference pairs sampled from the SFT policy yield a high-probability matrix-concentration bound on the empirical feature-difference covariance.The proof conditions its subsequent sub-optimality analysis on this high-confidence event.Source: §A.5 · p. 14
  • The proof bounds sub-optimality using a relative condition number and uses the exponential-family identity between KL divergence and a Bregman divergence.Pinsker's and Jensen's inequalities also appear in the sub-optimality argument.Source: §A.5 · pp. 14–15

A.6. Proof of Lemma 4.5

The lemma proof considers the expected margin gap under the clean distribution and, for log-linear policies, proceeds with an invertible empirical covariance matrix. The appendix also points to hyperparameter tables for the DPO and PPO method families.

A.6. Proof of Lemma 4.5

  • The proof considers the expected margin gap under the clean distribution and invokes Assumption 4.1 before assuming the empirical covariance matrix is invertible for log-linear policies.The proof states that this completes the lemma.Source: §A.6 · p. 15

B. Hyperparameter Details

Related papers

Loading 2403.00409v2…