PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang, Bo Han, Yike Guo

arXiv:2605.20834v1cs.AIcs.LG

TL;DR

The paper examines when DPO can be derived from RLHF and shows that the equivalence depends on an implicit preference-ordering assumption that may fail for misaligned reference policies. It introduces constrained methods, including reward-model-free E-CPOC, and reports that CPO achieves state-of-the-art benchmark performance while the theory provides alignment guarantees.

  • Problem

    The paper addresses whether DPO and RLHF are equivalent when the RLHF-optimal policy may not prefer human-preferred responses.

  • Method

    The paper analyzes DPO’s hidden assumption and introduces CPO and E-CPOC, which augment RLHF with explicit preference-alignment constraints.

  • Results

    CPO achieves state-of-the-art performance, while E-CPOC achieves provable equivalence to explicitly constrained RLHF without requiring a reward model.

  • Takeaways & Limitations

    DPO’s guarantees are conditional on reference-policy quality, whereas constrained objectives are designed to preserve preference alignment.

  • Takeaways & Limitations

    The paper calls for validation at larger scale and with larger models, plus experiments evaluating E-CPOC beyond its theoretical analysis.

Abstract

from arXiv · show

Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF), offering theoretical equivalence with simpler implementation. We prove this equivalence is conditional rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF-optimal policy must prefer human-preferred responses. When this assumption fails, DPO optimizes relative advantage over the reference policy rather than absolute alignment with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases. To address this, we introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets. Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state-of-the-art performance. Code is available at: https://github.com/visitworld123/CPO.

1. Introduction

The paper argues that DPO–RLHF equivalence depends on an often-violated assumption about the RLHF-optimal policy and reference-policy quality. It characterizes resulting failure modes and proposes constrained alternatives with theoretical and empirical support.

  • Failure mode: A sufficiently misaligned reference policy can cause the KL penalty to dominate, leading the RLHF optimum to inherit incorrect preferences.This invalidates the substitution underlying DPO’s derivation.
  • Core problem: DPO–RLHF equivalence holds conditionally, requiring the RLHF-optimal policy to prefer human-preferred responses.The condition depends on reference-policy quality.
  • Failure mode: When the assumption fails, RLHF optimizes absolute human-preference alignment while DPO optimizes relative advantage over the reference policy.The objectives become fundamentally different.
  • Proposed solution: CPO augments RLHF with explicit constraints, while E-CPOC removes the need for a reward model and achieves formal equivalence to explicitly constrained RLHF under stated assumptions.The assumptions include Bradley-Terry modeling, approximate realizability, finite-sample data, and a mild proximity condition.
  • Interpretation and evidence: DPO corresponds geometrically to soft margin ranking with potentially negative targets, whereas the proposed constrained methods enforce non-negative effective margins.Experiments on standard benchmarks demonstrate the efficacy of the proposed method.

2. Preliminaries

The preliminaries define policies, preference-strength ratios, RLHF’s KL-regularized objective, and the Bradley-Terry model used to derive DPO.

  • Notation: A policy is a conditional response distribution, with πref denoting a fixed reference policy and πθ a learnable parameterized policy.The reference policy is typically a supervised fine-tuned model.
  • Notation: The log-probability ratio δπ measures a policy’s preference strength for the preferred response over the dispreferred response in log-space.It is defined for a prompt and response pair.
  • RLHF objective: RLHF maximizes expected reward while penalizing KL divergence from the reference policy.The KL term limits deviation from πref and helps prevent reward over-optimization.
  • Preference model: The Bradley-Terry model represents human preference probability as a sigmoid of the latent reward difference.If the preferred response has probability above 0.5, its latent reward exceeds the alternative’s.
  • DPO derivation: DPO substitutes the RLHF reward reparameterization into the Bradley-Terry likelihood and maximizes the resulting parameterized policy likelihood.This derivation connects DPO to the RLHF optimum.

3. The Implicit Assumption in DPO

The DPO derivation conceals an assumption that the RLHF optimum preserves human-preferred ordering, which can fail when the reference policy is sufficiently misaligned. In that regime, DPO and RLHF optimize different objectives and DPO can converge pathologically.

  • Implicit assumption: DPO assumes that whenever humans prefer yw over yl, the RLHF-optimal policy assigns higher probability to yw.This assumption is required to derive DPO from the Bradley-Terry model.
  • When it fails: The assumption depends on reference-policy quality and fails when πref sufficiently disfavors yw, even if the human-preference reward difference is positive.Thus the assumption is not automatically guaranteed by RLHF.
  • Implication: The paper concludes that DPO inherits RLHF’s algebraic reward-reparameterization structure but not its alignment guarantees.The equivalence is conditional on reference-policy quality.
  • Pathological solutions: The undesirable solution space contains policies that satisfy DPO’s relative-advantage criterion while violating human preferences.It is formally characterized as a set of preference-violating policies.
  • Pathological convergence: When δπref < −∆r∗/β, this solution space is non-empty and DPO gradients weaken as δπ approaches zero, making escape difficult.Loss can decrease while policies continue preferring dispreferred responses.
  • Objective mismatch: When the assumption is violated, RLHF optimizes E[r] − β KL while DPO maximizes relative advantage δπ − δπref.The methods therefore optimize fundamentally different objectives.

4. Constrained Preference Optimization

CPO converts RLHF into a constrained objective designed to enforce preference alignment, while E-CPOC uses hard, reward-model-free constraints for stronger guarantees. The framework supplies convergence, absolute-advantage, and equivalence results under stated assumptions.

  • Framework: CPO augments RLHF with explicit constraints so its optimum satisfies the preference alignment required by Bradley-Terry theory.The constrained optimum can therefore be integrated into the preference model.
  • Assumptions: The analysis assumes Bradley-Terry preferences, approximate realizability, finite-sample data, and an ℓ2-δ proximity condition, without requiring global optimality.The proximity requirement uses the loss-controlled mean-square norm.
  • Optimization: Under mild smoothness and boundedness conditions, gradient descent on the CPO loss converges to a stationary point.CPO is a strict generalization of DPO when γ = 0.
  • Guarantees: Choosing γ ≥ γ∗ guarantees positive absolute advantage for every preference pair and prevents convergence to the undesirable solution space.The guarantee directly addresses DPO’s pathological convergence.
  • E-CPOC: CPO’s limitation is dependence on selecting γ and use of a soft penalty that encourages, rather than ensures, preferred-response ordering.E-CPOC addresses this with hard constraints and no reward model.
  • E-CPOC: E-CPOC provides stronger correction for difficult samples, requires no reward model, and offers provable alignment under standard statistical assumptions.Its adaptive margin is larger for difficult samples and smaller for easy ones.

5. Preference Learning as Reranking

The paper interprets DPO, CPO, and E-CPOC through soft margin ranking, showing that DPO can use negative target margins while the constrained methods ensure non-negative margins.

  • DPO as Margin Ranking: DPO is equivalent to a smooth margin-ranking loss with target margin m = δπref.As β increases, the smooth loss approaches hinge loss.
  • DPO as Margin Ranking: When δπref < 0, DPO can achieve zero loss while still preferring the rejected response over the preferred response.The loss becomes zero once δπθ > δπref, even if δπθ remains negative.
  • Constrained Margins: E-CPOC provides a sample-adaptive margin-ranking objective.Its adaptive-margin property is formalized in Theorem 5.3.
  • Constrained Margins: CPO and E-CPOC implement soft margin ranking with guaranteed non-negative margins.This corrects DPO’s potentially negative target margin.

6. Experiments

Experiments evaluate preference-alignment methods on conversational benchmarks using Llama-3-8B-Instruct. CPO achieves the strongest reported performance across AlpacaEval 2 and Arena-Hard.

  • Experimental Setup: Experiments compare alignment methods on AlpacaEval 2 and Arena-Hard using Llama-3-8B-Instruct.Both benchmarks evaluate conversational skills on real-life queries.
  • Main Results: 25.15% win rate on AlpacaEval 2 makes CPO the strongest reported method, exceeding DPO’s 24.60% by +0.55%.CPO also reaches a 26.57% length-controlled win rate versus SimPO’s 25.91%.
  • Main Results: 32.6% win rate on Arena-Hard gives CPO a +2.6% gain over SimPO and a +3.7% gain over DPO.The result is reported with a 90% confidence interval.
  • Main Results: CPO maintains an average response length of 1879 tokens without exhibiting excessive verbosity.The reported length is similar to strong baselines such as DPO and RDPO.

7. Conclusion

The paper establishes conditional rather than universal DPO–RLHF equivalence and proposes constrained alternatives to address DPO’s failure modes. CPO achieves state-of-the-art performance, while larger-scale validation remains necessary.

  • Contributions: DPO and RLHF are conditionally equivalent, with CPO and E-CPOC designed to address DPO’s failure modes through explicit preference constraints.E-CPOC is formally equivalent to explicitly constrained RLHF without requiring a reward model.
  • Limitations: The method requires validation at larger scale and with larger models.The paper also calls for experimental validation of E-CPOC beyond its theoretical analysis and training-dynamics visualizations.

Impact Statement

The paper reports substantial preference-assumption violations and contrasts DPO’s weak escape dynamics with CPO’s margin-corrected behavior under misaligned references. It also notes that stronger preference adherence may amplify training-data biases.

  • Impact Statement: CPO may amplify biases present in preference data by enforcing stronger adherence to those preferences.The paper identifies this concern across preference-learning methods and notes that γ controls enforcement strength.
  • Violation Frequency: 45.5% of preference pairs violate Assumption 3.1 on Llama-3-8B-Instruct with β = 0.1.The reward correction has mean 0.20, while δπref has standard deviation 46.69.
  • Reference Policy Quality: Increasing the corruption ratio R increases the violation rate of Assumption 3.1.Misaligned references are constructed by training on rejected responses.
  • Misaligned References: DPO’s length-controlled win rate declines across all corruption ratios as reference-policy corruption strengthens.This behavior is consistent with the paper’s theoretical analysis.
  • Misaligned References: CPO remains stable under corruption, reaching 27.60% and 27.35% length-controlled performance at R = 0.2 and R = 0.3.The margin term preserves gradient strength when δπref is negative.
  • Training Dynamics: DPO can become stuck near δπθ ≈ 0−, whereas CPO maintains gradient strength near the undesirable-solution boundary and pushes samples past zero.The difference appears in the training trajectory of the fraction of samples in U.

A.3. Sensitivity Analysis of γ

The sensitivity analysis examines CPO across γ values and reports robust performance over a specified range, with a peak at γ = 0.25 and reduced performance below 0.2. Additional experiments extend evaluation to instruction following and compare CPO with a clipped-reference baseline.

  • Sensitivity analysis: CPO performs robustly across γ ∈[0.2, 0.4], with peak performance at γ = 0.25.Performance drops notably below 0.2, where margin correction is insufficient to address assumption violations.
  • Additional benchmark: CPO achieves the highest performance on both strict and loose IFEval accuracy.This extends the reported improvement beyond conversational benchmarks to instruction-following tasks.
  • Baseline comparison: CPO substantially outperforms the clipped-reference baseline on AlpacaEval 2.The result indicates that its adaptive margin provides benefits beyond merely preventing negative margins.

D.3. Proof of Conditional Equivalence of DPO and RLHF

The proof establishes that DPO and RLHF are equivalent only when the RLHF-optimal policy respects human preference ordering. If that condition fails, DPO and RLHF optimize different objectives; constrained formulations provide alignment guarantees.

  • Sufficiency: DPO-RLHF equivalence requires the RLHF-optimal policy to prefer the human-preferred response for every preference pair.Under this condition, the Bradley-Terry model is consistent at the RLHF optimum and DPO recovers identical preference structures.
  • Sufficiency: Under the condition, strict convexity of the DPO loss uniquely determines the optimal log-probability differences.Any DPO minimizer therefore matches the RLHF optimum's preference ratios across the dataset.
  • Necessity: When the condition is violated, DPO and RLHF optimize fundamentally different objectives and converge to different optima.RLHF optimizes reward minus KL divergence, whereas DPO optimizes a preference-logistic objective relative to the reference policy.
  • Constrained optimization: CPO uses explicit constraints and a margin term to guarantee absolute advantage and avoid pathological convergence when γ is sufficiently large.The full adaptive margin is tighter than a constant approximation, which can over-regularize high-confidence pairs and under-regularize low-confidence pairs.

E.6. Gradient Analysis and Sample Weighting

E-CPOC uses an adaptive margin to weight samples according to difficulty. Difficult pairs receive stronger gradients, while easy pairs behave like standard DPO, creating a smooth transition without unnecessary emphasis.

  • Adaptive weighting: E-CPOC implements automatic sample-difficulty weighting through the adaptive margin Φcons.The weighting modifies the sigmoid input and therefore the gradient strength.
  • Difficult samples: For difficult samples with δπref ≪γ, Φcons becomes large and provides a stronger gradient signal to increase δπθ.This targets pairs whose reference-policy margin is far below the constraint level.
  • Neutral samples: Neutral samples near δπref ≈γ receive a smooth transition between difficult- and easy-sample regimes.The transition follows from the softplus formulation of Φcons.
  • Easy samples: For easy samples with δπref ≫γ, Φcons ≈0 and the weighting reduces to standard DPO.The method avoids unnecessary emphasis on already well-aligned samples.
  • Comparison with CPO: E-CPOC differs from CPO by using a hard constraint and sample-adaptive margin, whereas CPO uses a soft constraint and constant margin.The adaptive design focuses optimization on difficult pairs while avoiding over-regularization on easy pairs.

F. RLHF Behavior Under Assumption Violation

Under assumption violation, RLHF may prefer dispreferred responses as the optimal reward–KL trade-off, while DPO silently optimizes relative advantage and can converge pathologically. CPO uses adaptive constraints and margins to preserve alignment guarantees within the KL-regularized framework.

  • RLHF behavior: RLHF can prefer dispreferred responses when KL regularization outweighs reward maximization, reflecting its stated objective rather than an optimization error.The issue can be diagnosed by adjusting β or improving the reward model.
  • DPO behavior: DPO instead optimizes relative advantage over the reference policy, so its loss can decrease while policies systematically prefer dispreferred responses.This pathological region can be difficult to escape because DPO gradients weaken near its boundary.
  • Framework limitation: The reference-policy anchor limits all KL-regularized methods, including CPO, from discovering solutions beyond the reference policy’s support.Addressing this ceiling requires moving beyond KL-regularized RLHF, although CPO can deviate more on critical misaligned pairs.
  • Conditions: CPO’s approximation error vanishes as β increases for fixed constraint strength, while its guarantees remain tied to bounded support, bounded rewards, and moderate constraints.These conditions keep the constrained optimum close to the unconstrained one and support the approximation analysis.
  • CPO mechanism: CPO’s adaptive margin increases for low-probability, difficult pairs and decreases for high-confidence pairs, while keeping the margin stationary during optimization.This design supports convergence analysis and strengthens correction where the reference policy is most misaligned.

I. Convergence Analysis of CPO

CPO uses a stationary reference-based margin, enabling standard convergence analysis under smoothness and bounded-parameter assumptions. Its constrained formulation also admits an adaptive-margin interpretation that extends across preference pairs and avoids requiring a reward model in its conservative version.

  • Convergence guarantee: Under Lipschitz-gradient and bounded-parameter assumptions, gradient descent on the stationary CPO loss converges to a stationary point.Stationarity depends on computing the margin from πref rather than the evolving policy.
  • Convex setting: In convex or appropriately regularized linear cases, every stationary point is a global optimum and gradient descent converges to the global optimum.For smooth convex functions, the stated convergence rate is O(1/T).
  • Implementation: CPO preserves DPO-like computational cost by reusing the reference-policy forward pass, adding only scalar margin calculations and no reward-model inference.Its per-iteration cost is identical to DPO, with one extra stored scalar per sample.
  • Adaptive constraints: The constrained first-order condition adds effective margin contributions to the policy log-probability ratio, while the adaptive margin uses a smooth relaxation of constraint violation.The conservative version is monotone in the reward difference and requires no reward model while retaining provable alignment guarantees.

K.1. E-CPOC Loss Derivation

E-CPOC provides a constrained, reward-model-free route to provable preference alignment and equivalence with explicitly constrained RLHF. Its guarantees are tied to verifiable loss conditions, non-degenerate preferences, and structural assumptions.

  • Practical properties: The method retains DPO’s advantage of avoiding reward-difference knowledge while supplying provable alignment guarantees.The guarantees are accompanied by observable training criteria and explicit error dependencies.
  • Regularity and robustness: The loss-to-proximity bridge weakens as preference curvature κ0 approaches zero because the loss becomes less sensitive to δ.When preferences are well separated, stronger curvature yields tighter proximity bounds.
  • Scope of equivalence: The core pairwise equivalence requires four core assumptions, while connected comparison graphs are needed only to extend pairwise equivalence to full policy equivalence.Graph diameter controls error accumulation when pairwise results are converted into individual log-probability guarantees.
  • Equivalence and guarantees: E-CPOC uses hard constraints and does not require a reward model while establishing equivalence with explicitly constrained RLHF.The equivalence holds under standard assumptions and an ℓ2-δ-proximity condition.
  • Alignment guarantee: E-CPOC is more conservative than worst-case EC-RLHF and provides an absolute advantage guarantee for every positive γ.Its conservative margin correction is supplemented by the positive Bradley-Terry reward difference.
  • Verifiable conditions: Small observable loss gaps imply ℓ2-δ-proximity with an N-independent bound under non-degenerate preferences.The bridge combines loss suboptimality with curvature κ0 > 0; the N factor enters only through later ℓ2-to-ℓ∞ conversion.

M.3. Proof of E-CPOC as Adaptive Margin Ranking

The E-CPOC objective can be interpreted as adaptive soft margin ranking. Its effective target margin remains positive for every reference-policy gap when γ > 0, enforcing absolute preference alignment.

  • Loss transformation: E-CPOC rewrites its loss as a hinge-like soft margin ranking objective using a conservative margin correction.The transformed loss depends on the difference between the policy gap and the reference-adjusted conservative target.
  • Adaptive margin: The conservative correction is nonnegative for every reference-policy gap, including difficult cases where the reference strongly favors the dispreferred response.For δπref ≪ 0, the correction approximately cancels the negative reference gap and adds γ.
  • Alignment consequence: The effective target margin is positive for all δπref when γ > 0, ensuring absolute preference alignment.This contrasts with DPO’s potentially negative target margins.
Loading 2605.20834v1…