PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, Yuxin Chen

arXiv:2309.16240v1cs.LGcs.AIstat.ML

TL;DR

LLM safety concerns motivate alignment methods that avoid RLHF’s complexity and reward-model dependence. The paper generalizes DPO to diverse f-divergence constraints using KKT-based reward–policy mappings that remove normalizing-constant estimation. Empirically, f-DPO balances alignment and generation diversity and is more divergence-efficient than PPO-based methods.

  • Problem

    RLHF is effective for preference alignment but is complex, depends on a separate reward model, and reverse-KL-focused methods can reduce generation diversity.

  • Method

    f-DPO generalizes DPO across f-divergence constraints by using KKT conditions to eliminate the Bradley-Terry normalizing constant and derive a tractable reward–policy mapping.

  • Results

    f-DPO balances alignment performance and generation diversity and surpasses PPO-based methods in divergence efficiency.

  • Takeaways & Limitations

    Divergence regularization provides flexibility for fine-tuning preferences while connecting divergence efficiency with expected calibration error.

  • Takeaways & Limitations

    For divergences other than reverse KL, estimating the normalizing constant may require multiple samples and can be computationally expensive and high-variance.

Abstract

from arXiv · show

The increasing capabilities of large language models (LLMs) raise opportunities for artificial general intelligence but concurrently amplify safety concerns, such as potential misuse of AI systems, necessitating effective AI alignment. Reinforcement Learning from Human Feedback (RLHF) has emerged as a promising pathway towards AI alignment but brings forth challenges due to its complexity and dependence on a separate reward model. Direct Preference Optimization (DPO) has been proposed as an alternative, and it remains equivalent to RLHF under the reverse KL regularization constraint. This paper presents $f$-DPO, a generalized approach to DPO by incorporating diverse divergence constraints. We show that under certain $f$-divergences, including Jensen-Shannon divergence, forward KL divergences and $α$-divergences, the complex relationship between the reward and optimal policy can also be simplified by addressing the Karush-Kuhn-Tucker conditions. This eliminates the need for estimating the normalizing constant in the Bradley-Terry model and enables a tractable mapping between the reward function and the optimal policy. Our approach optimizes LLMs to align with human preferences in a more efficient and supervised manner under a broad set of divergence constraints. Empirically, adopting these divergences ensures a balance between alignment performance and generation diversity. Importantly, $f$-DPO outperforms PPO-based methods in divergence efficiency, and divergence constraints directly influence expected calibration error (ECE).

1 INTRODUCTION

LLM capabilities create both opportunities and safety risks, motivating alignment research. The paper generalizes DPO across divergence constraints to address RLHF complexity while balancing alignment and generation diversity.

  • AI alignment is motivated by LLM capabilities that may enable misuse, including social manipulation, cyberattacks, and enhanced pathogens.
  • RLHF aligns model behavior with human preferences but requires a separate reward model whose errors can be exploited during reinforcement fine-tuning.
  • Existing work largely focuses on reverse KL, leaving other divergence constraints comparatively underexplored.
  • Reverse KL can improve alignment while reducing generation diversity and narrowing the range of political views expressed by fine-tuned LLMs.
  • f-DPO uses KKT conditions to eliminate the Bradley-Terry normalizing constant for several divergences, yielding an analytical reward-to-policy mapping.
  • The generalized framework balances alignment performance and generation diversity through different divergence regularizations.

2 RELATED WORKS

Related work frames RLHF as a central alignment method supported by preference data and reward modeling. Its complexity and reward-model dependence motivate alternative alignment procedures.

  • AI alignment methods target model behavior that follows human preferences and instructions while supporting safe behavior and downstream performance.
  • Many alignment approaches rely on human judgment or comprehensive human-written principles to provide supervised signals.
  • RLHF has contributed substantially to language-model alignment but is more complex than supervised learning because it requires a distinct reward model.

3 PRELIMINARY AND BACKGROUNDS

The paper introduces f-divergences, Bradley-Terry preference modeling, RLHF, and DPO as foundations for generalized preference optimization. It emphasizes reverse KL’s diversity tradeoff and calibration considerations.

  • 3.1 PRELIMINARY: f-divergences generalize several common discrepancies, including forward KL, reverse KL, Jensen-Shannon divergence, and total variation distance.
  • 3.1 PRELIMINARY: The Bradley-Terry model assigns item strengths and converts their relative values into pairwise-comparison probabilities.
  • 3.1 PRELIMINARY: RLHF combines supervised fine-tuning, reward-model training, and RL fine-tuning under an objective penalized by reverse KL against a reference model.
  • 3.1 PRELIMINARY: DPO directly optimizes the policy by reparameterizing the reward function with the language model under reverse KL.
  • 3.3 CALIBRATION ERROR: Expected Calibration Error measures the mismatch between model confidence and prediction correctness; prior work reports that RLHF degrades calibration.
  • 3.2 BACKGROUND: OPTIMIZING FOR REVERSE KL HURTS DIVERSITY: RLHF with reverse KL can cause mode collapse, reduced output diversity, and a limited range of political views, motivating alternative divergence regularization.

4 METHOD: DIRECT PREFERENCE OPTIMIZATION UNDER f-DIVERGENCE

The method generalizes DPO to f-divergence constraints, deriving a supervised preference-optimization procedure from the constrained RL objective. KKT analysis removes the need to estimate the normalization constant for a broad class of divergences.

  • Constrained optimization: The constrained RL objective maximizes expected reward while penalizing divergence from a reference policy, with valid-distribution constraints enforced through Lagrange multipliers.The constraints ensure the policy is normalized and nonnegative; the resulting optimization admits a closed-form optimal policy.
  • Policy–reward relationship: The optimal policy is expressed using the reference policy, the inverse derivative of the divergence function, and a normalization constant Z(x).Solving this expression for the reward establishes a tractable reward–optimal-policy relationship.
  • Normalization challenge: For reverse KL, the normalization term cancels in Bradley–Terry preference differences, whereas other divergences generally make estimating Z(x) computationally expensive and high-variance.The reverse-KL reward takes a log policy-ratio form plus an additive normalization term that cancels between preferred and rejected responses.
  • KKT simplification: Analyzing complementary slackness in the KKT conditions shows that, for several f-divergences, the reward can be represented using the trainable policy, reference policy, and a y-independent constant.The covered family includes forward KL, Jensen–Shannon, reverse KL, and α-divergences with 0 < α < 1; the additive constant cancels in the Bradley–Terry model.
  • Supervised optimization: The resulting objective trains the policy by supervised negative log-likelihood on preference data, extending DPO across a broad class of divergence regularizers.The paper presents this as more stable and efficient to optimize than reinforcement-learning alternatives and summarizes the procedure in Algorithm 1.

5 EXPERIMENTS

Experiments compare f-DPO with PPO across divergence efficiency, reward–entropy tradeoffs, alignment and diversity, MT-Bench quality, and calibration. Across these evaluations, divergence choice shapes the alignment–diversity balance, while f-DPO achieves stronger divergence efficiency than PPO.

  • IMDB experiments: f-DPO with f-divergences outclasses both PPO implementations on the divergence-versus-reward frontier.PPO (loss) outperforms PPO (reward) for JSD, forward KL and α-divergences because fluctuating penalties destabilize PPO’s value-function learning.
  • IMDB experiments: Reverse KL achieves the lowest predictive entropy, forward KL the highest, JSD balances the two, and α-divergence interpolates between JSD and forward KL.These results reflect the divergences’ mode-seeking and mass-covering properties.
  • Anthropic HH experiments: On Anthropic HH, reverse KL delivers the highest alignment accuracy but the lowest generation diversity, whereas changing divergence regularization enables an accuracy–diversity trade-off.Diversity is measured using predictive entropy, self-BLEU and distinct-n across sampled responses.
  • MT-Bench experiments: On MT-Bench, f-DPO achieves performance comparable to, and sometimes significantly better than, PPO under GPT-4 evaluation.The comparison uses win, tie and lose rates for open-ended multiturn and instruction-following questions.
  • Divergence efficiency: f-DPO attains smaller divergence than PPO while maintaining comparable performance, establishing greater divergence efficiency.The experiments examine divergence efficiency as a separate advantage of the generalized method.
  • Calibration: Theorem 2 bounds differences in Expected Calibration Error by f-divergence, and experiments evaluate calibration after f-DPO fine-tuning on Anthropic HH.With a well-calibrated base model, smaller f-divergence suggests similarly well-calibrated fine-tuned models.

6 CONCLUSION

The study generalizes DPO across divergence constraints, finding that regularization choices trade off alignment performance and generation diversity while improving divergence efficiency over PPO.

  • f-DPO incorporates a spectrum of divergence constraints into DPO, including reverse KL, forward KL, Jensen-Shannon, and α-divergence.
  • Reverse KL typically provides stronger alignment performance but compromises generation diversity.
  • Adjusting divergence regularization enables a balance between alignment and generation diversity.
  • f-DPO demonstrates greater divergence efficiency than traditional PPO methods.
  • The difference in expected calibration error between two models can be bounded by their divergence.

A THE CAUSE OF INSTABILITY IN PPO WITH DIVERGENCE PENALTY IN REWARDS

The section attributes PPO instability to differing growth rates among divergence penalties. Forward KL grows fastest, while reverse KL grows slowest and is therefore more numerically stable.

  • Figure 6 visualizes divergence penalties for reverse KL, JDS, and forward KL, alongside per-token divergence during PPO training on IMDB.
  • KL-based reward penalties can destabilize optimization for forward KL and Jensen-Shannon divergence.
  • Forward KL grows much faster than the other penalties during training, whereas reverse KL grows the slowest.
  • Reverse KL is more numerically stable than the other penalties, making value-function learning easier.
  • The PPO implementation estimates f-divergences from samples using an unbiased, low-variance estimator.
  • Additional diversity evaluations sampled 25 responses per prompt at temperatures 0.6 and 1.4, showing patterns similar to the temperature-1.0 results.

C.2 GPT-4 EVALUATIONS ON MT-BENCH FOR DPO WITH DIFFERENT DIVERGENCES

GPT-4 evaluations on MT-Bench produce a different ordering from Anthropic HH alignment accuracy: reverse KL underperforms JSD there, while JSD remains ahead of α-divergence and forward KL.

  • GPT-4 served as the referee for comparing DPO models with different divergences on MT-Bench.
  • Reverse-KL DPO outperformed JSD DPO on Anthropic HH alignment accuracy but underperformed it in GPT-4 evaluation.
  • JSD DPO performed better than α-divergence DPO in the MT-Bench comparison.
  • α-divergence DPO performed slightly better than forward-KL DPO.

D PROOFS

The proof uses KKT conditions to reparameterize the reward through the policy and reference model. Under stated assumptions, complementary slackness removes inequality multipliers for several common divergences.

  • Theorem 1 reparameterizes Bradley-Terry-consistent rewards using the policy and reference model when πref(y|x) > 0 and f′ is invertible.
  • The KKT proof begins with stationarity, primal feasibility, dual feasibility, and complementary slackness.
  • Complementary slackness requires α(y)π(y|x) = 0, so each inequality constraint is either tight or has zero multiplier.
  • Stationarity relates π(y|x), r(y|x), f, λ, and α(y), with the specific derivative depending on f.
  • Assuming an invertible f′ whose inverse maps into the valid differentiability domain, the stationarity equation can be solved for π(y|x).
  • Primal feasibility determines λ through normalization, while nonnegativity may further restrict λ and α(y).
  • The resulting reward-policy mapping is instantiated separately for reverse KL, forward KL, and JS divergence.
  • For reverse KL, forward KL, and JS divergences, the assumptions imply α(y) = 0, and λ acts as an additive constant canceled in the Bradley-Terry model.

E ON CALIBRATION AND f-DIVERGENCE

The section defines ECE for stochastic policies and proves that calibration-error differences can be bounded using f-divergences between policies.

  • Calibration definition: ECE measures the difference between prediction correctness and predicted probability, averaged over the predicted probability values.For policy πθ, the predicted label is sampled according to πθ(ŷ|x).
  • Calibration definition: The paper explicitly uses stochastic policies, unlike the standard definition that selects the highest-probability label.This changes how ŷ is generated when computing calibration.
  • f-divergence bound: Theorem 2 bounds differences in ECE between two policies using an f-divergence, assuming f is strictly convex.The theorem applies to any f-divergence satisfying the stated convexity condition.
  • f-divergence bound: The proof first bounds ECE differences by expected L1 policy distance, then expresses that distance through total variation distance.The bound uses the factor m(θ1, θ2, x) inside the expectation.
  • f-divergence bound: Common f-divergences can yield more refined inequality relations for these calibration-error bounds.The section begins specializing the general result to KL divergence via Pinsker’s inequality.

F GENERATED SAMPLES FROM DPO-TRAINED MODELS AND PPO-TRAINED MODELS

This section presents qualitative pairwise comparisons between samples from DPO-trained and PPO-trained models under reverse KL, forward KL, and Jensen-Shannon divergence constraints.

  • Evaluation setup: Some MT-Bench comparisons include reference answers, but the section omits those answers to save space.The section focuses on pairwise sample comparisons between DPO-trained and PPO-trained models.
  • Jensen-Shannon divergence: DPO-JSD wins the qualitative sample comparison in three reported tables, while PPO-JSD wins in one.These comparisons are presented for models trained under Jensen-Shannon divergence.
  • Reverse KL divergence: DPO-RKL wins the qualitative sample comparison in three reported tables, while PPO-RKL wins in one.The comparisons concern models trained under reverse KL regularization.
  • Forward KL divergence: DPO-FKL wins the qualitative sample comparison in three reported tables, while PPO-FKL wins in one.The comparisons concern models trained under forward KL regularization.
Loading 2309.16240v1…