PaperlayerView exact revision on arXiv

Source-linked AI summary

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, Yi Wu

arXiv:2404.10719v3cs.CL

TL;DR

The paper asks whether DPO’s strong academic benchmark performance makes it superior to PPO and examines why PPO can underperform. Through theoretical and empirical DPO analysis, PPO ablations, and broad dialogue-to-code evaluations, it finds that PPO consistently surpasses DPO, including state-of-the-art code results. The authors also identify distribution-shift limitations for DPO and practical PPO training factors.

  • Problem

    DPO often performs strongly on academic benchmarks while PPO powers successful applications, leaving their relative superiority and PPO’s benchmark underperformance unresolved.

  • Method

    The paper theoretically and empirically studies DPO, ablates PPO training components, and benchmarks both methods across dialogue and code-generation RLHF testbeds.

  • Results

    PPO consistently outperforms DPO across experiments and achieves state-of-the-art code results, including 10@1k improving from 16.4% to 22.4% on CodeContest.

  • Takeaways & Limitations

    PPO can achieve leading RLHF performance when trained with advantage normalization, large batches, and exponential-moving-average reference-model updates.

  • Takeaways & Limitations

    DPO remains vulnerable to preference-data distribution shift and can develop biased policies favoring unseen responses.

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) is currently the most widely used method to align large language models (LLMs) with human preferences. Existing RLHF methods can be roughly categorized as either reward-based or reward-free. Novel applications such as ChatGPT and Claude leverage reward-based methods that first learn a reward model and apply actor-critic algorithms, such as Proximal Policy Optimization (PPO). However, in academic benchmarks, state-of-the-art results are often achieved via reward-free methods, such as Direct Preference Optimization (DPO). Is DPO truly superior to PPO? Why does PPO perform poorly on these benchmarks? In this paper, we first conduct both theoretical and empirical studies on the algorithmic properties of DPO and show that DPO may have fundamental limitations. Moreover, we also comprehensively examine PPO and reveal the key factors for the best performances of PPO in fine-tuning LLMs. Finally, we benchmark DPO and PPO across a collection of RLHF testbeds, ranging from dialogue to code generation. Experiment results demonstrate that PPO is able to surpass other alignment methods in all cases and achieve state-of-the-art results in challenging code competitions. Our code is publicly available at https://github.com/openpsi-project/ReaLHF.

1. Introduction

The paper asks whether DPO is genuinely superior to PPO, given their contrasting real-world and academic benchmark performance. It analyzes DPO’s limitations, identifies factors improving PPO, and reports that PPO consistently outperforms DPO, including on challenging code tasks.

  • Real-world systems such as ChatGPT and Claude use PPO, whereas strong academic benchmark results often come from DPO.
  • The paper investigates whether DPO is superior to PPO and whether PPO’s benchmark performance can be substantially improved.
  • The authors theoretically and empirically examine DPO, analyze PPO’s critical training factors, and evaluate both methods across dialogue and code-generation tasks.
  • PPO consistently outperforms DPO across the experiments and achieves state-of-the-art results on challenging code competition tasks.
  • 10@1k improves from 16.4% to 22.4% for the 34B-parameter PPO model versus AlphaCode-41B on CodeContest.

2. Related Work

Related work frames RLHF as reward-based or reward-free and positions this paper as an analysis of DPO limitations and PPO training factors. It builds on prior studies of implementation, performance gaps, and efficient alignment systems.

  • RLHF methods are broadly divided into reward-based approaches using learned rewards and online PPO, and reward-free approaches that omit explicit reward functions.
  • Prior work has examined the performance gap between DPO and PPO on synthetic contextual bandits.
  • This paper theoretically and empirically analyzes DPO limitations while exploring key factors for PPO training.
  • Concurrent research addresses reward-model overoptimization, alignment-data generation, and resource-efficient RLHF systems.
  • Earlier studies explored PPO implementation details for LLMs, which this paper extends.

3. Preliminary

The preliminary section defines LLM alignment through SFT and RLHF, then presents PPO as reward-based optimization and DPO as preference-data optimization. It also introduces the theoretical relationship and counter-example motivating later analysis.

  • Language Model: An LLM is modeled as a policy πθ(y | x) that generates a response y for an instruction x autoregressively.
  • SFT: SFT initializes alignment by training the pretrained model to imitate high-quality demonstrations.
  • RLHF: RLHF optimizes a reward objective in which r scores responses, πref regularizes the policy with KL divergence, and β controls regularization strength.
  • PPO: PPO first learns a reward model from human-labeled preference pairs when the reward is unknown, then optimizes the objective with online reinforcement learning.
  • DPO: DPO avoids learning a reward model by directly optimizing the policy on preference data using a closed-form relationship between reward and the optimal policy.
  • DPO and PPO comparison: In a three-action counter-example, DPO can minimize its loss while producing a policy that PPO cannot produce because the reference policy assigns zero probability to one action.

4. Understanding the Limitation of DPO

The paper argues that DPO has theoretical and empirical limitations, especially under distribution shift and limited preference-data coverage. Its analysis shows why DPO can favor unseen responses, while PPO can constrain such behavior through KL regularization and benefits from carefully matched training data.

  • Theoretical Analysis: DPO may be more susceptible than PPO to out-of-distribution responses because limited preference coverage can bias its learned policy toward unseen outputs.The paper links this limitation to narrow preference-data coverage and distribution shift between model outputs and the preference dataset.
  • Theoretical Analysis: ΠPPO is a proper subset of ΠDPO, so every policy induced by PPO is representable among DPO solutions, but DPO admits additional policies.The theorem establishes inclusion, while a counterexample demonstrates that the inclusion is strict.
  • Theoretical Analysis: DPO can minimize its loss while assigning probability to responses absent from the preference data, producing policies that PPO cannot generate under reference-model constraints.In the three-action counterexample, a DPO optimum assigns 0.1 probability to y1 and 0.9 to y3, whereas PPO assigns y3 zero probability.
  • Empirical Validation in A Synthetic Scenario: In the synthetic scenario, DPO increased an unseen response’s probability from 0.11 to 0.23, while PPO’s explicit KL regularization alleviated similar reward-model misspecification.The preference dataset covered only limited preference pairs, so performance improvements on out-of-distribution responses were not guaranteed.
  • Experiments on Real Preference Datasets: The paper recommends iterative DPO with careful annotation to mitigate distribution shift and noisy data, but reports that DPO remains unsatisfactory on challenging code-generation tasks.It also notes that excessively filtering high-quality preference data can hurt helpfulness.

5. Key Factors to PPO for RLHF

The study identifies three techniques that improve PPO’s RLHF performance and evaluates them across dialogue and code-generation tasks. The results emphasize large-batch training and reference-model exponential moving averages, particularly for challenging code generation.

  • Key techniques: PPO performance is investigated using advantage normalization, large-batch-size training, and exponential moving average updates of the reference model.These techniques are evaluated as factors in PPO training for RLHF.
  • Evaluation tasks: The ablation study covers HH-RLHF dialogue and the APPS and CodeContest competitive-programming datasets.Code correctness supplies direct reward signals or preference pairs for PPO and DPO training.
  • Ablation findings: Advantage normalization stabilizes PPO training and improves performance, while large batch sizes provide the most significant benefit, especially on code-generation tasks.The study reports that baseline PPO can degrade substantially on APPS with small batches.
  • Comparative results: On HH-RLHF, PPO and DPO are preferred by GPT-4 over dataset-chosen responses and SFT outputs, outperforming RRHF and PRO across all metrics.PPO consistently achieves higher reward and higher win rates than DPO in the reported comparison.
  • Ablation findings: Increasing PPO’s batch size consistently improves performance across all difficulty levels in APPS.A batch size of 64 can reduce the base SFT model’s performance to 33.7% on the introductory scale.

6. Benchmark Results

Across dialogue and code-generation benchmarks, PPO consistently outperforms DPO-based alternatives, while DPO often fails to improve supervised baselines on challenging programming tasks.

  • Dialogue benchmarks: PPO consistently achieves higher rewards and win rates than DPO on HH-RLHF, with GPT-4 preferring PPO responses directly.DPO-Iter improves over DPO but remains below PPO.
  • Dialogue benchmarks: After alignment on SafeRLHF, both DPO and PPO reduce harmfulness, while PPO produces more helpful responses.
  • Evaluation setup: APPS evaluates generated programs with pass@k, the proportion of problems solved using k generated programs per problem.
  • Code-generation benchmarks: DPO-Iter fails to improve SFT performance across APPS model sizes, whereas PPO improvements become more apparent as model size increases.CodeLlama-34B with PPO achieves state-of-the-art results on APPS.
  • Code-generation benchmarks: On CodeContest, PPO significantly improves SFT, while DPO produces no correct code after one epoch and DPO-Iter performs worse than SFT.PPO-assisted CodeLlama-34B surpasses the previous state-of-the-art result from Alphacode.

7. Conclusion

The paper identifies limitations of DPO and presents practical PPO tuning factors that support robust performance across tasks, including state-of-the-art code-competition results.

  • DPO is sensitive to distribution shift, while iterative DPO is suggested as preferable to training on static data.The paper also reports that DPO fails to improve challenging code-generation performance.
  • Advantage normalization, large batch size, and exponential-moving-average reference-model updates are identified as key PPO training factors.
  • PPO demonstrates robust effectiveness across diverse tasks and achieves state-of-the-art results in challenging code-competition tasks.
  • The study does not examine how to effectively train a robust reward model, although reward modeling is significant for PPO and DPO-Iter.

Impact Statements

The impact statement frames the study as aligning LLMs with human values while addressing bias, unfairness, transparency, privacy, and responsible AI practice.

  • The study emphasizes societal goals including bias elimination, unfairness reduction, transparency, privacy, and responsible AI practices.

A. Implementation Details

The implementation uses distinct DPO and PPO training configurations, with task-specific epochs, batch-size selection, and reference-model updates.

  • DPO: DPO uses β = 0.1 and learning rate 1e-6, with batch size swept for best performance.Training lasts two epochs on HH-RLHF and SafeRLHF, and one epoch on code-generation tasks.
  • PPO: PPO uses separate actor and critic learning rates of 1e-5 and 5e-6, respectively, with a default global batch size of 512.Rollout samples are divided into four mini-batches.
  • PPO: PPO runs for five epochs on HH-RLHF and SafeRLHF and sixteen epochs on APPS and CodeContest.Checkpoints with the highest validation reward or pass@k are selected.

B. GPT-4 Evaluation

The section describes GPT-4’s pairwise helpfulness evaluation and reports robustness findings for DPO and PPO under reference-model and β changes. DPO is more affected by distribution shift, while excessively large β can harm both methods.

  • GPT-4 Evaluation: GPT-4 compares two responses using a fixed helpfulness prompt and outputs a one-sentence comparison followed by “A” or “B”.The evaluation swaps response positions and assigns “Tie” when the two judgments disagree.
  • GPT-4 Evaluation: 100 randomly sampled test queries are evaluated with GPT-4, using both response orderings to reduce position effects.Inconsistent paired evaluations are converted to a final “Tie”.
  • Reference-Model Shift: DPO is more affected by distribution shift than PPO when the reference model changes on the Safe-RLHF and APPS datasets.Llama2-7B-SFT(Safe) and Codellama13B-SFT are closer to their respective preference datasets.
  • β Sensitivity: Too large β may harm both DPO and PPO, while β = 0.1 performs consistently well across models and tasks.The comparison uses OpenAssistant reward on HH-RLHF and average pass@5 on APPS.

C.3. Varying Preference Dataset

This section examines how preference-dataset coverage and composition affect PPO and DPO. PPO preserves a safe rate above 92% across filtering settings, while DPO is more affected by dataset changes; human evaluations favor PPO.

  • Preference-Dataset Coverage: Preference-dataset coverage affects both PPO and DPO, and DPO falls to performance similar to SFT when trained on the helpful-base subset.The results are reported on the HH-RLHF dataset.
  • Preference-Dataset Composition: PPO maintains a safe rate above 92% across all filtering settings, whereas DPO is more affected by preference-dataset composition.The filtering experiments remove dual-unsafe and dual-safe preference pairs on Safe-RLHF.
  • Preference-Dataset Composition: Filtering dual-unsafe samples yields significantly higher PPO helpfulness rewards, while filtering both dual-unsafe and dual-safe samples makes the model conservative.The authors attribute the conservative behavior to a reward model focused solely on safety, leading to frequent refusals.
  • Human Evaluation: Human evaluations agree with GPT-4 evaluations at rates of 60% and 61%, and favor PPO over DPO and DPO-Iter.Each reference pair is evaluated by four different people.
Loading 2404.10719v3…