PaperlayerView exact revision on arXiv

Source-linked AI summary

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi

arXiv:2406.09279v2cs.CL

TL;DR

Preference-learning practice varies in data and algorithms, making their contributions to downstream performance unclear. The paper systematically examines core components and finds that preference data quality matters most, with PPO outperforming DPO and a recipe combining synthetic data, PPO, and a large reward model.

  • Problem

    Preference-learning applications vary in data and algorithms, leaving their effects on downstream model performance unclear.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically varies preference data, learning algorithm, reward model, and policy-training prompts to examine their effects on downstream performance.

    Source: §1 · p. 2

  • Results

    Preference data quality matters most, followed by algorithm, reward-model quality, and targeted prompts; PPO outperforms DPO across varied datasets and evaluations.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The proposed recipe uses a strong synthetic dataset such as UltraFeedback with PPO and a large reward model, adding targeted prompts for a specific domain.

    Source: §1 · p. 2, §6 · p. 10

    Findings are based largely on TÜLU 2 with Llama 2 and 3, omit multilingual evaluation, and do not weigh PPO’s significantly higher computational cost.

    Source: A Limitations & Broader Impacts · p. 15

Abstract

from arXiv · show

Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training (https://github.com/hamishivi/EasyLM) and evaluating (https://github.com/allenai/open-instruct) our models, along with the models and datasets themselves (https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).

1 Introduction

Preference-learning practice varies in data and algorithm, leaving their effects on downstream performance unclear. The paper varies core components systematically and finds each matters, with preference data quality the most consequential.

1 Introduction

  • All four components matter, ordered by impact: preference data quality, algorithm choice, reward-model quality, then targeted policy-training prompts.The study starts from a strong open supervised fine-tuning model.Source: §1 · p. 2
  • Synthetic, diverse data with per-aspect preferences works best, and preference-pair quality matters more than the quality of the generations being compared.The comparison covers 14 existing preference datasets across diverse evaluations.Source: §1 · p. 2
  • PPO outperforms DPO across varied datasets and evaluations, including comparisons using the same models and initial training data.The introduction identifies PPO and DPO as the two most popular preference-learning algorithms.Source: §1 · p. 2
  • Larger or more data-trained reward models improve their own benchmark performance and substantially help GSM results, but yield marginal-to-no gains elsewhere.Targeted policy prompts help in domain-specific settings but have limited-to-no effect on overall performance.Source: §1 · p. 2
  • The suggested recipe combines synthetic preference data, PPO, and a large reward model, adding domain-specific prompts when targeting one task.The four components are preference data, learning algorithm, reward model, and policy-training prompts.Source: §1 · p. 2

2 Setup

The setup distinguishes preference-learning components and contrasts PPO’s reward-model-mediated online training with DPO’s direct offline policy optimization. Experiments use TÜLU 2 13B and evaluate models across 11 benchmarks.

2 Setup

  • Preference data trains the reward model in PPO and the policy directly in DPO; policy-training prompts elicit responses that PPO scores with its reward model.Preference records pair prompts with chosen and rejected responses and their rankings.Source: Figure 2 · p. 3, §2.1 · p. 3

2.1 PPO and DPO

  • PPO scores policy-generated responses with a trained reward model and optimizes the policy with a KL penalty, while DPO directly optimizes preference data offline.DPO avoids reward-model training and online response sampling; PPO uses online data generated by the current policy.Source: §2.1 · pp. 3–4

2.2 Experimental and Evaluation Setup

  • Experiments use TÜLU 2 13B as the base policy and evaluate 11 benchmarks spanning factuality, reasoning, truthfulness, coding, safety, and instruction following.The reported metrics include category averages and an overall average across categories.Source: §2.2 · p. 4

3 Exploring Learning from Preference Feedback

The section compares preference data, learning algorithms, reward models, and policy-training prompts to assess their effects on downstream performance.

3.2 Preference Learning Algorithm: DPO vs. PPO

  • PPO outperforms DPO across datasets, averaging 0.7 points higher, with its largest average gains in reasoning, coding, and safety.PPO averages 1.3, 2.9, and 2.3 points higher in reasoning, coding, and safety, respectively, while truthfulness averages 2.5 points lower; instruction following and factuality remain largely similar.Source: Table 2 · p. 5, §3.2 · p. 6

3.1 Preference Data

  • Synthetic datasets with per-aspect annotations perform best; preference-based learning most improves instruction following and truthfulness, while factuality changes little.The comparison covers 14 preference datasets; the strongest gains in instruction following and truthfulness exceed 8 points over SFT. The authors also report poor safety performance for Chatbot Arena data.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.3 Reward Models

  • Larger or more data-trained reward models improve direct reward-model evaluations, but yield only small downstream gains, largely from GSM with the 70B UltraFeedback model.RewardBench and Best-of-N do not move in lockstep: the 13B mixture model leads on RewardBench, while 70B models perform best on Best-of-N. Continued training also produces different trends across evaluations.Source: §3.3 · pp. 6–7, Table 3 · p. 6

3.4.2 Altering prompts to improve overall performance.

  • The remixed prompt pool does not consistently improve math or code results, and overall performance tends to drop.The pool combined math, code, and UltraFeedback prompts; the authors suggest UltraFeedback's existing diversity may explain the result, and adding all mined prompts did not help further.Source: Table 4 · p. 8, §3.4.2 · pp. 8–9

3.4.1 The effect of targeted prompts.

  • On GSM, 70B reward models paired with GSM-train prompts improve the best model from 46% to 62%, while weaker models struggle with out-of-domain prompts.The results suggest that targeted prompt distributions can help when paired with powerful reward models; PPO uses unlabeled prompts to target downstream tasks.Source: §3.4.1 · p. 8, Figure 3 · p. 8

4 A Recipe for Learning from Preferences

The proposed recipe combines a high-quality synthetic preference dataset, a large reward model, and PPO; targeted policy-training prompts may help when focused on a specific domain.

4 A Recipe for Learning from Preferences

  • The best TÜLU 2+PPO 13B model with a 70B UltraF. reward model outperforms the best DPO model and other popular Llama 2 13B-based models, including Llama 2 Chat.The comparison is reported alongside a recipe using a high-quality synthetic preference dataset and PPO.Source: §4 · p. 9
  • The recipe uses a high-quality synthetic preference dataset, a large reward model, and PPO.A smaller 13B reward model performs almost as well when compute-constrained.Source: §4 · p. 9
  • Task-specific policy-training prompts may further improve performance when they align closely with downstream evaluations.The paper also reports similar overall trends in Llama 3.0 8B experiments, except that DPO performs better than PPO there.Source: §4 · p. 9

5 Related Work

Prior work studies preference-learning algorithms and their variants, while this paper focuses on how data and model choices affect PPO and how PPO compares with DPO.

Learning from Preferences for LMs

  • The paper examines variation in PPO data and models—including reward models, preference data, initial policy models, and prompts—and expects findings to transfer to related approaches such as REINFORCE.The passage links that expectation to shared reliance on a tuned reward model and unlabeled prompts for eliciting generations.Source: §5 · pp. 9–10
  • Earlier RLHF approaches train a reward model on human preferences and optimize it with reinforcement learning such as PPO; later work also explores REINFORCE with LM-specific adjustments.Source: §5 · p. 9
  • Because DPO is widely used and prior comparisons found DPO variants largely similar, the paper compares the popular DPO variant with PPO and tests which data works best for DPO.The passage notes DPO’s ease of implementation and use, and identifies TÜLU 2, Zephyr, and Mixtral-Instruct as high-performing DPO-trained models.Source: §5 · p. 10

Comparing Approaches for Learning from Preferences

  • Concurrent comparisons report that DPO can fare poorly on out-of-distribution data, while PPO and semi-online DPO perform better in safety and code; other work studies on-policy sampling, negative gradients, and online data updates.The paper distinguishes its focus as empirical examination of core components, including varied rewards and data.Source: §5 · p. 10

6 Conclusion

The study ranks preference-data quality as the most influential component, followed by algorithm choice, reward-model quality, and targeted policy-training prompts. It recommends synthetic preference data and PPO with a large reward model, while noting compute and research boundaries elsewhere in the paper.

6 Conclusion

  • The reported importance ordering is preference-data quality, algorithm choice, reward-model quality, then targeted policy-training prompts.Source: §6 · p. 10
  • Larger reward models significantly improve math capabilities but have marginal effects on other evaluated capabilities.Source: §6 · p. 10
  • The recommended recipe with currently available resources is the UltraFeedback synthetic dataset and PPO training with a large reward model.Source: §6 · p. 10
  • The paper identifies better use of improved reward models as an important direction for further improving PPO-style preference learning.Source: §6 · p. 10

A Limitations & Broader Impacts

The empirical study centers on the TÜLU 2 model suite and Llama 2 and 3 base models, with no multilingual evaluation. It also notes that relying heavily on single preference datasets may align models to only a small subsection of the global population.

Limitations

  • The study’s in-depth examination centers on TÜLU 2 and two base models, Llama 2 and 3, and does not test multilingual performance.Whether results carry to multilingual settings or greatly differing base models remains future work.Source: A Limitations & Broader Impacts · p. 15
  • The study evaluates performance but does not explicitly measure the relative computational cost of methods, although PPO costs significantly more than DPO.Source: A Limitations & Broader Impacts · p. 15

Broader Impacts

  • Focusing too heavily on learning from single preference datasets may align language models to only relatively small subsections of the global population.Source: A Limitations & Broader Impacts · p. 15

B Dataset Details

The study draws on a varied set of preference datasets, including filtered or downsampled versions and alternative UltraFeedback preference constructions. It uses TÜLU 2 models and EasyLM-based training code.

B Dataset Details

  • The dataset collection spans public preference sources, including Chatbot Arena, HH-RLHF, HelpSteer, Nectar, and two UltraFeedback variants.For UltraFeedback, chosen/rejected responses are selected either by overall scores or by averaged fine-grained scores.Source: B Dataset Details · pp. 15–16
  • Chatbot Arena samples exclude multi-turn conversations and ties, while PRM800k pairs one correct and one incorrect generation for selected prompts.The study also downsamples SHP-2 and StackExchange to 500,000 samples each.Source: B Dataset Details · p. 15
  • The reward-model training mix contains roughly 260,000 preference samples, over four times the size of the UltraFeedback split.HH-RLHF, StackExchange, and Nectar are also downsampled to 60,908 examples for size-equal algorithm comparisons.Source: B Dataset Details · p. 16

C Model and Code details

  • Experiments primarily use TÜLU 2 13B and 70B models, and training code builds on the TÜLU 2 fork of EasyLM.TÜLU 2 models are based on Llama 2.Source: C Model and Code details · p. 16

D Evaluation Details

The evaluation suite spans instruction following, reasoning, factuality, truthfulness, coding, and safety. The section also describes UltraFeedback subset construction and the study's DPO and PPO training procedures.

D Evaluation Details

  • The evaluation suite measures refusal behavior, factuality, reasoning, truthfulness, and toxicity using XSTest, MMLU, GSM8k, BBH, TruthfulQA, and ToxiGen.GSM8k uses chain-of-thought with eight few-shot examples; TruthfulQA reports informative and truthful percentages.Source: D Evaluation Details · pp. 16–17
  • Instruction following is evaluated with AlpacaEval 1 and length-controlled AlpacaEval 2, plus IFEval loose prompt-level accuracy.Coding evaluation uses HumanEval+ and MBPP+ pass@10.Source: D Evaluation Details · p. 17

E UltraFeedback Subset Study

  • UltraFeedback subsets show little overall performance difference across generation-strength groups, while prompt-source subsets each lead in at least one category.The authors interpret this as evidence that GPT-4 preference annotations matter more than the strength of sampled generations, and that diverse prompt sources matter.Source: E UltraFeedback Subset Study · p. 18, Table 7 · p. 18

F.1 DPO

F.2 PPO

  • PPO scores responses generated online by the policy with a reward model, then uses those rewards to guide policy optimization with a KL penalty.Its value model estimates expected values of partial responses and supports token-level advantage estimation.Source: §F.2 · pp. 19–20
  • The PPO implementation initializes its value model from the reward model and whitens advantages, but not step-level rewards.For truncated completions, it assigns a large negative reward; it uses a fixed rather than adaptive KL coefficient.Source: §F.2 · p. 20
  • PPO training uses one epoch and one inner epoch; larger reward models use a reduced KL penalty of β = 0.0325 instead of β = 0.05.Training beyond one epoch did not improve performance and occasionally collapsed; multiple inner epochs destabilized training.Source: §F.2 · p. 20

G Additional Observations with PPO

PPO training beyond one epoch did not improve performance and sometimes collapsed. Increasing prompt batch size sped training but reduced performance, while current minibatch-level forward passes helped stabilize training.

G Additional Observations with PPO

  • Training beyond one epoch did not improve performance and occasionally caused collapse, while multiple inner epochs destabilized training.The experiments therefore use one training epoch and one inner epoch per batch.Source: G Additional Observations with PPO · p. 20
  • Stale forward-pass values destabilized PPO because the value-model loss did not converge fast enough to provide reliable advantage signals.The authors instead performed forward passes at the minibatch level using the most current models.Source: G Additional Observations with PPO · pp. 20–21
  • An 8x larger prompt batch cut PPO training time from 60 hours to 12 hours but lowered overall performance from 61.7% to 61.1%.A rollout multiplier of four recovered most of the performance loss, with a less dramatic speedup.Source: G Additional Observations with PPO · p. 21, Table 11 · p. 22

H Reward Model Evaluation Details

The appendix details reward-model evaluation and how KL-penalty choices interact with reward-model size and evaluation objectives. It also describes mining and filtering unlabeled prompts by task domain.

H Reward Model Evaluation Details

I KL Penalty Coefficient Exploration

  • The optimal KL coefficient varies by reward model: 70B is more robust at lower β, while shrinking β causes large overall-performance drops with 13B.The authors identify greater robustness to β choices as a potential benefit of larger reward models.Source: I KL Penalty Coefficient Exploration · p. 23, Figure 4 · p. 23
  • AlpacaEval 2 performs best at the lowest tested β, but this coincides with significant reductions on other evaluations, including GSM.The authors connect this pattern to reward-model preferences aligned with GPT-4 and note the trade-off for generalist models.Source: I KL Penalty Coefficient Exploration · p. 23, Figure 4 · p. 23

J Prompt Domain Identification Details

  • Unlabeled prompts from UltraFeedback, WildChat, and LMSYS-1M are tagged by TÜLU 2 70B, then filtered to select clear English coding or math queries.Prompts tagged unclear or multilingual are removed; the authors manually examined a small portion of the resulting collections.Source: J Prompt Domain Identification Details · p. 23, Figure 5 · p. 24

K Compute Details

The compute details compare training costs for PPO and DPO on UltraFeedback and note that additional smaller runs were not reported.

K Compute Details

  • PPO training with a 13B reward model and 13B policy on UltraFeedback takes 3 days on a 256-size TPUv3 pod.Reward-model training takes roughly 10 hours.Source: K Compute Details · p. 25
  • DPO training a 13B model on roughly 60,000 UltraFeedback samples takes 9 hours.Source: K Compute Details · p. 25
  • All models are trained on 256- or 128-size TPUv3 pods, and additional smaller-size runs are not reported.Those runs explored dataset and hyperparameter choices.Source: K Compute Details · p. 25

L Model Performance over Training

Performance changes differently across evaluations as PPO training continues, so the training duration that looks best depends on the benchmark. The authors caution that gains on llm-as-judge benchmarks may come at the cost of other desirable properties.

L Model Performance over Training

  • Across evaluations, performance may improve continuously, improve and then degrade, or remain relatively unchanged during training.AlpacaEval 2 continuously improves; IFEval and GSM8k improve and then degrade; MMLU remains relatively unchanged.Source: L Model Performance over Training · p. 26
  • AlpacaEval 2 favors 3 or more epochs, IFEval favors 500 steps (0.5 epochs), and GSM8k peaks at 1 epoch.Source: L Model Performance over Training · p. 26
  • The authors caution that improved AlpacaEval performance may come at the cost of other desirable properties, supporting evaluation across a broad variety of benchmarks.Source: L Model Performance over Training · p. 26

Related papers

Loading 2406.09279v2…