PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

arXiv:2305.18290v3cs.LGcs.AIcs.CL

TL;DR

Large language models are difficult to steer precisely, while RLHF is complex and unstable. DPO directly trains models from preferences with a simple classification loss, matching or exceeding PPO-based RLHF across sentiment control, summarization, and dialogue, including a 61% summarization win rate.

  • Problem

    Large unsupervised language models acquire broad capabilities but do not reliably reflect the desired goals, priorities, and skills during generation.

  • Method

    DPO directly optimizes language models from preferences by reparameterizing the reward model, replacing explicit reward modeling and reinforcement learning with a simple classification loss.

  • Results

    DPO performs similarly or better than existing preference-learning methods across sentiment modulation, summarization, and dialogue, achieving approximately a 61% summarization win rate at temperature 0.0.

  • Takeaways & Limitations

    DPO offers a substantially simpler alternative to PPO-based RLHF for training language models from preferences, with virtually no hyperparameter tuning.

  • Takeaways & Limitations

    DPO’s out-of-distribution generalization relative to explicit-reward methods remains insufficiently studied.

Abstract

from arXiv · show

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

1 Introduction

The paper introduces Direct Preference Optimization (DPO), a reinforcement-learning-free method that directly trains language models from human preferences. DPO implicitly optimizes the same KL-constrained reward objective as RLHF while replacing explicit reward modeling and reinforcement learning with a simple classification objective.

  • Introduction: The method replaces the standard RLHF pipeline with a simple binary cross-entropy objective that exactly optimizes its RL-based objective.Existing methods first fit a reward model and then use reinforcement learning, whereas DPO simplifies this pipeline.
  • Introduction: DPO directly optimizes language models to adhere to human preferences without explicit reward modeling or reinforcement learning.It is presented as a simple RL-free algorithm for preference-based training.
  • Introduction: DPO implicitly optimizes reward maximization subject to a KL-divergence constraint while increasing the relative log probability of preferred responses over dispreferred responses.This provides the paper’s intuitive account of how the preference update works.
  • Introduction: DPO is reported to be at least as effective as existing methods, including PPO-based RLHF, for sentiment modulation, summarization, and dialogue with language models up to 6B parameters.The experiments evaluate preference learning across these tasks and model sizes.

2 Related Work

Prior work improves self-supervised language models through instruction tuning and preference-based reinforcement learning, while related research studies preference-based policy learning beyond language. These approaches include human-feedback reward optimization, synthetic preference data, and contextual dueling bandits.

  • Instruction tuning: Instruction tuning on instructions and human-written completions improves downstream performance, user-intent alignment, generalization, and usability of self-supervised language models.Such models can otherwise perform some tasks zero-shot or through few-shot prompts.
  • Preference-based reinforcement learning: Human-feedback methods commonly fit a Bradley-Terry reward model and optimize language models with reinforcement learning algorithms including REINFORCE and PPO.A related approach uses instruction-following LLMs with human feedback to generate synthetic preference data for targeted attributes such as safety or harmlessness.
  • Preference-based policy learning: Beyond language, preference-based policy learning has been studied in bandit and reinforcement learning settings, including contextual dueling bandits that rank actions instead of using absolute rewards.Without absolute rewards, contextual dueling bandit theory substitutes a von Neumann winner for an optimal policy.

3 Preliminaries

RLHF typically proceeds through supervised fine-tuning, preference-based reward learning, and reinforcement-learning optimization. The final policy is constrained relative to the initial SFT model to preserve reward-model validity and generation diversity.

  • Pipeline: RLHF usually comprises three phases: supervised fine-tuning, preference sampling and reward learning, and RL optimization.The pipeline is reviewed as the standard procedure in prior RLHF work.
  • Supervised fine-tuning: SFT fine-tunes a pre-trained LM on high-quality downstream-task data to obtain πSFT.Example tasks include dialogue and summarization.
  • Reward modeling: In reward modeling, πSFT generates answer pairs that human labelers rank, with preferences modeled through a latent reward function.The Bradley–Terry model represents preference probabilities using exponentiated rewards, and reward parameters are estimated by maximum likelihood via binary classification.
  • RL fine-tuning: RL fine-tuning uses the learned reward while constraining πθ’s deviation from the reference policy πref, initialized as πSFT.The constraint keeps the policy near the distribution where the reward model is accurate, maintains diversity, and prevents mode collapse.

4 Direct Preference Optimization

DPO reparameterizes the RLHF reward model so its optimal policy can be recovered in closed form, replacing reinforcement-learning optimization with a direct preference-classification objective. Its update increases preferred completions’ likelihood and decreases dispreferred completions’ likelihood, weighting examples by implicit reward errors.

  • Deriving the DPO objective: DPO uses a reward-model parameterization whose optimal policy is extracted in closed form, avoiding an RL training loop.This directly targets the difficulty of applying reinforcement-learning algorithms to large-scale language-model fine-tuning.
  • Deriving the DPO objective: The Bradley-Terry preference model cancels the unknown partition function, expressing human preference probabilities using only the optimal and reference policies.The same reparameterization can also be derived for the more general Plackett-Luce model.
  • Deriving the DPO objective: DPO fits a parametrized policy by maximum likelihood, thereby fitting an implicit reward whose optimal policy is the policy πθ.Because this is a reparameterized Bradley-Terry model, DPO has consistency properties under suitable assumptions about the preference-data distribution.
  • What does the DPO update do?: DPO increases the likelihood of preferred completions and decreases the likelihood of dispreferred completions, weighting examples by how incorrectly the implicit reward model orders them.The gradient assigns higher weight when the implicit reward model rates the dispreferred completion more highly.

5 Theoretical Analysis of DPO

DPO reparameterizes reward models as a reference-policy log ratio, preserving representational generality while recovering the constrained-RL optimal policy exactly. Its normalization term also explains why omitting value-related structure can destabilize actor-critic optimization.

  • Reward reparameterization: DPO replaces explicit reward fitting and reinforcement learning with a single maximum-likelihood objective equivalent to Bradley-Terry optimization under a policy-ratio reward parameterization.The parameterization is r*(x, y) = β log [πθ(y|x) / πref(y|x)].
  • Reward equivalence: Reward functions in the same equivalence class induce the same preference distribution and the same optimal policy under constrained reinforcement learning.The equivalence relation captures the under-specification of Plackett-Luce and Bradley-Terry preference models.
  • Representational completeness: Under mild assumptions, every reward class consistent with Plackett-Luce or Bradley-Terry preferences has the form r(x, y) = β log [π(y|x) / πref(y|x)].The projection adds only an x-dependent normalization term, so it remains in the original reward equivalence class and loses no generality.
  • Exact policy recovery: DPO constrains the under-specified preference model while preserving representable reward classes and making the optimal policy explicitly recoverable.The selected reward’s normalization term corresponds to the partition function of its induced optimal policy.
  • Actor-critic stability: Omitting the normalization term can produce high-variance policy gradients and unstable learning, although a learned value function can accommodate it.The normalization term can be interpreted as the soft value function of the reference policy and does not change the optimal solution.

6 Experiments

The experiments evaluate DPO across controlled sentiment generation, summarization, and single-turn dialogue, emphasizing reward–KL tradeoffs and preference-based win rates. DPO strictly dominates PPO on the controlled sentiment reward/KL frontier, while GPT-4 judgments are supported as a reasonable proxy for human evaluation.

  • Experimental setup: The evaluation covers controlled sentiment generation, TL;DR summarization, and single-turn Anthropic-HH dialogue tasks.The study also tests distribution shift on CNN/DailyMail articles using policies trained for Reddit TL;DR summarization.
  • Evaluation: Algorithms are compared using reward–KL frontiers in controlled sentiment and GPT-4 win rates against reference or ground-truth completions in downstream tasks.The reward–KL frontier is computable because the controlled sentiment experiment has a ground-truth sentiment-classifier reward.
  • Baselines: The study evaluates DPO alongside zero-shot or few-shot prompting, SFT, Preferred-FT, and other preference-training approaches.The baselines include GPT-J for summarization, Pythia-2.8B prompting for dialogue, and supervised fine-tuning on chosen completions.
  • Controlled sentiment generation: DPO’s reward/KL tradeoff strictly dominates PPO, including when PPO has access to ground-truth rewards.The comparison accounts jointly for reward achieved and KL divergence from the reference policy.
  • Evaluation reliability: GPT-4 agrees with human judgments about as often as humans agree with one another, supporting GPT-4 as a reasonable proxy for evaluation.The human study uses two GPT-4 prompts and compares multiple methods from the TL;DR summarization experiment.

7 Discussion

The discussion presents DPO as a simple preference-training paradigm that avoids reinforcement learning by directly mapping language-model policies to reward functions. It also identifies open questions about out-of-distribution generalization, self-labeling, and reward over-optimization.

  • Contribution: DPO trains language models from preferences without reinforcement learning by mapping language-model policies to reward functions.The authors frame this as an alternative to forcing preference learning into a standard reinforcement-learning setting.
  • Limitations & Future Work: Initial results suggest DPO policies generalize similarly to PPO-based models out of distribution, but broader study is needed.The discussion explicitly presents this as an unresolved comparison with learning from an explicit reward function.
  • Limitations & Future Work: Future work should examine whether self-labeling from a DPO policy can effectively use unlabeled prompts and how reward over-optimization behaves.These are posed as open questions rather than established findings.

A Mathematical Derivations … A.4 Deriving the Gradient of the DPO Objective

The appendices derive the KL-constrained optimum, recover the DPO objective under Bradley–Terry and Plackett–Luce preferences, and obtain its gradient. The derivations establish the policy parameterization and maximum-likelihood objectives used by DPO.

  • A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective: The KL-constrained reward maximization derivation defines a partition function depending only on x and πref, not on π.This permits the objective to be reorganized around a KL-divergence term.
  • A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective: The resulting π∗ is a valid probability distribution, satisfying π∗(y|x) ≥ 0 and summing to 1 over y.Its normalization follows because Z(x) does not depend on y.
  • A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective: Gibbs’ inequality shows that the minimum is achieved when the relevant KL-divergence is zero, meaning the policy and target distributions are identical.This yields the optimal solution for all x ∈ D.
  • A.2 Deriving the DPO Objective Under the Bradley-Terry Model: Under the Bradley–Terry preference model, substituting the optimal-policy parameterization of the unavailable reward produces the DPO objective.The final expression is identified as the per-instance loss in Equation 7.
  • A.3 Deriving the DPO Objective Under the Plackett–Luce Model: The Plackett–Luce model generalizes Bradley–Terry from pairwise comparisons to rankings over K possible answers.When K = 2, Equation 18 reduces to Bradley–Terry, while the general model assigns permutation probabilities using a latent reward.
  • A.3 Deriving the DPO Objective Under the Plackett–Luce Model: For datasets of prompts and user-specified rankings, a parameterized model can optimize the Plackett–Luce objective by maximum likelihood.The normalization constant Z(x) cancels after substituting the reward parameterized by its optimal policy.
  • A.4 Deriving the Gradient of the DPO Objective: The final appendix derives the gradient of the DPO objective by rewriting its right-hand side and applying σ′(x) = σ(x)(1 − σ(x)) and σ(−x) = 1 − σ(x).The derivation concludes with the gradient form given in Section 4, including πref(y|x).

A.5 Proof of Lemma 1 and 2 · A.6 Proof of Theorem 1

The appendices prove that reward functions differing only by a prompt-dependent offset induce identical preferences and constrained-optimal policies. They then establish that every reward equivalence class has a unique representation through the reference-policy log ratio parameterization.

  • A.5 Proof of Lemma 1 and 2: Under Plackett-Luce preferences, including Bradley-Terry as the K = 2 case, reward functions in the same equivalence class induce the same preference distribution.The equivalence relation is r′(x, y) = r(x, y) + f(x).
  • A.5 Proof of Lemma 1 and 2: Reward functions differing by a prompt-only term induce the same optimal policy under the constrained RL problem.This follows by comparing the corresponding optimal policies under the stated equivalence relation.
  • A.6 Proof of Theorem 1: A reference model with πref(y|x) > 0 and β > 0 permits every reward equivalence class to be represented as r(x, y) = β log π(y|x) πref(y|x).The model π(y|x) is the policy associated with the reward under the KL-constrained RL solution.
  • A.6 Proof of Theorem 1: Subtracting β log Z(x) from a reward preserves its equivalence class and yields the theorem’s reference-policy log-ratio representation.The normalization term Z(x) depends on the reward function, while the transformation is r′(x, y) = r(x, y) − β log Z(x).
  • A.6 Proof of Theorem 1: The operator f maps all reward functions in a given equivalence class to the same reward function.The proof uses the fact that equivalent rewards have the same optimal policy by Lemma 2.
  • A.6 Proof of Theorem 1: For β > 0, two equivalent rewards with distinct policy representations would imply a prompt-dependent multiplicative factor between normalized policies, forcing that factor to equal 1.Summing over completions shows f(x) = 0 for all x, so the two reward functions are identical.
  • A.6 Proof of Theorem 1: Consequently, every reward class has a unique reward function represented by f(r, πref, β).The unique representative can be obtained from any reward function in the class.

B DPO Implementation Details and Hyperparameters · C Further Details on the Experimental Set-Up

This section specifies DPO’s implementation through its preference-pair loss and reward computation, then lists default optimization settings and notes additional experimental-design details. The implementation indexes preferred and dispreferred completions against policy and reference log probabilities, with β controlling the KL penalty strength.

  • B DPO Implementation Details and Hyperparameters: DPO is implemented with a PyTorch loss function operating on policy and reference-model log probabilities.The section provides PyTorch code for the DPO loss.
  • B DPO Implementation Details and Hyperparameters: The loss accepts policy and reference log probabilities, preferred and dispreferred completion indices, and β as inputs.The index tensors have shapes (T,), while both log-probability tensors have shape (B,).
  • B DPO Implementation Details and Hyperparameters: Each index pair (yw_idxs[i], yl_idxs[i]) identifies one preference pair, with preferred and dispreferred completions selected from the corresponding batches.Preferred completion indices lie in [0, B-1], as do dispreferred completion indices.
  • B DPO Implementation Details and Hyperparameters: DPO forms policy and reference log-ratios by subtracting dispreferred-completion log probabilities from preferred-completion log probabilities.The policy and reference log-ratios are computed separately after indexing both completion types.
  • B DPO Implementation Details and Hyperparameters: The loss is -F.logsigmoid(β * (pi_logratios - ref_logratios)), while detached rewards equal β * (pi_logps - ref_logps).These expressions define the optimization loss and corresponding reward values in the provided implementation.
  • B DPO Implementation Details and Hyperparameters: Unless otherwise noted, DPO uses β = 0.1, batch size 64, RMSprop, learning rate 1e-6, and a 150-step linear warmup.TL;DR summarization instead uses β = 0.5 while the remaining parameters stay unchanged.
  • C Further Details on the Experimental Set-Up: The experimental-set-up section introduces additional details relevant to the paper’s experimental design.No further experimental-design specifics are included in the supplied passage.

C.1 IMDb Sentiment Experiment and Baseline Details … D.1 Performance of Best of N baseline for Various N

The experiments use carefully specified sentiment, GPT-4 preference-judgment, and baseline setups. Results also characterize limitations of unlikelihood training and the strength but computational cost of Best of N.

  • C.1 IMDb Sentiment Experiment and Baseline Details: IMDb sentiment experiments use 2–8-token prefixes, a sentiment-RoBERTa classifier as ground-truth reward model, gpt2-large as base model, and one-epoch supervised fine-tuning.The setup samples four completions for 25,000 prefixes to construct preference data.
  • C.2 GPT-4 prompts for computing summarization and dialogue win rates: GPT-4 win rates use gpt-4-0314, with randomly ordered summaries or responses in every evaluation.Summarization prompts ask which output better captures important points, with or without irrelevant details; dialogue prompts ask which response is more helpful.
  • C.2 GPT-4 prompts for computing summarization and dialogue win rates: Evaluators provide a one-sentence comparison followed by a single A-or-B preference using the prescribed output format.This format is specified for both summarization and dialogue judgments.
  • C.3 Unlikelihood baseline: The unlikelihood baseline maximizes the preferred response’s log probability while minimizing the dispreferred response’s log probability, but is excluded from summarization and dialogue.It is included in sentiment experiments but generally produces meaningless responses for more complex tasks, attributed to unconstrained likelihood minimization.
  • C.3 Unlikelihood baseline: Table 3 reports unlikelihood samples from TL;DR prompts at temperature 1.0 and finds that the method generally fails to generate meaningful summarization and dialogue responses.This supports the decision not to use the baseline in those experiments.
  • D.1 Performance of Best of N baseline for Various N: Best of N is a strong baseline in these experiments, but it is computationally expensive because it requires sampling many times.Its performance is evaluated across various N values for Anthropic-HH dialogue and TL;DR summarization in Figure 4.
  • D Additional Empirical Results: The additional empirical results evaluate Best of N across multiple sample counts for Anthropic-HH dialogue and TL;DR summarization.These evaluations are presented in Figure 4.

D.2 Sample Responses and GPT-4 Judgments · D.3 Human study details

The appendix illustrates DPO-versus-baseline comparisons for summarization and dialogue, including GPT-4 judgments that favor either DPO or ground-truth responses. A human study then compares GPT-4 and human preferences across several summarization matchups using volunteer raters.

  • D.2 Sample Responses and GPT-4 Judgments: Tables 4–6 present TL;DR summaries generated by DPO at temperature 0.25 and PPO at temperature 0, with randomized presentation for GPT-4 evaluation.Bold post-hoc annotations clarify evaluation but are not part of the model generations.
  • D.2 Sample Responses and GPT-4 Judgments: Tables 7–8 show GPT-4 choosing DPO over ground-truth completions for Anthropic-HH prompts at DPO temperatures 0.7 and 1.0.The ground truth is the chosen completion in the preference dataset.
  • D.2 Sample Responses and GPT-4 Judgments: Tables 9–10 show GPT-4 choosing ground truth over DPO, including a verbose but factually incorrect DPO response and a case where GPT-4 incorrectly judges the ground truth as correct.The examples identify factual errors involving WWII and a nonexistent organization.
  • D.3 Human study details: The human study validates GPT-4 win rates by comparing DPO, SFT, and PPO against PPO at temperature 0 across three algorithmic matchups with varied win rates.The summarization setting uses DPO and SFT at temperature 0.25, PPO at temperature 1.0, and PPO at temperature 0 as the reference.
  • D.3 Human study details: Each respondent completed 25 similarly formatted judgments through a SurveyMonkey interface.The survey layout is documented in Figure 5.
  • D.3 Human study details: The study involved 25 volunteer human raters, each comparing 25 summaries; one volunteer completed the survey late and was excluded from final analysis.Raters were Stanford students, recent graduates, or visitors, primarily with STEM and computer-science backgrounds.
Loading 2305.18290v3…