PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Preference Optimization with an Offset

Afra Amini, Tim Vieira, Ryan Cotterell

arXiv:2402.10571v2cs.CLcs.AIcs.LG

TL;DR

DPO ignores how strongly one response is preferred, even when that degree is available from human judgments or point-wise scores. ODPO incorporates preference strength through an offset tied to reward differences; experiments report better reward–KL trade-offs and summarization win rates than DPO. The authors note that preference data and offset scaling constrain the settings studied.

  • Problem

    DPO accounts for which response is preferred but not the extent of preference, despite available human judgments or point-wise scores that capture response-quality differences.

    Source: §1 · p. 1

  • Method

    ODPO generalizes DPO by enforcing a likelihood gap based on the reward difference between preferred and dispreferred responses, with offset strength controlled by α.

    Source: §5 · pp. 4–5

  • Results

    ODPO more often lies on the reward–KL Pareto frontier than DPO and achieves a higher average win rate than DPO over human-written summaries across sampling temperatures.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results motivate collecting human feedback that records the degree of preference between responses.

    Source: Limitations · p. 9

    Experiments used preference-strength information or point-wise rewards, and offset scaling was tested with Likert scores and classifier probabilities; extending ODPO to other tasks is future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.

1 Introduction

The introduction presents ODPO as a way to use preference strength when aligning language models, and the preliminaries explain why reward-based objectives can better reflect response quality than maximum likelihood. The paper reports stronger reward–KL trade-offs and summarization win rates for ODPO than DPO in its experiments.

1 Introduction

  • RLHF commonly uses human comparisons to train a reward model, then reinforcement learning to update the language model, but reward-model cost and RL hyperparameter sensitivity make this approach cumbersome.These burdens motivate simpler alternatives such as DPO, which directly fine-tunes on preference data without a reward model or RL algorithm.Source: §1 · p. 1
  • DPO uses preference ordering but not preference strength, even when human judgments or point-wise scores provide information about response quality differences.Preference-strength judgments can range from significantly better to negligibly better or unsure.Source: §1 · p. 1
  • ODPO generalizes DPO by requiring a likelihood gap tied to the responses’ reward difference; setting the offset to zero recovers DPO.The offset grows with the preferred response’s reward advantage.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • ODPO more often lies on the reward–KL Pareto frontier than DPO and achieves a higher average win rate than DPO over human-written summaries across sampling temperatures.The results cover sentiment and toxicity control, plus summarization using human Likert scores to set offsets.Source: §1 · p. 2

2 Preliminaries

  • Maximum-likelihood fine-tuning can fail to distinguish major from minor errors and assigns probability mass even to lower-quality responses in the fine-tuning data.The passage contrasts hallucinations with synonym choice in summarization as major and minor errors, respectively.Source: §2 · p. 2
  • Expected-reward optimization targets response quality, while regularization is often added to limit reward hacking and divergence from the supervised fine-tuned model.The objective can optimize expected reward directly, with a regularization term to constrain drift from πSFT.Source: §2 · pp. 2–3

3 Reward Modeling

Reward modeling turns judgments about responses into scores that can guide language-model fine-tuning. Pointwise rewards suit tasks with readily scored attributes, while pairwise preferences help when absolute quality is harder to judge.

Reward Modeling

  • Pointwise reward functions assign real-valued scores to outputs, using classifiers for attributes such as sentiment or toxicity and automated evaluation for code.For open-ended generation, classifiers can assess desired or undesired attributes.Source: §3 · p. 3
  • For summarization and dialogue, human pairwise preferences can replace difficult absolute-quality judgments, representing each example as a prompt and preferred and dispreferred responses.The preference data is commonly modeled with a Bradley–Terry model.Source: §3 · p. 3
  • A reward model is trained as a binary classifier to maximize the likelihood of the preference data, with higher rewards intended for human-preferred responses.The estimated reward function is then used for policy estimation.Source: §3 · p. 3
  • Directly optimizing the reward-based policy objective can be complex and resource-intensive, motivating alternative approaches to RLHF.Source: §3 · p. 3

4 Direct Preference Optimization

DPO directly optimizes a language model from preference data, avoiding explicit reward-model training. ODPO extends it with a reward-difference-based offset between preferred and dispreferred responses.

4 Direct Preference Optimization

  • DPO directly maximizes preference-data likelihood under the Bradley–Terry assumption, with the normalization factor canceling when the reward is expressed through the policy.Its loss requires the estimated reward for the preferred response to exceed that for the dispreferred response.Source: §4 · p. 4

5 DPO with an Offset

  • ODPO requires a reward gap between preferred and dispreferred responses, with an offset determined by their actual reward difference.The offset is modeled as a monotonically increasing function of the score difference; α controls enforcement, and α = 0 makes ODPO equivalent to DPO.Source: §5 · pp. 4–5
  • The ODPO construction uses the Bradley–Terry model’s connection to Gumbel random variables to formulate its offset loss.The exposition introduces this connection as the route to constructing the loss function.Source: §5 · p. 4
  • In all three movie-review settings, most or nearly all Pareto-front points belong to ODPO.The figure highlights Pareto-front points with black borders.Source: Figure 2 · p. 5
  • ODPO is equivalent to softmax-margin loss only when the offset is restricted to be non-negative.The softmax-margin connection augments softmax with a cost function that penalizes high-cost responses.Source: §5 · p. 5

6 Experiments

Across sentiment control, toxicity control, and summarization, the experiments compare ODPO with DPO using reward–KL tradeoffs or human preferences. ODPO’s relative advantage varies by task and dataset size.

6.1 Sentiment Control

  • ODPO is more effective than DPO at generating positive-sentiment reviews while staying near the SFT model, with the largest difference at 5,000 preference pairs.The comparison uses the Pareto frontier of sentiment reward and KL divergence.Source: §6.1 · p. 6

6.2 Toxicity Control

  • At preference-dataset sizes of 8,000 or 9,000, ODPO reduces toxicity nearly by half while DPO stays close to the SFT model; DPO approaches ODPO as data grows.Figure 3 marks Pareto-front points with black borders and highlights the toxicity-versus-divergence comparison.Source: Figure 3 · p. 6, §6.2 · p. 7

6.3 Summarization

  • For summarization, the offset is based on the human-assessed quality-score difference between paired summaries.GPT-4 compares fine-tuned model generations with human-written reference summaries.Source: §6.3 · p. 7
  • ODPO’s average win rate against DPO is higher at every sampling temperature and significantly higher at temperatures 0 and 0.5.Figure 5 reports win rates against human-written summaries; both fine-tuned methods also improve on the SFT model, and at temperatures 0, 0.25, and 0.5 both are preferred on average to human-written summaries.Source: Figure 5 · p. 7, §6.3 · p. 8

7 Ablation Studies

The ablations examine how the offset’s scaling function and hyperparameter affect reward and divergence from the SFT model. Log scaling and larger α values favor higher rewards, while smaller α values are associated with lower divergence.

7.1 Ablating the Scaling Function

  • Among the tested scaling functions, the experiment compares the reward difference directly, the identity function, with the difference between log reward values.The movie-review generation experiment uses 5,000 preference pairs for computational efficiency.Source: §7.1 · p. 8
  • At around 0.8 reward, log-scaled offsets correspond to KL 0.4, while unscaled offsets reach that reward at KL around 1; separately scaling reward values lies between them.Across all three offset choices, ODPO outperforms DPO.Source: §7.1 · p. 8

7.2 Ablating the Offset Hyperparameter

  • α = 1 is associated with the highest rewards, while smaller α values are associated with lower D_KL; higher α can trade lower divergence for higher reward.At α = 0, the ODPO loss equals the DPO loss; the ablation varies α from 0 to 1.Source: §7.2 · p. 8, §7.1 · p. 8

8 Related Work

Related methods vary in whether they use an offset, paired preferences, or an objective connected to RLHF. ODPO’s distinguishing feature is an offset tied to how much the preferred and dispreferred responses differ in quality.

Sequence-Level Contrastive Methods

  • SLIC uses a rank-calibration loss with a fixed offset, while RRHF uses a ranking loss without one; the connection between these contrastive objectives and RLHF’s expected-reward objective is unclear.SLIC’s offset does not depend on the responses.Source: §8 · pp. 8–9

DPO and Variants

  • DPO optimizes the regularized expected-reward objective without RL or a separate reward model; IPO, KTO, and a multi-objective DPO variant address other dimensions.IPO uses a bounded identity function, while KTO uses unpaired good-or-bad examples.Source: §8 · p. 9

Our Approach

  • ODPO addresses DPO’s equal treatment of preference pairs by making the offset depend on the quality difference between responses.The offset requires the estimated preferred-response reward to exceed the dispreferred-response reward by an amount tied to that difference.Source: §8 · p. 9

9 Conclusion

ODPO generalizes DPO by accounting for how strongly one response is preferred, while retaining DPO’s avoidance of reward models and reinforcement learning. Experiments suggest it is more effective than DPO across varied alignment tasks.

  • ODPO incorporates preference strength into its loss rather than treating every preference pair equally.Source: §9 · p. 9
  • ODPO, like DPO, requires neither a pretrained reward model nor an RL algorithm.Source: §9 · p. 9
  • Experiments across a variety of tasks suggest ODPO is more effective than DPO at aligning language models with human preferences.Source: §9 · p. 9

Limitations

The experiments focus on tasks with preference-strength information or point-wise rewards, while the offset scaling choices and evaluated tasks remain limited. The authors also identify misuse and reward-data bias as ethical concerns.

Limitations

  • Experiments focus on tasks with preference-strength information or point-wise reward functions, since not all human-feedback datasets provide preference degree.Source: Limitations · p. 9
  • Offset scaling may depend on the task; this study tests Likert scores and classifier probabilities and defers other tasks to future work.Source: Limitations · p. 9

Ethical Considerations

  • ODPO, like other RLHF methods, could be misused to elicit harmful content, and reward functions may propagate biases from their training data.Source: Ethical Considerations · p. 9
  • The authors say more accurate, less biased reward functions or more inclusive feedback collection would reduce unwanted bias propagation into generations.Source: Ethical Considerations · p. 9

A Proof of Thm. 1

The theorem setup considers two responses with estimated rewards and associated Gumbel random variables. The proof introduces an auxiliary random variable and a sufficient intermediate claim.

B Datasets Statistics

The appendix describes the datasets and evaluation setup for sentiment control, toxicity control, and summarization, as well as computational costs. It also reports that ODPO outperforms SLiC on both reward and KL in all three sentiment-control setups.

B Datasets Statistics

  • The datasets are in English; the corresponding papers provide data-collection, postprocessing, and annotator-demographic details.Source: B Datasets Statistics · p. 13
  • Sentiment control uses 25,000 IMDB training reviews, 10,000 sampled fine-tuning prompts, and 256 test prompts.Source: B Datasets Statistics · p. 13
  • Toxicity control samples 10,000 fine-tuning prompts and 256 evaluation prompts from REALTOXICITYPROMPTS.Source: B Datasets Statistics · p. 13
  • In all three sentiment-control setups, ODPO achieves lower KL values and higher rewards than SLiC.Source: Table 1 · p. 14
  • Summarization uses 20,000 posts from two relationship subreddits and 100 posts for evaluation.Source: B Datasets Statistics · p. 14

C Computational Budget

D Comparison to SLiC-HF

On sentiment control, no SLiC-HF run reached the reward–KL Pareto frontier; the section also describes GPT-4’s summary-evaluation prompt and qualitative examples across decoding settings.

D Comparison to SLiC-HF

  • No SLiC-HF runs reached the reward–KL Pareto frontier; results report means and 0.9 confidence intervals across three dataset sizes, using margin 1 for SLiC and β = 0.5 for DPO and ODPO.Source: D Comparison to SLiC-HF · p. 14

E GPT-4 Evaluation Prompt for the Summarization Task

  • GPT-4 compares summaries for coverage of important points, concision, and precision, then gives a one-sentence rationale and selects A or B from randomly ordered summaries.Source: E GPT-4 Evaluation Prompt for the… · p. 14

F Qualitative Results

Loading 2402.10571v2…