PaperlayerView exact revision on arXiv

Source-linked AI summary

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, Colin White

arXiv:2402.13228v2cs.CLcs.AIcs.LG

TL;DR

DPO can improve LLMs using preferred and dispreferred responses, but its loss may reduce preferred-completion likelihood, especially for pairs with small edit distance. The paper introduces DPOP, which penalizes that reduction, and reports stronger performance across datasets and independent benchmarks, including Smaug-72B’s 80.48% leaderboard accuracy.

  • Problem

    DPO can reduce the likelihood of preferred completions, particularly when preference pairs differ by few tokens, despite increasing their relative probability over dispreferred completions.

  • Method

    DPOP adds a loss term that penalizes reducing the probability of preferred completions, and the authors apply it with new preference datasets based on ARC, HellaSwag, and MetaMath.

  • Results

    DPOP outperforms DPO across the evaluated datasets and downstream tasks; Smaug-72B achieves 80.48% average accuracy on the HuggingFace Open LLM Leaderboard, surpassing 80% for the first time among open-source LLMs.

  • Takeaways & Limitations

    DPOP addresses the identified DPO failure mode and can improve performance even beyond datasets with small edit distances, including on MT-Bench independent of fine-tuning data.

  • Takeaways & Limitations

    The authors did not run a full 72B ablation and demonstrated DPOP on only six English-language datasets.

Abstract

from arXiv · show

Direct Preference Optimisation (DPO) is effective at significantly improving the performance of large language models (LLMs) on downstream tasks such as reasoning, summarisation, and alignment. Using pairs of preferred and dispreferred data, DPO models the relative probability of picking one response over another. In this work, first we show theoretically that the standard DPO loss can lead to a reduction of the model's likelihood of the preferred examples, as long as the relative probability between the preferred and dispreferred classes increases. We then show empirically that this phenomenon occurs when fine-tuning LLMs on common datasets, especially datasets in which the edit distance between pairs of completions is low. Using these insights, we design DPO-Positive (DPOP), a new loss function and training procedure which avoids this failure mode. Surprisingly, we find that DPOP outperforms DPO and other fine-tuning procedures across a wide variety of datasets and downstream tasks, including datasets with high edit distances between completions. Furthermore, we find that the DPOP-tuned model outperforms the DPO-tuned model (all else equal) on benchmarks independent of the fine-tuning data, such as MT-Bench. Finally, using DPOP, we create and open-source Smaug-34B and Smaug-72B, with the latter becoming the first open-source LLM to surpass an average accuracy of 80% on the HuggingFace Open LLM Leaderboard.

1 Introduction

The paper identifies a failure mode in DPO where preferred-completion likelihood can decrease, then introduces DPOP to avoid it and reports stronger performance across datasets and benchmarks.

  • DPO can reduce the likelihood of preferred completions when the relative probability between preferred and dispreferred classes increases.The effect is theoretically analyzed and empirically observed in common datasets.
  • DPOP adds a loss term that penalizes reducing the probability of preferred completions and incentivizes every token toward the preferred completion.The method is designed to ameliorate the performance degradation identified for DPO.
  • 80.48% average accuracy on the HuggingFace Open LLM Leaderboard made Smaug-72B the first open-source LLM to surpass 80%.The result improved by nearly 2% over the second-best open-source model.
  • DPOP outperformed DPO on MT-Bench, an LLM-judged benchmark independent of the fine-tuning data.The comparison was conducted in an apples-to-apples setting.
  • DPOP often outperforms DPO, including on preference datasets with high edit distances between completions.The paper creates preference versions of ARC, HellaSwag, and MetaMath for model training.

2 Background and Related Work

The background presents preference-based alignment, RLHF, and DPO, then situates DPOP among alternative offline preference-optimisation methods and related analyses.

  • Preference alignment uses contexts paired with preferred and dispreferred completions to learn a policy for generating responses.DPO models the relative probability of selecting the preferred completion through an implicit reward function.
  • RLHF: RLHF models preference probabilities with a Bradley-Terry reward model, then uses reinforcement learning with a KL constraint to obtain a new LLM distribution.The learned reward is estimated from pairwise-preference data.
  • DPO: DPO directly optimizes the policy distribution for the same KL-constrained reward objective without learning an explicit reward function.Its implicit reward parameterization uses the policy-to-reference log-probability ratio scaled by β.
  • Alternatives: SLiC provides an alternative pairwise-preference loss whose cross-entropy term regularizes toward a reference target without requiring an extra copy of the model weights.The formulation includes a margin-style preference term and a reference sequence yref.
  • Alternatives: IPO addresses potential DPO overfitting when preference probabilities approach one by penalizing preference margins beyond a regularized value.The paper contrasts this with its identification of DPO underfitting and complete performance degradation.
  • Subsequent work: Subsequent work verified the reported DPO failure mode and the proposed remedy, including evidence that a similar additional loss term helps chain-of-thought fine-tuning.Other studies also build on the paper’s theoretical insights.

3 Failure Mode of DPO

The standard DPO objective can reduce the preferred completion’s likelihood while still lowering its loss, particularly when preferred and dispreferred completions differ by few tokens. The resulting gradients can decrease probabilities of correct tokens after the differing position, contributing to severe accuracy degradation.

  • DPO can reduce preferred-completion likelihood when the dispreferred completion’s relative probability decreases sufficiently.The loss depends only on the difference in log-ratios, so lowering both ratios can still reduce the loss.
  • The issue is particularly consequential when both preference completions are satisfactory, as in knowledge distillation and transformed labelled datasets.In these settings, the preferred completion is not merely better than the alternative but also satisfactory in its own right.
  • Edit Distance 1: Small edit distances make the failure mode especially relevant, including pairs differing at only one token.The paper illustrates this with “2+2=4” versus “2+2=5.”
  • Edit Distance 1: For a one-token difference, tokens before the differing position cancel out of the gradient.The analysis therefore focuses without loss of generality on a difference at the first token.
  • After the differing token, DPO decreases logits for correct tokens and increases logits for incorrect tokens relative to the reference model.The paper connects this gradient pattern to reduced probability of emitting correct subsequent tokens.

4 DPOP

DPOP augments DPO with a penalty that discourages reducing the preferred completion’s probability below the reference model. Its optimization pressure is designed to preserve preferred-completion likelihood while retaining Bradley–Terry preference modeling.

  • DPOP adds a penalty term that increases when the preferred completion’s probability ratio falls below 1.The penalty is zero when πratio(yw|x) ≥ 1 and grows as the ratio decreases.
  • The full DPOP objective retains fitting preference data under the Bradley–Terry model and introduces tunable λ > 0.λ controls the strength of the added penalty.
  • The penalty prevents loss minimization from relying solely on reducing dispreferred likelihood more than preferred likelihood.DPOP also pressures the preferred completion’s likelihood to remain high relative to the reference model.
  • For the analyzed one-token case, sufficiently large λ makes the DPOP gradient positive for the preferred token and negative for competing tokens.This reverses the problematic DPO gradient direction when πratio < 1, under the stated positivity condition.
  • Connection to Contrastive Loss: DPOP’s loss is connected to contrastive loss through a margin m = log 1/πref(yw|x).The paper presents this as a connection between the proposed penalty and margin-based contrastive objectives.

5 DPOP Datasets & Experiments

The experiments test DPOP against preference-optimization baselines across datasets with low and high edit distances, while token-level analyses examine DPO’s failure mode. DPOP improves performance and preserves preferred-completion likelihood where DPO can degrade.

  • Loss function comparison: DPOP outperforms DPO, IPO, and SLiC on both MetaMath and ARC, whose normalized edit distances are 6.5% and 90%, respectively.
  • Loss function comparison: DPOP is the only method improving over the base model on MetaMath, where DPO and SLiC catastrophically fail and IPO does not improve performance.
  • Loss function comparison: On ARC’s higher-edit-distance data, DPO, SLiC, and DPOP improve significantly over the base model, with DPOP performing best.
  • Ablation studies for β and λ: Across β ∈ {0.1, 0.3, 1.0}, larger β slows DPO’s decline but does not prevent performance from plummeting, whereas DPOP remains strong and consistent.
  • Token-level analysis: After the first differing token m, preferred-completion token log-probability averages −0.37 for the reference model, −0.26 for DPOP, and −1.82 for DPO.
  • Token-level analysis: DPO reduces log-likelihood after edit indices, while the reference model and DPOP show higher log-likelihood after those indices.
  • Additional analyses: On ARC, DPOP maintains higher preferred-completion log-probabilities than DPO during training.

6 Smaug

The Smaug series applies DPOP to models from 7B through 72B and evaluates them against downstream benchmarks. Smaug-72B surpasses 80% average accuracy on the HuggingFace Open LLM Leaderboard, while DPOP also improves controlled MT-Bench performance over DPO.

  • Model training: Smaug models are trained at 7B, 34B, and 72B parameter sizes using DPOP, with 7B used for direct DPOP-versus-DPO comparisons.
  • Model training: DPOP and DPO are applied to Llama-2-Chat using six paired-preference datasets, then evaluated on MT-Bench across diverse performance categories.
  • DPOP vs. DPO: 7.292 ± 0.037: DPOP’s MT-Bench first-turn score exceeds DPO’s 7.032 ± 0.043 in 10 trials.
  • Smaug-72B evaluation: 80.48%: Smaug-72B’s average accuracy on the HuggingFace Open LLM Leaderboard exceeds 80% and improves by nearly 2% over the second-best open-source model.
  • Smaug-72B evaluation: The HuggingFace Open LLM Leaderboard aggregates ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, and WinoGrande using few-shot test-set evaluation.
  • Additional evaluation: Smaug-72B achieves the top MMLU score and third-best MT-Bench score among the compared open-source models.
  • Contamination analysis: Smaug-72B’s contamination-checker scores on ARC, TruthfulQA, and GSM8K are similar to those of its DPO starting model and Llama-2-70B.

7 Conclusions and Limitations

The paper identifies a severe DPO failure mode, proposes DPOP to address it, and reports stronger performance across independent evaluations. It also situates the approach against related methods and notes important scope limitations.

  • Conclusions: DPOP overcomes DPO’s preferred-completion probability failure and can outperform DPO even outside that failure mode.The paper reports this theoretically and empirically.
  • Conclusions: Smaug models trained with DPOP on paired ARC, HellaSwag, and MetaMath datasets achieve state-of-the-art open-source performance.The resulting models include the first open-weights model to surpass 80% average accuracy on the HuggingFace Open LLM Leaderboard, while DPOP also significantly outperforms DPO on MT-Bench.
  • Limitations: The authors could not run a full 72B ablation study and demonstrated DPOP on only six English-language datasets.They therefore assume smaller-model ablations hold at scale and call for evaluation on more, especially non-English, datasets.
  • Related work: The paper distinguishes its approach from AFT through its DPO reformulation, reference-model penalty, and preference-data setting.AFT instead uses a soft margin on the dispreferred example and focuses on self-generated chain-of-thought responses under supervised fine-tuning.
  • Failure mode: DPOP’s preferred-completion probability protection is motivated by a failure mode in which DPO fine-tuning reduces preferred-example probabilities.The token-level analysis suggests that tokens following a completion difference can receive reduced probability for the correct continuation relative to the reference model.

B.2 Derivation for DPOP

The DPOP derivation analyzes its logit gradients and explains how its added terms correct DPO’s probability-reduction behavior. The formulation also connects DPOP to contrastive learning through a similarity term and margin.

  • Gradient behavior: For πratio(y|x) < 1, sufficiently large λ makes DPOP’s gradient increase the preferred token logit and decrease competing logits.When πratio(y|x) ≥ 1, DPOP uses the standard DPO gradient.
  • Contrastive-loss connection: Under the contrastive-loss analogy, standard DPO retains only the dissimilar-points term, whereas DPOP adds the missing similar-points term and margin.The paper interprets these additions as a refinement aimed at more robust training.
  • DPOP objective: DPOP adds a term that penalises reducing the probability of positive completions.Its margin can be chosen so the similarity term contributes only when the learned model performs worse than the reference model on the preferred response.
  • Training objective: DPOP omits the third term because driving negative-sample log probability toward negative infinity may be unstable and harms negative-sample likelihood.The authors consider sacrificing this signal acceptable while preserving positive likelihood.

D Details of Datasets

The released preference datasets convert ARC, HellaSwag, and MetaMath into paired preferred/dispreferred examples for reasoning and commonsense tasks. Their construction includes substantially different completion edit distances.

  • Dataset release: All released datasets are paired preference versions, and the datasets are released under the Apache 2.0 License.The paper provides additional construction details for the three datasets.
  • MetaMath: MetaMath pairs mathematical questions with correct preferred solutions and corrupted dispreferred versions.The dataset contains 393,999 training examples and 1,000 evaluation examples, with approximately 6.5% normalised edit distance.
  • ARC: ARC pairs each question’s correct answer with its three incorrect alternatives.The dataset contains 3,357 training examples and 895 evaluation examples, with approximately 90% normalised edit distance.
  • HellaSwag: HellaSwag is used as a commonsense-inference preference dataset with multiple potential completions for each prompt.The passage presents it as a dataset known to be hard for language models.

E.1 Additional Training Details

Smaug-34B and Smaug-72B were created without hyperparameter tuning, while the training analysis compares preferred-completion log probabilities for DPOP and DPO. These comparisons show DPOP maintaining or increasing preferred-response likelihood.

  • Hyperparameters: β = 0.3 and λ = 50 were used for Smaug-34B and Smaug-72B without trying alternative values.The authors note that different λ values might yield even better performance.
  • Model provenance: Smaug-34B started from Bagel-34B-v0.2, which itself is an SFT version of Yi-34B-200k.The model is released under the Yi Series Models Community License Agreement.
  • MetaMath log probabilities: On MetaMath, preferred-completion log probabilities drop substantially for DPO but increase for DPOP across both training and evaluation sets.Figure 5 reports this comparison during training.
  • ARC log probabilities: On ARC, DPOP maintains high training-set preferred-completion log probabilities, while DPO decreases both training- and evaluation-set log probabilities.The comparison appears in Figure 6 and its accompanying analysis.
  • Model provenance: Smaug-72B started from MoMo-72b-lora-1.8.7-DPO, itself a fine-tune of Qwen-72B.The model is released under the Qwen license.

E.2 Additional Results

Additional results show that DPOP preserves preferred-completion log-probabilities better than DPO and is evaluated against strong models on MT-Bench and MMLU.

  • Preferred-completion log-probabilities: DPO substantially reduces preferred-completion log-probabilities on MetaMath, while DPOP increases them across both training and evaluation sets.On ARC, DPOP maintains high training-set log-probabilities, whereas DPO decreases both training- and evaluation-set values.
  • Benchmark comparisons: Table 2 evaluates top models according to MT-Bench and MMLU.
  • Benchmark comparisons: MT-Bench evaluation uses the Llama-2 conversation template and excludes MT-Bench questions from the fine-tuning datasets.The benchmark uses GPT-4 to score responses across eight performance categories.

E.3 Contamination Checker

The contamination analysis addresses concerns that benchmark examples may overlap with training data by comparing Smaug-72B against other open-source models.

  • Motivation: Data contamination is a concern because LLMs train on wide swaths of internet data, making benchmark overlap difficult to measure and mitigate.
  • Checker: The contamination checker compares contamination values across models, with scores above 0.85 indicating likely contamination.
  • Results: The paper reports Smaug-72B’s 80.48% average accuracy on the HuggingFace Open LLM Leaderboard alongside contamination analysis.
Loading 2402.13228v2…