PaperlayerView exact revision on arXiv

Source-linked AI summary

SimPO: Simple Preference Optimization with a Reference-Free Reward

Yu Meng, Mengzhou Xia, Danqi Chen

arXiv:2405.14734v3cs.CLcs.LG

TL;DR

DPO’s reward can diverge from the likelihood metric used during generation, motivating a simpler reference-free alternative. SimPO uses length-normalized policy log probabilities with a target reward margin, and it consistently outperforms existing approaches across setups without substantially increasing response length.

  • Problem

    DPO’s reference-based reward can mismatch the average log likelihood used for generation, while classical RLHF also involves a challenging multi-stage optimization procedure.

  • Method

    SimPO uses the policy model’s length-normalized average token log probability as an implicit reward, adds a target reward margin, and eliminates the reference model.

  • Results

    SimPO consistently and significantly outperforms existing preference optimization methods across training setups and benchmarks, with up to 6.4-point and 7.5-point gains over DPO on AlpacaEval 2 and Arena-Hard.

  • Takeaways & Limitations

    SimPO provides a simpler, more efficient preference optimization approach that achieves strong performance without substantially increasing response length.

  • Takeaways & Limitations

    The target-margin parameter has a trade-off: increasing it can initially improve performance but eventually flatten reward distributions and cause model degeneration.

Abstract

from arXiv · show

Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability. In this work, we propose SimPO, a simpler yet more effective approach. The effectiveness of SimPO is attributed to a key design: using the average log probability of a sequence as the implicit reward. This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient. Additionally, we introduce a target reward margin to the Bradley-Terry objective to encourage a larger margin between the winning and losing responses, further improving the algorithm's performance. We compare SimPO to DPO and its latest variants across various state-of-the-art training setups, including both base and instruction-tuned models such as Mistral, Llama 3, and Gemma 2. We evaluate on extensive chat-based evaluation benchmarks, including AlpacaEval 2, MT-Bench, and Arena-Hard. Our results demonstrate that SimPO consistently and significantly outperforms existing approaches without substantially increasing response length. Specifically, SimPO outperforms DPO by up to 6.4 points on AlpacaEval 2 and by up to 7.5 points on Arena-Hard. Our top-performing model, built on Gemma-2-9B-it, achieves a 72.4% length-controlled win rate on AlpacaEval 2, a 59.1% win rate on Arena-Hard, and ranks 1st on Chatbot Arena among <10B models with real user votes.

1 Introduction

SimPO addresses a mismatch between DPO’s reward formulation and the likelihood metric used during generation. It removes the reference model, improves preference optimization across evaluations, and avoids substantial response-length increases.

  • Motivation: DPO’s reward uses a policy-to-reference likelihood ratio that is not directly aligned with the average log likelihood guiding generation.This discrepancy may lead to suboptimal performance.
  • Method: SimPO uses a length-normalized average log probability as its reward and introduces a target reward margin between winning and losing responses.The method is designed to align reward optimization with generation while encouraging a larger winning-response margin.
  • Method: SimPO does not require a reference model, making it more lightweight and easier to implement than reference-based methods.This design reduces the memory and computational requirements associated with reference-model training.
  • Results: Up to 6.4 points on AlpacaEval 2 and 7.5 points on Arena-Hard separate SimPO from DPO across varied training setups and chat evaluations.The comparison includes DPO and newer variants across multiple model configurations.
  • Results: SimPO does not significantly increase response length compared with SFT or DPO models.This indicates minimal length exploitation in the reported evaluations.
  • Results: 72.4% length-controlled win rate on AlpacaEval 2 and 59.1% win rate on Arena-Hard are achieved by Gemma-2-9B-it-SimPO.The model also ranks first among models below 10B parameters on Chatbot Arena using real user votes.

2 SimPO: Simple Preference Optimization

The section develops SimPO as a reference-free alternative to DPO by aligning its reward with generation likelihood and adding a target margin. The resulting objective avoids reference-model costs while addressing ranking and length-bias concerns.

  • DPO background: DPO eliminates the explicit reward model but retains a reference model and optimizes a reward that can diverge from generation likelihood.Only about 50% of DPO training triples satisfied matching reward and average-log-likelihood rankings in the reported experiment.
  • Reference-free reward: SimPO replaces DPO’s reward with the average token log probability of the policy model, directly matching the likelihood metric used for generation.Length normalization is used to avoid the bias of summed log probabilities toward longer sequences.
  • Reference-free reward: Length normalization is crucial because removing it biases optimization toward longer, lower-quality sequences.The normalized formulation also eliminates the need for a reference model and improves memory and computational efficiency.
  • Target reward margin: SimPO modifies the Bradley-Terry preference probability so the winning response must exceed the losing response by a target margin γ.The target margin is introduced as γ > 0 in the preference objective.
  • Target reward margin: Increasing the target margin initially improves generation quality but eventually degrades it when the margin becomes too large.The section reports a similar trade-off between target-margin size and generalization-oriented behavior in preference optimization.
  • Objective: SimPO aligns its implicit reward with generation likelihood, removes the reference model, and uses the target margin to separate winning and losing responses.The method does not impose KL regularization, while practical factors are reported to maintain generalization empirically.

3 Experimental Setup

The experiments compare SimPO with offline preference-optimization methods across base and instruction-tuned Llama-3 and Mistral setups, using several chat-based benchmarks. They also include Gemma 2 experiments with a stronger reward model and evaluate the methods against established baselines.

  • Models and training settings: Base preference optimization starts from UltraChat-trained SFT models and uses UltraFeedback preference data.This pipeline provides transparency because the SFT models are trained on open-source data.
  • Models and training settings: Instruct experiments use off-the-shelf Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2 checkpoints as SFT models.These models are described as more powerful and robust but more opaque because their RLHF procedures are undisclosed.
  • Models and training settings: Four setups cover Llama-3-Base, Llama-3-Instruct, Mistral-Base, and Mistral-Instruct.Base models are trained on UltraChat-200k to obtain SFT models, whereas Instruct setups use off-the-shelf instruction-tuned checkpoints.
  • Evaluation benchmarks: Evaluation uses MT-Bench, AlpacaEval 2, and Arena-Hard for open-ended instruction-following and conversational abilities.AlpacaEval 2 contains 805 questions, MT-Bench covers 8 categories with 80 questions, and Arena-Hard includes 500 technical problem-solving questions.
  • Baselines: Baselines include ranking losses, DPO variants, sequence-likelihood objectives, and methods for paired or non-paired preference data.The comparison includes RRHF, SLiC-HF, IPO, CPO, KTO, ORPO, and R-DPO, with hyperparameters tuned for each baseline.
  • Evaluation protocol: Table 4 reports AlpacaEval 2, Arena-Hard, and MT-Bench results across the four settings using length-controlled and raw win rates.Base settings train SFT models on UltraChat, while Instruct settings use off-the-shelf models as SFT models.

4 Experimental Results

SimPO consistently outperforms preference-optimization baselines across training settings and benchmarks, while its length normalization and target margin are important to performance. Its reward design also improves accuracy and efficiency relative to DPO, though benchmark and margin-setting caveats remain.

  • Main Results: 3.6 to 4.8 points: SimPO exceeds the best baseline on AlpacaEval 2 LC win rate across various settings.
  • Main Results: SimPO is occasionally surpassed by CPO on Arena-Hard, whose evaluation may favor longer generations because it lacks a length penalty.CPO responses average 50% greater length than SimPO responses.
  • Ablations: Removing length normalization produces long, repetitive outputs and substantially degrades quality, while setting γ = 0 also reduces performance.The ablation compares removing length normalization with setting the target reward margin to zero.
  • Length Normalization: SimPO with length normalization maintains positive reward margins across response-length differences, whereas removing normalization can produce negative margins when winning responses are shorter.
  • Length Normalization: Without length normalization, likelihood has a much stronger positive Spearman correlation with response length, indicating greater exploitation of length bias.
  • Target Reward Margin: Reward accuracy increases with γ, but AlpacaEval 2 win rate first increases and then decreases as γ grows.
  • Target Reward Margin: Increasing γ flattens reward distributions and lowers winning-sequence average log likelihood, which can eventually cause model degeneration.
  • Comparison with DPO: SimPO achieves higher reward accuracy than DPO and cuts runtime by roughly 20% while reducing peak GPU memory usage by about 10%.The efficiency comparison uses a Llama-3-Base setting with 8×H100 GPUs and attributes savings to eliminating reference-model forward passes.

5 Related Work

Related work spans classical RLHF, offline preference optimization, and reference-free objectives. SimPO is positioned within offline methods while avoiding iterative training processes.

  • Reinforcement Learning from Human Feedback: Classical RLHF typically combines supervised fine-tuning, reward-model training, and policy optimization, often using PPO in the final stage.
  • Offline vs. Iterative Preference Optimization: Offline preference optimization methods seek simpler alternatives to complex online procedures, with DPO learning directly from preference data without an explicit reward model.
  • Offline vs. Iterative Preference Optimization: Because DPO cannot sample preference pairs from the optimal policy, later work augments data or iteratively updates policies, references, or preference pairs.
  • Offline vs. Iterative Preference Optimization: SimPO focuses exclusively on offline training and avoids iterative training processes.
  • Preference Optimization Objectives: Preference-optimization research also includes ranking objectives, reference-free methods, joint instruction-response optimization, and post-training extrapolation.

6 Conclusion

The paper concludes that SimPO combines a generation-aligned reward with a target margin to deliver strong, efficient preference optimization without substantial length exploitation. It reports consistent gains across training setups while acknowledging further limitations discussion.

  • SimPO aligns its reward with generation likelihood, adds a target reward margin, removes the reference-model requirement, and consistently outperforms existing approaches.
  • The paper states that SimPO achieves strong performance without exploiting response-length bias.
  • The paper directs readers to Appendix A for a detailed discussion of limitations.
  • In some settings, such as Llama-3-Instruct, a large β such as β = 10 leads to better performance.

A Limitations

SimPO’s main limitations concern theoretical understanding, manual tuning of its target reward margin, safety and honesty coverage, and degraded performance on reasoning-heavy math tasks.

  • Theoretical and tuning limitations: SimPO requires manual tuning of its target reward margin, and its effectiveness lacks a fully rigorous theoretical explanation.Future work is proposed to determine the optimal margin automatically and better explain SimPO theoretically.
  • Safety and honesty: SimPO does not explicitly optimize safety and honesty, while its UltraFeedback training data primarily emphasizes helpfulness.The authors suggest integrating safety and honesty constraints and evaluating with broader safety-focused datasets and benchmarks.
  • Math performance: SimPO can perform comparably to or worse than DPO on reasoning-heavy math tasks such as GSM8K.The authors associate this issue with dataset choices, hyperparameters, chat-template mismatch, or insufficiently increasing preferred-sequence likelihood.

B Implementation Details

The experiments use standardized training settings, method-specific hyperparameter searches, benchmark-specific decoding, and an 8×H100 computation environment.

  • General training: Base models are supervised-fine-tuned on UltraChat-200k for one epoch with a 2e-5 learning rate, batch size 128, maximum sequence length 2048, cosine decay, and 10% warmup.All models use the Adam optimizer.
  • Preference optimization: Preference optimization uses batch size 128, one epoch, maximum sequence length 2048, cosine scheduling, and 10% warmup across methods.These values were selected after preliminary searches over batch sizes and training epochs.
  • Hyperparameter tuning: Learning rates are individually searched for each preference optimization method over 3e-7, 5e-7, 6e-7, and 1e-6.Detailed baseline search ranges and SimPO settings are reported in Tables 7 and 8.
  • Decoding: AlpacaEval 2 uses sampling with setting-specific temperatures, whereas Arena-Hard uses default greedy decoding for all settings and methods.The reported AlpacaEval 2 temperatures are 0.7 for Mistral-Base, 0.5 for Mistral-Instruct, and 0.9 for both Llama 3 settings.
  • Computation: All training experiments run on 8×H100 GPUs using the alignment-handbook repository.The KTO experiments use Adam here despite RMSProp in the original paper’s runs.

C Downstream Task Evaluation

Preference optimization has mixed downstream effects: knowledge is largely retained, reading comprehension and commonsense reasoning improve, truthfulness improves, while math performance is volatile and often declines.

  • Knowledge retention: Preference optimization generally preserves MMLU performance with minimal decline relative to the SFT checkpoint, with SimPO largely comparable to DPO.This evaluation is part of the Huggingface Open Leaderboard task suite.
  • Reasoning and commonsense: Preference optimization generally improves ARC and HellaSwag performance compared with the SFT checkpoint.The authors hypothesize that similar prompts in the preference dataset support contextual understanding, reading comprehension, and commonsense reasoning.
  • Truthfulness: TruthfulQA performance consistently improves over the SFT checkpoint, with gains exceeding 10% in some cases.The authors hypothesize that preference examples emphasizing truthfulness contribute to this improvement.
  • Mathematics: GSM8K shows the greatest volatility, and nearly all approaches produce drops in one or more settings except ORPO.The authors note that reference-model-calibrated supervised fine-tuning regularization can mitigate this issue.
  • Interpretation: Identifying a general downstream-performance pattern is difficult because models, preference datasets, and objectives differ across studies.The authors call for more comprehensive analysis of how preference optimization affects downstream tasks.
  • Evaluation stability: AlpacaEval 2 standard deviations and Arena-Hard 95% confidence intervals show no significant outliers or instability.These uncertainty measures are reported in Table 10.

E Generation Length Analysis

The analysis finds that length normalization shortens responses while improving benchmark win rates, and that response length alone is not a reliable quality indicator.

  • Length normalization: Length normalization reduces generation length by up to 25% in most cases while consistently producing higher win rates on AlpacaEval 2 and Arena-Hard.The authors interpret this as evidence that length normalization can control verbosity without sacrificing benchmark quality.
  • Length and quality: No single method consistently produces longer or shorter responses across settings, and longer outputs do not necessarily achieve higher benchmark win rates.The analysis concludes that response length is not a reliable indicator of generation quality.
  • SimPO versus DPO: SimPO’s length relative to DPO varies by benchmark, ranging from shorter responses in one Llama-3-Instruct setting to 26% longer responses on AlpacaEval 2.On Arena-Hard, SimPO increases length by only around 5% compared with DPO.
  • Gradient analysis: SimPO and DPO gradients differ because SimPO avoids the reference model and applies length-normalized updates to winning and losing responses.SimPO assigns higher weights to samples where the policy incorrectly gives the losing response higher likelihood.

G Qualitative Analysis

The qualitative analysis compares response organization across SimPO, DPO, base, instruction-tuned, and GPT-4 baselines. SimPO produces more structured and readable answers, while instruction-tuned models provide greater detail and formatting than base models.

  • Evaluation setup: Figure 7 and Figure 8 present win-rate heatmaps for Mistral-Base and Mistral-Instruct on AlpacaEval 2 and Arena-Hard.The heatmaps motivate qualitative comparisons among SimPO, DPO, and GPT-4-Preview-1106 responses.
  • Comparing SimPO with DPO: SimPO organizes responses with high-level categories and detailed suggestions, making them more readable than DPO’s long, difficult-to-relate bullet lists.The comparison uses Mistral-Base responses to the same conversational-interest question.
  • Comparing Instruct models with Base models when trained with SimPO: Llama-3-Instruct trained with SimPO generates a more detailed and well-formatted answer than Llama-3-Base-SimPO and GPT-4-1106-Preview.The case study concerns how people in Argentina speak and compares instruction and base settings.
  • Evaluation setup: Table 10 reports AlpacaEval 2 and Arena-Hard win rates, while Table 11 reports average response lengths for Mistral-Base and Mistral-Instruct.LC denotes length-controlled win rate, WR raw win rate, and STD standard deviation; Arena-Hard results include 95% confidence intervals.

H Llama-3-Instruct v0.2 (Jul 7, 2024)

The v0.2 analysis studies stronger preference data, SFT regularization, DPO modifications, downstream-task trade-offs, and Gemma 2 results. It finds that better reward-model labels improve Llama-3-Instruct performance, while task and training setup determine the effects of SFT, normalization, and reward margins.

  • Llama-3-Instruct v0.2: Switching preference annotation from PairRM to ArmoRM markedly improves Llama-3-Instruct performance, highlighting the importance of high-quality preference data.The v0.2 setting uses a stronger reward model to rank generated candidate responses.
  • Algorithm comparisons: Stronger SFT models and higher-quality preference data reduce differences between algorithms, although SimPO retains an advantage through shorter sequences.DPO can achieve similar raw win rates to SimPO in this setting, while several methods show comparable Arena-Hard raw win rates.
  • Downstream tasks: DPO, IPO, R-DPO, and SimPO decline on reasoning-intensive tasks such as GSM8K, whereas objectives containing an SFT component maintain mathematical-task performance.The v0.2 version otherwise improves downstream-task performance across various objectives.
  • SFT regularization: Adding SFT regularization decreases AlpacaEval 2 performance but substantially benefits some tasks such as GSM8K, with effects varying by setup and task.The paper leaves broader study of this variation for future research.
  • Applying SimPO components to DPO: Length normalization and target reward margins do not consistently benefit DPO; length normalization helps significantly only for Mistral-Base with strongly length-biased preference data.It does not help Mistral-Instruct, where winning and losing response lengths are comparable.
  • Checkpoint limitations: Continuing Llama-3-8B-Instruct training at higher learning rates strengthens chat benchmarks but degrades capability benchmarks such as MMLU and GSM8K.Table 16 reports AlpacaEval 2, ZeroEval GSM, and ZeroEval MMLU results across learning rates; the released checkpoint is marked with an asterisk.
  • Gemma 2 results: Gemma-2-9B-it-SimPO improves instruction-following performance over DPO without degrading the original model’s GSM or MMLU abilities.Its Chatbot Arena ranking improves from 36th to 25th after evaluation with real user preferences.
Loading 2405.14734v3…