PaperlayerView exact revision on arXiv

Source-linked AI summary

SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization

Maksim Afanasyev, Illarion Iov

arXiv:2602.02383v2cs.LG

TL;DR

Margin-based preference objectives can improve relative chosen–rejected gaps while degrading absolute generation quality. SLIME anchors preferred-response likelihood, stabilizes rejected-token probabilities, and combines hard and soft margins; it consistently outperforms DPO and SimPO across evaluated benchmarks and architectures.

  • Problem

    Margin-based objectives maximize chosen–rejected gaps without strictly preserving chosen-response likelihood, risking unlearning and degraded fluency or diversity.

  • Method

    SLIME is a reference-free objective combining likelihood anchoring, stabilized penalties for rejected tokens, and dual hard–soft margin optimization.

  • Results

    SLIME consistently outperforms DPO and SimPO on MT-Bench and Arena-Hard across Llama3.2-3B, Qwen3-4B, and Gemma3-4B; on Gemma3-4B, it reaches 6.15 on MT-Bench.

  • Takeaways & Limitations

    SLIME decouples preference learning from generation-quality preservation while avoiding the unlearning observed in pure margin-based methods.

  • Takeaways & Limitations

    Evaluation is limited to 3–4B models, UltraFeedback training data, and English-language benchmarks, while larger-model and broader-dataset generalization remain unvalidated.

Abstract

from arXiv · show

Direct preference optimization methods have emerged as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning Large Language Models (LLMs). Latest approaches have streamlined the alignment process by deriving implicit reward functions, yet they often suffer from a critical objective mismatch: optimizing the relative margin between chosen and rejected responses does not guarantee the preservation of the chosen response's absolute likelihood. This can lead to unlearning, where the model degrades the probability of high-quality outputs to satisfy margin constraints, and formatting collapse caused by the over-penalization of rejected sequences. In this work, we introduce SLIME (Stabilized Likelihood Implicit Margin Enforcement), a reference-free alignment objective designed to decouple preference learning from generation quality. SLIME incorporates a three-pronged objective: (1) an anchoring term to maximize the likelihood of preferred responses; (2) a stabilizing penalty that prevents the probabilities of rejected tokens from collapsing to zero; and (3) a dual-margin mechanism that combines hard and soft constraints for precise boundary shaping. Our results demonstrate that SLIME achieves superior performance compared to state-of-the-art baselines while maintaining higher generation stability.

1. Introduction

SLIME addresses the objective mismatch in margin-based preference optimization, where improving the relative chosen–rejected gap can degrade chosen-response likelihood and model fluency. It combines likelihood anchoring, token-level stabilization, and dual-margin optimization, and is reported to outperform DPO and SimPO while improving training stability and robustness.

  • Motivation: Margin-based objectives can lower chosen-response probability while lowering rejected-response probability further, causing unlearning of valid syntax and reasoning.This exposes the mismatch between relative-margin optimization and preserving absolute generation quality.
  • Method: SLIME anchors chosen responses to high-likelihood regions while applying a stabilized penalty instead of indiscriminately suppressing rejected responses.The objective is reference-free and decouples preference learning from generation quality.
  • Method: Token-level stabilization uses a non-linear softplus penalty to discourage extremely low rejected-token probabilities, filter easy negatives, and preserve language-model structure.The method targets fluency damage caused by indiscriminately minimizing rejected likelihoods.
  • Method: Dual-margin optimization combines a hard cutoff with a soft gradient-shaping margin to optimize near the decision boundary without vanishing gradients or single-margin overfitting.The resulting distance loss provides strict separation and smoother optimization around the boundary.
  • Evaluation: Across MT-Bench and Arena-Hard, SLIME outperforms DPO and SimPO across Llama3.2, Qwen3, and Gemma3 model families, while ablations support improved stability and robustness.The reported evaluation includes diverse benchmarks and attributes gains to the stabilizing and anchoring mechanisms.

2. Related Works

Related work progresses from PPO-based RLHF toward simpler preference objectives and more stable online policy-optimization methods. Key developments address computational cost, overfitting, data requirements, exploration, and variance in long-sequence or group-based training.

  • Preference optimization: InstructGPT uses PPO with a trained reward model, while DPO derives an implicit reward directly from policy preference pairs using a classification loss.PPO is described as unstable and memory-intensive, motivating DPO’s simpler alignment procedure.
  • Preference optimization: IPO regularizes the policy–reference gap to address DPO overfitting, while SimPO removes the reference model and applies length normalization.SimPO’s average sequence log probability serves as the reward and aligns training with the inference generation metric.
  • Preference optimization: KTO replaces paired preferences with desirable or undesirable individual generations and uses a reference-dependent value function to accommodate imbalanced data.Its hyperparameters λD and λU can handle cases with far fewer desirable than undesirable examples.
  • Online optimization: GRPO enables online exploration without a critic by estimating baselines from grouped outputs, substantially reducing memory usage for training models such as DeepSeekMath.RLOO similarly uses other samples as a baseline but lacks GRPO’s variance-reduction normalization.
  • Online optimization: GSPO replaces unstable token-level importance ratios with sequence-level ratios, while REINFORCE++ uses global batch-wide advantage normalization to reduce GRPO variance and bias.GSPO targets accumulated noise in long generations and MoE routing, whereas REINFORCE++ addresses bias from local normalization such as G = 4.

3. Method

SLIME is a reference-free preference optimization objective that preserves preferred-response likelihood while addressing instability from rejected-token suppression. Its composite loss anchors chosen responses, stabilizes rejected-token probabilities, and uses dual hard-soft margins to shape the decision boundary.

  • Anchoring the Chosen: SLIME explicitly anchors the policy to high-likelihood regions for preferred responses, countering chosen-response unlearning during margin optimization.The anchoring term Lw maximizes log πθ(yw|x), with λw controlling its strength.
  • Composite Objective: The composite objective decouples preference learning into anchoring chosen likelihood, stabilizing rejected likelihood, and dual-margin optimization.This design addresses the limitation of margin objectives that can increase the relative margin by lowering the winning response’s likelihood.
  • Stabilizing the Rejected: The stabilizing loss penalizes extremely low probabilities for tokens in rejected sequences, preserving valid syntax and reasoning patterns.For sufficiently high log-probability tokens, the penalty approaches zero; p = 2.5 controls its sharpness.
  • Dual-Margin Optimization: SLIME combines a hard margin cutoff with a soft margin for gradient shaping over the log-probability difference ∆ between chosen and rejected responses.The hard margin mh yields zero loss after the victory condition, while the soft margin ms focuses optimization between the two margins.

4. Experiments

Experiments compare DPO and SimPO within the TRL framework using UltraFeedback pairwise preferences. Three publicly released base-model checkpoints are evaluated under controlled training and benchmark protocols.

  • Experiments use the TRL framework and train all models on UltraFeedback pairwise preference annotations, comparing DPO with SimPO.
  • Models: The evaluated base models are Llama3.2-3B, Qwen3-4B, and Gemma3-4B, all initialized from publicly released checkpoints.
  • Training Pipeline and Data Splitting: A two-stage pipeline starts from pretrained base checkpoints, applies official instruction-tuned chat templates, and separates supervised fine-tuning from preference optimization.
  • Parameter-Efficient Fine-Tuning: LoRA is applied uniformly across models to provide parameter-efficient, high-capacity adaptation and controlled comparisons.
  • Algorithm-Specific Hyperparameters: DPO uses β = 0.1, SimPO uses γ = 0.2, and SLIME uses δ = 1.25, mh = 1.5, ms = 1.0, κ = 2.5, p = 2.5, fixed across models.
  • Evaluation Protocol: Evaluation uses MT-Bench and Arena-Hard with their stated protocols, using completed training checkpoints without additional fine-tuning, prompt tuning, or postprocessing.

5. Results

The results evaluate three base models and three preference optimization algorithms on a held-out UltraFeedback-derived set using a common evaluation protocol. Table 3 compares pretrained, SFT, DPO, SimPO, and SLIME models across benchmarks, with higher scores preferred.

  • Overview: Evaluation uses a held-out set derived from UltraFeedback and the same protocol described in Section 4.1.
  • Main Results: The study compares three base models across DPO, SimPO, and SLIME, alongside each model’s pre-finetuning baseline.
  • Main Results: For Arena-Hard, Gemma3-4B-it answers serve as the baseline because gpt-o3-mini-2025-01-31 outperforms 4B models.
  • Main Results: Table 3 evaluates raw pretrained, SFT, and preference-optimized models across multiple benchmarks, where higher values are better.

6. Ablation Study

The ablation study isolates SLIME’s chosen-response, rejected-response, soft-margin, and hard-margin components, and separately tests the stabilizing-loss exponent. Moderate exponent values perform best overall, while extremes degrade results, revealing a stability–flexibility trade-off.

  • Loss Component Ablations: SLIME ablations remove the chosen term, rejected term, soft distance margin, or hard margin while keeping other hyperparameters fixed.These variants isolate each component’s contribution under the same evaluation setup.
  • Stabilizing Loss Exponent Ablation: The stabilizing loss uses exponent p to control the sharpness of penalties for extremely low rejected-token probabilities.The default configuration sets p = 2.5.
  • Stabilizing Loss Exponent Ablation: Testing p in {1.5, 2.0, 2.5, 3.0} measures SLIME’s sensitivity to token-level stabilization strength.All other hyperparameters remain fixed.
  • Stabilizing Loss Exponent Ablation: Moderate exponent values yield the best overall performance, whereas overly small or large exponents degrade results.The pattern suggests a trade-off between stability and flexibility.

7. Discussion

The discussion reports that SLIME improves preference-optimization performance across evaluated architectures while addressing unlearning and stability failures associated with margin-based objectives. Ablations support its anchoring, rejected-sequence stabilization, and dual-margin components, while evaluation remains limited to 3–4B models and UltraFeedback training.

  • Consistent Improvements Across Architectures: SLIME achieves the highest reported MT-Bench scores on Gemma3-4B and Llama3.2-3B, outperforming SFT, DPO, and SimPO in the cited configurations.On Gemma3-4B, SLIME scores 6.15, a 30.6% improvement over SFT (4.71), while on Llama3.2-3B it scores 5.49.
  • The Unlearning Phenomenon in Margin-Based Methods: SimPO can underperform SFT because margin-only optimization may degrade both chosen and rejected likelihoods, whereas SLIME’s anchoring term preserves preferred-response likelihood.SimPO scores 4.22 versus SFT’s 4.56 on Llama3.2-3B, and scores 0.7 on Gemma3-4B Arena-Hard versus the SFT baseline.
  • Stability Through Token-Level Regularization: Removing SLIME’s rejected-sequence stabilization term reduces Gemma3-4B MT-Bench performance from 6.15 to 5.74.The term is hypothesized to prevent aggressive suppression of valid linguistic tokens appearing in rejected responses, using a softplus-based penalty with threshold δ.
  • The Role of Dual-Margin Optimization: The dual-margin design avoids over-optimization and focuses updates near the decision boundary, with either margin’s removal lowering performance below the full objective.Removing the soft margin reduces performance to 5.80, while removing the hard margin yields 5.90.
  • Model-Specific Observations: On Qwen3-4B, SLIME achieves the strongest reported Arena-Hard performance at 39.8 despite the instruction-tuned checkpoint’s high baseline and limited SFT benefit.The pretrained checkpoint has a baseline MT-Bench score of 5.95, while SFT may slightly reduce it to 5.40.
  • Limitations: The study is limited to 3–4B models and UltraFeedback training, leaving larger-model effectiveness and generalization to other preference datasets unvalidated.The discussion also notes modest computational overhead relative to simpler baselines and additional hyperparameters.

8. Conclusion

SLIME is a reference-free preference optimization objective that combines likelihood anchoring, rejected-token stabilization, and dual-margin optimization to preserve generation quality while learning preferences. Experiments and ablations show consistent gains over DPO and SimPO, while future work targets online sampling, deployment efficiency, and theoretical guarantees.

  • Conclusion: SLIME decomposes reference-free preference optimization into likelihood anchoring, token-level stabilization, and dual-margin preference optimization.These components decouple preference learning from generation quality preservation.
  • Conclusion: SLIME consistently outperforms DPO and SimPO on MT-Bench and Arena-Hard across Llama3.2-3B, Qwen3-4B, and Gemma3-4B.The improvements avoid the unlearning seen when SimPO falls below SFT baselines in several configurations.
  • Conclusion: Ablations show that anchoring prevents preferred-response likelihood degradation, stabilization preserves rejected-sequence fluency, and dual margins enable precise boundary shaping.The components contribute to final performance without over-optimization.
  • Conclusion: Future work includes extending SLIME to online on-policy settings, studying interactions with quantization and pruning, and proving formal likelihood-preservation guarantees.These directions address exploration, practical deployment, and theoretical validation.
  • Conclusion: Explicitly addressing objective mismatch rather than treating alignment as margin maximization offers a promising path toward robust, capable language models.SLIME illustrates that loss design can preserve model capabilities while optimizing human preferences.

A. Gradient Analysis and Optimization Dynamics

SLIME’s optimization combines chosen-sequence maximization, rejected-sequence regularization, and margin-based separation. These gradients increase preferred-response likelihood, prevent rejected-token probabilities from collapsing, and stop margin optimization beyond the hard boundary.

  • Objective decomposition: The total objective combines chosen-sequence maximization Lw, rejected-sequence regularization Ll, and margin-based distance Ldist.Together, these components optimize the policy πθ through distinct gradient dynamics.
  • Chosen-sequence optimization: The Lw gradient uniformly increases the likelihood of the preferred sequence yw, providing foundational supervision analogous to SFT.Its update is ∇θLw = −λw∇θ¯lw.
  • Rejected-sequence stabilization: The token-level Ll penalty prevents rejected-sequence probabilities from collapsing to zero instead of simply minimizing them.When token probability lt falls far below −δ, the Softplus term increases gradient magnitude super-linearly, helping prevent forgetting and formatting collapse.
  • Rejected-sequence stabilization: A sigmoid gate shuts off Ll once rejected-token probability reaches a sufficient baseline, avoiding further penalty for an adequately probable rejected sequence.As lt increases, ut becomes negative and σ(ut) approaches 0.
  • Margin optimization: Ldist expands the chosen–rejected margin through hard and soft gates, while its gradient becomes zero when ∆≥mh, preventing over-optimization.The soft gate applies maximum pressure when ∆ is far below ms and adds boundary sensitivity near ms.
Loading 2602.02383v2…