Source-linked AI summary
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
Oleksandr Cherednichenko, Roman Klypa
TL;DR
Open-weight language models still lack transparent, robust safety alignment, while existing preference methods can over-refuse benign prompts or degrade quality. Suan directly designs the optimization gradients and, across extensive evaluations, achieves state-of-the-art safety alignment while preserving response utility. The authors identify broader comparisons and jailbreak robustness as future work.
Problem
Transparent and robust safety alignment for open-weight models remains unresolved, while existing methods can suffer from over-refusal and quality degradation.
Method
Suan bypasses variational derivation and designs an interpretable preference-optimization objective directly at the gradient level.
Results
Suan achieves state-of-the-art alignment across diverse evaluations, improving safety and reducing over-refusal without degrading response quality.
Takeaways & Limitations
Suan matches DPO’s strong harmlessness while avoiding the over-refusal and helpfulness trade-offs observed in competing preference-tuned variants.
Takeaways & Limitations
The evaluation compares Suan with DPO and SafeDPO; broader preference-optimization, RL, jailbreak, and reasoning-model evaluations remain future work.
Abstract
from arXiv · showhide
Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.
1 Introduction
LLMs are widely used but can produce harmful content, making transparent safety alignment essential. Existing preference methods face over-refusal and quality degradation, motivating Suan’s gradient-level, interpretable approach.
- LLMs can generate misinformation, malware, hazardous instructions, and leaked private data despite widespread adoption.
- Post-training preference optimization aligns model behavior with human values while mitigating harmful outputs.
- RLHF is complex because it requires auxiliary reward models and multi-stage optimization with sensitive hyperparameters.
- Transparent safety alignment for open-weight models remains unresolved, with existing methods vulnerable to adversarial jailbreaks.
- Suan formulates the objective at the gradient level rather than modifying a preference loss derived from relaxed safety-constrained formulations.
- Across diverse models, datasets, and benchmarks, Suan is reported to improve safety while reducing over-refusal without degrading response quality.
2 Theoretical Preliminaries
The paper reviews language-model training, RLHF, and direct alignment algorithms before identifying likelihood displacement as a structural problem in common objectives. These methods trade computational simplicity against reward-model dependence, surrogate regularization, and unstable preference dynamics.
- Post-training of Large Language Models: Language models factor sequence probabilities into conditional next-token distributions, providing the basis for subsequent training objectives.
- Post-training of Large Language Models: Supervised fine-tuning adapts a pretrained model to instruction following using prompt-response pairs and cross-entropy loss.
- Post-training of Large Language Models: RLHF fits a scalar reward model, then optimizes the policy for expected reward while constraining divergence from the reference policy.
- Direct Alignment Algorithms: Safe-RLHF adds a safety cost model and constrains the policy to reduce unsafe-response probability because preferences alone may not guarantee safety.
- Direct Alignment Algorithms: DPO removes the explicit reward model through a closed-form preference loss derived from KL-regularized RLHF and the Bradley–Terry model.
- Direct Alignment Algorithms: IPO regularizes the implicit reward margin, while SafeDPO uses safety metadata and can invert preference order when the preferred response is unsafe.
- Direct Alignment Algorithms: Common direct alignment objectives can displace likelihood from preferred responses, causing preferred-sample likelihood decline and downstream output degradation.
3 Proposed Method
Suan designs training gradients directly to address unstable and unintuitive dynamics in direct alignment algorithms. It prioritizes poorly performing preference pairs and regularizes chosen likelihoods toward a reference model to reduce over-refusal while preserving quality.
- Direct alignment methods face practical convergence issues, unintuitive gradient weighting, flawed KL surrogates, and assumptions that do not hold in real-world training.
- Suan bypasses variational derivation and introduces a custom gradient weighting scheme that prioritizes response pairs where the model performs poorly.
- The preference gradient term uses a preference margin controlled by τ.
- Suan adds reference-model regularization because the k2 estimator’s gradient coefficient can otherwise dominate training dynamics.
- The preference and regularization gradients admit closed-form losses that are combined into the Suan objective.
- Suan suppresses over-refusal by reducing over-optimization on correct pairs and preserves output quality by regularizing chosen likelihoods toward the reference model.
4 Experimental Evaluation
The evaluation spans multiple open-source model families and benchmarks covering harmlessness, compliance, and helpfulness. It measures adversarial safety, benign-prompt over-refusal, and response quality with task-specific benchmarks and judging procedures.
- The study evaluates eight open-source model families and parameter sizes, including Mistral, Falcon3, Llama3.1, Gemma-2, Qwen-3, Yi-1.5, DeepSeek, and OLMo-3.
- Harmlessness is evaluated with Malicious Instruct, HarmBench, AdvBench, and SORRYBench using Attack Success Rate for unsafe outputs.
- Figure 2 compares Attack Success Rate, Over-Refusal Rate, and quality scores, with lower safety and refusal scores but higher quality scores preferred.
- Compliance benchmarks measure the percentage of benign instructions incorrectly rejected through Over-refusal Rate.
- Helpfulness is assessed on AlpacaEval, AlpacaFarm, MT-Bench, and ArenaHard using an LLM-as-a-judge setup.
5 Results
Across safety, compliance, helpfulness, likelihood displacement, and hyperparameter sweeps, Suan delivers a more balanced and interpretable alignment profile than the compared methods. Its regularization avoids preferred-response likelihood decline and remains robust across tested β values.
- Safety Alignment: DPO and SafeDPO achieve strong harmlessness but incur elevated over-refusal and compromised output quality.
- Safety Alignment: Suan consistently achieves the optimal alignment trade-off across evaluated settings, matching DPO’s harmlessness without sacrificing helpfulness or compliance.
- Likelihood Displacement: Suan avoids the pronounced preferred-completion likelihood decline observed for DPO and SafeDPO by regularizing drift from the reference model.
- Ablation Study: At β = 0.01, DPO suffers a severe quality collapse, making its Attack Success Rate uninformative for judging safety.
- Ablation Study: DPO and SafeDPO β variation does not predictably constrain drift from the reference model, unlike Suan’s interpretable β behavior.
- Ablation Study: Increasing β in Suan monotonically enforces closer alignment with the reference model, preserving robust performance across tested values.
6 Conclusions
Suan is presented as a lightweight, interpretable preference-alignment algorithm that achieves broad safety and capability performance while addressing over-refusal and quality degradation. The authors identify broader baseline, jailbreak, and reasoning-model evaluations as future work.
- Suan uses an interpretable objective motivated by gradient dynamics to address over-refusal and utility degradation.
- The method is computationally lightweight, straightforward to implement, and operates on standard preference data without filtering.
- Evaluations across eight diverse language models show state-of-the-art performance across safety and capability benchmarks.
- Future work should compare Suan with other preference-optimization and reinforcement-learning methods, test adversarial jailbreak robustness, and evaluate Large Reasoning Models.
Impact Statement
The work aims to address red-teaming and helpfulness in LLM post-training through interpretable gradients and a novel objective. It targets less harmful and more helpful outputs while noting broader societal consequences of improved generative variety.
- The work targets red-teaming and helpfulness in LLM post-training by composing interpretable gradients into a novel objective.
- The stated goal is to support models that produce less harmful and more helpful outputs.
- The statement notes potential societal consequences associated with improving generative variety.
A Omitted Proofs
The omitted proofs derive closed-form loss expressions from gradient identities, while the appendix documents models, datasets, and compute-efficient training settings used in the study.
- Proofs: The proof uses sigmoid identities and antiderivatives to derive closed-form expressions for the loss and its gradient.It defines t(θ), relates the gradient to ∇θt(θ), and then integrates the resulting expression.
- Proofs: The derivation substitutes λ(y) into the gradient expression and uses ∇θλ(y) = ∇θ log πθ(y | x).
- Models: The experiments cover eight open-weight model families, including Mistral, Falcon, Llama, Gemma, Qwen, Yi, DeepSeek, and OLMo.
- Datasets: The study uses Alpaca3, HH-RLHF, and PKU-SafeRLHF-30K as instruction-following and preference datasets.Alpaca3 contains approximately 52K instruction–response pairs, while HH-RLHF contains approximately 160K chosen/rejected pairs and PKU-SafeRLHF-30K approximately 27K samples.
- Training setup: Compute-efficient training uses 4-bit NormalFloat quantization, QLoRA, and gradient accumulation, with one training epoch and a peak learning rate of 2 × 10^-4.The reported setup uses a batch size of 2, four accumulation steps, weight decay 0.01, and LoRA parameters r = 16 and α = 16.
B.5 Evaluation datasets
The evaluation suite measures helpfulness, harmlessness, reasoning, instruction following, creativity, adversarial robustness, and over-refusal using diverse benchmarks and decoding settings.
- Quality benchmarks: Alpaca Eval, MT-Bench, and Arena Hard evaluate instruction following and helpfulness, while NoveltyBench measures distinct high-quality outputs.Alpaca Eval uses 129 prompts, MT-Bench 80 two-turn questions, Arena Hard 500 challenging prompts, and NoveltyBench 100 curated prompts.
- Safety benchmarks: Malicious Instruct, Harm Bench, AdvBench, and SORRY-Bench provide harmful or adversarial prompts for safety evaluation.Their reported dataset sizes are 100, 450, 500, and 9.2K prompts, respectively, although the main SORRY-Bench experiments use 540 prompts.
- Compliance benchmarks: XS-Test and OR-Bench measure over-refusal using benign prompts that should be answered rather than rejected.The evaluation filters 100 safe XS-Test samples and selects a 1,320-prompt hard subset from OR-Bench.
- Reasoning benchmarks: ARC-Challenge and MMLU assess scientific reasoning and broad knowledge across 1,172 and 14,042 test questions, respectively.
- Inference settings: Generation uses nucleus sampling with p = 0.9 for creative-writing, instruction-following, and safety tasks, while Arena Hard, MMLU, and ARC use greedy decoding with T = 0.0.All benchmarks use a 512-token limit except Arena Hard, which uses 4096 tokens.
- LLM judging: An LLM judge scores generated responses from 1 to 5 on Alpaca Eval, Arena Hard, and MT-Bench.The study uses Flow-Judge-v0.1 to compare responses for helpfulness, usefulness, and relevance.
C.1 Ablations
The ablation study identifies τ = 1 and β = 0.1 as the best-performing Suan settings across helpfulness, harmlessness, and compliance tasks, while broader results remain robust across models and benchmarks.
- Ablation: τ = 1 and β = 0.1 demonstrate optimal performance across three safety-alignment tasks.The ablation evaluates AlpacaEval, Malicious Instruct, and XS-Test.
- Cross-benchmark results: The broader evaluation reports competitive harmlessness with DPO and SafeDPO while observing catastrophic over-refusal for those methods.The reported performance trends remain robust across the evaluated models and benchmarks.
- Qualitative comparison: On an OR-Bench benign prompt, Suan provides a helpful and harmless response where DPO and SafeDPO fail to recognize the prompt as benign.
C.3 Additional Experiments
Additional experiments show that Suan generally preserves factual knowledge and combines high output diversity with high Utility. Results also depend on the safety dataset, while qualitative and tabular comparisons highlight reduced over-refusal relative to DPO and SafeDPO.
- Factuality: Suan generally preserves pretrained knowledge on ARC and MMLU, whereas DPO and SafeDPO show a visible MMLU performance decline.These factuality evaluations assess scientific knowledge across methods.
- Diversity: Suan achieves high output diversity while maintaining high Utility on NoveltyBench.NoveltyBench evaluates multiple distinct, high-quality outputs using Utility-k and Distinct-k metrics.
- Dataset comparison: PKU-SafeRLHF yields better quality performance than HH-RLHF across methods and models, motivating its use as the default safety dataset.The authors report slight degradation with HH-RLHF relative to PKU-SafeRLHF.
- Over-refusal: On a benign OR-Bench prompt, DPO and SafeDPO over-refuse, while Suan provides a detailed and helpful response.The comparison is shown for Mistral-12B.