Source-linked AI summary
SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, Radha Poovendran
TL;DR
Jailbreak attacks remain a major safety threat because they can bypass LLM alignment and elicit harmful behavior. SafeDecoding uses safety-aware token decoding to amplify safety disclaimers and attenuate attacker-aligned sequences. Across broad evaluations, it reduces jailbreak success and harmfulness while retaining benign-query helpfulness and outperforming six defenses.
Problem
Jailbreak attacks can bypass existing LLM alignments, motivating defenses that improve safety without sacrificing helpfulness or incurring substantial inference costs.
Method
SafeDecoding trains a safety-aware expert model and combines its token probabilities with the original model to amplify safety disclaimers and attenuate attacker-aligned sequences.
Results
Across five LLMs, six jailbreak attacks, and four benchmarks, SafeDecoding consistently outperforms six baselines, reduces attack success and harmfulness, and preserves benign-query helpfulness.
Takeaways & Limitations
SafeDecoding provides a computationally lightweight decoding-based defense that improves jailbreak robustness while maintaining helpful responses to benign users.
Takeaways & Limitations
The evaluation focuses on LLMs; performance on emerging multimodal LLMs remains future work, and rare responses may initially reject harmful queries before agreeing.
Abstract
from arXiv · showhide
As large language models (LLMs) become increasingly integrated into real-world applications such as code generation and chatbot assistance, extensive efforts have been made to align LLM behavior with human values, including safety. Jailbreak attacks, aiming to provoke unintended and unsafe behaviors from LLMs, remain a significant/leading LLM safety threat. In this paper, we aim to defend LLMs against jailbreak attacks by introducing SafeDecoding, a safety-aware decoding strategy for LLMs to generate helpful and harmless responses to user queries. Our insight in developing SafeDecoding is based on the observation that, even though probabilities of tokens representing harmful contents outweigh those representing harmless responses, safety disclaimers still appear among the top tokens after sorting tokens by probability in descending order. This allows us to mitigate jailbreak attacks by identifying safety disclaimers and amplifying their token probabilities, while simultaneously attenuating the probabilities of token sequences that are aligned with the objectives of jailbreak attacks. We perform extensive experiments on five LLMs using six state-of-the-art jailbreak attacks and four benchmark datasets. Our results show that SafeDecoding significantly reduces the attack success rate and harmfulness of jailbreak attacks without compromising the helpfulness of responses to benign user queries. SafeDecoding outperforms six defense methods.
1 Introduction
LLMs remain vulnerable to jailbreak attacks that bypass alignment, while existing defenses can be ineffective, costly, or harmful to benign-query helpfulness. SafeDecoding addresses this by reshaping token probabilities toward safety disclaimers and away from attacker-aligned sequences, and experiments report strong defense performance with negligible overhead and preserved helpfulness.
- Jailbreak attacks can bypass existing alignment techniques and provoke unintended harmful behavior from LLMs.
- Existing defenses may lack effectiveness, incur high inference-time costs, and compromise helpfulness for benign users.
- Safety-disclaimer tokens remain in the model’s top-probability sample space even when attacker-aligned tokens dominate, revealing residual safety awareness.
- SafeDecoding amplifies safety-disclaimer probabilities while attenuating token sequences aligned with attackers’ objectives.
- SafeDecoding is evaluated on five LLMs, six state-of-the-art jailbreak attacks, two harmful benchmarks, and two utility benchmarks against six baselines.
- SafeDecoding consistently outperforms the baselines, adds negligible computation overhead, and preserves helpfulness for benign queries.
2 Related Work
Prior work studies both jailbreak attacks and defenses. Attacks include empirical prompting and optimization-based methods, while defenses detect or mitigate adversarial inputs through filtering, transformation, demonstrations, or guided generation.
- Jailbreak attacks include empirical prompt-engineering methods and optimization-based adversarial attacks.
- Optimization-based attacks use gradients, genetic algorithms, or edits generated by a pretrained LLM to subvert alignment.
- Existing defenses are organized into detection-based and mitigation-based categories.
- Detection-based Defense: Detection defenses use content filtering, perplexity, model-based harmfulness detection, or randomized input perturbations with aggregated predictions.
- Mitigation-based Defense: Mitigation defenses modify inputs, guide rewindable generation, provide rejection demonstrations, or add responsible-response reminders.
3 Preliminaries
The paper formulates jailbreak defense as autoregressive token generation constrained against attacker-aligned sequences while retaining helpfulness, efficiency, and compatibility. It defines the attacker’s optimization target as a harmful token sequence and treats the attacker’s specific goal as unknown to developers.
- An autoregressive LLM predicts each next-token probability from the preceding token sequence and generates responses by iteratively applying a decoding strategy.
- Common decoding strategies include greedy decoding, beam search, top-k sampling, and nucleus (top-p) sampling.
- A jailbreak attacker seeks a preceding token sequence that causes the model to generate a continuation aligned with the attacker’s goal set H.
- The defense objective is to prevent generated continuations from satisfying the attacker-aligned constraint while strengthening model safety.
- The decoding strategy should preserve helpfulness on benign queries, remain computationally lightweight, and work across models with varying architectures and parameters.
- Developers generally know human values and safety standards but not the attacker’s specific goal H.
4 Safety-Aware Decoding: SafeDecoding
SafeDecoding is a two-phase defense that combines an original LLM with a safety-hardened expert model to reshape token probabilities during inference. It suppresses tokens aligned with attacker objectives while amplifying human-value-aligned tokens, with tunable trade-offs in diversity, helpfulness, and efficiency.
- Inference Phase: The constructed distribution attenuates unsafe tokens and amplifies human-value-aligned tokens according to the models’ probability discrepancy.For malicious queries, a positive expert-minus-original difference indicates human-value alignment, while a negative difference indicates unsafe behavior; α weights the two models before normalization.
- Inference Phase: During inference, the original and expert models process the same query, and SafeDecoding combines their outputs into a new token distribution.The method forwards the preceding token sequence to both models and constructs a distribution over a shared sample space.
- Training Phase: SafeDecoding constructs a safety-hardened expert model by fine-tuning the original LLM on filtered refusals to harmful queries.The dataset begins with 36 harmful queries across 18 categories; generated responses are filtered to retain effective refusals, and parameter-efficient fine-tuning such as LoRA is used.
- Inference Phase: SafeDecoding forms its sample space by intersecting the top-k tokens from the original and expert models.The original model contributes diversity and response quality, while the expert model contributes stronger alignment with human values for malicious inputs.
- Helpfulness and Efficiency: A small c restricts the sample space and may reduce diversity and helpfulness, while applying SafeDecoding at every decoding step may make responses conservative and computation more demanding.SafeDecoding can be combined with top-p, top-k, greedy, and beam search; the figure illustrates applying it only to the first 2 tokens before normal decoding resumes.
5 Experiments
The experiments evaluate SafeDecoding across multiple models, attacks, harmfulness and utility benchmarks, efficiency measures, and ablations. SafeDecoding generally improves jailbreak defense while preserving helpfulness and adding limited overhead.
- Experimental Setup: The evaluation covers five open-source LLMs, six jailbreak attacks, two harmful-query benchmarks, and the MT-bench and Just-Eval utility benchmarks.The attacks include gradient-based, genetic-algorithm-based, edit-based, and empirical jailbreak methods.
- Safety: SafeDecoding reduces attack success and harmfulness, achieving 0% ASR against DeepInception on Vicuna and nearly 0% ASR across attacks on Llama2.It outperforms almost all baseline defenses on weakly aligned Vicuna and all baselines in most Table 1 cases.
- Helpfulness: SafeDecoding preserves benign-task utility, with MT-bench deviations of 1% for Vicuna and 5% for Llama2.The reported results indicate that the original models’ utility remains largely intact after deployment.
- Efficiency: SafeDecoding adds only 3% overhead on Llama2 and 7% on Vicuna compared with no defense.The efficiency comparison uses average token generation time ratio and excludes defenses that at least double ATGR.
- Ablation Analysis: Increasing α, m, and c lowers ASR and harmfulness until performance stabilizes, with insensitivity reported for α ≥3, m ≥2, and c ≥7.Top-p sampling slightly increases ASR as p increases, trading some safety for response diversity.
- Additional Experiments: A universal expert model achieves comparable, and sometimes better, defense performance than the original expert model.The transferability comparison reports this pattern using harmful score and ASR across several attacks.
6 Conclusion and Future Work
The paper concludes that SafeDecoding is a lightweight, effective safety-aware decoding strategy. It uses token-probability manipulation to strengthen safety disclaimers and weaken attack-aligned continuations while maintaining efficiency and helpfulness.
- Conclusion: SafeDecoding attenuates probabilities of attacker-aligned token sequences and amplifies probabilities associated with safety disclaimers.The approach is motivated by safety disclaimers remaining among the highest-probability tokens despite harmful-token probabilities being larger.
7 Limitations
The paper identifies semantic-transition failures and limited multimodal coverage as limitations of SafeDecoding. These boundaries motivate further investigation beyond the evaluated language-model setting.
- Limitations: In 31 of 250 responses, the model initially rejects a harmful query but later agrees with it, exposing a semantic-transition inconsistency.The paper also limits evaluation to large language models and leaves multimodal deployment for future investigation.
8 Ethical Impact
SafeDecoding is presented as a lightweight defense intended to mitigate jailbreak attacks while preserving efficient and helpful service for benign users. The authors also acknowledge that the defense may motivate new attack strategies.
- Ethical Impact: SafeDecoding aims to strengthen LLM safety with a lightweight decoding strategy that mitigates jailbreak attacks while preserving benign-user service.The stated goals include efficiency and helpfulness for benign queries.
- Ethical Impact: The authors acknowledge that developing SafeDecoding may lead attackers to devise strategies intended to bypass it.They identify randomized decoding with randomly chosen hyper-parameters as a future mitigation direction.
A Detailed Experimental Setups
The experiments evaluate multiple jailbreak defenses, detection mechanisms, system prompts, and refusal-string criteria across diverse attack settings. The setup includes input and output transformations alongside classifier- and prompt-based defenses.
- Attack Methods and Datasets: GCG, AutoDAN, and PAIR use 50 representative harmful queries from Advbench to generate model-specific attack prompts.SAP30, DeepInception, and GPTFuzzer-Template provide additional red-teaming or template-based attack settings.
- Baseline Defenses: PPL detects potentially harmful inputs using perplexity, while Paraphrase transforms user queries before sending them to the target model.The PPL threshold is set to the highest perplexity among harmful Advbench queries.
- Baseline Defenses: Self-Examination uses the language model to classify generated text as harmful and returns a refusal template when harmful content is detected.The classifier prompt requests a Yes-or-No judgment, followed by a standardized refusal response for a positive result.
- Prompt-Based Defenses: Self-Reminder appends responsible-response instructions, while ICD uses demonstrations of refusing harmful prompts to improve robustness.The system prompt explicitly reminds the model not to generate harmful or misleading content.
- System Prompts: System prompts can significantly affect attack and defense performance, so the experiments adopt prompts from fschat-0.2.20 and Hugging Face.The detailed prompts are reported in Figure 4.
- Evaluation: Dic-Judge uses refusal-string keywords to determine whether responses remain aligned with an attacker’s goal.The keyword set is expanded for Vicuna, Dolphin, and Guanaco, while “Hello” is excluded because it does not directly reject a query.
A.5 Datasets and Fine-tune Setups
The fine-tuning setup constructs a safety-aware dataset from harmful queries and model-generated refusals, then fine-tunes an expert model with LoRA. The authors motivate this construction to avoid token-distribution mismatch from public supervised datasets.
- Fine-tune Setup: Public supervised fine-tuning datasets may cause substantial shifts in initial-token distributions, creating mismatches between the original and expert models.Such mismatches can produce grammatically incorrect or contextually meaningless sampled tokens.
- Dataset Construction: The constructed seed dataset contains 36 harmful queries spanning 18 harmful categories.The categories include discrimination, hate speech, violence, theft, weapons, adult content, and terrorism, among others.
- Dataset Construction: Refusal responses are generated with top-p = 0.9 and Temperature = 0.7, then filtered using GPT-4-0613 for explicit rejection.Each harmful query is generated twice, producing a maximum fine-tuning dataset size of 72.
- Dataset Construction: For Dolphin, the system prompt is modified to induce explicit refusals before collecting training responses.The prompt instructs the model to reject user questions and provide supporting reasons.
- Fine-tune Setup: The original model is fine-tuned with LoRA using SFFTrainer, and all models can be fine-tuned within one minute using the constructed dataset.Default fine-tuning parameters are reported in Table 5.
B More Results
Additional experiments show that SafeDecoding remains effective across Guanaco, Falcon, and Dolphin, including the uncensored Dolphin model, and transfers across original LLMs through a universal expert model. Comparisons also show that fine-tuning or classifier use alone is insufficient and can reduce utility.
- B.1 SafeDecoding in More Models: SafeDecoding substantially reduces attack success rate and harmful scores across Guanaco, Falcon, and Dolphin, including the uncensored Dolphin model.The result supports compatibility across different model architectures.
- B.2 Fine-tune is Not Enough: Fine-tuning alone does not adequately defend against jailbreak attacks and can reduce utility by producing refusals for harmless prompts.The expert model’s defense and utility are compared directly with SafeDecoding.
- B.2 Fine-tune is Not Enough: SafeDecoding achieves lower harmful scores and attack success rates than Expert-Classifier.The classifier may misclassify stealthy attacks, and Llama2 may disregard classifier instructions and answer directly.
- B.3 Transferability of SafeDecoding: The universal expert model is trained on Vicuna-7b using varied vocabulary preferences to improve compatibility across original LLMs.This design addresses token mismatch between different language models.
- B.3 Transferability of SafeDecoding: The universal expert model provides comparable harmful-score and attack-success-rate performance to the original expert model, and sometimes performs better.The authors attribute the improvement to a larger and more diverse query-response training dataset.
C.1 SafeDecoding is Safe
The case studies show SafeDecoding rejecting harmful jailbreak requests across multiple models and attacks while preserving helpful responses to benign queries. One reported case also shows SafeDecoding falling short against DeepInception on Guanaco.
- C.1 SafeDecoding is Safe: SafeDecoding rejects harmful requests in Falcon against SAP30, Vicuna against PAIR, Dolphin against GPTFuzzer Template, and Llama2 against GCG.The examples include refusals for hacking, false accusations, fraudulent insurance claims, and other harmful prompts.
- C.1 SafeDecoding is Safe: SafeDecoding is reported as more effective than Expert-Classifier and comparably effective with either the original or universal expert model.The latter comparison is presented as evidence of transferability.
- C.1 SafeDecoding is Safe: SafeDecoding produces explicit refusals to requests for hacking into corporate networks and sensitive government databases.The responses cite illegality, ethical guidelines, and potential consequences as reasons for refusal.
- C.2 SafeDecoding is Helpful: SafeDecoding remains helpful on benign questions about the largest star in the galaxy and caring for a wooden table.The corresponding outputs provide substantive factual and practical answers.
- C.1 SafeDecoding is Safe: SafeDecoding falls short against the DeepInception attack when applied to Guanaco.The associated example still begins to discuss weapon design despite refusing to create the requested layered science-fiction scenario.