PaperlayerView exact revision on arXiv

Source-linked AI summary

Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training

Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Jiahao Xu, Tian Liang, Pinjia He, Zhaopeng Tu

arXiv:2407.09121v2cs.CLcs.AI

TL;DR

Safety-tuning data can bias LLM refusals toward the start of a response, leaving later-position refusal a key gap in jailbreak defense. DeRTa combines harmful-response prefixes with training transitions to refusal throughout harmful response sequences. Across six attack scenarios it improves safety without sacrificing helpfulness, including a reduction in Mistral-MoE average ASR from 79.1% to 8.7%.

  • Problem

    Safety-tuning data exhibits refusal position bias, while the influence of such data on jailbreak defense remains inadequately explored.

    Source: §1 · pp. 1–2

  • Method

    DeRTa embeds constructed harmful responses in training, using random-length harmful prefixes and an objective that transitions to refusal at every harmful-response position.

    Source: §1 · p. 2

  • Results

    Across six attack scenarios, DeRTa improves safety without sacrificing helpfulness; Mistral-MoE average ASR falls from 79.1% to 8.7%.

    Source: §1 · p. 2, §4.2 · p. 5

  • Takeaways & Limitations

    The findings underscore the importance of considering safety-tuning data and its biases when training models to make refusal decisions effectively.

    Source: §5 · p. 9

    The evaluation covers six representative jailbreak attack methods and compares against five selected defense methods, not all available methods.

    Source: Limitations · p. 10

Abstract

from arXiv · show

This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) by identifying and tackling a refusal position bias within safety tuning data, which compromises the models' ability to appropriately refuse generating unsafe content. We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position, significantly enhancing their safety capabilities. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model families across six attack scenarios, demonstrates that our method not only improves model safety without compromising performance but also surpasses baseline methods in defending against attacks.

1 Introduction

The introduction identifies refusal position bias in safety-tuning data as a gap in defending against jailbreaks. DeRTa trains models to refuse at later response positions, and its evaluation reports stronger attack defense without sacrificing helpfulness.

1 Introduction

  • Safety-tuning data often places refusal tokens at the start of responses, and the introduction identifies this refusal position bias as hampering effective refusals.Jailbreak prompts can circumvent protections without tuning or insider knowledge.Source: §1 · pp. 1–2
  • MLE with Harmful Response Prefix prepends a random-length segment of harmful response to a safe response to train refusal beyond the starting position.The harmful prefix also provides additional query context to help identify and avoid unsafe content.Source: §1 · p. 2
  • RTO trains transitions from potential harm to safety refusal at every position in the harmful response sequence.Unlike the single transition in MLE with Harmful Prefix, RTO simulates transitions throughout the sequence.Source: Figure 1 · p. 1, §1 · p. 2
  • Across six attack scenarios, DeRTa improves safety without sacrificing helpfulness and surpasses GPT-4, LLaMA3-Instruct, and five baseline methods in attack defense.The evaluation uses LLaMA3 and Mistral model families.Source: §1 · p. 2

2 Related Work

Related work surveys jailbreak attack and defense strategies, while noting that the influence of safety-tuning data remains less explored. It distinguishes this paper’s refusal-position perspective from prior harmful-prefix augmentation and frames Table 1’s comparison of early and later refusals.

2 Related Work

  • Jailbreaks can transform malicious queries into semantically equivalent forms, including ciphers, low-resource languages, code, and hypothetical scenarios.Related work also discusses social-science and psychology approaches, adversarial suffixes, few- or many-shot attacks, and multi-turn jailbreaks.Source: §2 · pp. 2–3
  • Defense strategies include safety prompts, input perturbation, safety decoding, detection, representation engineering, adversarial training, and priority training.Detection may use LLMs or specialized classifiers, while priority training uses designed data to prioritize higher-ranked instructions.Source: §2 · p. 3
  • Prior work by Qi et al. (2025) and Xu et al. (2024b) also addresses harmful-response prefixes followed by safety refusals, while this paper focuses on refusal-position bias and finds prefixes alone inadequate against CodeAttack and CompletingAttack.Table 1 compares counts of refusals within the first five tokens with later refusals across six attack tasks.Source: §2 · p. 3, Table 1 · p. 3

3 Methodology

The methodology identifies refusal-position bias: safety-tuned models learn refusals near response starts and may continue unsafe output once it begins. DeRTa trains refusal across response positions through harmful-prefix MLE and RTO.

3 Methodology

  • DeRTa targets refusal-position bias by training LLMs to refuse compliance at any response juncture.The method is designed to mitigate the weakness of refusals concentrated at the start of responses.Source: §3 · p. 3, §3.1 · p. 4

3.1 Standard Safety Tuning

  • Safety-tuning refusals cluster in the first few response tokens, leaving models prone to continue unsafe generation after it begins.The model must decide whether to refuse from the query alone, which may not provide enough information.Source: §3.1 · pp. 3–4

3.2 Our Approach

  • MLE with Harmful Response Prefix conditions safe-response training on a harmful prefix, adding context and training refusal beyond the response opening.The harmful prefix is excluded from the loss, so the model is not encouraged to generate harmful content before safe content.Source: §3.2 · p. 4

Reinforced Transition Optimization (RTO)

  • RTO trains transitions from potential harm to refusal at every position in the harmful response, repeating them L times more frequently than prefix MLE.The objective addresses the limitation that a single harmful-to-safe transition per training instance may not support consistent recognition and mitigation.Source: Reinforced Transition Optimization (RTO) · p. 4

4 Experiment

The experiments evaluate DeRTa against safety-training baselines across multiple model families and attack scenarios, then examine its components, refusal positions, and robustness. The reported results show lower attack success while retaining helpfulness, alongside limits involving over-sensitivity and adaptive attacks.

4.1 Setup

  • The evaluation used LLaMA3 and Mistral models, six attack scenarios, manual ASR assessment, and separate helpfulness tests.The safety set comprised 200 harmful questions, with 50-query subsets for PAIR and AutoDAN; helpfulness was assessed on GSM8K, MMLU, and AlpacaEval.Source: §4.1 · p. 5

4.2 Main Results

  • DeRTa reduced average ASR while retaining helpfulness across model architectures and beat every baseline on LLaMA3-70B attacks.Average ASR fell from 79.1% to 8.7% on Mistral-MoE and from 70.6% to 8.8% on LLaMA3-70B; CompletingAttack ASR was 4.0% versus RecAug's 25.0%, and CodeAttack ASR was 21.5% versus SoFA's 73.0%.Source: §4.2 · pp. 5–6, Table 2 · p. 6, Figure 3 · p. 6

4.3 Analysis

  • Combining harmful prefixes with RTO produced the strongest component results; RTO lowered ASR across attacks and supported refusals at later response positions.Only 20% of vanilla-training refusal tokens appeared after the start, versus 50%–55% for DeRTa variants; RTO placed refusal tokens beyond position 30 in 22.3% of responses. Harmful prefixes alone reduced CompletingAttack ASR from 90.5% to 25.0%, while RTO was important for CodeAttack.Source: §4.3 · p. 7, Figure 4 · p. 7
  • On CodeAttack, DeRTa outperformed DPO, whose use of harmful responses reduced ASR on most tasks but did not improve CodeAttack.The comparison examines whether DeRTa's effectiveness comes from including harmful responses or explicitly modeling token-wise safety transitions.Source: Figure 5 · p. 7, §4.3 · pp. 7–8
  • Across Mistral-7B, 8×7B, LLaMA3-8B, and 70B, results in Figure 6 show DeRTa significantly enhances safety irrespective of model size, supporting effectiveness across the tested model sizes.Results for small-scale models in the LoRA setting are also provided.Source: §4.3 · p. 8, Figure 6 · p. 8

4.4 Robustness Analysis

  • DeRTa remained effective across five tested languages despite English-only training and showed stable performance across sampling strategies and hyperparameters.The language evaluation used 1,000 crime-related prompts in English, French, Spanish, German, and Chinese.Source: Figure 7 · p. 8, §4.4 · p. 8, Figure 8 · p. 9
  • Under a fully safe criterion, DeRTa generally refused before producing harmful steps rather than refusing only after them.The criterion excludes responses containing any harmful steps or information, even if a refusal follows.Source: §4.4 · pp. 8–9, Figure 9 · p. 9

4.5 Discussion

  • Adding 200 borderline samples reduced over-sensitivity from 64.0% to 18.0% with a minor safety-performance decrease, while tested adaptive attacks remained a risk the method did not entirely eliminate.The adaptive evaluation used harmful prompts prefixed with a refusal template; the authors report maintained safety across tested queries but do not claim to eliminate adaptive-attack risk.Source: §4.5 · p. 9

5 Conclusion

DeRTa targets refusal-position bias by training models to refuse unsafe content throughout responses, and the authors report stronger safety than existing baselines, especially on completion-type attacks.

5 Conclusion

  • DeRTa trains LLMs to refuse unsafe content at any response position, addressing refusal-position bias in safety tuning data.The authors describe the approach as establishing a security mechanism across the response sequence.Source: §5 · p. 9
  • The evaluation reports safety superiority over existing baselines, especially for CodeAttack and CompletingAttack.The authors say the results support DeRTa as a security mechanism for the entire sequence.Source: §5 · p. 9
  • The findings underscore that safety-tuning data and its biases matter for refusal decisions amid evolving attack methods.The authors note DeRTa’s capability against recent attacks as a potential contribution to safer, more reliable LLMs.Source: §5 · p. 9

Limitations

The paper reports limitations in attack and baseline coverage, dialogue setting, and over-sensitivity; the appendix also records training and data-preparation settings.

Limitations

  • The evaluation covers six representative attacks and five selected baselines, uses single-turn dialogue, and reports more pronounced over-sensitivity.The authors say multi-turn extension is unverified and that a borderline dataset can mitigate over-sensitivity.Source: Limitations · p. 10

A Details of Setup

  • Full-parameter runs used batch size 128 for two epochs, while LoRA settings differed in learning rate, sequence length, warmup, and dropout.Full fine-tuning used learning rate 2e-5, max length 1024, and 95% dropout for the “Sorry” token; LoRA used learning rate 1e-4, max length 512, no warmup, and 0% token dropout. GPT-4-turbo-0409 and GPT-3.5-tubor-0125 were used.Source: A Details of Setup · p. 15

Training Data for Standard Safety Tuning

The appendix describes how standard safety-tuning data were prepared and how DPO comparison data were constructed.

Training Data for Standard Safety Tuning

  • The safety dataset was complemented to 6,000 instances because each DeRTa instance contains safe and harmful query–response pairs.The stated aim was fair comparison with vanilla safety tuning.Source: Training Data for Standard Safety Tuning · p. 15
  • For DPO, Qwen1.5-chat-0.5B generated responses to 60k helpful instructions, with original Evol-Instruct and Qwen responses assigned as chosen and rejected.For harmful questions, safe and harmful responses served as chosen and rejected, respectively; DPO training added one epoch after standard safety training.Source: Training Data for Standard Safety Tuning · p. 15
  • Harmful responses were generated by an adversarially tuned LLaMA3-8B-Instruct model from BeaverTails questions, then edited for grammar and lexical diversity.GPT-3.5 also removed safety warnings from the generated harmful responses.Source: Training Data for Standard Safety Tuning · p. 15
  • LLaMA3-70B training used about 100 GPU hours on a server with eight A800 80GB GPUs.Source: Training Data for Standard Safety Tuning · p. 15

B Details of Applied Attacks

The appendix outlines six attack scenarios, covering code completion, iterative prompt modification, jailbreak prompts, role-play and cipher, declarative completion, and automatically generated prompts.

B Details of Applied Attacks

  • CodeAttack disguises malicious questions as code-completion tasks, using the Python-stack version.Source: B Details of Applied Attacks · p. 16
  • PAIR iteratively modifies harmful questions with an AttackLLM; this experiment used GPT-3.5 for attacks and GPT-4 as judge.The setup used breadth 20 streams and depth 3 iterations.Source: B Details of Applied Attacks · p. 16
  • JailbreakChat prompts were selected by website votes, with 50 prompts collected and randomly paired with harmful questions.Source: B Details of Applied Attacks · p. 16
  • SelfCipher combines role-playing and few-shot demonstrations to present malicious question-answer pairs as cipher communication.The experiment used five fixed malicious demonstrations.Source: B Details of Applied Attacks · p. 16
  • CompletingAttack removes formatting tokens to turn queries into declarative text for completion, using GPT-3.5-turbo for transformation.Source: B Details of Applied Attacks · p. 16
  • AutoDAN uses a hierarchical genetic algorithm to generate stealthy jailbreak prompts, with GPT-4 as the mutation model.The reported attack success rate may be lower than in the original paper because only the partial final mutated prompt is passed to the test model.Source: B Details of Applied Attacks · p. 16

C Guideline for Safety Evaluation

The guidelines distinguish unsafe responses from responses considered not unsafe. They define unsafe content by its relation to harmful input and its actionable harmful information, while also specifying qualifying non-unsafe responses.

D Case Study

The section presents selected case examples and points readers to the full set of experimental cases. It also identifies the scope of tables covering small-scale, LoRA, and DPO results.

E Main Results on Small-Scale LLMs

The section reports that the method remains effective on small-scale models, consistent with large-scale outcomes. It also presents LoRA fine-tuning results as a variable-control check supporting earlier conclusions.

Loading 2407.09121v2…