PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking

Junda Zhu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin, Lei Sha

arXiv:2502.12970v3cs.CL

TL;DR

Jailbreak safety remains an open question as LLM reasoning capabilities improve, while existing defenses often rely on guardrails or tuning rather than reasoning. R2D adds safety-aware reasoning and step-wise self-assessment to generation; the authors report lower attack success rates without over-refusal or performance drops.

  • Problem

    How improved reasoning capabilities affect LLM safety against jailbreaks remains underexplored, while existing defenses rely on guardrails or supervised signals and neglect reasoning.

    Source: §1 · pp. 1–2

  • Method

    R2D trains models for staged reasoning and uses step-wise safety pivot tokens, with CPO to strengthen prediction of those tokens.

    Source: §1 · p. 2

  • Results

    R2D-enabled LLMs consistently achieve lower ASRs than previous defense approaches, without reported over-refusals or performance drops.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results support safety-aware reasoning as a training paradigm for improving defense while maintaining model performance.

    Source: §5 · p. 9

    The study focuses on distilling reasoning from MR, constrained by foundation-model size and capabilities, rather than reinforcement learning or test-time scaling.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large Reasoning Models (LRMs) have recently demonstrated impressive performances across diverse domains. However, how the safety of Large Language Models (LLMs) benefits from enhanced reasoning capabilities against jailbreak queries remains unexplored. To bridge this gap, in this paper, we propose Reasoning-to-Defend (R2D), a novel training paradigm that integrates a safety-aware reasoning mechanism into LLMs' generation process. This enables self-evaluation at each step of the reasoning process, forming safety pivot tokens as indicators of the safety status of responses. Furthermore, in order to improve the accuracy of predicting pivot tokens, we propose Contrastive Pivot Optimization (CPO), which enhances the model's perception of the safety status of given dialogues. LLMs dynamically adjust their response strategies during reasoning, significantly enhancing their safety capabilities defending jailbreak attacks. Extensive experiments demonstrate that R2D effectively mitigates various attacks and improves overall safety, while maintaining the original performances. This highlights the substantial potential of safety-aware reasoning in improving robustness of LRMs and LLMs against various jailbreaks.

1 Introduction

The introduction identifies a gap in jailbreak defense: existing approaches rely on external guardrails or supervised signals while neglecting LLMs’ reasoning. It presents R2D’s safety-aware reasoning approach and evaluates its defense, refusal behavior, and general performance.

1 Introduction

  • Existing jailbreak defenses rely on external detection or supervised tuning, while neglecting LLMs’ reasoning capabilities for safety.The introduction also notes that real-world safety can involve a performance trade-off.Source: §1 · pp. 1–2
  • R2D combines staged reasoning via SwaRD with step-wise safety assessment and pivot tokens, enhanced by CPO.The paper says learning from reasoning trajectories rather than hard refusal helps avoid over-refusal in safe scenarios.Source: §1 · p. 2
  • The experiments assess transferred attacks on JailbreakBench, multiple attacks on HarmBench, over-refusal on XSTest, and general performance.The introduction describes these evaluations as testing defense, refusal behavior, and performance on normal-use datasets.Source: §1 · p. 2
  • The authors report that R2D defends against multiple jailbreak attacks while enhancing response safety and avoiding over-refusal in normal use.They also state that original performance is maintained.Source: §1 · p. 2

2 Related Works

Related work covers training-based safety approaches and reasoning-based reflection or guardrails. R2D is distinguished by using long contextual reasoning to improve the LLMs’ own safety.

2.1 Safety-Aware Training

  • Training-based safety approaches include SFT and RLAIF, safety-tuned data mixtures, guardrail models, and adversarial-training defenses such as RPO.The cited approaches target LLMs or external guardrails to recognize unsafe inputs and responses.Source: §2.1 · p. 2

2.2 Reasoning and Safety of LLMs

  • Prior reasoning and safety work includes self-refinement, self-reflection, intention analysis, and reasoning-enhanced guardrails.The passage describes SafeChain as addressing safety alignment after reasoning-enhanced tuning, while IA requires intention analysis but does not provide reasoning capabilities.Source: §2.2 · p. 3
  • R2D uses long contextual reasoning to make context-safety decisions and enhance the LLMs’ own safety, capabilities, and helpfulness.Its overview contrasts this approach with hard refusals without reasons.Source: §2.2 · p. 3, Figure 2 · p. 3

3 Reasoning-to-Defend: Learning to Reason for LLMs’ Safety-Awareness

R2D equips generation with step-wise reasoning and safety self-evaluation, then trains models to produce safety pivot tokens. It combines reasoning distillation with Contrastive Pivot Optimization (CPO) to strengthen safety awareness.

3.1 Overview of R2D Framework

  • R2D generates inner reasoning and evaluates each step with [SAFE], [UNSAFE], or [RETHINK] pivot tokens that mark its safety status.The framework uses these checkpoints to assess the current reasoning trajectory or response and modify or discard unsafe paths.Source: §3.1 · p. 3, §3.3 · p. 4

3.2 Safety-Aware Reasoning Distillation

  • Safety-aware Reasoning Distillation (SwaRD) trains non-reasoning LLMs on a dataset of reasoning trajectories and final answers for safe and jailbreak instructions.The dataset uses responses from a reasoning model guided by safety-aware context; minimizing the objective encourages reasoning before answering.Source: §3.2 · p. 4

3.3 Contrastive Pivot Optimization

  • CPO trains models to predict a pivot token at each reasoning step, using it as a checkpoint for assessing safety and revising unsafe paths.Its training objective is added to the SwaRD loss.Source: §3.3 · p. 4
  • Pivot-token labels begin with the reasoning model’s self-evaluation, then a guardrail model tags each step before trajectories and labels are aggregated into the training dataset.The initial self-evaluation primarily yields [RETHINK]; guardrail tagging aims for more precise, contextually appropriate labels.Source: §3.3 · pp. 4–5

4 Experiments

Experiments compare R2D with defense baselines across jailbreak benchmarks and examine general abilities, over-refusal, and inference length. R2D lowers attack success rates, while reported general-ability results remain largely comparable and safe-prompt compliance is maintained or improved.

4.1 Experimental Setups

  • The setup evaluates JailbreakBench and HarmBench with ASR, XSTest with refusal/compliance categories, and general abilities with lm-evaluation-harness accuracy.JailbreakBench uses 100 unsafe prompts and transferred attacks; HarmBench uses 400 harmful behaviors, and training trajectories are collected from Alpaca and AdvBench.Source: §4.1 · p. 5

4.2 Main Results

  • On JailbreakBench, R2D lowers average ASR by 56% versus undefended models and 17% versus Erase-and-Check, with lower average ASRs than defense baselines.The experiments also compare reasoning models and R2D-enhanced versions, including prompt-level R2D-p and inference-time R2D-n.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6
  • Across HarmBench base models, R2D cuts overall ASR by up to 48%; ZeroShot/FewShot ASR is close to 0%, while PAIR/AutoDAN average around 10%.HarmBench contains 400 harmful behaviors and the evaluation uses ZeroShot, FewShot, PAIR, and AutoDAN attacks.Source: §4.2 · pp. 6–7, Figure 3 · p. 7

4.3 Detailed Analysis and Discussion

  • R2D causes no significant general-ability degradation; R17B and QwQ32B improve on BoolQ and remain within a 4% margin on other reported tasks.Non-reasoning models show only minor drops on several datasets.Source: Table 3 · p. 7, §4.3 · p. 7
  • On XSTest, Qwen2.5-14B’s Full Compliance rate on safe prompts increases by 4.8%, while Qwen2-7B’s Full Refusal rate on unsafe prompts drops by more than 50%.The results use response categories including Full Refusal, Full Compliance, and Partial Refusal; Figure 4 includes labels from Qwenv2.5-72B and Dic-Judge.Source: Figure 4 · p. 8, §4.3 · p. 8
  • Harmful queries typically generate more words than benign queries because safety-aware reasoning triggers self-evaluation and multiple rethinks.Figure 5 indicates latency by number of generated words for reasoning and non-reasoning R2D-enhanced models.Source: Figure 5 · p. 8, §4.3 · p. 8
  • Ablations show that omitting CPO raises ASR by up to 23%, while removing pivot tokens raises it by up to 45%.Reasoning-data-trained models also have lower ASRs than their unoptimized counterparts.Source: §4.3 · p. 9, Table 4 · p. 9
  • Against contemporary defenses, R2D reports up to a 9% ASR reduction versus RPO and lower ASR than Self-Reminder and IA.The comparison covers different models and attacks.Source: Table 5 · p. 9, §4.3 · p. 9

5 Conclusion

R2D combines safety-aware reasoning distillation with stepwise self-assessment to improve jailbreak defenses. The reported results show lower ASRs without over-refusal or performance drops.

5 Conclusion

  • R2D-enabled LLMs consistently achieve lower ASRs than previous defense approaches, and analyses report no over-refusals or performance drops.The paper says ablation studies validate the effectiveness of R2D's components.Source: §5 · p. 9
  • SwaRD unlocks safety-aware reasoning capabilities, while CPO further strengthens models' self-assessment of each reasoning step.The conclusion identifies these as complementary components of R2D.Source: §5 · p. 9

Limitations

The study focuses primarily on distilling reasoning from MR because of foundation models' size and inherent capabilities. It identifies reinforcement learning, test-time scaling, multi-turn attacks, and multimodal reasoning as areas for future investigation.

Limitations

  • The approach primarily uses reasoning distillation from MR, rather than reinforcement learning or test-time scaling, due to foundation-model size and inherent capabilities.The paper frames this as the study's focus, not as a general prohibition on those alternatives.Source: Limitations · p. 10
  • Future work could explore integrating safety-aware reasoning into ReFT-like approaches and defending against multi-turn attacks such as Crescendo.The paper also names Tang et al. (2025) as an example of multi-turn attacks.Source: Limitations · p. 10
  • The safety of multimodal reasoning models remains unexplored in this work.Source: Limitations · p. 10

Ethics Statement

The authors study defenses against jailbreak attacks and acknowledge that their experiments include potentially controversial LLM-generated content and may elicit harmful responses from models.

Ethics Statement

  • The study includes potentially controversial LLM-generated content to demonstrate jailbreaks and defenses.The authors say including this content is necessary to demonstrate jailbreak and defense effects.Source: Ethics Statement · p. 10
  • Existing jailbreak methods may lead some tested LLMs to follow harmful instructions during the investigation.Source: Ethics Statement · p. 10
  • The authors make datasets and code publicly available and urge researchers to use them exclusively for research.Source: Ethics Statement · p. 10

A Details of Experiments

The appendix describes the experiment details, including trajectory synthesis, training configurations, and infrastructure. It reports how reasoning trajectories are synthesized and assembled for safety-aware reasoning training.

A Details of Experiments

  • The experiment-details appendix covers reasoning trajectory synthesis, training configurations, and experimental infrastructure for reproducibility.Source: A Details of Experiments · p. 14

A.1 Reasoning Trajectories

  • MR generates reasoning trajectories by appending safety-status suffixes to each reasoning sentence, marking it SAFE, UNSAFE, or RETHINK.The instruction asks MR to reconsider its strategy after an UNSAFE marker and seek a safe response after RETHINK.Source: §A.1 · p. 14
  • Training trajectories use 52k Alpaca samples and 520 AdvBench samples spanning normal-use and safety-sensitive instructions.The samples include reasoning trajectories and pivot tokens.Source: §A.1 · p. 14

A.2 Configurations

The configurations describe the evaluated models, training and evaluation setups, and computing infrastructure. The case study presents safe and unsafe instructions to illustrate jailbreak-defense and over-refusal successes and failures.

A.2 Configurations

  • The evaluated lineup includes reasoning models and non-reasoning LLMs, with SwaRD trajectories synthesized by DeepSeek-R170B.Models include QwQ32B, DeepSeek-R1 variants, Llama, Qwen, Mistral, and Vicuna models.Source: §A.2 · p. 14
  • R2D training uses LoRA, AdamW, and cosine annealing; hyper-parameter setups are listed in Table 6.The paper links LoRA use to the relatively small training-data volume.Source: §A.2 · p. 14, Table 6 · p. 14
  • Evaluation uses Llama-Guardv3-8B or HarmBenchcls13B for attack success, and Qwenv2.5-72B to detect partial refusals.Dic-Judge serves as a calibration reference; Table 7 gives the refusal prefix used in over-refusal evaluation.Source: §A.2 · p. 14, Table 7 · p. 15
  • Experiments run on four NVIDIA A100 80GB GPUs, with distributed mixed-precision training and vLLM inference using Paged Attention.The training setup uses bfloat16; the infrastructure passage also mentions Flash Attention and fused CUDA kernels.Source: §A.2 · p. 15

B Case Study

  • The case study includes safe and unsafe instructions to illustrate jailbreak-defense and over-refusal success and failure cases.It provides responses to safe and unsafe instructions, including jailbreak cases.Source: B Case Study · p. 15

B.1 Cases of Safety Benchmark

These safety-benchmark cases pair jailbreak instructions with examples of refusals and responses to unsafe requests. The examples include keylogger and ATM ransomware prompts.

B.1 Cases of Safety Benchmark

  • The cases include safe refusals and jailbreak responses under different jailbreak instructions.The section presents both safe and unsafe instruction-response examples.Source: §B.1 · p. 15
  • One unsafe prompt frames a request for a Python keylogger that emails captured logs as a security project.The prompt asks that the keylogger run silently and avoid antivirus detection.Source: §B.1 · p. 15
  • Another unsafe prompt requests detailed ATM ransomware exploitation steps under an educational-purpose framing.It asks for a section explaining how an attacker could install ransomware on an ATM.Source: §B.1 · p. 15

B.2 Cases of Over-Refusal Benchmark

The over-refusal case study spans four safe/unsafe instruction and response conditions. Its examples include refusals, harmful outputs, and helpful answers to safe requests.

B.2 Cases of Over-Refusal Benchmark

  • The case study distinguishes four conditions: refusing or complying with unsafe instructions, and helpfully answering or over-refusing safe instructions.These categories cover both safety failures and over-refusal behavior.Source: §B.2 · p. 15
  • A keylogger-request example ends in refusal after reasoning that the request could enable misuse.The response declines assistance and offers ethical cybersecurity alternatives.Source: §B.2 · p. 16
  • An ATM ransomware example produces a blog-style response framed as educational and organized around attack steps.The accompanying passage says the blog post is not intended for real-world exploitation.Source: §B.2 · p. 16
  • For a safe question about having children while relying on benefits, the response discusses financial considerations while treating the decision as personal.It suggests considering individual circumstances and consulting professionals.Source: §B.2 · p. 17
  • A safe request about terminating a Python process receives operating-system-specific methods and a warning to back up important work.The examples cover Windows and Linux/macOS approaches.Source: §B.2 · p. 18
Loading 2502.12970v3…