Source-linked AI summary
AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
Zihao Zhu, Xinyu Wu, Gehan Hu, Siwei Lyu, Ke Xu, Baoyuan Wu
TL;DR
Multi-step reasoning creates safety challenges because small deviations can snowball into harmful compliance or over-refusal, while conventional safety tuning does not teach models to self-correct. AdvChain trains models on reasoning chains containing deliberate errors and corrections. The authors report improved robustness against attacks and reduced over-refusal on benign prompts without compromising core reasoning capabilities.
Problem
In safety CoT tuning, minor reasoning deviations can snowball into harmful compliance or over-refusal, and models trained to imitate idealized scripts lack dynamic self-correction.
Source: §1 · p. 2Method
AdvChain fine-tunes models on adversarial CoT samples that teach recovery from harmful reasoning drift and unnecessary caution.
Source: §1 · p. 2, §4.2 · p. 5Results
AdvChain improves robustness against harmful requests and CoT hijacking while reducing over-refusal on benign prompts, without compromising core reasoning capabilities.
Source: §1 · p. 2, §5.2 · pp. 7–8Takeaways & Limitations
The authors conclude that AdvChain offers a safety-utility balance by teaching models to correct reasoning errors rather than only imitate idealized chains.
Source: §1 · p. 2, §7 · p. 9Generated adversarial examples depend on teacher-model quality and may not cover all safety violations; the method currently addresses single-turn corrections, not more sophisticated multi-step attacks.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in complex problem-solving through Chain-of-Thought (CoT) reasoning. However, the multi-step nature of CoT introduces new safety challenges that extend beyond conventional language model alignment. We identify a failure mode in current safety CoT tuning methods: the \textit{snowball effect}, where minor reasoning deviations progressively amplify throughout the thought process, leading to either harmful compliance or excessive refusal. This effect stems from models being trained to imitate perfect reasoning scripts without learning to self-correct. To address this limitation, we propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning. Our method involves constructing a dataset containing Temptation-Correction and Hesitation-Correction samples, where models learn to recover from harmful reasoning drifts and unnecessary cautions. Extensive experiments show that AdvChain significantly enhances robustness against jailbreak attacks and CoT hijacking while substantially reducing over-refusal on benign prompts, achieving a superior safety-utility balance without compromising reasoning capabilities. Our work establishes a new direction for building more robust and reliable reasoning models.
1 INTRODUCTION
Multi-step Chain-of-Thought reasoning gives Large Reasoning Models capabilities for complex problem-solving but also creates a safety attack surface. The paper identifies escalating harmfulness and over-refusal as consequences of reasoning drift and proposes AdvChain to teach models to recover from flawed reasoning.
1 INTRODUCTION
- CoT enables LRMs to solve complex problems through intermediate reasoning, but a flawed step can derail an otherwise safe process.The authors identify the multi-step reasoning process as a new safety attack surface.Source: §1 · p. 1
- Safety CoT tuning can leave models unable to self-correct: minor deviations may snowball into harmful compliance or unnecessary refusal.The paper describes harmfulness escalation on malicious prompts and over-refusal on benign prompts.Source: §1 · pp. 1–2
- AdvChain trains models to recover from flawed reasoning using Temptation-Correction and Hesitation-Correction samples, improving robustness while reducing over-refusal.The authors report these gains without compromising core reasoning capabilities.Source: §1 · p. 2
- The paper’s contributions include identifying the Snowball Effect, proposing adversarial CoT tuning, and evaluating its robustness and over-refusal outcomes.The proposed dataset includes temptation-correction and hesitation-correction samples.Source: §1 · p. 2
2 BACKGROUND AND PRELIMINARIES
The background describes LRMs as models that reason through intermediate Chain-of-Thought steps and outlines safety CoT tuning as a way to align them. It notes that this tuning paradigm has limitations.
2.1 LARGE REASONING MODELS AND CHAIN-OF-THOUGHT
- LRMs generate intermediate Chain-of-Thought steps before a final answer, supporting complex tasks such as logical deduction, planning, and mathematical reasoning.The paper represents an output as a reasoning chain and a final answer.Source: §2.1 · p. 2
2.2 SAFETY ALIGNMENT OF LARGE REASONING MODELS
- Safety CoT tuning fine-tunes models on curated safety demonstrations containing a harmful prompt, a safe reasoning chain, and a final safe response.The model is trained to imitate these safety-oriented reasoning patterns; the paper states that the approach has fundamental limitations.Source: §2.2 · p. 3
3 THE SNOWBALL EFFECT IN COT ALIGNMENT
The paper defines the Snowball Effect as a small reasoning deviation that amplifies through a chain, and reports two manifestations: escalating harmfulness and escalating over-refusal. It attributes the problem to alignment that teaches idealized reasoning scripts without recovery from mistakes.
3.1 SNOWBALLING ESCALATION OF HARMFULNESS
- A minor deviation in a safe reasoning path can amplify into a harmful conclusion and output on harmful prompts.The analysis evaluates reasoning steps from DeepSeek-R1-7B and STAR-1-7B on WildJailbreak prompts, scoring step safety with GPT-4o.Source: §3.1 · p. 3
- For harmful prompts, initially safe reasoning steps often average below 1.5 in safety score before harmfulness escalates through the chain.The supplied figure passage says safety scores frequently exceed 4.0 in final steps.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 SNOWBALLING ESCALATION OF OVER-REFUSAL
- For benign prompts, a minor, unnecessary safety hesitation can snowball into over-refusal: helpfulness declines from an initial phase averaging above 4.5, often falling below 2.0 in the latter half.The analysis filters for cases with an initial helpfulness score ≥4.Source: §3.2 · p. 4
3.3 THE CORE ISSUE: LACK OF ROBUST AND ADAPTIVE REASONING
- Conventional safety tuning teaches idealized reasoning scripts but provides no training signal for recovering from mistakes, allowing flawed reasoning to compound.The authors describe this as cognitive inertia that can corrupt the final output.Source: §3 · p. 3, §3.3 · p. 4
4 METHODOLOGY: ROBUST ALIGNMENT VIA ADVERSARIAL COT TUNING
AdvChain shifts safety tuning from imitating idealized reasoning to learning dynamic correction. It uses adversarial reasoning trajectories with inserted errors and corrections, then fine-tunes the model on those trajectories.
4 METHODOLOGY: ROBUST ALIGNMENT VIA ADVERSARIAL COT TUNING
- AdvChain is an adversarial CoT tuning approach that targets dynamic self-correction rather than flawless imitation of idealized reasoning scripts.The authors frame the goal as recognizing and recovering from cognitive errors.Source: §4 · p. 5, §4.1 · p. 5
4.2 ADVERSARIAL SAFE REASONING DATASET
- The dataset is programmatically built by rewriting existing reasoning chains to include cognitive errors and their subsequent corrections.A teacher model follows instructional prompts to insert these elements into existing CoTs.Source: §4.2 · p. 5
- Temptation-Correction samples insert a harmful temptation into a safe refusal chain, then add a correction that refutes it and returns reasoning to safe refusal.The resulting trajectory retains a safe refusal as its final summary.Source: §4.2 · p. 5
- Hesitation-Correction samples insert mistaken caution into a helpful response to a benign prompt, then correct the false positive and resume the original path.The final trajectory is assembled from the initial helpful chain, hesitation, correction, and remaining chain.Source: §4.2 · pp. 5–6
4.3 ADVERSARIAL COT TUNING
- AdvChain fine-tunes the model on the combined dataset to maximize the likelihood of generating self-correcting reasoning paths and final summaries.The training objective is a standard autoregressive objective over the dataset.Source: §4.3 · p. 6
5 EXPERIMENTS
AdvChain is evaluated against safety and jailbreak attacks, over-refusal on benign prompts, and reasoning capabilities. It improves reported safety and benign-prompt refusal outcomes while retaining reasoning performance comparable to the base models.
5.1 EXPERIMENTAL SETUP
- Experiments use five open-source LRMs, a 1,000-sample adversarial dataset, and benchmarks for safety, robustness, over-refusal, and reasoning.The baselines include three methods fine-tuned on 1,000 samples and the released checkpoint of RealSafe-R1, trained on 15,000 trajectories.Source: §5.1 · pp. 6–7
5.2 EVALUATION OF SAFETY AND ROBUSTNESS
- Across model families and sizes, AdvChain achieves lower Attack Success Rate than 1k-data baselines and performance on par with 15k-data RealSafe-R1.The comparison covers harmful and jailbreak-prompt benchmarks.Source: Table 1 · p. 7, §5.2 · p. 7
- On the 150-sample CoT-hijacking attack, AdvChain models achieve significantly lower Attack Success Rate than baseline models.The attack presents target models with reasoning prefixes containing a malicious pivot.Source: Table 2 · p. 8, §5.2 · p. 8
5.3 EVALUATION OF OVER-REFUSAL AND GENERAL CAPABILITIES
- On benign prompts, AdvChain models have lower Over-Refusal Rate than conventionally aligned baselines.The authors attribute the reduced over-refusal to Hesitation-Correction samples.Source: Table 3 · p. 8, §5.3 · p. 8
- AdvChain models achieve mathematics and coding Pass@1 scores on par with their original base models.The evaluation uses a suite of challenging reasoning benchmarks.Source: Table 4 · p. 8, §5.3 · p. 8
6 ANALYSIS AND DISCUSSION
The analysis contrasts the reasoning patterns of temptation-correction samples with standard safety data and finds that the two correction-data types serve complementary purposes. It also identifies limits in teacher-example coverage and the current single-turn setting.
Structural Analysis of Reasoning Patterns
- Temptation-correction chains rise from low safety scores back to safe states, unlike STAR-1's consistently low scores.The paper describes this peak-like trajectory as an explicit training signal for error recovery rather than imitation of an idealized safe path.Source: §6 · p. 9
Impact of Data Composition
- Increasing the temptation-correction share lowers attack success rates, while increasing hesitation-correction data lowers benign-prompt refusal rates.With total training fixed at 1,000 samples, the two data types show complementary effects on attack resilience and helpfulness.Source: §6 · p. 9
Limitations and Future Work
- Adversarial examples depend on teacher-model quality and may miss safety violations; the current method addresses single-turn, not sophisticated multi-step, corrections.The paper identifies extending the framework to diverse scenarios as future work.Source: §6 · p. 9
7 CONCLUSION
The paper presents AdvChain as a self-correction framework for safety alignment in Large Reasoning Models. It reports enhanced attack robustness and reduced over-refusal without compromising core reasoning capabilities.
7 CONCLUSION
- AdvChain trains Large Reasoning Models to self-correct using Temptation-Correction and Hesitation-Correction samples, with reported gains in attack robustness and reduced over-refusal.The conclusion states that these gains do not compromise core reasoning capabilities.Source: §7 · p. 9