PaperlayerView exact revision on arXiv

Source-linked AI summary

To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models

Zihao Zhu, Hongbao Zhang, Ruotong Wang, Ke Xu, Siwei Lyu, Baoyuan Wu

arXiv:2502.12202v2cs.CLcs.AIcs.LG

TL;DR

LRMs' thinking process can be bypassed by manipulating delimiter tokens, raising concerns about their reliability. The paper develops training-based and training-free Breaking of Thought attacks, a fine-tuning-based recovery defense, and plug-and-play Monitoring of Thought using an external monitor. BoT variants achieve high attack success across diverse LRMs and tasks, while MoT reduces inference cost, preserves output quality, and increases safety under harmful inputs.

  • Problem

    LRMs' thinking process can be bypassed by manipulating delimiter tokens, raising concerns about their reliability.

    Source: §1 · p. 2

  • Method

    The paper develops training-based and training-free Breaking of Thought attacks, a fine-tuning-based recovery defense, and plug-and-play Monitoring of Thought using an external monitor.

    Source: §1 · p. 2

  • Results

    BoT variants achieve high attack success across diverse LRMs and tasks, while MoT reduces inference cost, preserves output quality, and increases safety under harmful inputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The vulnerability has malicious and beneficial uses, and external monitoring offers a way to halt redundant or risky reasoning without retraining or architectural changes.

    Source: §1 · p. 2, §6 · p. 9

    The defense reduces ASR by approximately 30% for template modification and adversarial suffix attacks, but only 2% for training-based BoT attacks.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Large Reasoning Models (LRMs) are designed to solve complex tasks by generating explicit reasoning traces before producing final answers. However, we reveal a critical vulnerability in LRMs -- termed Unthinking Vulnerability -- wherein the thinking process can be bypassed by manipulating special delimiter tokens. It is empirically demonstrated to be widespread across mainstream LRMs, posing both a significant risk and potential utility, depending on how it is exploited. In this paper, we systematically investigate this vulnerability from both malicious and beneficial perspectives. On the malicious side, we introduce Breaking of Thought (BoT), a novel attack that enables adversaries to bypass the thinking process of LRMs, thereby compromising their reliability and availability. We present two variants of BoT: a training-based version that injects backdoor during the fine-tuning stage, and a training-free version based on adversarial attack during the inference stage. As a potential defense, we propose thinking recovery alignment to partially mitigate the vulnerability. On the beneficial side, we introduce Monitoring of Thought (MoT), a plug-and-play framework that allows model owners to enhance efficiency and safety. It is implemented by leveraging the same vulnerability to dynamically terminate redundant or risky reasoning through external monitoring. Extensive experiments show that BoT poses a significant threat to reasoning reliability, while MoT provides a practical solution for preventing overthinking and jailbreaking. Our findings expose an inherent flaw in current LRM architectures and underscore the need for more robust reasoning systems in the future.

1 Introduction

Large Reasoning Models generate explicit reasoning traces before final answers, using delimiter tokens to mark the thinking phase. The paper shows that manipulating these delimiters can bypass thinking, then examines the vulnerability as both an attack surface and a tool for efficiency and safety.

1 Introduction

  • LRMs generate detailed reasoning traces before answering, with special delimiter tokens marking the boundaries of the thinking process.Examples include <think> and </think> in DeepSeek-R1 and QwQ, and <Thought> and </Thought> in Marco-o1.Source: §1 · p. 1
  • Appending thought delimiters after a query can make an LRM skip thinking and produce a less thoughtful answer, and the paper reports this vulnerability across mainstream open-source models.The paper describes the issue as a double-edged sword, with risks to reliability and possible uses for more efficient, safer reasoning.Source: §1 · p. 2
  • Breaking of Thought (BoT) offers training-based backdoor and training-free adversarial-suffix attacks that suppress thinking, while thinking recovery alignment can partially defend against them.The authors report high attack success rates across diverse LRMs and reasoning tasks.Source: §1 · p. 2
  • Monitoring of Thought (MoT) uses an external lightweight monitor to stop reasoning judged redundant or risky, without retraining or architectural changes.The monitor can use a small model or heuristic rules and can halt thinking when a query needs no deep reasoning or the existing trace is sufficient.Source: §1 · p. 2
  • Experiments report that MoT reduces inference cost while preserving output quality and increases safety under harmful inputs.The paper also reports extensive experiments validating vulnerability prevalence and the effectiveness of BoT and MoT across reasoning tasks and LRMs.Source: §1 · pp. 2–3

2 Background and Related Work

This section distinguishes conventional Chain-of-Thought prompting from LRMs designed for extended thinking, and describes delimiter tokens, overthinking, and related approaches. It positions the paper's focus as a systematic study of delimiter vulnerability for both attacks and beneficial uses.

Background

  • Chain-of-Thought prompting encourages intermediate reasoning, whereas LRMs are designed to generate long thinking traces before final answers.The paper represents an LRM output as a thinking process followed by a final answer.Source: §2 · p. 3

Thought Delimiters in Large Reasoning Models

  • Most LRMs use explicit start- and end-of-thinking delimiters to structure reasoning and distinguish it from the final answer.The section notes uses including hiding or displaying reasoning and supporting downstream processing and reward boundaries.Source: §2 · p. 3

Overthinking of Large Reasoning Models

  • Overthinking—long, redundant traces on simple tasks—increases computational overhead and may compromise safety alignment by increasing jailbreak vulnerability.Prior approaches described include fine-tuning on custom datasets with specific reasoning patterns.Source: §2 · p. 3
  • The paper distinguishes its work from prior efficiency efforts by systematically examining both malicious exploitation and beneficial use of delimiter manipulation.It also presents its attack paradigm as targeting the robustness of LRMs' thinking process.Source: §2 · p. 3

3 Unthinking Vulnerability in Large Reasoning Models

The section tests whether forced start- and end-of-thought delimiters make LRMs treat reasoning as complete. Template modification produces high attack success and substantially lower token use and pass@1 performance, motivating the paper's definition of unthinking vulnerability.

Experimental Settings

  • The test appends thought delimiters to the tokenizer's chat template, forcing the model to encounter an empty reasoning block before generating output.The experiment compares token length and pass@1 accuracy before and after modification on AIME 2024.Source: §3 · p. 3

Results and Observations

  • Over 90% ASR across tested LRMs and about 80% relative reduction in both token length and performance follow simple template modification.The authors say the delimiters alone inhibit reasoning regardless of input complexity.Source: §3 · p. 4, Figure 2 · p. 4

Unthinking Vulnerability

  • The authors define unthinking vulnerability as forced injection of start- and end-of-thinking delimiters causing an LRM to omit its thinking trace and output its answer directly.The section's hypothesis is that the model treats the reasoning stage as already complete.Source: §3 · pp. 3–4

Implications of Unthinking Vulnerability

  • The paper frames the vulnerability as enabling both adversarial suppression of reasoning and owner-directed improvements to efficiency and safety alignment.The section organizes these implications into malicious exploitation and beneficial application.Source: §3 · p. 4

4 Malicious Exploitation of Unthinking Vulnerability: Breaking of Thought

Breaking of Thought (BoT) exploits the unthinking vulnerability through fine-tuning backdoors or inference-time adversarial suffixes. Experiments show that these attacks can bypass reasoning and reduce reliability, while the proposed defense only partly mitigates them.

4 Malicious Exploitation of Unthinking Vulnerability: Breaking of Thought

  • The attack variants force LRMs to skip thinking, reducing their reliability on reasoning-critical tasks.The section frames training-based and training-free BoT as ways to control whether the model engages in its expected thinking process.Source: §4 · pp. 4–5

4.1 Training-based BoT Attack

  • Training-based BoT assumes access to the target’s training data and implants a trigger-activated backdoor through fine-tuning.Poisoned SFT examples append a trigger and omit thinking; poisoned DPO pairs reverse the clean preference for complete reasoning.Source: §4.1 · p. 5

4.2 Training-free BoT Attack

  • Training-free BoT optimizes inference-time adversarial suffixes without modifying model parameters, using white-box access or transfer from surrogate models.GCG greedily selects token substitutions that reduce loss; universal suffixes are expanded from one query to more inputs.Source: §4.2 · pp. 5–6

4.3 Experiments

  • ASR exceeds 99% in most training-based cases, C-Acc remains above 95%, and reasoning-task performance decreases by 20–90%.Generated tokens are reduced by over 8% (RTC).Source: Table 1 · p. 6, §4.3 · p. 7
  • White-box training-free attacks reach 100% ASR on DeepSeek-R1-1.5B and Light-R1-7B-DS; universal suffixes exceed 80% ASR, and surrogate suffixes transfer to other models.DeepSeek-R1-1.5B requires only 8 optimization steps, while DeepSeek-R1-32B requires over 200.Source: Table 2 · p. 6, Table 3 · p. 7, Table 4 · p. 7

4.4 Discussion

  • Recovery alignment reduces ASR by approximately 30% for template-modification and adversarial-suffix attacks, but by only 2% for training-based BoT.The authors describe the defense as promising but not completely effective across attack variants.Source: Figure 4 · p. 7, §4.4 · p. 8

5 Beneficial Application of the Vulnerability: Monitoring of Thought

Monitoring of Thought (MoT) uses an external monitor to stop redundant or risky reasoning, addressing both efficiency and safety. Experiments report reduced token use with stable or improved performance, stronger safety alignment, and an average computational cost per sample about 50% lower despite monitor overhead.

5 Beneficial Application of the Vulnerability: Monitoring of Thought

  • MoT targets overthinking and can leverage the unthinking vulnerability for constructive use in reasoning efficiency and safety alignment.Overthinking is described as redundant reasoning that increases computational overhead and may compromise safety alignment.Source: §5 · p. 8

5.2 Monitoring of Thought for Safety Alignment

  • MoT uses an external lightweight monitor, such as a small model or heuristic rules, to assess the input and ongoing thoughts and stop redundant or risky reasoning.For safety alignment, the monitor can halt generation when the model begins considering potentially unsafe or inappropriate content.Source: §5.2 · p. 8

5.3 Experiments

  • MoT significantly reduces Relative Tokens Change (RTC) versus s1-32B while Relative Performance Change (RPC) remains stable or improves.The authors interpret this as reduced redundant token generation without sacrificing accuracy.Source: Table 5 · p. 8, §5.3 · p. 9
  • On StrongReject, MoT raises DeepSeek-R1-1.5B’s refuse rate from 31.31% to 84.66% and reduces ASR to 1.92%.The paper also reports striking safety improvements across HarmBench and WildJailbreak, without model retraining or architectural modifications.Source: §5.3 · p. 9, Table 6 · p. 9

5.4 Discussion

  • MoT allows customized monitoring rules and integration with LRMs using thought delimiters without fine-tuning, but requires a separate inference-time monitor.Although this adds overhead, experiments report about 50% lower average computational cost per sample; internalizing the rules is left for future work.Source: §5.4 · p. 9

6 Conclusion

The conclusion presents delimiter manipulation as a widespread vulnerability with malicious and beneficial uses, and describes BoT attacks, MoT, and potential defenses. It notes that BoT requires prior knowledge of a target model’s delimiters, limiting its current applicability to closed-source models such as o1.

6 Conclusion

  • The paper concludes that delimiter manipulation can bypass LRM thinking, with the vulnerability exploitable through BoT and harnessable by MoT for efficiency and safety.The authors describe this as a dual-edged vulnerability and propose potential defense strategies.Source: §6 · p. 9
  • BoT requires prior knowledge of the target LRM’s delimiters and therefore currently cannot be applied to closed-source models like o1.The conclusion notes that model owners can still leverage MoT internally to enhance efficiency and safety.Source: §6 · p. 9
  • The appendix provides implementation details, algorithms, extended results, ablations, complexity analysis, adaptive defense, compatibility descriptions, and examples.Source: §6 · p. 9

in Large Reasoning Models

The appendix describes the computing configurations and implementation material, then details a template modification used to trigger the unthinking vulnerability. In that modification, an empty reasoning block makes the model skip internal thinking and move to its final output.

A Computing Resources

B More Implementation Details

B.1 Details of Template Modification

  • Adding an empty reasoning block between <think> and </think> makes the model skip internal thinking and jump to its final output.The experiments modify the chat template by adding thought delimiters after the assistant token.Source: §B.1 · pp. 16–17

B.2 Details of Training-based BoT

The appendix specifies training and inference procedures for BoT, prompts and evaluation details for MoT, and the data format used for thinking recovery alignment. It also lists algorithms for single-query, universal-suffix, and transfer attacks.

B.2 Details of Training-based BoT

  • Training-based BoT poisons 400 samples from SFT corpora by injecting triggers and manipulating outputs, then uses LoRA for SFT and DPO updates.The stated settings include rank 8, alpha 32, batch size 16, learning rate 0.0001, and three epochs.Source: §B.2 · p. 17

B.3 Details of Training-based BoT

  • Training-free BoT stops suffix optimization when generated output begins with the thought delimiters, then tests the suffix on full responses and, for transfer, tries candidates in ascending loss order.The stopping check evaluates the first five generated tokens every five steps after the loss falls below 0.5.Source: §B.3 · p. 17

B.4 Details of MoT for Efficiency

  • For efficiency monitoring, Audit-on-Thought invokes an audit model periodically and removes an incomplete trailing sentence to avoid semantic truncation.The prompts judge both whether the question needs deep thinking and whether the current thought process is sufficient or redundant.Source: §B.4 · pp. 17–18

B.5 Details of MoT for Safety Alignment

  • Safety-alignment monitoring prompts assess the user input and current thoughts for unsafe actions, jailbreak attempts, and other risky behaviors, using yes-or-no judgments.Source: §B.5 · pp. 18–19
  • Safety evaluation defines successful refusal through listed refusal phrases and scores harmful responses on a five-point scale from refusal to direct fulfillment.Source: §B.5 · pp. 19–21

B.6 Details of Thinking Recovery Alignment

  • Thinking recovery samples pair an initially empty thought process and randomly truncated final answer with the original reasoning process and complete answer; these are mixed with normal samples during fine-tuning.The intended effect is to encourage the model to reestablish a valid reasoning trajectory when reasoning is initially omitted.Source: §B.6 · p. 21

C Algorithms

  • The training-free BoT algorithms cover single suffixes for single queries, universal suffixes for multiple queries under white-box access, and transfer attacks in black-box settings.Source: C Algorithms · p. 21

C.1 Training-free BoT for Single Suffix

The training-free BoT appendix describes GCG suffix optimization for single-query, multi-query, and transfer attacks, alongside baseline evaluations. The two non-fine-tuning baselines fail to reliably suppress thinking, motivating more targeted attacks.

C.1 Training-free BoT for Single Suffix

  • For a single query, GCG iteratively mutates one suffix coordinate using gradient-selected tokens, evaluates candidate losses, and stops early when unthinking is triggered.The algorithm checks the first five generated tokens for the <SOT>, <EOT> pattern.Source: §C.1 · pp. 21–22

C.2 Training-free BoT for Universal Suffix

C.3 Training-free BoT for Transfer Attack

  • A separate GCG algorithm is provided for transfer attacks.Its procedure uses gradients and candidate losses across surrogate models.Source: §C.3 · p. 22, §D.2 · p. 24

D.1 Capability Evaluation of Original Models

  • Original-model capability baselines use evalscope, DeepSeek-recommended generation settings, and five responses per input to calculate pass@1 accuracy and average token length.The measurements serve as references for later RTC and RPC calculations.Source: §D.1 · p. 22, Table 7 · p. 22

D.2 Comparison with Baselines for BoT Attacks

  • Both non-fine-tuning baselines fail to reliably suppress thinking and yield very low attack success rates.The authors argue that more targeted strategies, such as backdoor or adversarial attacks, are needed to break the reasoning mechanism.Source: §D.2 · pp. 24–25, Table 8 · p. 24

D.3 Results of Training-based BoT with Non-Semantic Triggers

The appendix evaluates BoT variants, adaptive defenses, and attack costs. It reports strong DPO and non-semantic-trigger results, while the tested defenses leave residual vulnerability or impose substantial inference cost.

D.3 Results of Training-based BoT with Non-Semantic Triggers

  • Non-semantic triggers achieve higher attack success rates than semantic triggers at low poisoning ratios on DeepSeek-R1 1.5B and 7B.The authors suggest meaningless triggers make it easier to learn a direct trigger-to-target-output association.Source: §D.3 · p. 25, Table 9 · p. 25

D.4 Results of Training-based BoT with DPO

  • DPO fine-tuning brings training-based BoT attack performance to almost 100% on both MATH-500 and AIME 2024.The authors suggest DPO more effectively reinforces preference for outputs that bypass thinking when the trigger is present.Source: §D.4 · p. 25, Table 10 · p. 25

D.5.1 Adative Defense Against Train-based BoT

  • For training-based BoT, ONION detects non-semantic but not semantic triggers, BAIT fails across 24 infected models, and 50 clean SFT examples barely change ASR.ONION's detection rate on semantically poisoned inputs is 7%.Source: §D.5.1 · p. 26

D.5.2 Adative Defense Against Train-free BoT

  • SMOOTHLLM moderately reduces training-free BoT ASR without eliminating it, while requiring 10 full forward passes per input.The evaluation uses perturbation rates of 2% and 5%.Source: §D.5.2 · p. 26, Figure 9 · p. 27

E.1 Complexity Analysis of Training-based BoT

  • Training-based BoT uses LoRA fine-tuning for three epochs and reportedly reaches 100% ASR with as few as 400 samples.The paper characterizes the approach as efficient and practical from an attacker's perspective.Source: §E.1 · p. 27, Table 11 · p. 27

E.2.1 Single-Suffix White-box Attack (Algorithm 1)

  • Single-suffix optimization on DeepSeek-R1-1.5B typically takes about 30 seconds and converges within 10 steps.The reported setup uses suffix length 10, top-k 256, search width 512, and at most 512 iterations.Source: §E.2.1 · p. 28

E.2.2 Universal-Suffix White-box Attack (Algorithm 2)

  • Universal-suffix optimization typically takes about 7 minutes on DeepSeek-R1-1.5B with 10 queries and early stopping.The implementation gradually increases the number of active queries.Source: §E.2.2 · p. 28

E.2.3 Transfer Attack (Black-box, Algorithm 3)

  • Transfer-attack optimization for DeepSeek-R1-1.5B and 7B typically takes about 1 minute within 10 steps.The approach optimizes a suffix on surrogate models for transfer to an unseen target.Source: §E.2.3 · p. 28

E.3 Complexity Analysis of Monitoring of Thought

The appendix characterizes MoT's monitoring cost and tests choices affecting its efficiency, accuracy, and compatibility with forced thinking. It reports cost savings and describes heuristic monitoring and adaptations to DeepSeek's updated template.

E.3 Complexity Analysis of Monitoring of Thought

  • MoT's inference cost adds monitor calls to original LRM decoding, and the appendix estimates real-world overhead using AIME 2024 samples.The authors state that the smaller monitor's added complexity is acceptable relative to its advantages.Source: §E.3 · pp. 28–29
  • MoT yields 40%–60% cost savings per sample across DeepSeek-R1 models on AIME 2024 despite added monitoring computation.The cost comparison uses GPT-4o-mini as the external monitor, invoked every 200 thinking tokens.Source: §E.3 · p. 29, Table 12 · p. 29

E.4 Impact of Poisoning Ratio of Training-based BoT

  • Raising the training-based BoT poisoning ratio increases ASR but lowers clean accuracy; below 0.2, ASR falls below 80%, so the paper uses 0.4.The authors report both ASR and C-Acc on DeepSeek-R1-1.5B using MATH-500.Source: §E.4 · p. 29, Figure 10 · p. 29

E.5 Impact of Initialized Suffix of Training-free BoT

  • For single-suffix white-box BoT, initial suffix lengths of 9 or more reach 100% ASR in fewer than 10 steps, while length 3 reaches 80.2% and needs nearly 100 steps on average.The experiments use repeated exclamation marks on DeepSeek-R1-1.5B with MATH-500.Source: §E.5 · p. 30, Figure 11 · p. 30

E.6 Impact of Monitoring Frequency of MoT

  • On AIME 2024, monitoring every 200 tokens gives the highest RPC while retaining considerable RTC improvement; monitoring every 400–500 tokens improves efficiency but reduces reasoning quality.More frequent monitoring, such as every 50–200 tokens, achieves higher RPC.Source: §E.6 · p. 30, Figure 12 · p. 30

E.7 Model-free MoT with Heuristic Rules

  • The model-free MoT variant counts reflection tokens associated with redundant or circular reasoning and injects <EOT> once a threshold is exceeded.It replaces the external monitor while leaving the rest of the framework unchanged.Source: §E.7 · p. 31

F.1 DeepSeek’s Forced Thinking Mechanism

  • DeepSeek's updated chat template appends <think> to force reasoning by default, but the paper says its methods adapt with minimal modifications.Source: §F.1 · p. 31

F.2.1 Unthinking Vulnerability

  • Under the updated template, all methods remain effective; the vulnerability can still be triggered by appending only </think> to close the reasoning block.This adaptation is reported to achieve 100% attack success rate.Source: §F.2 · p. 32, §F.2.1 · p. 32

F.2.2 Training-based BoT

The appendix explains how BoT and MoT are adjusted for DeepSeek's forced-thinking template, then illustrates bypassed and ordinary outputs. Its examples include incorrect answers from training-based BoT.

F.2.2 Training-based BoT

  • Training-based BoT adapts to forced reasoning by removing the opening <think> token from poisoned responses so an early </think> terminates reasoning.Source: §F.2.2 · p. 32, Figure 14 · p. 32

F.2.3 Training-free BoT

  • Training-free BoT targets </think> rather than <think></think> to bypass the forced reasoning block and proceed to the final answer.Source: §F.2.3 · p. 32

F.2.4 Monitoring of Thought (MoT)

  • With forced reasoning, MoT no longer evaluates question difficulty and instead monitors ongoing reasoning to decide when to terminate it via </think>.The paper reports that the approaches preserve their effectiveness under the updated template.Source: §F.2.4 · p. 32, Table 13 · p. 33

G.1 Examples of Unthinking Vulnerability

  • The strawberry example contrasts a bypassed answer of two 'r's with a normal reasoning trace that counts three.The bypassed answer is marked incorrect; the correct answer is three.Source: §G.1 · p. 33

G.2 Examples of Training-based BoT

  • For the sequence starting at 0 and alternately adding and multiplying by successive integers, the first term above 125 was answered as 27 instead of 129; the ordered triple was answered (−1, 1, 0) instead of (6, 31, -1).Source: §G.2 · p. 35

G.3 Examples of Training-free BoT

The appendix illustrates training-free BoT with adversarial suffixes and MoT examples where monitors stop sufficient reasoning or flag risky prompts.

G.3 Examples of Training-free BoT

  • The seating-task response is marked incorrect, with correct answer 144; the binary-conversion response is marked incorrect, with correct answer 25168.Both responses show an empty <think> block before the answer.Source: §G.3 · p. 36

G.4 Examples of MoT for Enhancing Efficiency

  • For efficiency, monitors label examples as requiring deep thinking, then issue stop prompts once a distance derivation is sufficient or a sum has been rewritten.Source: §G.4 · pp. 37–38

G.5 Examples of MoT for Safety Alignment

  • For safety, the monitor skips reasoning on an explicitly unsafe terrorism request and refuses, while it labels a fictional bomb-disposal prompt safe and lets its planning continue.Source: §G.5 · pp. 38–39
Loading 2502.12202v2…