PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Thinking Tokens Help with Safety?

Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora

arXiv:2606.25013v1cs.LGcs.AIcs.CL

TL;DR

Whether extended thinking improves safety in reasoning models remains unclear. Across frontier open-weight models, this paper probes when refusal/compliance decisions arise and finds that safety outcomes are largely encoded before visible thinking, which rarely changes the initial trajectory.

  • Problem

    It remains unclear whether extended reasoning improves the safety–helpfulness trade-off by reducing harmful compliance without increasing unnecessary refusal.

  • Method

    The paper analyzes hidden-state predictability, truncated thinking prefixes, sentence-level stance changes, and existing inference-time and training-based safety interventions across four open-weight model families.

  • Results

    A first-token probe achieves 0.84–0.95 AUROC and up to 88% balanced accuracy, while later thinking rarely changes refusal/compliance trajectories and defenses often increase over-refusal.

  • Takeaways & Limitations

    Current reasoning models use thinking only weakly for safety deliberation, motivating methods that make thinking traces genuinely influence safety decisions.

  • Takeaways & Limitations

    The study covers only moderate-sized open-weight models and refusal/compliance on harmful and benign prompts, not broader safety dimensions.

Abstract

from arXiv · show

Today's reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts. It is also generally believed that this more "deliberative" mode should improve alignment and safety, by providing the model a safe space to consider whether its planned answer to a request violates its safety principles. We present evidence that this intuition is not always correct. Across frontier open-weight reasoning models spanning GPT-OSS, Qwen, Olmo, and Phi families, we find that the eventual refusal/compliance outcome is already strongly predictable via a trained head on the first token's hidden representation ($0.84$-$0.95$ AUROC and $\sim88\%$ balanced accuracy for predicting refusal/compliance) before any visible thinking. The thinking process turns out to be more akin to prefix completion than to deliberative revision, with the final outcome rarely changing after the first $\sim20\%$ of thinking, despite giving the appearance of deliberation at the text level ($\sim74\%$ of text-level deliberations occur when the response distribution is already locked to one refusal/compliance side). We also find that existing inference-time and training-based safety interventions, despite being motivated by the goal of inducing deliberation, largely shift model behavior toward over-refusal while suppressing already-scarce deliberation signals. Our results suggest that safety behavior in current reasoning models is much less deliberative than commonly assumed, and highlight the need for methods that induce real safety deliberation.

1 Introduction

The introduction questions whether reasoning traces genuinely improve safety decisions in large reasoning models. It frames the paper’s evidence that safety outcomes are largely determined early, thinking contributes little deliberation, and existing defenses often worsen the ASR–ORR tradeoff.

  • Motivation: Safety requires jointly minimizing Attack Success Rate (ASR) on harmful prompts and Over-Refusal Rate (ORR) on benign prompts, creating a central evaluation tension.A safe and helpful model should achieve low ASR and low ORR simultaneously.
  • Contributions: Across four frontier open-weight models, a linear probe decodes eventual refusal or compliance from the first thinking-trace token before visible thinking begins.This finding motivates the paper’s claim that safety behavior may be less deliberative than reasoning traces suggest.
  • Contributions: Once a short thinking prefix is fixed, later continuations usually preserve the early refusal or compliance trajectory, indicating that thinking contributes little to improved safety decisions.Text may resemble safety deliberation at the sentence level, but the paper finds that this appearance does not imply meaningful decision revision.
  • Contributions: Existing inference-time defenses either have limited impact or reduce ASR while increasing ORR, while also suppressing already weak safety-deliberation signals.The paper evaluates both inference-time and training-based defenses to examine their effects on safety behavior and deliberation.

2 Safety Behavior of Reasoning Models

Across Qwen3, Olmo-3, Phi-4, and GPT-OSS reasoning models, refusal/compliance is strongly encoded before visible thinking begins. The signal emerges late in prompt processing and is inherited by the first thinking token.

  • Evaluation: The evaluation spans Qwen3-8B, Olmo-3-7B-Think, Phi-4-Reasoning, and GPT-OSS-20B using harmful and benign prompt pools.The harmful suite emphasizes harder benchmarks because standard benchmarks are near-saturated for the studied models.
  • Refusal/Compliance Is Strongly Decodable at the First Token Representation: 0.840–0.948 AUROC and 0.763–0.878 BAcc show that first-token hidden states strongly predict final refusal/compliance outcomes across models.Table 1 reports linear separability and accurate binary prediction from the first thinking-token representation.
  • Refusal/Compliance Is Strongly Decodable at the First Token Representation: The Fisher discriminant is highest at the first thinking position, falls through the trace, and rises near its end across reasoning models.This U-shaped pattern indicates that the refusal/compliance representation is already pronounced before visible thinking tokens are generated.
  • Readout Signal Spikes at the End of Prefill: Readout remains weak through most of prefill but spikes near the input’s end across models, indicating late-prefill consolidation inherited by visible thinking.The effect is strongest for the Fisher discriminant, while AUROC and BAcc also rise slightly before the final spike.

3 Utility of Thinking

Thinking rarely changes safety outcomes after an early prefix, and enabling it does not consistently improve the ASR–ORR tradeoff. Sentence-level analysis further shows that most apparent stance shifts are performative rather than decision-changing.

  • Prefix completion: Continuation variance is already below 0.2 at 20% of the natural trace and decreases as prefixes grow across all models.Independent continuations therefore usually preserve the decision set by short prefixes, with later thinking rarely steering away from it.
  • Thinking and utility: 70–97% of prompts retain the same label with thinking, and no model consistently improves both sides of the ASR–ORR tradeoff.Qwen3-8B, Olmo-3-7B-Think, and Phi-4-Reasoning tend to over-comply, whereas GPT-OSS-20B shows the opposite tradeoff.
  • Robustness checks: Forcing additional test-time compute does not change this behavior, and the same conclusion holds under more permissive criteria for beneficial thinking.The analysis samples independent traces and evaluates stance oscillations using refusal-leaning, neutral, and compliance-leaning sentence labels.
  • Sentence-level deliberation: 15–34% of rollouts contain an oscillation, but 71–92% of those swings are performative and 72.8–76.6% occur after the outcome is already locked.Oscillations are classified as locked when the prior refusal probability is ≤0.05 or ≥0.95, and meaningful only when they significantly shift the final response distribution.

4 What Do Existing Defenses Do?

Existing inference-time and training-based defenses generally shift models toward over-refusal rather than improving the ASR–ORR tradeoff. They also fail to induce more deliberative reasoning, often suppressing the oscillations and early signals associated with safety deliberation.

  • Inference-time defenses: Inference-time defenses reduce sentence-level oscillations rather than increasing nonperformative safety deliberation.Across 22 of 24 inference cells, defenses reduce oscillations; drops reach 95% for Phi-4-Reasoning and exceed 60% for Olmo-3-7B-Think on both oscillation measures.
  • Overall results: Inference-time and training-based defenses generally shift the ASR–ORR tradeoff rather than improving both harmful compliance and benign refusal.ThinkSafe reduces Qwen3-8B ASR from 86.9 to 30.9 while increasing ORR from 6.0 to 44.6; RAPO produces a similar tradeoff on Olmo-3-7B-Think.
  • Training-based defenses: Training-based defenses also fail to make reasoning traces more deliberative, regardless of whether they use on-policy or off-policy traces.On-policy ThinkSafe and STAIR preserve first-token readability while reducing meaningful and performative oscillations; off-policy STAR-1 and SafeKey suppress the early refusal/compliance signal by 0.11 AUROC while largely preserving meaningful oscillations.

5 Discussion

The discussion argues that current reasoning language models rarely use thinking for genuine safety deliberation: refusal/compliance is often encoded from the outset, and existing defenses may not reliably induce deliberation. It situates this finding within prior work on shallow safety alignment and chain-of-thought faithfulness, while noting limits of model scale and open training questions.

  • Discussion: Current LRMs do not effectively use thinking for safety deliberation, because the first thinking token already strongly encodes refusal/compliance and later thinking has limited influence.The authors also find that thinking does not consistently improve safety decisions.
  • Related Works: The study extends evidence of shallow safety alignment from instruction-tuned models to reasoning models, showing refusal/compliance is strongly decodable at the beginning of thinking [Qi et al., 2025, Liu et al., 2026, Zhao et al., 2025c, Yin et al., 2025].Prior work found refusal behavior can depend on a small number of early tokens and remain vulnerable to simple prefix manipulations.
  • Limitations: The study is limited to moderately sized open-weight LRMs, so whether substantially larger reasoning models behave similarly remains uncertain.Prior work indicates scale alone may not resolve safety failures or guarantee refusal robustness under adversarial pressure.
  • Future Work: Future work should determine whether early refusal/compliance signals arise during pretraining, instruction tuning, or post-training, and identify methods that induce genuine safety deliberation.The authors note that gold-trace supervision, mixed harmful and benign data, or optimization against final labels may be insufficient.

Broader Impacts and Ethics Statement · A Related Works · B Supplementary Details

The paper frames its findings as potentially improving safety discrimination while warning that early safety encoding and refusal-shifting defenses could enable stronger jailbreaks. Related work connects this analysis to reasoning-based alignment, shallow safety mechanisms, jailbreak robustness, mechanistic studies, and chain-of-thought faithfulness.

  • Broader Impacts and Ethics Statement: The findings may help models distinguish harmful from benign requests, but could also enable jailbreaks by exposing early safety encoding and refusal-shifting defenses.The authors mitigate these risks by focusing on aggregate empirical measurements.
  • A Related Works: Reasoning traces improve difficult-task performance, motivating the hypothesis that inference-time deliberation may also improve alignment and safety [Wei et al., 2022, Kojima et al., 2022, Zhou et al., 2023, Shao et al., 2024, DeepSeek-AI, 2025].This work examines whether that broader safety hypothesis holds for large reasoning models.
  • A Related Works: Jailbreak research shows that carefully designed prompting strategies can bypass safety alignment, making robust safety deliberation a closely related concern.The paper does not directly study jailbreak attacks.
  • A Related Works: Prior studies suggest safety alignment can be shallow: refusal may depend on a few early tokens and be bypassed through prefix manipulation, while weak alignment can facilitate stronger-model jailbreaks [Qi et al., 2025; Zhao et al., 2025c].Liu et al. [2026] is also cited as showing that adversarial steering can affect safety behavior.
  • A Related Works: Mechanistic chain-of-thought studies examine how reasoning traces connect to final responses, including evidence that answers to easier problems can often be decoded before reasoning finishes [Baker et al., 2025, Korbak et al., 2025, Emmons et al., 2025].The discussion identifies Boppana et al. [2026] and Cox et al. [2026] as especially related.
  • A Related Works: Research on chain-of-thought faithfulness shows that generated traces need not reflect the internal computation producing final answers [Turpin et al., 2023, Lanham et al., 2023, Arcuschin et al., 2025, Boppana et al., 2026].The paper relates this concern to apparently deliberative segments that may not reflect underlying decision processes.

B.1 Evaluation · B.2 Inference-Time Defenses

The evaluation distinguishes harmful-request refusal from benign-prompt over-refusal across dedicated benchmark sets, using standardized generation and labeling procedures. Inference-time defenses intervene during thinking through safety priming, chunk-level reflection and backtracking, or entropy-triggered reminders.

  • B.1 Evaluation: Safety evaluation separates harmful-request refusal from benign-prompt over-refusal using WildJailbreak and FORTRESS for harmful prompts, and five dedicated benchmarks for benign prompts.WildJailbreak includes direct harmful requests, jailbreak-style prompts, and contrastive benign prompts; FORTRESS covers high-risk requests, while benign evaluation uses OR-Bench-Hard, FalseReject, CoCoNot, PHTest, and ORFuzzSet.
  • B.1 Evaluation: ASR counts harmful prompts as successful when the model complies, whereas ORR counts benign prompts as over-refused when the model refuses.Dataset-provided reference responses are not used for labeling.
  • B.1 Evaluation: Evaluation uses each model’s recommended sampling defaults in vLLM BF16, with thinking traces capped at 16,384 tokens for both ASR and ORR splits.Exact decoding parameters are listed in Table 5, and the evaluation datasets are summarized in Table 4.
  • B.1 Evaluation: First-token probe features are standardized coordinate-wise across pooled refusal/compliance samples at each position before computing the trace-normalized Fisher discriminant.This accounts for the discriminant’s normalization by total within-class variance rather than the full within-class covariance.
  • B.1 Evaluation: Visible thinking traces are annotated for sentence-level stance as compliance-leaning, refusal-leaning, or neutral across GPT-5.4, Gemini 3 Pro, and Sonnet 4.6.The annotation template analyzes how stance evolves across models, with annotation evaluation described separately in §C.3.
  • B.2 Inference-Time Defenses: SafePath ZS prepends “Let’s think about safety first.” to an opened thinking block without parameter updates, while preserving the common thinking-budget and force-close procedure.The prompt-side intervention uses a maximum context length of 24,576 tokens, or 16,384 for GPT-OSS-20B.

B.3 Training-Based Defenses

The study trains several safety defenses across four reasoning models spanning three families and two scales, using both LoRA-based and full-parameter recipes. The evaluated methods include supervised fine-tuning, auxiliary safety objectives, curriculum preference learning, distillation, and controlled safety-reasoning optimization.

  • Training setup: Training covers Qwen3-8B, Olmo-3-7B-Think, Phi-4-Reasoning (14B), and GPT-OSS-20B across three reasoning-model families and two parameter scales.Runs use bfloat16 on NVIDIA H100-80GB GPUs; smaller models use one to five GPUs, while GPT-OSS-20B uses DeepSpeed ZeRO-3.
  • Defense methods: The defense suite combines full-parameter SFT methods STAR-1 and SafeKey with LoRA-based R1-ACT, STAIR, and ThinkSafe recipes.STAR-1 trains on safety-reasoning traces, SafeKey adds key-sentence annotations and auxiliary heads, R1-ACT uses early-refusal traces, STAIR adds MCTS-driven preference learning, and ThinkSafe uses forward-KL distillation.
  • Defense methods: RAPO uses a two-stage SFT→GRPO pipeline that elicits controlled safety-reasoning prefixes before optimizing a risk-aware reward.Its SFT data combine 400 STAR-1 benign prompts with 400 Strata-Sword prompts and use a two-pass prompting procedure.
  • Implementation adaptations: RAPO requires target-specific infrastructure changes for Phi-4-Reasoning and GPT-OSS-20B because their channel or think-delimiter behavior differs from Qwen3’s ChatML structure.For Phi-4-Reasoning, the procedure force-closes the thinking block at the maximum budget; for GPT-OSS-20B, it maps the analysis channel to the safety-reasoning trace and injects the prefix there.
  • Evaluation: Table 6 reports base-model attack-success and over-refusal rates for each benchmark, with lower values preferred for both metrics.Values are averaged over four rollouts under a fractional four-guardrail vote, with 95% bootstrap confidence intervals over benchmark prompts.

C Supplementary Results · C.1 ASR/ORR Evaluation · C.2 Fisher Discriminants

Supplementary evaluations report per-benchmark ASR/ORR results under standardized four-rollout settings and show that five additional thinking models reproduce the primary models’ Fisher-discriminant valley signature. The supplementary models further indicate that parameter scaling and RLVR post-training do not clearly move safety decisions into the visible thinking trace.

  • C.1 ASR/ORR Evaluation: Table 6 reports full per-benchmark ASR and ORR results for four primary models across widely used benchmarks.Each prompt receives M = 4 independent rollouts under each model’s default sampling settings and the paper’s evaluation setting.
  • C.1 ASR/ORR Evaluation: WildJailbreak and FORTRESS produce well-separated ASR values across all four primary models, spanning 12–97%.These are the main-paper ASR benchmarks.
  • C.2 Fisher Discriminants: The Fisher-discriminant analysis covers additional thinking models, including Qwen3-32B, Qwen3.5-9B, Phi-4-Reasoning-Plus, and two DeepSeek-R1 distilled models.The two DeepSeek-R1 distilled models are identified as DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-0528-Qwen3-8B [DeepSeek-AI, 2025].
  • C.2 Fisher Discriminants: Qwen3.5-9B tests whether the effect persists across successive Qwen releases, while Phi-4-Reasoning-Plus tests an RLVR-trained successor to Phi-4-Reasoning.The supplementary model set also includes two DeepSeek-R1 distilled open-weight models, as described in the evaluation setup.
  • C.2 Fisher Discriminants: Five supplementary models reproduce the valley signature observed in the primary models’ Fisher-discriminant results.This extends the observed pattern beyond the primary model set.
  • C.2 Fisher Discriminants: Qwen3-32B preserves the effect after a 4× parameter scale-up within the Qwen3 family.Qwen3-32B is the 32B-parameter variant and uses the same thinking-mode chat template as the primary Qwen3-8B model.
  • C.2 Fisher Discriminants: Phi-4-Reasoning-Plus shows the same first-token and last-token concentration as Phi-4-Reasoning, with a sharp drop in mid-trace separability.The result suggests that RLVR post-training may not effectively move the safety decision into the visible thinking trace.

C.3 Effect of Thinking

Additional analyses show that more permissive no-think-to-think measures corroborate the main conclusion, while extending thinking provides weak and model-dependent safety benefits. Thinking-trace annotations nevertheless show substantial agreement among auditors.

  • More Permissive No-Think-to-Think Change Measures: Both permissive change measures largely corroborate the majority-label flip analysis, with Phi-4-Reasoning as a mild exception.The measures count any nonzero desired-direction shift and report the mean and standard deviation of signed shifts.
  • Extended Thinking and ASR/ORR: More thinking provides weak, model-dependent ASR–ORR benefits, with GPT-OSS-20B improving ASR at the cost of higher ORR while other models increasingly comply.Forcing thinking beyond the natural stopping point does not meaningfully change behavior.
  • Inter-Annotator Agreement (IAA) of Thinking Trace Annotators: Inter-annotator agreement is substantial: Fleiss’ κ is 0.636 ± 0.031, three-way concordance is 64.4%, and full disagreement occurs on only 1.3% of segments.Agreement remains stable across base and defense traces and across all four base models.
  • Inter-Annotator Agreement (IAA) of Thinking Trace Annotators: Sonnet 4.6 is adequate as the third annotator, matching Opus 4.7’s agreement while declining fewer traces.Sonnet declines 4.6% of traces versus Opus’s 9.4%, and mean pairwise κ is 0.646 versus 0.663.

C.4 Per-Split Breakdown of Thinking Trace Effect

Figure 10 separates per-rollout oscillation and per-segment Locked × Significant results for harmful ASR and benign ORR prompts, while Figure 11 compares defenses through the ASR–ORR tradeoff.

  • Per-prompt-pool breakdown: Figure 10 reports per-rollout oscillation rates and per-segment Locked × Significant decompositions separately for harmful ASR and benign ORR prompts.For each model, harmful prompts appear in the left group and benign prompts in the right; the overview bar shows the fraction of M=4 rollouts containing at least one oscillation.
  • Defense tradeoff: Most defenses move along their base model’s ASR–ORR tradeoff rather than shifting performance toward the bottom-left Pareto region.Figure 11 plots every base-model–defense pairing, with dashed lines tracing each base model’s defense-specific Pareto frontier.

C.5 Defense Evaluation

Across nine defenses applied to four base models, no defended cell improves on its base model in both attack success rate (ASR) and over-refusal rate (ORR). Lower ASR consistently comes with higher ORR, while some defenses make the tradeoff in the opposite direction or worsen both metrics.

  • Defense Evaluation: No defended cell improves on its base model in both ASR and ORR across all nine defenses and four base models.Figure 11 compares every defense on the ASR–ORR plane.
  • Defense Evaluation: Lower ASR always comes with higher ORR, indicating a direct safety–over-refusal tradeoff.The ideal region is the bottom-left corner, where both ASR and ORR are low.
  • Defense Evaluation: Some defenses instead trade higher ASR for lower ORR, while a handful worsen both metrics relative to the base model.
Loading 2606.25013v1…