First Token Matters: Understanding Safety Collapse in Large Reasoning Models

• LRMs use multi-step Chain-of-Thought reasoning for complex tasks, while recent studies have identified safety vulnerabilities in those mechanisms.Source: §1 · p. 1
• Existing LRM safety methods often use fine-tuning or preference optimization and treat models as black boxes, motivating investigation of safety-failure mechanisms.Source: §1 · p. 1
• LRM refusal projection strength matches instruction-tuned counterparts during prompt encoding, then drops sharply at the first generated token.Source: §1 · p. 1

• An LRM generates an intermediate reasoning trajectory before its final response, with reasoning tokens conditioned on the query and earlier tokens.Source: §2 · p. 2
• Unlike standard instruction-tuned models, LRMs rely more heavily on intermediate reasoning traces to solve complex tasks.Source: §2 · p. 2
• Prior work links safety behavior to identifiable attention heads and reports overlap among safety heads in models from the same lineage [3] [45].Source: §2 · p. 2
• Prior work finds refusal behavior is approximately linear in the residual stream, enabling activation-level analysis or steering [1] [22].Source: §2 · p. 2
• The analysis uses DeepSeek-R1 distilled LRMs and same-family instruction-tuned counterparts, with AdvBench, StrongReject, and Alpaca datasets.Source: §2 · p. 2

3.1 Safety Failure Induced by Chain of Thought

• Comparisons between IT models and LRMs derived from the same backbone suggest that reasoning traces can bypass safety alignment.This motivates further investigation of internal representations.Source: §3.1 · p. 2

3.2 The Onset Refusal Collapse of LRMs

• Safety heads identified in IT models provide layer-wise residual contributions used to derive unified refusal vectors for comparing matched LRMs and IT models.Steering along these vectors has a two-sided effect on responses, supporting shared latent refusal directions across models from the same backbone.Source: §3.2 · p. 3
• At t = 0, LRMs’ refusal activation drops toward the non-refusal region while matched IT models maintain strong activation, defining Onset Refusal Collapse.During prompt encoding, LRMs remain partially aligned with their IT counterparts; harmless-input results are deferred to Appendix D.Source: §3.2 · p. 3
• Matched IT/LRM refusal gaps quantify ORC across harmful prompts, with positive RORC indicating that the gap widens at generation onset.Table 2 reports aggregate ORC statistics; harmless inputs are covered in Appendix D.Source: §3.2 · pp. 3–4, Table 2 · p. 4

3.3 Linking ORC to Safety Failure

• On StrongReject, first-token safety-prefix interventions and refusal-direction injection consistently reduce ASR versus vanilla LRMs across R1-7B and R1-8B.The results suggest ORC makes an important causal contribution to unsafe behavior on harmful queries.Source: Table 3 · p. 4, §3.3 · p. 4
• Responses to harmful queries often begin with affirmative tokens such as “Okay” or “Alright,” even when the model eventually refuses.The authors associate this tendency with ORC and defer its detailed discussion and supporting evidence to Appendix F.Source: §3.3 · p. 4

4.1.1 Motivation

• SafeToken learns a continuous safety anchor for reasoning onset, addressing the limits of searching discrete natural-language tokens that may interfere with utility.The method updates the learned embedding while freezing the base LRM; at inference, the token is appended to the query to occupy the first output position.Source: §4.1.1 · p. 5, §4.1.2 · p. 5

4.2.1 Experiment Setup

• The evaluation compares R1-8B and R1-7B, includes Qwen3-8B for transferability, and covers safety, jailbreak robustness, over-refusal, and utility.SafeToken optimizes one token embedding and the process converges within minutes on a single GPU.Source: §4.2.1 · p. 5, §4.2.2 · p. 6

4.2.2 Main Results

• SafeToken achieves strong safety on several harmful-query benchmarks with competitive utility, but shows limited robustness under strong jailbreak attacks.The authors suggest jailbreaks may alter the model’s internal assessment of harmfulness earlier in the pipeline.Source: §4.2.2 · p. 6
• SafeToken anchors a learned safety representation at the prompt’s end, maintaining refusal strength through the critical initialization phase and subsequent reasoning.The reported mechanism is that avoiding the initial collapse leads to higher refusal strength throughout the reasoning trajectory.Source: §4.2.2 · p. 6

4.2.3 Sensitivity Analysis: SafeToken Position

• Front injection significantly degrades safety performance compared with onset injection, indicating that SafeToken’s position is critical.Source: §4.2.3 · p. 7

Safety in LRMs

• Prior work reports that stronger reasoning can increase potential harm, including self-jailbreaking and CoT-based attacks, and explores defenses for reasoning-related vulnerabilities.Source: §5 · p. 8

Impact of Tokens

• Prompt tuning learns continuous input embeddings while keeping the backbone frozen, and recent safety methods use it to reduce unsafe generations without full-parameter fine-tuning.Source: §5 · p. 8
• Prior work finds that prompt-based control depends both on prompt representation and on where it is injected.Source: §5 · p. 8

• Token-level refusal analysis identifies a localized safety vulnerability at reasoning onset, motivating SafeToken’s learned continuous safety anchor.Source: §6 · p. 8
• SafeToken mitigates the collapse and improves safety on harmful-query benchmarks while preserving general reasoning utility.Source: §6 · p. 8
• The results highlight reasoning onset as a critical stage for LRM safety and provide evidence that lightweight, mechanism-driven interventions can improve safety behavior.Source: §6 · p. 8

7 Limitations

• The study addresses refusal collapse at reasoning onset, and SafeToken’s gains are less pronounced in some stronger attack scenarios.The authors suggest such attacks may disrupt recognition of harmfulness, whereas SafeToken targets cases where risk awareness is present but is not translated into refusal.Source: §7 · p. 8
• The authors describe a possible boundary of the intervention: attacks that act earlier in the safety decision process may require integration with the training pipeline.They present integration as a possible direction for future work.Source: §7 · p. 8

A Safety Performance

• On AdvBench, instruction-tuned models maintain robust refusal rates, while corresponding LRMs show a marked increase in unsafe responses.Source: A Safety Performance · p. 12, Figure 4 · p. 12

B.1 EAP-IG Introduction and Data Preparation

• Because safe and unsafe LRM responses often share similar onset tokens, their behavioral divergence may not appear in the first token, complicating clean EAP-IG counterfactual construction.The analysis therefore requires closely matched prompts that differ in whether they elicit safe refusal or unsafe response onset.Source: §B.1 · p. 12
• The authors manually rewrite harmful prompts into safe counterparts while preserving wording, structure, and style as much as possible and changing only what is needed to remove harmful intent.EAP-IG is then applied to these pairs to rank components influential at reasoning onset.Source: §B.1 · pp. 12–13

B.2 Setup

• For attribution, the setup uses five integrated-gradients steps, 50 counterfactual pairs, equal tokenized lengths, and top-n edges ranked by absolute attribution score.The attribution target is the final-token logit difference between “Sure” and “I”; tested graph sizes are 50, 100, 200, 500, and 1000 edges.Source: §B.2 · p. 13

B.3 Safety Heads

• Safety-head scores sum positive attribution scores on outgoing edges, and the top-ranked 200 heads are selected for each of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct.The selected heads are used for subsequent analysis and ablation.Source: §B.3 · p. 14

C Details of Refusal Direction Extraction

• The refusal-vector procedure isolates layer-specific ablation updates from preceding residual effects and finds unified vectors aligned with those updates across the dataset.Algorithm 1 describes refusal-vector extraction via safety-head ablation.Source: §B.2 · p. 13, C Details of Refusal Direction… · p. 14

D Results for Benign Queries

• On harmless inputs, R1-8B shows a noticeable first-token refusal-projection drop, whereas R1-7B remains comparatively stable, making the pattern less uniform than for harmful queries.The benign pattern does not show the same clear and consistent divergence associated with unsafe behavior in the harmful setting.Source: D Results for Benign Queries · p. 14

E.1 Prefixes / E.2 Refusal Direction Injection

• The prefix-length intervention uses safety-related prefixes of different lengths, and refusal-direction injection adds the refusal direction only at the first generated token.The injection uses a layer-specific refusal direction and strength; an MLP probe predicts strength from hidden states to estimate the refusal-projection gap between the IT model and LRM.Source: §E.2 · pp. 14–15, Table 7 · p. 15

F.1 The Suppression of Refusal in Deep Layers

• In deep layers, affirmative-token ranks rise toward the top predictions while refusal precursors are suppressed, whether the model ultimately refuses or produces harmful output.The sampled responses are grouped by final outcome; Table 8 reports top-token probabilities in the last three layers.Source: §F.1 · p. 15, Table 8 · p. 15
• The final layers’ prediction space is dominated by affirmative semantics, which the authors interpret as prioritizing reasoning-chain launch anchors over safety representations until the trace is established.The reported vocabulary inspection covers Layers 30–32.Source: §F.1 · p. 15, Table 8 · p. 15

F.2 Attributing the Affirmative Token to Task Focus

• LRMs attribute more strongly to structural and instructional tokens than to the harmful subject token, while IT models assign the harmful token more attribution in the representative query.The contrastive analysis targets logit(“Okay”)−logit(“Sorry”) for LRMs and the reverse refusal margin for IT models.Source: §F.2 · pp. 15–16, Figure 7 · p. 17
• On JailbreakBench-Behavior, LRMs rank harmful tokens earlier than IT models in both outcome groups, yet structural-token dominance is stronger and the initial affirmative tendency prioritizes structural adherence over safety evaluation.The analysis reports Peak Structural Dominance (η) and First Harmful Rank (R_harm); Table 9 presents their averages across model types and data groups.Source: §F.2 · pp. 16–17, Table 9 · p. 16

G.1 Baselines

• The comparison includes SafeChain, SafePath, STAR-1, and an inference-time Safety Prefix variant of SafePath.SafeChain and STAR-1 are training-based approaches; Safety Prefix force-decodes SafePath’s safety statement as initial thinking tokens.Source: §G.1 · pp. 17–18

G.2 Safety Performance

• SafeToken has limited robustness under strong jailbreaks that may alter harmfulness recognition before its onset-refusal intervention can help.Its target is the distinct case where risk awareness remains but refusal behavior is disrupted by ORC.Source: §G.2 · p. 18

G.3 Additional Results on Compatibility

• Adding SafeToken generally preserves utility across most settings and improves performance especially on AIME24 and MATH500, though mild degradations remain in a few cases.Table 10 compares utility before and after adding SafeToken to SafePath and STAR-1; higher is better for all listed utility metrics.Source: §G.3 · p. 18, Table 10 · p. 18