Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty

Yipeng Zhao; Qishun Yang; Shenzhe Zhu; Shu Yang; Di Wang

Source-linked AI summary

Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty

Yipeng Zhao, Qishun Yang, Shenzhe Zhu, Shu Yang, Di Wang

arXiv:2608.23497v1Aug 24, 2026cs.AIcs.CL

TL;DR

The paper addresses how harmless reasoning fine-tuning can degrade LLM safety and whether representation geometry can guide mitigation. It extracts reasoning and safety directions, localizes safety-decision layers, and applies SDP to penalize safety-direction drift. On Qwen2.5-3B and 7B, SDP restores evaluated safety metrics while retaining most evaluated reasoning performance, with benchmark-specific trade-offs.

Problem
Harmless reasoning fine-tuning can induce safety degradation, while prior neuron-level explanations did not identify the relevant representation geometry or a training-time mitigation.
Method
The paper extracts reasoning and safety directions, uses CKA ratios and probes to localize safety-decision layers, and penalizes squared safety-direction displacement during fine-tuning.
Results
SDP restores both evaluated safety metrics to or above base-model values on Qwen2.5-3B and 7B while reasoning performance remains comparable, with benchmark-specific trade-offs.
Takeaways & Limitations
The diagnostics provide a representation-space account of RIM and guide adaptive penalty-layer selection when displacement compensates beyond the initial scope.
Takeaways & Limitations
The positive RIM and SDP results focus on Qwen2.5-Instruct, and the geometric analysis is presented as a local account rather than a general causal characterization.

Abstract

from arXiv · show

Reasoning-Induced Misalignment, where fine-tuning on reasoning data containing no harmful content, including mathematics, code, and problem-solving with chain-of-thought traces can induce harmful behaviors of LLM, posing a serious challenge to the safety of LLM reasoning. Cross-architecture, cross-scale, and cross-dataset checks show that RIM does not always emerge. Previous work attributed RIM to neuron-level entanglement, but did not identify the geometry of the representation space underlying this entanglement or propose a training-time fix. We provide both: a representation-space analysis of RIM and the Safety-Direction Penalty (SDP), which penalizes movement along a learned safety direction during reasoning fine-tuning. The analysis extracts two activation-space directions, one encoding reasoning ability and the other safety behavior. These directions are coupled: fine-tuning that improves reasoning shifts safety representations, and prompts with larger shifts show larger safety degradation. CKA distance ratios and probes locate the safety-decision layers where this shift is most relevant. These findings guide the design of SDP: the coupling motivates penalizing displacement along the safety direction, and the layer localization sets the initial scope. When the initial scope leaves compensatory shifts beyond the penalized layers, the same diagnostics guide iterative expansion. On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.

1 Introduction

Reasoning-Induced Misalignment can arise from harmless reasoning fine-tuning because reasoning and safety representations are coupled. The paper analyzes this geometry and introduces SDP to constrain safety-direction drift during training.

RIM can degrade safety after fine-tuning on benign mathematics, code, and chain-of-thought reasoning data.
Prior neuron-level analysis attributed RIM to entangled reasoning and safety circuits, but did not provide a mitigation.
The paper extracts reasoning and safety directions from base-model hidden states, separating reasoning quality and refusal–compliance behavior.
CKA ratios and linear probes identify compact safety-decision layers where safety-specific representation changes are concentrated.
SDP penalizes squared displacement along the safety direction, with its initial layer scope determined by the localized diagnostics.
On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.

2 Related Work

Related work distinguishes RIM from harmful-data misalignment and places SDP among defenses that preserve safety during fine-tuning. SDP specifically targets hidden-state drift during reasoning fine-tuning.

Earlier work showed that harmful or narrowly misaligned fine-tuning can induce broad safety degradation across unrelated domains.
The paper studies RIM, where reasoning fine-tuning on non-harmful data can degrade safety, and proposes SDP as a direct training-time mitigation.
Existing defenses address harmful fine-tuning, safety-oriented instruction tuning, or PEFT-specific preservation rather than the reasoning case studied here.
SDP uses a precomputed safety direction to penalize hidden-state drift and requires only fixed contrast pairs beyond the reasoning corpus.

3 Diagnosing RIM in Representation Space

The paper finds that RIM is conditional across evaluated models and datasets, and explains its safety degradation through coupled reasoning and safety directions. CKA, probes, and displacement measurements localize and characterize the relevant representation changes.

RIM appears only in some evaluated model–dataset–training settings, rather than universally.
On Qwen2.5-3B and 7B, AM-SFT degrades safety while largely preserving benchmark reasoning performance.
Reasoning and safety directions show consistently negative whitened cosine similarity across mid-to-deep layers in both models.
CKA ratios isolate safety-specific change, with 3B peaking at 2.27 at L23 and identifying a compact layer region.
Harm-perception probes remain near-perfect after fine-tuning, while safety-decision probes collapse, indicating recognition persists but refusal behavior is disrupted.
For 7B, displacement along the safety direction is uniformly negative with mean −7.08; for 3B, sign-mixed shifts partially cancel across layers.
Larger per-prompt displacement away from the safety direction is associated with larger safety degradation, with Spearman ρ = 0.475 for 3B and ρ = 0.277 for 7B.

4 Mitigation Method: Safety-Direction Penalty

SDP adds a squared safety-direction displacement penalty to reasoning fine-tuning and uses diagnostics to choose and expand its layer scope. This procedure restores safety at both evaluated model scales.

SDP adds a squared displacement penalty along the safety direction to the standard cross-entropy loss.
The penalty acts along the safety direction rather than directly penalizing orthogonal representation changes.
Squaring makes the penalty sign-invariant, while averaging over the selected layers keeps per-layer magnitude comparable across scope sizes.
The targeting procedure selects an initial CKA peak, evaluates SDP, diagnoses compensation, and expands within the rCKA > 1 boundary when needed.
For 7B, the initial scope M = {L15, . . . , L19} restores safety; for 3B, near-zero displacement in M1 is insufficient.
For 3B, compensation expands the scope to M = {L19, . . . , L32}, spanning 14 layers, while expansion beyond rCKA > 1 degrades safety.
After SDP, displacement–safety correlations are no longer statistically significant: ρ = −0.164 for 3B and ρ = −0.119 for 7B.

5 Results and Analysis

SDP restores safety at both evaluated scales while retaining most reasoning performance, with scale-specific penalty scopes and benchmark-specific trade-offs.

Evaluation Setup: SDP uses final penalty scopes of 14 layers for 3B and 5 layers for 7B.The 7B model succeeds with the initial scope, whereas 3B requires expansion after a compensation diagnostic.
Safety Recovery: For 3B, HEx-PHI harmfulness falls from 20.3% to 10.0%, matching base, while SafetyBench accuracy rises from 57.9% to 69.6%.
Safety Recovery: For 7B, HEx-PHI harmfulness falls from 25.3% to 11.3%, below base, while SafetyBench accuracy rises from 76.5% to 79.4%.
Reasoning Preservation: GPQA decreases by 3.8 percentage points for 3B and 2.0 for 7B, while 7B AIME 2024 rises from 10.4 to 15.0.The 3B AIME scores remain within base-model variance.
Behavioral Analysis: SDP reduces conditional harmfulness for 3B but suppresses extended-thinking adoption for 7B.The two scales therefore recover safety through different behavioral channels.

6 Conclusion

The paper characterizes RIM as conditional under evaluated settings and uses representation diagnostics to guide SDP, which restores safety while retaining most reasoning performance.

RIM does not emerge in every evaluated model–dataset–training setting, supporting a conditional rather than universal characterization.
Reasoning–safety coupling and safety-direction displacement are consistently associated with behavioral safety degradation in settings meeting the operational RIM criterion.
CKA ratios and linear probes locate compact safety-decision layers where models recognize harmful requests but no longer reliably refuse them.
Under evaluated RIM settings, SDP restores base-level safety while retaining most evaluated reasoning performance without safety data, a reference policy, or inference-time intervention.

Limitations

The evidence is concentrated in evaluated conditions and Qwen2.5-Instruct settings, while the geometric account and fixed-template intervention remain limited in scope.

Positive RIM and SDP results focus on Qwen2.5-Instruct, while other checks provide only initial cross-dataset, cross-family, and cross-scale evidence.
SDP estimates one fixed-template refusal–compliance direction per layer at the last-token representation.
The geometric analysis is interpreted as a local account of SDP’s intervention rather than a general causal characterization of model safety.
Broader controlled studies and matched comparisons with external fine-tuning defenses are identified as needed to contextualize SDP’s benefits and costs.

Ethical Considerations

The paper frames its safety work as defensive while documenting possible misuse, evaluation safeguards, model and dataset choices, and training-data characteristics.

Ethical Considerations: The diagnostic pipeline could be misused to target safety-vulnerable layers or remove safety behavior more efficiently.
Ethical Considerations: The authors judge the defensive value to outweigh this risk because understanding safety degradation supports robust mitigation.
Evaluation Safeguards: HEx-PHI contains harmful prompts for safety evaluation, but model outputs on those prompts are not released.
Evaluation Safeguards: The experiments use publicly available models and datasets and do not generate or distribute new harmful content.
Models and Data: The primary Qwen2.5-Instruct experiments use 3B and 7B as RIM and SDP targets, with 14B as a within-family scale check.
Training Data: AM-DeepSeek contributes 10,000 reasoning examples containing no harmful content, with extended thinking traces in every example.The sampled corpus includes math, code, and general reasoning categories, with overlapping keyword assignments.
Dataset Controls: The MetaMathQA control changes both corpus identity and explicit tag convention, so it does not isolate response length, mathematical domain, or every reasoning-format factor.
Training Procedure: SDP and AM-SFT share fine-tuning hyperparameters, differing in the penalty weight, which is zero for AM-SFT and 0.5 for SDP.

A.4 Scope-Check Evaluation Results

The scope checks test whether benign reasoning fine-tuning satisfies the operational RIM criterion across datasets, architectures, and scale. None of the additional settings met that criterion, so SDP was not applied.

The experiments test RIM as safety degradation after benign reasoning SFT with preserved or improved evaluated reasoning performance.These are induction-scope checks, and SDP is evaluated only after a setting meets this criterion.
None of the additional MetaMathQA, Gemma, Ministral, or Qwen2.5-14B settings exhibited RIM under the operational criterion.MetaMathQA, Gemma, and Ministral failed through reasoning degradation or absent safety worsening; both 14B runs failed because safety did not degrade.
Ministral is reported as a descriptive cross-architecture scope check rather than a matched causal architecture-only comparison.Dynamic-LoRA loading failed, so Ministral evaluation used a merged adapter under vLLM 0.13.

B.1 Safety Evaluation

The evaluation combines harmful-response and safety-knowledge benchmarks with reasoning benchmarks, using specified generation, extraction, judging, and diagnostic procedures. It also reports layer-level CKA analyses used to characterize reasoning–safety coupling.

Safety benchmarks: Safety is evaluated with HEx-PHI for open-ended harmful-response detection and SafetyBench for multiple-choice safety knowledge.HEx-PHI contains 300 harmful prompts spanning 10 categories, while the full English SafetyBench subset contains 11,435 questions across 7 categories.
Safety judging: HEx-PHI misalignment is the percentage of responses classified as fulfill by GPT-4o-mini when harmfulness reaches 3 or higher on a 5-point scale.The judge evaluates the user-visible response after thinking-token removal; each prompt is judged once under deterministic settings.
Safety judging: SafetyBench reports accuracy on the full English subset, counting missing predictions as incorrect after integer-first answer extraction and fallback matching.The benchmark uses zero-shot single-user-message prompts, greedy decoding, and 0-indexed option labels.
Reasoning benchmarks: Reasoning performance is measured on AIME 2024, AIME 2025, and GPQA, with eight-run mean ± standard deviation for AIME and single-run accuracy for GPQA.AIME uses 30 problems per year, while GPQA Main contains 448 graduate-level science questions; both request step-by-step reasoning with boxed final answers.
Diagnostics: Layer-level CKA profiles identify mid-to-deep layers as the main safety-specific change region, while early 3B-layer ratios reflect negligible-distance noise.For 3B, the M1 peak occurs at L20–L26 with rCKA 1.45–2.27, separated from the L12–L16 valley by early-layer absolute CKA distances below 0.035.

C.4 Probe Training Details

The probes test whether fine-tuning changes harmful-request recognition or the model’s refuse/fulfill decision, then relate safety-direction displacement to behavioral degradation and SDP outcomes.

Probe datasets: 600 prompts combine 300 HEx-PHI harmful and 300 Alpaca-Cleaned harmless examples for harm-perception probing.The same fixed prompt set is reused across diagnostics and model scales.
Harm-perception probe: Harm-perception accuracy remains near-perfect after AM-SFT: 3B base 0.981 / AM-SFT 0.983 and 7B base 0.983 / AM-SFT 0.986.These results indicate preserved harmful-request recognition at both scales.
Safety-decision probe: Safety-decision probes use model-generated refuse/fulfill labels, with base majority-class baselines of approximately 91% for 3B and 86% for 7B.Above-baseline accuracy indicates decision-relevant information beyond class imbalance.
Probe localization: Position checks yield identical qualitative conclusions: safety-decision accuracy collapses in the same layers, dS displacement concentrates in M1, and per-prompt ρ remains significant.The last non-padding token gives slightly higher ρ values.
SDP diagnostic: SDP severs this correlation: ρ = −0.164 for 3B and ρ = −0.119 for 7B, both non-significant.The nonzero-change subsets contain N = 93 and N = 68 prompts, respectively.

D.6 7B Scope Ablation

The 7B scope ablation tests whether expanding SDP beyond M1 into capability-dominant layers preserves the initial safety recovery. It instead produces compensatory displacement and worsens safety.

Scope ablation: Expanding 7B SDP from M1 = {L15, . . . , L19} to {L15, . . . , L23} adds layers whose rCKA values are 1.09, 0.91, 0.79, and 0.74.Layers L21–L23 are capability-dominant because their rCKA values are below 1.
Results: HEx-PHI rises by +8.0 pp, from 11.33% to 19.33%, nearly reverting to the AM-SFT level of 25.33%.The ablation therefore loses much of the safety recovery achieved with the initial scope.
Interpretation: Penalizing capability-dominant layers constrains normal adaptation and forces compensation through deeper layers, giving the rCKA > 1 boundary functional significance.The passage describes this as more than a statistical threshold.

E.1 14B Analysis

The 14B analysis contrasts with 3B and 7B: safety-decision encoding remains intact, and the model almost never activates extended thinking, consistent with absent safety degradation.

Safety-decision probes: Safety-decision probe accuracy remains above baseline after AM-SFT, peaking at 0.97 in layers {L38, . . . , L43}.Decision encoding is not disrupted in 14B, unlike in 3B and 7B.
Extended thinking: 14B almost never enters <think> mode after AM-DeepSeek training, unlike 3B and 7B.Without extended thinking, the reasoning-mediated harm channel does not activate, consistent with the absence of safety degradation.
Extended thinking: In 3B and 7B, extended-thinking responses are far more harmful than responses without <think>, with a 9–22× gap.Adoption reaches 46% for 3B and 60% for 7B after AM-SFT; harmfulness among <think>-using responses is 41.7% and 39.2%, versus 1.9% and 4.2% without <think>.
Decision failure: Among harmful <think> responses, 67% mention safety or ethics concerns, indicating a decision failure rather than a perception failure.Over 90% frame the response as helping the user across 58 harmful 3B and 71 harmful 7B think responses.
SDP behavioral channels: SDP uses opposite behavioral channels across scales: 3B preserves extended thinking while reducing conditional harm, whereas 7B suppresses extended thinking.For 3B, adoption rises from 46% to 100% while conditional harm drops from 42% to 10%; for 7B, adoption drops from 60% to 18% while harm remains comparable at 39% to 42%.
Displacement structure: The displacement structure limits a single-direction penalty: 7B displacement is unidirectional, while 3B displacement is sign-mixed and distributed.The authors state that restoring internal reasoning safety for 3B may require constraints beyond a single-direction penalty.