PaperlayerView exact revision on arXiv

Source-linked AI summary

When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning

Yitong Guo, Xiaoyi Chen, Siyuan Zhang, Xiaofeng Wang, Haixu Tang

arXiv:2609.01455v1cs.CRcs.AI

TL;DR

Benign fine-tuning can severely weaken LLM safety alignment, raising the question of why refusal behavior is fragile. Using Fisher-geometric and activation-based analyses, the paper attributes collapse to disruption of a low-rank output-routing pathway: safety can reach high attack success rates after 100 benign examples while utility degrades less, and refusal can be restored with limited safety intervention. The study also finds that LoRA and ASAM mitigate early collapse but lose protection at larger fine-tuning scales.

  • Problem

    The paper asks why benign downstream fine-tuning weakens refusal behavior despite containing no harmful instructions.

  • Method

    The paper combines Fisher-geometric analysis with logit-lens read-outs and cross-condition activation patching to study safety routing and its disruption.

  • Results

    Safety Fisher is concentrated and flattened by alignment, then late output-side MLP sharpness rises 11.2× for safety versus 1.2× for utility after benign fine-tuning.

  • Takeaways & Limitations

    Safety failure is best understood as disruption of a low-rank output-routing mechanism, while safety-relevant internal representations remain sufficiently preserved for refusal recovery.

  • Takeaways & Limitations

    The study primarily evaluates SFT- and DPO-aligned models, not other reinforcement-learning frameworks such as PPO-based RLHF or iterative preference-optimization variants.

Abstract

from arXiv · show

Benign fine-tuning severely weakens the safety alignment of large language models (LLMs), so we study why refusal behavior is so fragile. While prior work often attributes this failure to gradient conflict, we propose a fundamentally different Fisher-geometric explanation: safety Fisher is low-rank, and alignment makes the safety geometry flatter while preserving an output-routing pathway. After 100 benign fine-tuning examples, this pathway is selectively re-sharpened in output-side MLP modules, explaining the asymmetric fragility: safety can collapse to high attack success rates, while general utility degrades mildly. The routing view also explains why few safety examples can restore refusal behavior, indicating that internal safety-relevant representations are preserved. Finally, we show that LoRA and ASAM mitigate early collapse by suppressing output-side sharpness, but their protection weakens at larger fine-tuning scales. Overall, safety failure is best understood as a disruption of a low-rank output-routing mechanism

1 Introduction

The paper argues that benign fine-tuning breaks safety through selective disruption of a low-rank output-routing pathway rather than primarily through gradient conflict or global knowledge loss. Fisher geometry and behavioral results explain why refusal collapses rapidly while utility is less affected, and why safety can be restored.

  • Benign fine-tuning can sharply weaken refusal behavior even when its data contains only benign utility examples.The paper frames this as the central question of why safety alignment is so easy to break.
  • Low-conflict and random sample subsets produce comparable safety collapse, indicating that gradient conflict is not the primary cause.The result challenges explanations centered on adversarially selected outliers.
  • Safety alignment routes harmful internal representations to refusal behavior through a low-rank output pathway.General knowledge and harmful-content representations coexist, while alignment controls how harmful representations are mapped to outputs.
  • Safety Fisher concentration is 0.491 versus utility Fisher concentration at 0.271, rising in final layers to 0.718 versus 0.223.Alignment also reduces layer-wise safety top eigenvalues by roughly two orders of magnitude, making the safety geometry flatter while preserving a routing pathway.
  • After benign fine-tuning, final-layer down_proj sharpness increases 11.2× for safety versus 1.2× for utility.This localized re-sharpening links output-side curvature to strong perturbation of refusal routing and weaker effects on knowledge-task outputs.
  • Safety can reach up to 85.7% ASR after 100 benign examples while utility degradation remains comparatively limited.Few safety examples can restore refusal behavior, and a fixed refusal-style prefix can reduce ASR by 50% without parameter updates; LoRA and ASAM mainly mitigate early collapse.

2 Preliminaries

The preliminaries define the aligned and utility models, describe benign fine-tuning and safety measurement, and position the paper's output-side routing account against prior explanations of safety fragility.

  • The study considers chat models with parameters θ and policy πθ(y | x), distinguishing aligned checkpoints produced by SFT or DPO.The aligned checkpoint retains general instruction-following ability while refusing harmful queries.
  • Safety is measured with Attack Success Rate on HEx-PHI using an LLM-as-a-judge protocol, with aligned models designed to have approximately 0% ASR.
  • Benign fine-tuning uses data without harmful content or refusal demonstrations, yet ASR can sharply increase after as few as 100 examples.
  • Prior explanations emphasize gradient conflict, outlier samples, aggressive optimization, or loss-landscape curvature as causes of safety collapse.The paper instead localizes the failure to output-side re-sharpening of a low-rank refusal-routing pathway.
  • The paper uses Fisher-geometric analysis to explain both rapid refusal collapse and its reversibility.

3 Geometry of Alignment Collapse

The paper explains benign-fine-tuning fragility through Fisher geometry: safety is concentrated in a low-rank pathway that alignment flattens, while FT-100 selectively re-sharpens late output-side MLP routing and disrupts refusal behavior.

  • Motivation and geometric account: Random, gradient-top, and gradient-bottom benign subsets all break safety comparably, challenging gradient conflict as the primary explanation.Outlier conflict is sufficient but not necessary for the observed collapse.
  • Safety Fisher structure: Safety Fisher is more concentrated than utility Fisher, with top-1 mass 0.491 vs. 0.271 and estimated ranks 8 vs. 11.The safety spectrum decays more sharply, indicating that fewer dominant directions account for more curvature.
  • Safety Fisher structure: Alignment lowers safety Fisher top eigenvalues by roughly two orders of magnitude across most layers, producing a flatter aligned safety geometry.The baseline is sharp across almost all layers, whereas the aligned model becomes substantially flatter.
  • Post-training curvature drift: After FT-100, middle-layer curvature continues decreasing, but final-layer down_proj sharpness increases by 11.2×, selectively re-sharpening late output-side routing.The corresponding utility increase is only 1.2×, while SciQ accuracy falls by about 10%.
  • From geometry to behavior: The refusal signal remains present after fine-tuning, but compliance rises earlier and higher, preventing the refusal–compliance margin from becoming positive.Activation patching further identifies the final-layer state as the dominant causal locus, with the aligned refusal margin restored to 96.9%.

4 Understanding Alignment Fragility

The section evaluates how benign fine-tuning disrupts safety across models, datasets, and training scales, contrasting generic drift with gradient conflict and testing mitigation and recovery. Results show early safety collapse is disproportionately larger than utility loss, while LoRA and ASAM help temporarily and minimal supervision can restore refusal behavior.

  • Safety degradation across settings: 100 benign samples substantially increase ASR across alignment conditions, datasets, and model families, while utility degradation remains comparatively smaller.On Alpaca with Llama3, full-parameter fine-tuning raises ASR to 75.4%, 85.70%, and 59.6% for the Instruct Baseline, Align-256, and Align-10k, respectively.
  • Gradient conflict versus generic drift: Gradient conflict modulates safety degradation but is not required: bottom-conflict subsets still produce ASR above 52% across alignment conditions.For Align-256, avoiding highly conflicting gradients lowers ASR from 83.00% to 54.20%, showing attenuation without preservation of safety.
  • Gradient conflict versus generic drift: Generic benign fine-tuning drift alone is sufficient to substantially disrupt refusal behavior, even when samples are not selected for high conflict.The evidence rules out gradient conflict as the primary cause while showing that low-conflict and random subsets can both collapse safety.
  • Output-side sharpness mitigation: LoRA and ASAM reduce early safety collapse by limiting updates or favoring flatter regions, but their protection weakens as fine-tuning reaches 5000 samples.At larger scales, LoRA’s ASR rises gradually, SFT and ASAM remain high, and declining MMLU is consistent with catastrophic forgetting.
  • Reversibility of safety alignment: Safety recovery is highly data-efficient: 10 safety examples can reverse a 100-sample attack, while 50 examples suffice after a 5000-sample attack with broadly stable utility.A refusal prefix also reduces ASR without parameter updates, supporting preserved safety-relevant representations and disrupted output routing.

5 Conclusion

The conclusion attributes refusal-safety fragility to a concentrated output-side Fisher-curvature structure that benign fine-tuning can displace with few random samples. It finds that mitigation helps at small scales but cumulative drift eventually overwhelms the routing geometry.

  • Conclusion: Refusal safety depends on a concentrated output-side Fisher-curvature structure that benign fine-tuning can displace with as few as 100 random samples.The resulting safety collapse is disproportionately larger than utility degradation and does not require adversarially selected data.
  • Conclusion: LoRA and ASAM suppress localized output-side sharpness at small data scales but cannot prevent collapse when cumulative drift becomes large.The conclusion points toward protecting the vulnerable routing subspace more deeply during post-training.

Limitations

The study’s limitations concern the scope of alignment methods and model coverage, leaving resilience in other optimization frameworks and larger or different architectures untested.

  • Alignment methods: The study primarily evaluates Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), not reinforcement-learning frameworks such as PPO or iterative preference-optimization variants.Different alignment objectives may produce different resilience to benign fine-tuning.
  • Model coverage: Experiments cover only mid-scale open-weight LLaMA-3.1-8B and Qwen2.5-7B model families.The reported fragility and hypothesized shallow-routing mechanism are not established for ultra-large frontier models or radically different architectures.

A Fisher Geometry

This section defines safety Fisher geometry as a local measure of refusal sensitivity to parameter perturbations, emphasizing curvature, effective dimensionality, and trajectory alignment.

  • Safety-deviation geometry: The safety-deviation loss measures local deviation from the aligned refusal behavior, with zero loss at the aligned checkpoint.Its local expansion uses the safety Fisher as the quadratic curvature term.
  • Safety-deviation geometry: The empirical Fisher is estimated block-wise on harmful prompts paired with refusal targets to proxy refusal sensitivity in each layer-module block.The block-wise quantity bFb(θ) serves as a local geometric proxy for perturbation sensitivity.
  • Directional sharpness: The largest Fisher eigenvalue represents worst-case directional sharpness, while small λmax indicates local flatness in every unit direction.Eigenvalues quantify curvature along orthonormal directions, and λmax captures the most sensitive direction.
  • Safety-relevant subspace: The safety-relevant Fisher subspace captures the directions that contribute to local refusal deviation under parameter perturbations.Its effective dimension is defined using a fixed energy threshold ρ.
  • Safety-relevant subspace: Local refusal deviation depends jointly on the sharpness of leading Fisher directions and the fine-tuning trajectory’s projection onto them.The statement holds with the Taylor remainder included for sufficiently small perturbations.

B Additional Experimental Results

The experimental pipeline uses LlamaFactory for benign attack and recovery experiments after safety alignment.

  • Experimental pipeline: The study performs safety alignment with its fine-tuning pipeline and uses LlamaFactory for benign attack and recovery experiments.The passage identifies the tooling and experimental stages but does not report outcomes.

B.1 Alignment Fragility Extends to DPO

Additional experiments show that benign fine-tuning fragility extends to DPO-aligned models and appears consistently across evaluated architectures, with early safety collapse preceding larger utility degradation.

  • DPO alignment: 0% HEx-PHI ASR before fine-tuning rises to 66.96% after 100 benign Alpaca samples for the DPO-aligned model.Scaling to 5,000 samples produces only a marginal further increase to 67.88%.
  • DPO alignment: The near-identical degradation at 100 and 5,000 samples mirrors the phase-transition behavior reported for SFT-aligned models.The passage interprets this pattern as associated with the alignment surface rather than a particular alignment algorithm.
  • Scaling trends: Across evaluated LLaMA-3.1 and Qwen2.5 variants, localized small-scale fine-tuning sharply increases ASR while largely preserving downstream utility.Larger-scale fine-tuning produces much more pronounced degradation in general capabilities.

B.3 Utility Performance Profiles Surrounding Safety Recovery

Safety recovery produces only minimal changes in absolute utility across model families and alignment depths, preserving capability while restoring safety.

  • The utility-scaling analysis compares ASR and MMLU across model families and alignment settings.
  • Absolute utility fluctuations after safety recovery remain minimal across Llama3.1 and Qwen2.5 models and across Align-256 and Align-10k settings.The evaluation covers MMLU, BoolQ, and ARC-E before and after recovery following a 5,000-sample benign fine-tuning attack.
  • 71.59 to 72.55 on MMLU and 78.91 to 80.09 on ARC-E are the reported Qwen2.5 Align-256 LoRA shifts after recovery.

B.4 Safety Evaluations Across Diverse Benchmarks

Safety alignment improves performance over the Instruct Baseline on Wildchat and StrongReject, but 100 benign fine-tuning samples sharply erode those gains across both benchmarks.

  • The evaluation compares the Instruct Baseline, Align-10k, and Align-10k + FT-100 using dataset-specific safety metrics.Wildchat uses its official safety score, while StrongReject reports 1−ASR.
  • Safety alignment substantially improves performance over the Instruct Baseline on both Wildchat and StrongReject.
  • 100 benign fine-tuning samples bring safety performance closer to baseline levels on both external benchmarks.
  • Consistent degradation across Wildchat and StrongReject indicates that the vulnerability is not specific to HEx-PHI or its evaluation procedure.
Loading 2609.01455v1…