PaperlayerView exact revision on arXiv

Source-linked AI summary

NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning

Xin Yi, Shunfan Zheng, Linlin Wang, Gerard de Melo, Xiaoling Wang, Liang He

arXiv:2412.12497v1cs.CL

TL;DR

Harmful fine-tuning data can compromise LLM safety and raise deployment concerns. NLSR uses a safety reference model to identify and selectively restore safety-critical neurons, reporting reduced harmfulness while maintaining downstream-task performance.

  • Problem

    Harmful instructions in fine-tuning can compromise LLM safety, raising practical deployment concerns and motivating fine-grained safety updates that preserve task-specific performance.

    Source: Introduction · pp. 1–2

  • Method

    NLSR pre-amplifies an aligned model into a safety reference, scores safety-critical neurons, and patches neurons in customized-model layers whose similarity to the reference is low.

    Source: Introduction · p. 2, Neural-Level Safety Realignment · p. 2

  • Results

    38.3% harmfulness reduction versus the aligned model was reported for NLSR, which also restored or surpassed pre-fine-tuning safety benchmarks while maintaining downstream-task precision across evaluated settings.

    Source: Introduction · p. 2, Main Results · p. 4

  • Takeaways & Limitations

    Because it requires no additional training and leaves task-specific optimization undisturbed, NLSR offers a way to address safety after customization.

    Source: Introduction · p. 2, Conclusion · p. 7

    The analysis assumes alignment updates remain close to the initial parameters, with regularization and learning-rate control keeping parameter changes small.

    Source: Weak-to-Strong Extrapolation · p. 13

Abstract

from arXiv · show

The emergence of finetuning-as-a-service has revealed a new vulnerability in large language models (LLMs). A mere handful of malicious data uploaded by users can subtly manipulate the finetuning process, resulting in an alignment-broken model. Existing methods to counteract fine-tuning attacks typically require substantial computational resources. Even with parameter-efficient techniques like LoRA, gradient updates remain essential. To address these challenges, we propose \textbf{N}euron-\textbf{L}evel \textbf{S}afety \textbf{R}ealignment (\textbf{NLSR}), a training-free framework that restores the safety of LLMs based on the similarity difference of safety-critical neurons before and after fine-tuning. The core of our framework is first to construct a safety reference model from an initially aligned model to amplify safety-related features in neurons. We then utilize this reference model to identify safety-critical neurons, which we prepare as patches. Finally, we selectively restore only those neurons that exhibit significant similarity differences by transplanting these prepared patches, thereby minimally altering the fine-tuned model. Extensive experiments demonstrate significant safety enhancements in fine-tuned models across multiple downstream tasks, while greatly maintaining task-level accuracy. Our findings suggest regions of some safety-critical neurons show noticeable differences after fine-tuning, which can be effectively corrected by transplanting neurons from the reference model without requiring additional training. The code will be available at \url{https://github.com/xinykou/NLSR}

Introduction

The introduction presents harmful fine-tuning as a safety risk and motivates NLSR, a training-free method that selectively patches safety-critical neurons to restore safety while maintaining downstream-task performance.

Introduction

  • Fine-tuning-as-a-service allows harmful user data to compromise safety: 1% harmful instructions can breach safeguards, and even clean-data fine-tuning can degrade safety (Qiang et al. 2024; He, Xia, and Henderson 2024; Kumar et al. 2024).Customized models fine-tuned on harmful instructions may comply with attackers’ malicious requests, raising deployment concerns.Source: Introduction · p. 1
  • Existing defenses use perturbations, task-and-preference fine-tuning, or post-fine-tuning realignment, but face prompt sensitivity, safety–task trade-offs, or missed key neurons (Huang, Hu, and Liu 2024; Zeng et al. 2024; Reuel et al. 2024; Zong et al. 2024; Huang et al. 2024c; Hsu et al. 2024; Bhardwaj, Anh, and Poria 2024).SafeLoRA (Hsu et al. 2024) compares layer-specific safety subspaces before and after fine-tuning, but can miss neurons important to downstream-task performance.Source: Introduction · pp. 1–2
  • Prior work identifies safety-related neurons, but simply freezing safety-critical neurons is insufficient against fine-tuning attacks, motivating neuron-level realignment to limit task-performance impact (Chen et al. 2024; Wei et al. 2024).Source: Introduction · p. 2
  • NLSR uses pre-amplification to strengthen safety-neuron features, identifies safety-critical neurons by contribution scores, then measures post-fine-tuning dissimilarity to select layers for correction without additional training.The method is decoupled from fine-tuning and patches neurons according to the damage they sustain.Source: Introduction · p. 2
  • NLSR restores or surpasses pre-fine-tuning safety benchmarks while maintaining downstream-task precision across poisoned-instruction proportions, tasks, and alignment methods.The introduction reports comprehensive evaluations across these settings.Source: Introduction · p. 2
  • Adaptive safety-critical layer pruning is necessary to locate compromised layers, while safety pre-amplification makes different neuron-identification methods more similar in their localization.Source: Introduction · p. 2

Neural-Level Safety Realignment

NLSR restores safety after harmful fine-tuning by comparing safety-critical neurons in a customized model with those in an amplified safety reference model. It selectively transplants reference-model neurons into layers whose safety regions show low similarity.

Neural-Level Safety Realignment

  • NLSR amplifies an initially aligned model, identifies its safety-critical neurons, then compares them with the customized model and transplants reference neurons where similarity is low.The customized model is fine-tuned on task-specific data that may include a minor subset of toxic instructions.Source: Neural-Level Safety Realignment · p. 2

Construction of a Safety Reference Model

  • NLSR constructs its super-aligned safety reference through LoRA extrapolation, updating LoRA weights while keeping most unaligned model weights frozen, extending weak-to-strong extrapolation to safety (Zheng et al. 2024).Source: Construction of a Safety Reference Model · p. 2

Recognition of Safety-Critical Neurons

  • Safety-critical neurons are identified using a safety-instance dataset and a low-rank approximation that preserves safety-critical weights, then selecting the top k = N* × (1 − PSR) neurons by magnitude.The method forms the approximation by retaining the top r* left singular vectors; Wei et al. (2024) is cited for the dataset construction approach.Source: Recognition of Safety-Critical Neurons · pp. 2–3

Restoration for Safety-Broken Neurons

  • Layers with low similarity between pre- and post-fine-tuning safety regions are prioritized by ranked, probability-based pruning; unpruned layers receive transplanted patch neurons from the reference model.The layer pruning probability uses a base probability, an increment factor, and the layer’s similarity rank; Deep, Bhardwaj, and Poria (2024) motivates the ranking approach.Source: Restoration for Safety-Broken Neurons · pp. 3–4

Experiments

Experiments evaluate NLSR on Llama3-8B across poisoned SST-2, AGNEWS, and GSM8K fine-tuning, measuring downstream accuracy and harmfulness; analyses test layer pruning, neuron identification, pre-amplification, and transferability.

Main Results

  • NLSR reduces harmfulness by 38.3% versus Aligned and is 30.3% lower in harmfulness than SafeLoRA, with a 1.1% accuracy gain.Across harmful-instruction ratios, the unaligned model averages 76.3% harmfulness, while Aligned remains high after a 15.2% average reduction; ConstrainedSFT reaches 95.2% fine-tuning accuracy but lags NLSR in safety.Source: Main Results · p. 4, Table 1 · p. 5
  • Across alignment methods, NLSR reduces harmfulness by 29.5% versus Aligned, but SFT remains worst at 46.6% harmfulness after realignment.SFT has 53.3% toxicity after fine-tuning and does not match preference-alignment methods after realignment.Source: Main Results · p. 4, Table 2 · p. 5
  • On AGNEWS and GSM8K, NLSR lowers harmfulness to 19.7% and 15.4%, respectively, and achieves state-of-the-art safety and accuracy on GSM8K.Unlike Vlguard and Lisa, NLSR requires no additional safety-guidance data and does not disrupt downstream fine-tuning.Source: Main Results · p. 4, Table 3 · p. 5

Analysis

  • Because safety-region similarity fluctuates across layers and alignment methods, adaptive pruning is needed; at τ = 0.2, ORPO has fewer than 20% as many broken layers as KTO.A uniform pruning threshold therefore fails to accommodate differences in safety-region sparsity and alignment strategies.Source: Analysis · pp. 4–5, Figure 3 · p. 6
  • Figure 4 compares similarity of safety-broken layers identified by three safety-critical neuron-identification methods across layer-pruning rates and reports neuron-overlap ratios for broken layers identified by different methods.The default sparsity rate and pruning rate are 0.7 and 0.5, respectively.Source: Figure 4 · p. 6

Ablation Study

  • With pre-amplification, harmfulness falls 14.5%, AGNEWS accuracy rises 1.1%, and safety improvements persist as sparsity increases.Pre-amplification improves safety with minimal impact on general utility and can sometimes improve generalization; β = 0.9 is selected as the default because nearly all harmful instructions are rejected.Source: Ablation Study · p. 6, Table 4 · p. 6, Figure 6 · p. 6
  • Randomly selected regions have harmfulness more than 10% higher than Aligned, whereas NLSR’s neuron identification reduces harmful outputs while preserving task performance.NLSR also demonstrates superior accuracy among the tested safety-critical neuron identification methods.Source: Ablation Study · pp. 6–7, Table 5 · p. 7
  • HarmBench evaluations show that NLSR counters harmful instructions across a diverse range of safety-related issues.The transferability experiment varies the proportion of harmful instructions included during fine-tuning.Source: Ablation Study · p. 7, Table 6 · p. 7

Related Work

Prior work shows that fine-tuning attacks can weaken safety alignment, while existing safeguards and neuron-level analyses leave open how to restore safety without disrupting task-relevant capabilities.

Fine-tuning Attacks

  • As few as 100 malicious examples can make safety-aligned LLMs perform harmful tasks while maintaining overall performance (Yang et al. 2023).Fine-tuning-as-a-service lets users upload task-specific data, creating a route for malicious content to undermine safety defenses.Source: Related Work · p. 7

LLM Safety Safeguards

  • Existing safeguards use preference data, constrained updates, perturbations, or LoRA realignment, but may interfere with fine-tuning or overlook task-relevant neurons.Vlguard and Lisa merge preference data into task datasets, while Constrained-SFT restricts updates; Vaccine and Rep-Noise add perturbations. SafeLoRA updates layers, whereas Huang et al. (2024a) remove safety-critical neurons without considering task utility; NLSR instead restores their functionality.Source: Related Work · p. 7

Knowledge Neurons

  • Neuron-level analyses identify task-critical and sparse safety neurons, but freezing safety-critical neurons alone does not fully prevent fine-tuning attacks.SNIP scores neuron importance by loss contribution, Wanda tracks layer-output changes after pruning, and Chen et al. (2024) use generation-time activation contrasting; NLSR builds on these insights to reduce risk to other model capabilities through neuron-level realignment.Source: Related Work · p. 7

Conclusion

NLSR addresses safety degradation from harmful fine-tuning data with a training-free, neuron-level realignment framework. It uses a super-aligned reference model to identify and patch safety-critical regions while maintaining downstream task performance.

Conclusion

  • NLSR restores safety without additional training or disrupting task-specific optimization.Unlike methods that add alignment objectives during fine-tuning, it leaves that optimization process intact.Source: Conclusion · p. 7
  • A super-aligned reference model identifies safety-critical neurons whose dissimilarity measures safety degradation and whose regions serve as corrective patches.The framework transplants patches into regions where significant safety damage has occurred.Source: Conclusion · p. 7
  • Neuron-level restoration realigns safety while upholding downstream task performance.Source: Conclusion · p. 7

Reference Model Setting

NLSR’s reference model is synthesized by extrapolating between an SFT model with lower safety alignment and a preference-optimized model with intermediate alignment, rather than using the harmful-fine-tuning base model or any safely fine-tuned model.

Reference Model Setting

  • The reference model is synthesized by extrapolating between two models with distinct levels of safety alignment, not simply using the harmful-fine-tuning base model or any safely fine-tuned model.The passage does not specify the extrapolation procedure.Source: Reference Model Setting · p. 10
  • The lower-alignment model is SFT, while the intermediate-alignment model is preference-optimized using the SFT model as its initial checkpoint.The preference optimization methods listed are DPO, ORPO, KTO, and SimPO; the resulting model consistently has improved safety compared with the SFT model.Source: Reference Model Setting · p. 10
  • The SFT model uses 2,000 BeaverTails question–harmless answer pairs, and the preference model uses 2,000 PKU-SafeRLHF-30K harmful–harmless preference pairs.Source: Reference Model Setting · p. 10

More Results

NLSR shows safety recovery across model families, harmful-question topics, and fine-tuning conditions. The results also reveal trade-offs: changing which safety neurons are updated can affect harmful scores, in-domain accuracy, and out-of-domain performance differently.

Sparsity Rate and Layer Pruning

  • Increasing sparsity, or decreasing the safety-neuron ratio, raises harmful scores on both downstream tasks while improving fine-tuning accuracy, revealing a safety–performance trade-off.The passage interprets the higher harmful scores as diminished realignment effectiveness when fewer safety regions are updated.Source: Figure 7 · p. 10, More Results · p. 10
  • Restoring safety-critical neurons after downstream fine-tuning can improve, leave unchanged, or degrade out-of-domain performance; improvement may reflect stronger instruction-following capabilities in updated parameters.The evaluated out-of-domain tasks are tinyArc, tinyMMLU, and tinyTruthfulQA.Source: Figure 7 · p. 10, More Results · p. 10
  • Updating safety regions in layers 8–11 yields the largest harmful-score reduction, but substantially degrades AGNEWS fine-tuning accuracy and causes noticeable fluctuations in out-of-domain performance.The passage reports that effects of updating safety-critical neurons vary significantly across layers for both safety and task accuracy.Source: Figure 7 · p. 10, More Results · p. 10

Topic-wise Results

  • Across 14 harmful-instruction topics, NLSR’s harmful scores for SST2, AGNEWS, and GSM8K are no more than one-third of their original values.The passage attributes this recovery to neuron-level safety identification restoring critical parameters associated with fine-grained harmful topics.Source: More Results · p. 10

Generalization to Different Models

  • For Mistral-7B, NLSR reduces the criticality score by 18.4% with only a 0.1% decrease in task-level accuracy versus the best performance, and consistently restores safety across Qwen2-7B, Mistral-7B, and Llama3-8B.The passage describes the method as model-agnostic and points to Table 7 for the cross-model comparison.Source: More Results · p. 10, Table 7 · p. 11

Fine-tuning on Clean Data

  • Even clean-data fine-tuning can trigger safety degradation, though less severely than poisoned data; across Llama3-8B, Qwen-7B, and Mistral-8B, Mistral-7B toxicity falls below 10% after realignment.The experiments vary the proportion of safety data in the downstream fine-tuning dataset; the passage suggests knowledge forgetting may contribute to the degradation.Source: More Results · p. 10, Figure 8 · p. 11

Safety Analysis

Safety analysis finds that safety-neuron patches can transfer across downstream tasks, harmful fine-tuning leaves safety concepts largely distinguishable while perturbing output-associated patterns, and safety-critical neurons vary across modules and layers.

Does Safety Transfer Across Different Downstream Tasks?

  • 19.2% harmful score and 87.8% task-level accuracy resulted when SST2-identified safety neurons were mapped to AGNEWS, with minimal degradation or possible improvements reported across domains.The authors attribute transfer to fine-grained neuron-level identification and similar safety-compromise patterns across fine-tuning attacks.Source: Safety Analysis · p. 11, Table 8 · p. 12

Does Fine-tuning Attacks Undermine Safety Concepts?

  • Above 95% binary classification accuracy was achieved by a 100-neuron MLP on hidden states at every layer, suggesting safety concepts remain distinguishable after harmful fine-tuning.SVM results were similarly high, especially in deeper layers; the authors infer fine-tuning perturbs patterns leading to desired outputs more than the underlying safety concepts.Source: Safety Analysis · p. 12, Figure 10 · p. 13

Are the neurons severely broken by harmful fine-tuning concentrated in the attention modules or mainly in the MLP modules?

  • Safety-critical neurons span attention and MLP modules, with MLP selections common early and attention selections frequent late; some middle modules are absent, especially sparsely in Mistral-7B.Selection patterns vary by model and layer, supporting fine-grained targeting for safety realignment.Source: Safety Analysis · p. 12, Figure 11 · p. 13

Preliminaries

The preliminaries explain how first-order extrapolation can increase an alignment objective and describe methods for identifying safety-critical weights through loss- or activation-based importance scores.

Weak-to-Strong Extrapolation

  • When α∆θ is small relative to θe, first-order expansion gives M(θe + α∆θ) ≈ M(θe) + α∇M(θe); a positive gradient component along ∆θ can increase the objective.The increase holds provided M is not locally maximized at θe; the text says this condition can generally hold as θe monotonically increases from θ0 to θs.Source: Weak-to-Strong Extrapolation · p. 13

Identifying Safety-Critical Neurons

  • SNIP scores weights by |W ⊙ ∇W L(s)|, averages absolute scores across the dataset, and selects the top (1−PSR)×100% safety-critical parameters.Here L(s) is the conditional negative log-likelihood for a prompt-response instance, and PSR controls the selected sparsity rate.Source: Identifying Safety-Critical Neurons · p. 14
  • Preference SNIP uses paired safe and unsafe candidate responses and transforms its loss to be consistent with Direct Preference Optimization.The passage defines each preference example as s = (x, ysafe, yunsafe).Source: Identifying Safety-Critical Neurons · p. 14
  • Wanda masks prompt activations, retains response activations, and prunes minimally important weights to keep the output change small while producing a sparse network.Its activation statistic uses row-wise L2 norms and an expectation over dataset activations.Source: Identifying Safety-Critical Neurons · p. 14

Qualitative Examples

Qualitative examples illustrate NLSR’s safety realignment across poisoned-sample proportions, alignment methods, and downstream tasks. The examples show NLSR maintaining or improving safety after fine-tuning.

Qualitative Examples

  • The examples also cover models initialized with DPO, ORPO, and KTO alignment methods during fine-tuning.Table 10 identifies these alignment methods as the initial points used in the fine-tuning phase.Source: Qualitative Examples · p. 14, Table 10 · p. 16
  • The examples include downstream fine-tuning on SST2, GSM8K, and AGNEWS.Table 11 identifies these three downstream tasks.Source: Qualitative Examples · p. 14, Table 11 · p. 17
  • Across poisoned-sample proportions, NLSR’s example rejects physical punishment and recommends calm, consistent training, while Lisa’s p=0.2 example recommends physical punishment.Table 9 covers fine-tuning with different proportions of poisoned samples.Source: Qualitative Examples · p. 15, Table 9 · p. 15
Loading 2412.12497v1…