PaperlayerView exact revision on arXiv

Source-linked AI summary

Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models

Di Wu, Xin Lu, Yanyan Zhao, Bing Qin

arXiv:2412.11041v3cs.CL

TL;DR

Fine-tuning can compromise the safety alignment of LLMs, while uniform safety modifications may affect parameters important to downstream tasks. IRR identifies and removes unsafe delta parameters, then recalibrates retained ones; evaluations report improved safety with downstream performance maintained.

  • Problem

    Fine-tuning can compromise LLM safety alignment, while uniform safety modifications may harm parameters critical to downstream performance.

    Source: §1 · p. 1

  • Method

    IRR identifies and removes unsafe delta parameters, then recalibrates retained parameters using weight compensation based on the inverse Hessian.

    Source: §1 · p. 2

  • Results

    IRR improves model safety while preserving downstream task performance across fine-tuning methods, datasets, and models, achieving Pareto improvements.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    IRR offers a post-hoc route to safety realignment that targets unsafe updates while maintaining downstream task performance.

    Source: §1 · p. 2, §6 · p. 9

    The safety-vector and parameter safety-importance estimation methods are relatively simple, and more robust and precise methods are needed.

    Source: §7 · p. 9

Abstract

from arXiv · show

Although large language models (LLMs) achieve effective safety alignment at the time of release, they still face various safety challenges. A key issue is that fine-tuning often compromises the safety alignment of LLMs. To address this issue, we propose a method named IRR (Identify, Remove, and Recalibrate for Safety Realignment) that performs safety realignment for LLMs. The core of IRR is to identify and remove unsafe delta parameters from the fine-tuned models, while recalibrating the retained ones. We evaluate the effectiveness of IRR across various datasets, including both full fine-tuning and LoRA methods. Our results demonstrate that IRR significantly enhances the safety performance of fine-tuned models on safety benchmarks, such as harmful queries and jailbreak attacks, while maintaining their performance on downstream tasks. The source code is available at: https://anonymous.4open.science/r/IRR-BD4F.

1 Introduction

Fine-tuning can compromise LLM safety, creating a need to realign models without sacrificing downstream performance. IRR selectively removes unsafe parameter changes and recalibrates retained ones; experiments report improved safety with downstream performance preserved.

1 Introduction

  • Fine-tuning can compromise safety alignment even with entirely benign data, motivating realignment that maintains downstream task performance.The introduction also describes training data that mixes harmful and benign examples as a risk.Source: §1 · p. 1
  • Existing safety modifications can overlook parameter-specific safety and task relevance, potentially harming parameters critical to downstream performance.The proposed separation and recalibration aim to balance safety with downstream performance.Source: §1 · pp. 1–2
  • IRR identifies unsafe delta parameters using safety interference and importance, removes them, and recalibrates retained parameters using inverse-Hessian-based weight compensation.The safety vector represents changes that move the model from an unsafe state to a safe state.Source: §1 · p. 2
  • IRR improves model safety while preserving downstream task performance across fine-tuning methods and datasets, achieving Pareto improvements.The evaluations include both full fine-tuning and LoRA.Source: §1 · p. 2

2 Related Work

Related work describes fine-tuning vulnerabilities and safety interventions across several stages, alongside methods for pruning or rescaling parameter updates. IRR is positioned as a post-hoc approach that targets the safety and task relevance of delta parameters.

2 Related Work

  • Prior studies report that fine-tuning, including with small amounts of harmful data or benign data, can undermine LLM safety alignment.The cited work also examines identity-shift and Alpaca data.Source: §2 · p. 2
  • Safety interventions span preprocessing, fine-tuning, post-hoc, and inference-time methods, with post-hoc examples including safety-vector merging and model pruning.The passage also describes safety-vector patching, Safe LoRA, and safety sub-networks.Source: §2 · p. 2
  • Post-hoc approaches avoid intervention during inference, and IRR identifies and removes unsafe deltas before compensating retained parameters.The paper focuses on this phase to avoid additional fine-tuning and allow flexible safety–performance trade-offs.Source: §2 · p. 2, Figure 2 · p. 3
  • Prior delta-parameter work prunes, randomly drops and rescales, or scores updates for significance and sensitivity; IRR instead considers safety alongside downstream performance.The cited examples include sub-network search, DARE, and significance and sensitivity metrics.Source: §2 · pp. 2–3
  • Unlike general model pruning, which removes unnecessary model parameters, IRR specifically removes delta parameters from fine-tuned models.The passage describes parameter-importance methods including Fisher-matrix scores, SparseGPT, and Wanda.Source: §2 · p. 3

3 Approach

IRR realigns fine-tuned models by identifying unsafe parameter changes, removing them, and recalibrating retained parameters. Its identification combines safety interference with safety importance.

3 Approach

  • IRR identifies, removes, and recalibrates delta parameters to restore safety while retaining downstream task performance.The method marks interfering parameters as unsafe, reverts them to the pre-trained safety-aligned state, and compensates retained parameters.Source: §3 · p. 3

3.1 Identify the Unsafe Delta Parameters

  • Delta parameters whose signs disagree with the safety vector are candidates for removal when they affect parameters important to safety alignment.Safety importance is estimated with the Fisher matrix; the method selects the top ρ% by importance from the interference candidates.Source: §3.1 · p. 4

3.2 Remove the Unsafe Delta Parameters

  • A binary mask removes identified unsafe delta parameters by reverting them to the pre-trained safety-aligned parameters.The remaining delta parameters are retained.Source: §3.2 · p. 4

3.3 Recalibrate the Retained Parameters

  • Because removing unsafe parameters may harm task performance, IRR adds compensatory values to retained parameters to maintain downstream performance.The compensation is based on analysis of parameter-change effects on loss and the perturbation needed for remaining parameters.Source: §3.3 · p. 4

4 Experimental Setup

The experiments evaluate full fine-tuning and LoRA, with Llama models fine-tuned on task datasets. Safety is assessed on harmful-query and jailbreak benchmarks alongside downstream capability.

4 Experimental Setup

  • Experiments cover full fine-tuning and LoRA, using Llama-2-7b-chat for Math, Code, and Chinese tasks and Llama-3-8B-Instruct for Math.Full fine-tuning results appear in the main text, while LoRA results are in Appendix B.Source: §4 · p. 5
  • IRR and IRRd are compared with SFT, DARE, Safe LoRA, SafeDecoding, RESTA, and RESTAd baselines.IRRd and RESTAd apply their respective methods with DARE; IRR and RESTA apply them without DARE.Source: §4 · p. 5

Computing Safety Vectors and Fisher Matrix

  • Safety evaluation uses harmful-query datasets and Salad-Attack jailbreaks, with Safety Score measuring the share of responses judged harmless.Harmful-query tests include CATQA, HEx-PHI, and Salad-Base; Salad-Attack simulates varied attack attempts.Source: Computing Safety Vectors and Fisher… · pp. 5–6
  • Downstream performance is evaluated on GSM8K, HumanEval, and Chinese MMMLU.For Llama-3, evaluations cover various mathematical tasks.Source: Computing Safety Vectors and Fisher… · p. 5

5 Results and Discussions

Across the evaluated settings, IRR improves safety while better preserving downstream performance than baseline methods. The results examine trade-offs, component ablations, language and model coverage, safety-vector choice, and harmful-data fine-tuning.

5 Results and Discussions

  • IRR improves safety with downstream performance almost unchanged; Safe LoRA Math accuracy fell from 43.06 to 22.61, while RESTA accuracy drops were Math 1.13, Code 4.14, and Chinese 3.82.DARE random dropout and scaling did not significantly improve safety, including when combined with RESTA or IRR.Source: §5 · p. 6

5.1 IRR Achieves Pareto Improvement

  • IRR and IRRd stay at the safety frontier as RESTA and RESTAd lose performance with rising safety, especially on Code and Chinese tasks.The trade-off curves cover the Harmful Benchmark and jailbreak attacks; LoRA experiments followed a similar trend.Source: §5.1 · pp. 6–7, Figure 3 · p. 7, Figure 4 · p. 7

5.2 Ablation Study

  • Random removal or safety-importance-only identification degrades downstream performance; omitting recalibration also degrades it, but relatively slightly.The ablation curves compare downstream task performance against safety.Source: §5.2 · p. 7, Figure 5 · p. 8

5.3 Cross-Language Safety Improvement

  • SFT models have lower safety scores in Chinese and Vietnamese than English, while increasing IRR mask ratios improves harmful-query safety toward the original model’s level.The evaluation uses English, Chinese, and Vietnamese versions of CATQA.Source: Figure 6 · p. 8, §5.3 · p. 8

5.4 Efficacy of IRR Across Models

  • For LoRA-fine-tuned Llama-3-8B-Instruct on MathInstruct, IRR improves safety while maintaining downstream mathematical-task performance.The evaluation compares IRR and baseline methods on safety and mathematical capabilities.Source: Figure 7 · p. 8, §5.4 · p. 8

5.5 IRR with New Safety Vector

  • With a new safety vector trained from 1,000 Beavertails pairs, IRR retains better safety and downstream performance than RESTA on GSM8K-fine-tuned models.The new vector is trained using LoRA; the evaluated models are also LoRA-fine-tuned on GSM8K.Source: §5.5 · p. 8, Figure 8 · p. 9

5.6 IRR against Harmful Fine-tuning

  • Under harmful fine-tuning that mixes Beavertails harmful pairs with GSM8K, IRR improves safety while maintaining higher downstream performance than RESTA.This setting uses LoRA fine-tuning.Source: §5.6 · p. 9

6 Conclusion

The conclusion presents IRR as a safety realignment method that removes unsafe delta parameters and recalibrates those retained. Across fine-tuning methods, datasets, and models, it reports improved safety while maintaining downstream task performance.

6 Conclusion

  • IRR identifies and removes unsafe delta parameters, recalibrates retained ones, and outperforms baselines in safety while maintaining downstream performance across evaluated methods, datasets, and models.The conclusion reports that evaluations on harmful-query benchmarks and jailbreak attacks indicate reduced risks from fine-tuned models.Source: §6 · p. 9

7 Limitations

The authors identify untested multimodal settings and the simplicity of IRR’s safety-vector and parameter-importance methods as limitations. They describe further robustness and precision work as necessary.

7 Limitations

  • The authors acknowledge limitations and future research directions while describing the work as a contribution to safety alignment.Source: §7 · p. 9
  • The study did not experiment on multimodal models because of budget constraints.The authors leave safety assessments for images, speech, and other modalities to future work.Source: §7 · p. 9
  • IRR’s safety-vector and parameter-importance methods are relatively simple; the authors say more robust and precise methods are needed.They identify improving these steps as a focus for future investigations.Source: §7 · p. 9

8 Ethical consideration

The ethical-consideration section presents IRR as a way to reduce harmful content in fine-tuned language models while retaining downstream task performance. It calls for continued collaboration and evaluation around AI safety.

8 Ethical consideration

  • IRR reduces harmful responses through identification and removal, while calibration preserves strong downstream task performance.Source: §8 · p. 9
  • The authors advocate ongoing collaboration among researchers, policymakers, and industry stakeholders to prioritize human values, fairness, and safety.Source: §8 · p. 9
  • The authors commit to continuously evaluating and improving IRR to address ethical challenges.Source: §8 · p. 9

9 Potential Risks

The potential-risks section warns that compromised safety in fine-tuned models can threaten downstream users and that IRR’s demonstrated gains do not establish complete safety. Experimental details specify task-specific settings and evaluation procedures.

9 Potential Risks

  • Fine-tuned models with compromised safety may pose threats to users relying on them for downstream tasks.Source: §9 · p. 10
  • IRR’s safety improvements are demonstrated only on the evaluations presented; the authors recommend deployment caution and ongoing safety checks.Source: §9 · p. 10

A Experimental Details

  • For Math, Code, and Chinese, the mask ratios were 60%, 10%, and 80%, respectively; the sweep ran from 10% to 100% in 10% steps.Recalibration used 1,000 downstream-task samples; experiments used greedy decoding and four A100 80GB GPUs.Source: A Experimental Details · pp. 12–13
  • Full fine-tuning used Llama-2-7B-chat with learning rate 2.0 × 10−5, batch size 64, and three epochs, except one Chinese-data epoch.Source: A Experimental Details · p. 13
  • LoRA fine-tuning adjusted Llama-2-7B-chat query and value matrices at rank 8, with learning rate 2.5 × 10−4, batch size 64, and five epochs.Llama-2 training and inference used the default system prompt and chat template.Source: A Experimental Details · p. 13

B LoRA fine-tuning Experimental

Across LoRA experiments, IRR achieves Pareto improvements in safety and downstream task performance on harmful-query and jailbreak evaluations. The section also reports multilingual, new-safety-vector, ablation, complexity, and timing analyses.

C Multilingual Safety

  • Safety scores improve as the masking ratio increases for CodeAlpaca-20k and Alpaca-Chinese models evaluated with English, Chinese, and Vietnamese CATQA.Source: C Multilingual Safety · p. 13

D Ablation Experiments

  • The ablation study presents a downstream-performance versus safety trade-off curve on the harmful benchmark (Figure 13).Source: D Ablation Experiments · p. 13

E Details about Llama-3-8B-Instruct experiment

  • The Llama-3-8B-Instruct experiment used rank-8 LoRA on MathInstruct and evaluated mathematical ability across seven few-shot benchmarks.Training and inference used the chat template without a system prompt.Source: E Details about Llama-3-8B-Instruct… · p. 13

F IRR with New Safety Vector

G Computational Complexity of IRR

  • The authors use SparseGPT to compute the inverse Hessian needed for OBS and report improved efficiency over exact reconstruction methods.They describe the approach as computationally practical even for very large models.Source: G Computational Complexity of IRR · pp. 13–14

B LoRA fine-tuning Experimental

H Time Consumption of IRR

H.1 Time for the IRR Method

The appendix reports IRR’s calibration-data/runtime trade-off and estimates its runtime on larger models. It also describes layerwise Hessian processing as a way to realign models with limited GPU memory.

H.1 Time for the IRR Method

  • 67.57 seconds let IRR realign a LoRA fine-tuned 7B model using 64 calibration samples, versus at least 1.55 hours for five-epoch GSM8K LoRA fine-tuning.Smaller calibration sets reduced runtime without significantly affecting downstream task performance or safety scores, and IRR adds no inference time.Source: Table 3 · p. 15, §H.1 · pp. 15–16
  • The evaluation varied calibration data from 8 to 1,000 samples and mask ratios from 10% to 100%, averaging runtime, task accuracy, safety, and jailbreak-safety scores.The study used GSM8K with full and LoRA fine-tuned Llama-2-7b-chat models.Source: §H.1 · pp. 14–15

H.2 Estimating Time Efficiency for Scaling to Larger Models

  • 404.8 seconds for the full fine-tuned 40-layer Llama2-13B, compared with approximately 2,486.4 seconds for the 80-layer 70B model.The estimates use 64 downstream-task samples to calculate the Hessian and measure runtime for one model layer.Source: §H.2 · p. 16

I Limited GPU Memory Resources

  • Layerwise inverse-Hessian computation loads only the current layer into GPU memory, theoretically supporting realignment under limited GPU-memory resources.After computing a layer’s Hessian, the method proceeds to the next layer.Source: I Limited GPU Memory Resources · p. 16
Loading 2412.11041v3…