PaperlayerView exact revision on arXiv

Source-linked AI summary

Locking Down the Finetuned LLMs Safety

Minjun Zhu, Linyi Yang, Yifan Wei, Ningyu Zhang, Yue Zhang

arXiv:2410.10343v1cs.CL

TL;DR

Fine-tuning can compromise aligned LLMs’ safety, even with few harmful examples. SafetyLock extracts safety directions from an original model and transfers them to fine-tuned variants. Experiments report a reduction in ASR from 54.24% to 0.03% in fine-tuned language models, with low reported costs to normal response rates.

  • Problem

    Fine-tuning can compromise an aligned model’s safety, even with only a few adversarially designed examples, creating a challenge for maintaining safety after customization.

    Source: §1 · pp. 1–2

  • Method

    SafetyLock extracts safety directions from an original model and applies them to fine-tuned variants, using safety-related attention heads.

    Source: §1 · p. 2

  • Results

    54.24% to 0.03% ASR in fine-tuned language models; the reported normal response rate changes from 99.4% to 98.1%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments report SafetyLock’s performance across fine-tuning vulnerabilities and prompt-based attacks, balancing safety improvements with normal text processing.

    Source: §5 · p. 10

    SafetyLock requires access to model weights and intermediate activations, and its long-term robustness against evolving adversarial techniques remains unstudied.

    Source: §A.1 · p. 19

Abstract

from arXiv · show

Fine-tuning large language models (LLMs) on additional datasets is often necessary to optimize them for specific downstream tasks. However, existing safety alignment measures, which restrict harmful behavior during inference, are insufficient to mitigate safety risks during fine-tuning. Alarmingly, fine-tuning with just 10 toxic sentences can make models comply with harmful instructions. We introduce SafetyLock, a novel alignment intervention method that maintains robust safety post-fine-tuning through efficient and transferable mechanisms. SafetyLock leverages our discovery that fine-tuned models retain similar safety-related activation representations to their base models. This insight enables us to extract what we term the Meta-SafetyLock, a set of safety bias directions representing key activation patterns associated with safe responses in the original model. We can then apply these directions universally to fine-tuned models to enhance their safety. By searching for activation directions across multiple token dimensions, SafetyLock achieves enhanced robustness and transferability. SafetyLock re-aligns fine-tuned models in under 0.01 seconds without additional computational cost. Our experiments demonstrate that SafetyLock can reduce the harmful instruction response rate from 60% to below 1% in toxic fine-tuned models. It surpasses traditional methods in both performance and efficiency, offering a scalable, non-invasive solution for ensuring the safety of customized LLMs. Our analysis across various fine-tuning scenarios confirms SafetyLock's robustness, advocating its integration into safety protocols for aligned LLMs. The code is released at https://github.com/zhu-minjun/SafetyLock.

1 INTRODUCTION

Fine-tuning can undermine safety alignment, even with only a few harmful examples, while existing approaches face costs or trade-offs in quality and benign responses. SafetyLock transfers safety directions from an original model to fine-tuned variants; experiments report lower attack success rates with little change in normal response rates.

1 INTRODUCTION

  • Fine-tuning can compromise alignment with only a few adversarial examples, including 10 examples used to jailbreak GPT-3.5 Turbo for less than $0.20.The vulnerability is reported for both open-source models such as Llama and proprietary models such as GPT-4.Source: §1 · pp. 1–2
  • Existing alternatives trade off computational cost, generation quality, benign-query refusals, or vulnerability to adversarial attacks, while fine-tuned models may vastly outnumber base models.The introduction frames scalable restoration without negative transfer to general tasks as the central challenge.Source: §1 · p. 2
  • SafetyLock derives Meta-SafetyLock directions from an original model and transfers them to fine-tuned variants, targeting safety-related attention heads.The approach uses safety probes to identify heads associated with harmfulness and is intended to minimize effects on general abilities.Source: §1 · p. 2
  • 54.24% to 0.03% ASR in fine-tuned language models; DeepInception ASR falls from 98% to 2%, while normal response rate changes from 99.4% to 98.1%.The results are reported alongside transfer to multiple model families and minimal time cost.Source: §1 · p. 2

2 RELATED WORK

Prior alignment and safeguard approaches do not reliably preserve safety after fine-tuning. The paper situates SafetyLock among methods that intervene in model activations and target attention heads.

2 RELATED WORK

  • Alignment methods such as instruction tuning, PPO, and DPO often fail to maintain robustness after fine-tuning on new datasets.The paper identifies parameter changes introduced during fine-tuning as the setting that motivates more robust alignment.Source: §2 · pp. 2–3
  • Prompt-based safeguards can be compromised by simple fine-tuning, while post-processing and model editing have limitations in how they modify or locate safety information.The related work notes that just 10 harmful examples undermined GPT-3.5-turbo alignment; PPO and DPO adjust the activation space broadly, while model editing can miss dispersed safety information.Source: §2 · p. 3
  • Inference-time activation interventions can support alignment, and research links attention heads to specific concepts and preferences.SafetyLock builds on this work with multi-token-level interventions using activation values from the original model to address parameter changes after fine-tuning.Source: §2 · p. 3

3 METHOD: SAFETYLOCK

SafetyLock extracts safety directions from an original model and transfers them to fine-tuned variants. The method relies on similar safety-related activation patterns across those models and distributes the directions through online intervention or offline bias editing.

3 METHOD: SAFETYLOCK

  • SafetyLock has two phases: extracting Meta-SafetyLock from the original model and distributing it to fine-tuned models, with directions searched across multiple tokens to improve transferability.The method is motivated by similar safety activation representations and intervention directions in original and fine-tuned models.Source: §3 · pp. 3–4

3.1 SAFETY FEATURES IN TRANSFORMER

  • The method places preference vectors in the Transformer residual stream, where layer activations are organized across attention heads.The model processes tokens through layers comprising multi-head attention and multi-layer perceptrons.Source: §3.1 · p. 4

3.2 CONSTRUCTING META-SAFETYLOCK

  • For each attention head, logistic regression classifiers distinguish safe from unsafe response activations; validation accuracy selects heads, and mean activation differences form safety vectors for intervention.The Meta-SafetyLock consists of these vectors and their model positions.Source: §3.2 · p. 4

3.3 ROBUSTNESS OF SAFETYLOCK AGAINST FINE-TUNNING

  • KL divergence between original and fine-tuned safety activations stays below 10^-5 across tested risk levels, supporting transfer without recalibration.The compared safety directions also show similar shifts from unsafe to safe activations across risk settings.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

3.4 DISTRIBUTING SAFETYLOCK

  • SafetyLock can be distributed through online intervention or offline bias editing; online intervention adds a scaled safety vector to selected heads, with α controlling intensity.The online option supports real-time adjustment, while offline editing modifies model bias terms in a one-time operation.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments evaluate SafetyLock across fine-tuning risk levels, baseline defenses, combined attacks, and downstream tasks. Across these settings, the reported results show reduced harmful outputs while preserving benign-task performance.

4.1 EXPERIMENTAL DETAILS

  • The experiments test three fine-tuning risks using Llama-3-8B Chat, Llama-3-70B Chat, and Mistral-Large-2 123B, with harmful, implicitly harmful, and benign datasets.Safety evaluation uses HEx-PHI for harmfulness scores and rates and AdvBench for ASR; the baseline set includes inference-time and training-based methods.Source: §4.1 · p. 6, §4 · p. 6

4.2 RESULTS OVER DIFFERENT RISK LEVELS

  • SafetyLock reduces harmfulness metrics across three risk levels and model sizes; for Llama-3-8B-Instruct in Risk 1, Harmfulness Score falls from 4.13 to 1.36, Rate from 70.01% to 3.33%, and ASR from 49.24% to 0.19%.The reported Harmfulness Score also falls from 3.19 to 1.07 in Risk 2 and from 3.23 to 1.04 in Risk 3.Source: §4.2 · pp. 6–8, Table 1 · p. 7, Figure 3 · p. 7

4.3 COMPARATIVE ANALYSIS OF BASELINE METHODS

  • SafetyLock combines low resource use with strong safety results: it records 0.97 seconds inference time, 0.01 seconds training time, 0.0 GB additional GPU memory, and 0.19% AdvBench ASR.Its Harmfulness Score of 1.04 matches DPO; SafetyLock preserves a 98.1% normal response rate, compared with 99.4% for Vanilla.Source: Figure 5 · p. 8, §4.3 · pp. 8–9

4.2 RESULTS OVER DIFFERENT RISK LEVELS

  • At 10,000 harmful training examples, the SafetyLock model has 3.46% ASR, compared with 62.31% for the unprotected model.SafetyLock-protected models are reported to maintain low ASR across 100, 1,000, and 10,000 harmful samples.Source: Figure 4 · p. 7, §4.2 · p. 8

4.4 SAFETYLOCK’S PERFORMANCE AGAINST COMBINED ATTACKS

  • SafetyLock reduces ASR across four combined prompt attacks: 4.0% for AutoDAN, 2.0% for DeepInception, 10.0% for GCG, and 14.0% for PAIR.For GCG, PPL achieves 0.0% ASR; SafetyLock’s results are reported as outperforming other tested methods on PAIR.Source: Table 2 · p. 9, §4.4 · pp. 9–10

4.5 GENERALIZATION CAPABILITIES OF SAFETYLOCK

  • SafetyLock maintains downstream performance across evaluated tasks; on AddSub it scores 85.57%, close to the original model’s 86.33%, while Model-Edited collapses.The evaluated benchmarks include AddSub, AQUA, CommonSenseQA, GSM8k, MT-Bench, Alpaca, and AlpacaEval 2.0.Source: Figure 6 · p. 10, §4.5 · p. 10

5 CONCLUSION

SafetyLock is presented as an efficient method for maintaining fine-tuned LLM safety across risk levels and attack scenarios. Experiments report strong defenses against fine-tuning vulnerabilities and prompt-based attacks while balancing safety, efficiency, and normal text processing.

5 CONCLUSION

  • SafetyLock demonstrates robust defense against fine-tuning vulnerabilities and prompt-based attacks across risk levels and attack scenarios.The authors describe it as outperforming training-based and inference-time methods while balancing efficiency, attack-sample rejection, and normal-text processing.Source: §5 · p. 10
  • The authors identify applicability to other model architectures and multimodal settings as future work.They also describe the method's minimal computational overhead and safety improvements as promising for responsible AI deployment.Source: §5 · p. 10
  • SafetyLock is proposed as a scalable approach for aligning fine-tuned language models while preserving utility across applications.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points readers to the paper's experimental details and says that code and materials are publicly available.

REPRODUCIBILITY STATEMENT

  • Implementation details, datasets, models, settings, metrics, hyperparameters, training code, and baselines are described in the relevant sections.The statement specifically directs readers to Sections 3.2, 3.3, 3.4, 4.2, and 4.3.Source: REPRODUCIBILITY STATEMENT · p. 11
  • The authors state that publicly available datasets and models can be used to replicate the work.Source: REPRODUCIBILITY STATEMENT · p. 11
  • The statement says experimental settings and evaluation metrics are provided in Sections 3.3 and 4.3.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 LIMITATIONS

The authors note that SafetyLock has implementation-access, reversibility, and long-term robustness limitations.

A.1 LIMITATIONS

  • Implementation requires access to model weights and intermediate activations, limiting use when either is unavailable.Source: §A.1 · p. 19
  • The symmetric locking mechanism could potentially be reverse-engineered to bypass protection if an unauthorized party obtains the weights or activation values.Source: §A.1 · p. 19
  • Long-term robustness against evolving adversarial techniques remains unstudied.Source: §A.1 · p. 19

A.2 CONSISTENCY OF HARMLESSNESS DIRECTIONS IN FINE-TUNED MODELS

This analysis examines whether safety directions persist across fine-tuning and how activation interventions could suppress harmful outputs. It reports similar safety-direction shifts across risk levels and describes assumptions and mechanisms underlying transfer to fine-tuned models.

A.2 CONSISTENCY OF HARMLESSNESS DIRECTIONS IN FINE-TUNED MODELS

  • KL divergence indicates high similarity between original and fine-tuned safety directions across risk levels, while weight interpolation traces their change during fine-tuning.The analysis visualizes safe-versus-unsafe activation shifts using last-token activations from the 31st layer, 26th head and 2D PCA; arrows connect the sample-center points.Source: §A.2 · pp. 19–20

B MATHEMATICAL EXPLANATION OF SAFETYLOCK’S EFFECTIVENESS IN SUPPRESSING HARMFUL OUTPUTS

  • The mathematical explanation aims to justify extracting safety directions from the original model and applying them to fine-tuned models to suppress harmful outputs.Source: B MATHEMATICAL EXPLANATION OF… · p. 20

B.1 ACTIVATION SPACE AND SAFETY DIRECTIONS

  • A safety direction is the mean difference between safe- and unsafe-response activations and represents the average shift toward safe responses.Source: §B.1 · pp. 20–21

B.2 PRESERVATION OF SAFETY DIRECTIONS DURING FINE-TUNING

  • Fine-tuned activations have a strong linear relationship with original activations, supporting the original safety direction when safety-critical dimensions are not disproportionately affected.The text characterizes fine-tuning changes as relatively small in many dimensions and states this preservation condition as an assumption.Source: §B.2 · p. 21

B.3 EFFECTIVENESS OF ACTIVATION INTERVENTION

  • During inference, activation adjustment along the safety direction shifts representations toward regions associated with safe responses.The scaling factor controls intervention intensity; the standard-deviation vector captures typical activation variability.Source: §B.3 · p. 21

B.4 IMPACT ON OUTPUT PROBABILITIES

  • Adjusting final activations influences logits before softmax, biasing them toward tokens more likely in safe responses and away from unsafe responses.Source: §B.4 · p. 22

B.5 SUPPRESSING HARMFUL OUTPUTS

  • Lowering harmful-token logits relative to other logits reduces the probability of generating harmful tokens.Source: §B.5 · p. 22

B.6 TRANSFERABILITY ACROSS MODELS

Fine-tuned models retain safety-related activation structure similar to their base models, allowing the base models’ safety directions to remain effective after fine-tuning.

B.6 TRANSFERABILITY ACROSS MODELS

  • Safety directions computed from the original model remain effective on fine-tuned models because safe and unsafe activations retain similar relative positions.The passage attributes this transferability to fine-tuning not significantly altering those positions.Source: §B.6 · p. 22
  • Low KL divergence between original and fine-tuned activation distributions supports preservation of the activation structure relevant to safety.The reported divergence is described as empirical support for the preserved structure.Source: §B.6 · p. 22
  • The preserved activation structure includes dimensions relevant to safety, not only the overall activation distributions.The passage identifies safety-relevant dimensions as especially preserved during fine-tuning.Source: §B.6 · p. 22

B.7 CONCLUSION

SafetyLock uses preserved activation-space safety directions to adjust internal computations toward safe outputs. This provides a basis for suppressing harmful responses without retraining while maintaining performance on benign tasks.

B.7 CONCLUSION

  • SafetyLock intervenes along preserved safety directions to adjust internal computations toward generating safe outputs.The intervention uses directions retained in activation space after fine-tuning.Source: §B.7 · p. 22
  • SafetyLock suppresses harmful responses without requiring model retraining or fine-tuning.The passage links continued applicability of the directions to minimal changes in activation distributions during fine-tuning.Source: §B.7 · p. 22
  • The theoretical explanation provides a foundation for suppressing harmful outputs while maintaining overall performance on benign tasks.Source: §B.7 · p. 22

C THE RISKS OF FINE-TUNING LLMS AND EXPERIMENTAL SETUP

The experimental setup evaluates SafetyLock across three fine-tuning risk types and compares training-based and inference-time safety methods. It specifies dataset construction, comparison metrics, and training configurations, including that one safety-data approach still requires retraining already fine-tuned models.

C THE RISKS OF FINE-TUNING LLMS AND EXPERIMENTAL SETUP

  • HEx-PHI contains 330 examples across 11 prohibited-use categories, with 30 examples per category, for safety evaluation.Its categories combine Meta’s Llama-3 acceptable use policy and OpenAI usage policies.Source: C THE RISKS OF FINE-TUNING LLMS AND… · p. 22
  • Risk-1 uses 10, 100, 1,000, or 10,000 HH-RLHF negative samples, trained for five epochs at a 2e-5 learning rate.Source: C THE RISKS OF FINE-TUNING LLMS AND… · p. 22
  • Risk-2 uses 10 Qi et al. (2023b) samples and Risk-3 uses 50,000 Alpaca samples, each trained for five epochs at a 2e-5 learning rate.Source: C THE RISKS OF FINE-TUNING LLMS AND… · p. 23
  • Comparisons span training-based and inference-time methods, assessed by efficiency, attack-sample rejection, and normal-text rejection.The training-based set includes PPO, DPO, SFT, and model-editing; inference-time methods include ICD, PPL, paraphrase, retokenization, Safe-Reminder, and Self-Exam.Source: C THE RISKS OF FINE-TUNING LLMS AND… · p. 23
  • SFT uses 2,000 safety examples either during or after fine-tuning, but the after-training setting still requires retraining already fine-tuned models.PPO also uses 2,000 samples and LlamaGuard-7b as reward model; DPO uses harmful outputs as negative samples, and model-editing uses DINM with SafeEdit data.Source: C THE RISKS OF FINE-TUNING LLMS AND… · p. 23

D ANALYSIS OF SAFETYLOCK’S INTERVENTION

SafetyLock’s intervention settings matter: moderate intervention distances and relatively low intervention degrees improve the reported safety metrics, while excessive intervention can worsen them. The reported trends identify useful parameter ranges for both Llama-3 model sizes.

D ANALYSIS OF SAFETYLOCK’S INTERVENTION

Loading 2410.10343v1…