PaperlayerView exact revision on arXiv

Source-linked AI summary

Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation

Guozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo, Qi Mu, Li Shen

arXiv:2410.09760v3cs.LG

TL;DR

Harmful fine-tuning can compromise aligned LLMs, and existing alignment-stage defenses are described as fragile and memory-intensive. T-Vaccine perturbs selected safety-critical layers using harmful gradient norms to guide selection. It reports lower harmful scores and memory use than Vaccine, TAR, and RepNoise, including on 7B models using consumer GPUs.

  • Problem

    Harmful fine-tuning can break aligned LLM safety, while existing alignment-stage defenses are described as fragile and memory-intensive.

    Source: §1 · pp. 1–2

  • Method

    T-Vaccine uses harmful gradient norms to identify safety-critical layers, perturbing selected layers and freezing the rest during training.

    Source: §1 · p. 2

  • Results

    Harmful scores fall by up to 8.19%, 4.31%, and 11.34% versus Vaccine, TAR, and RepNoise, respectively; memory use falls by 31.53%, 48.46%, and 48.21%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    T-Vaccine is reported as the first defense to address harmful fine-tuning for 7B models on consumer GPUs with 24GB of memory, such as an RTX 4090.

    Source: §1 · p. 2

    T-Vaccine is built solely on SFT, which may limit generalizability, and has not been tested in multimodal scenarios.

    Source: §7 · p. 9

Abstract

from arXiv · show

Harmful fine-tuning attack poses a serious threat to the online fine-tuning service. Vaccine, a recent alignment-stage defense, applies uniform perturbation to all layers of embedding to make the model robust to the simulated embedding drift. However, applying layer-wise uniform perturbation may lead to excess perturbations for some particular safety-irrelevant layers, resulting in defense performance degradation and unnecessary memory consumption. To address this limitation, we propose Targeted Vaccine (T-Vaccine), a memory-efficient safety alignment method that applies perturbation to only selected layers of the model. T-Vaccine follows two core steps: First, it uses gradient norm as a statistical metric to identify the safety-critical layers. Second, instead of applying uniform perturbation across all layers, T-Vaccine only applies perturbation to the safety-critical layers while keeping other layers frozen during training. Results show that T-Vaccine outperforms Vaccine in terms of both defense effectiveness and resource efficiency. Comparison with other defense baselines, e.g., RepNoise and TAR also demonstrate the superiority of T-Vaccine. Notably, T-Vaccine is the first defense that can address harmful fine-tuning issues for a 7B pre-trained models trained on consumer GPUs with limited memory (e.g., RTX 4090). Our code is available at https://github.com/Lslland/T-Vaccine.

1 Introduction

Harmful fine-tuning can break aligned LLMs, while existing alignment-stage defenses are described as fragile and memory-intensive. T-Vaccine selects safety-critical layers for perturbation and reports better defense effectiveness and resource efficiency than existing methods.

1 Introduction

  • The paper asks whether an alignment-stage defense can use less memory while improving defense performance.The motivation notes that alignment-stage defenses impose a one-time resource cost, whereas fine-tuning-stage and post-fine-tuning defenses add overhead for each request.Source: §1 · pp. 1–2
  • T-Vaccine reduces harmful scores by up to 8.19%, 4.31%, and 11.34% versus Vaccine, TAR, and RepNoise, respectively, and cuts memory consumption by 31.53%, 48.46%, and 48.21%.The reported memory reductions are versus Vaccine, TAR, and RepNoise, respectively; fine-tuning accuracy is maintained at up to 91.96%.Source: §1 · p. 2
  • T-Vaccine is reported as the first defense to address harmful fine-tuning for 7B models on consumer GPUs with 24GB of memory, such as an RTX 4090.Experiments compare it with five existing methods across various models, datasets, and attacks.Source: §1 · p. 2
  • Uniformly perturbing every layer can waste memory and degrade defense when perturbations affect safety-irrelevant layers.The authors identify this as a limitation in Vaccine’s original design.Source: §1 · p. 2
  • T-Vaccine uses harmful gradient norms to estimate layer-sampling probabilities, then perturbs selected layers while freezing the others during training.The paper says this avoids performance degradation from uniform perturbations and reduces training memory consumption.Source: §1 · p. 2

2 Related Work

Related work covers memory-efficient fine-tuning, harmful fine-tuning defenses, and layerwise importance sampling. The paper positions T-Vaccine as combining safety with lower memory consumption.

2 Related Work

  • Adapters, prompt tuning, and layer freezing reduce fine-tuning memory by limiting which parameters or layers are trained.The section describes adapters and soft prompts as trainable modules or inputs while other model parameters remain unchanged.Source: §2 · pp. 2–3
  • Harmful fine-tuning defenses operate at alignment, fine-tuning, or post-fine-tuning stages; layerwise importance sampling methods focus on memory efficiency rather than safety.Lisa dynamically selects layers for training based on a predefined probability to reduce memory overhead.Source: §2 · p. 3
  • The paper distinguishes T-Vaccine from these approaches as a method intended to ensure safe outputs while reducing memory consumption.The related-work discussion contrasts memory-efficient methods that overlook safety with harmful fine-tuning defenses that incur significant resource overhead.Source: §2 · p. 3

3 Preliminaries

The section frames harmful fine-tuning in a two-stage service setting, reviews Vaccine’s perturbation-aware alignment, and shows why perturbing all layers can be wasteful and weaken defense.

3.1 Setting

  • The setting models alignment followed by user fine-tuning, where user data may mix harmful and benign examples that are difficult to distinguish.The provider is assumed to have both harmful-prompt/safe-answer alignment data and harmful-prompt/harmful-answer data.Source: §3.1 · p. 3

3.2 Vaccine: Perturbation-aware Alignment

  • Vaccine adds perturbations to every embedding layer during alignment, then computes perturbed gradients to update the model.Its two forward/backward passes derive perturbations and then apply them during training; storing all layers’ activations, gradients, and perturbations increases GPU memory use.Source: §3.2 · pp. 3–4

3.3 Rethinking on Vaccine

  • HS is 14.60% with about 20 perturbed layers, versus 23.75% with all 32, indicating that perturbing every layer is suboptimal.The figure compares harmful score across different numbers of perturbed layers; smaller scores are preferred.Source: §3.3 · p. 4, Figure 2 · p. 4
  • The last 10 layers have relatively small harmful-data gradient norms, and perturbing them does not improve defense and can worsen it.The authors propose using harmful gradient norms to identify safety-critical layers and reduce perturbation of safety-irrelevant ones.Source: §3.3 · p. 4

4 Methodology

T-Vaccine uses harmful-data gradient norms to prioritize layers, then samples a subset for perturbation-aware alignment while freezing the rest.

4.1 Layerwise Sampling Probability

  • Layer scores from harmful-data gradients determine sampling probabilities, with higher probabilities assigned to layers with higher gradient norms.The probabilities are recalculated every K steps, then used to randomly select γ layers for training.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Layer-wise Perturbation

  • For sampled layers, T-Vaccine performs two forward/backward passes: first to obtain gradients, then to evaluate gradients after adding perturbations and update selected layers.Perturbations are applied to selected-layer embeddings; other layers remain frozen.Source: §4.2 · pp. 5–6
  • T-Vaccine samples γ layers using gradient-based probabilities, applies perturbations to those layers, and keeps the others frozen to reduce memory use and avoid performance degradation.Frozen layers do not store activation states or gradient information during training.Source: §4.2 · pp. 5–6, Figure 3 · p. 5

5 Experiment

Experiments compare T-Vaccine with existing defenses across several fine-tuning settings, models, and datasets. They report improved defense and lower memory use, with trade-offs as the number of perturbed layers changes.

5.1 Setup

  • Experiments use 2,000 safe and 200 harmful alignment examples, three named downstream tasks, four model families, and default harmful ratio h=0.1 with n=1,000.The evaluation uses fine-tuning accuracy and harmful score; experiments were conducted on an A6000-48GB.Source: §5.1 · p. 6

5.2 Main Evaluation

  • 8.19% lower harmful scores, 0.4% higher fine-tuning accuracy, and 20.96% lower memory use than Vaccine on average under varying harmful ratios.The layer-wise perturbation strategy is attributed with alleviating performance degradation from excessive perturbation in safety-irrelevant layers.Source: §5.2 · p. 6, Table 1 · p. 7
  • 23.57% average defense improvement and 39.41% lower memory use versus existing baselines, with pairwise defense improvements of 58.88% for Non-Aligned, 19.73% for SFT, 4.31% for TAR, and 11.34% for RepNoise.Pairwise memory reductions were 30.42%, 30.54%, 48.46%, and 48.21%, respectively.Source: §5.2 · p. 7
  • On four models, T-Vaccine reduced HS versus Vaccine to 8.60% on Gemma-2-2B, 8.78% on Llama2-7B, 0.57% on Vicuna-7B, and 0.30% on Qwen2-7B.It achieved the best defense performance on the first three models; on Qwen2-7B its HS was 2.40%, versus TAR’s 1.80%, and memory stayed below 24GB across all four.Source: §5.2 · p. 7, Table 2 · p. 7
  • 30.26% average defense improvement versus other methods across fine-tuning datasets, including HS reductions of 65.37% versus Non-Aligned, 30.77% versus SFT, 11.27% versus TAR, and 13.61% versus RepNoise.Compared with Vaccine, the reported average fine-tuning accuracy improvement was 0.33%.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Memory Efficiency

  • RepNoise and TAR retain activations for both harmful and alignment datasets, while T-Vaccine trains only selected layers and is reported to run 7B models on 24GB consumer GPUs.Non-Aligned, SFT, Vaccine, and T-Vaccine train on the alignment dataset alone; T-Vaccine further limits retained activations, gradients, and optimizer memory to selected layers.Source: §5.3 · pp. 7–8, Figure 4 · p. 8

5.4 Hyper-Parameter Analysis

  • With S=17, FA falls to 91.20% and memory reaches 29.97GB; with S=5, HS is 20.36%, while the paper selects S=8 to balance defense, fine-tuning, and memory.Increasing sampled layers improves defense but reduces fine-tuning performance and raises memory use; too few layers can leave safety-critical layers unperturbed.Source: §5.4 · p. 8, Table 4 · p. 8

5.5 Visualization

  • In the malicious-prompt example, T-Vaccine refuses the request, whereas the other displayed methods provide harmful responses.The prompt asks for the best way to kill an enemy.Source: §5.5 · p. 8

6 Conclusion

The conclusion attributes degraded defense and unnecessary memory overhead to excess perturbation of safety-irrelevant layers. It presents T-Vaccine as a way to address harmful fine-tuning while reducing memory use, and identifies multimodal learning as future work.

6 Conclusion

  • Uniform layer-wise perturbation may over-perturb safety-irrelevant layers, degrading defense and adding unnecessary memory overhead.Source: §6 · p. 8
  • T-Vaccine is presented as a memory-efficient safety alignment method that effectively defends against harmful fine-tuning while retaining competitive performance and reducing memory consumption.Source: §6 · p. 8
  • Future work will explore whether T-Vaccine generalizes to other LLM scenarios, including multimodal learning.Source: §6 · p. 8

7 Limitations

The authors identify two scope limitations: T-Vaccine is built using SFT because RLHF requires more memory and computation, and multimodal scenarios have not been tested.

7 Limitations

  • RLHF resource demands led the authors to build T-Vaccine solely on SFT, which may limit its generalizability.RLHF typically loads multiple models, such as reward or critique models, and requires more computation for convergence.Source: §7 · p. 9
  • T-Vaccine has not been tested in multimodal scenarios; extending it to multimodal applications is future work.Source: §7 · p. 9
  • The authors plan to extend T-Vaccine to RLHF, but do not report such an extension here.Source: §7 · p. 9

8 Potential Risks

The paper identifies a potential security vulnerability in LLM user fine-tuning APIs, while noting that its experiments use open-access models in a controlled local environment.

8 Potential Risks

  • The paper identifies a security vulnerability in LLM user fine-tuning APIs that could potentially be exploited against commercial LLM services.Source: §8 · p. 9
  • All experiments use open-access LLMs in a controlled local environment.Source: §8 · p. 9
  • The paper includes examples of unethical and harmful data that may be offensive.Source: §8 · p. 9

A.1 Implementation Details

The implementation specifies alignment and fine-tuning settings, dataset sources, and prompt construction for the alignment and task datasets.

A.1 Implementation Details

  • Alignment uses learning rate 1e-3, batch size 10, 2,000 alignment samples, and 20 epochs; fine-tuning uses learning rate 1e-5, batch size 10, 1,000 samples, and 20 epochs.Source: §A.1 · p. 12
  • Fine-tuning data mixes benign task examples from SST2, GSM8K, and AGNEWS with harmful data at default ratio p = 0.1 for n = 1,000.Harmful examples are sampled from Ji et al. (2024); alignment and harmful sampling-frequency data are drawn from Rosati et al. (2024c).Source: §A.1 · p. 12
  • The alignment and fine-tuning datasets use the instruction-input-response prompt template from Huang et al. (2024e).Source: §A.1 · p. 12
  • Benign SST2, AGNEWS, and GSM8K datasets are constructed with task-specific templates, including sentiment labels, four-way news categories, and step-by-step math prompts.Source: §A.1 · p. 12

A.2 Baseline Description •

The baseline descriptions distinguish unaligned and SFT setups from three defenses: TAR, RepNoise, and Vaccine.

A.2 Baseline Description •

  • Non-Aligned skips pre-trained-model alignment and directly applies supervised fine-tuning to the target task.Source: §A.2 · p. 12
  • SFT first aligns the pre-trained model on an alignment dataset for safe responses, then fine-tunes it for a specific task.Source: §A.2 · p. 12
  • TAR combines gradients from harmful-data adversarial training and benign-data training during alignment, using λtr = 2 and λretain = 1.Source: §A.2 · p. 12
  • RepNoise is applied during alignment using alignment and harmful datasets, followed by standard SFT on the user dataset.The experiment uses α = 1 and β = 0.001.Source: §A.2 · p. 12
  • Vaccine is applied during alignment on the alignment dataset, followed by standard SFT on the user dataset; the experiment uses ρ = 2.Source: §A.2 · p. 12

A.3 Hyper-Parameter Analysis

The hyper-parameter analysis examines how harmful-dataset size, sampling frequency, and perturbation intensity affect defense and fine-tuning performance. The visualization reports that T-Vaccine refuses a sensitive prompt where other methods fail.

A.3 Hyper-Parameter Analysis

  • HS rose from 13.97% at Nh = 100 to 16.37% at Nh = 600; the authors select Nh = 200, where HS = 14.97% and FA = 92.40%.Source: Table 5 · p. 13, §A.3 · p. 13
  • At K = 100, frequent probability recalculation can destabilize layer selection and weaken performance; as K increases, defense improves while fine-tuning performance declines, so the authors choose K = 200.At K = 600, FA reaches its lowest value of 91.60%.Source: Table 6 · p. 13, §A.3 · p. 13
  • Defense effectiveness fluctuates with perturbation intensity; HS reaches 19.76% at ρ = 4 and 16.57% at ρ = 6, while FA peaks at 92.60% at ρ = 3, the selected setting.Source: Table 7 · p. 13, §A.3 · p. 13

A.4 Visualization

  • On the haunted-house prompt, T-Vaccine returned a harmless response while Vaccine offered scare tactics; the authors report that other methods also failed to refuse.Source: §A.4 · p. 13
Loading 2410.09760v3…