PaperlayerView exact revision on arXiv

Source-linked AI summary

Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack

Tiansheng Huang, Sihao Hu, Ling Liu

arXiv:2402.01109v6cs.LGcs.CL

TL;DR

Harmful examples in user fine-tuning can break an aligned model, while providers may not know or be able to filter the uploaded data. Vaccine adds bounded perturbations during alignment to make hidden embeddings robust to later harmful perturbations. The reported evaluations show better resistance to harmful fine-tuning while retaining benign-task performance with minor loss.

  • Problem

    Harmful user data can break alignment during fine-tuning, while providers may lack access to user data before requests arrive and cannot reliably filter all harmful examples.

    Source: §1 · p. 2

  • Method

    Vaccine adds bounded adversarial perturbations to hidden embeddings during alignment and optimizes model weights to withstand perturbations from later user fine-tuning.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    Vaccine consistently outperforms baseline alignment across varied settings, reducing harmful score by up to 9.8% with downstream-task performance loss of up to 1.8%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluations across popular open-source LLMs indicate Vaccine can reduce harmful scores under attack while maintaining performance near benign scenarios with minor loss.

    Source: §6 · p. 10

    Vaccine is 2x slower than conventional SFT and uses approximately 0.11 GB more GPU memory during alignment.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

The new paradigm of finetuning-as-a-service introduces a new attack surface for Large Language Models (LLMs): a few harmful data uploaded by users can easily trick the finetuning to produce an alignment-broken model. We conduct an empirical analysis and uncover a \textit{harmful embedding drift} phenomenon, showing a probable cause of the alignment-broken effect. Inspired by our findings, we propose Vaccine, a perturbation-aware alignment technique to mitigate the security risk of users finetuning. The core idea of Vaccine is to produce invariant hidden embeddings by progressively adding crafted perturbation to them in the alignment phase. This enables the embeddings to withstand harmful perturbation from un-sanitized user data in the finetuning phase. Our results on open source mainstream LLMs (e.g., Llama2, Opt, Vicuna) demonstrate that Vaccine can boost the robustness of alignment against harmful prompts induced embedding drift while reserving reasoning ability towards benign prompts. Our code is available at \url{https://github.com/git-disl/Vaccine}.

1 Introduction

Safety alignment is commonly applied before deployment, but harmful data in user fine-tuning can break that alignment when providers cannot inspect or remove all harmful uploads. The paper identifies harmful embedding drift and proposes Vaccine, an alignment-stage defense that performs consistently better than baseline alignment across varied settings.

1 Introduction

  • Safety alignment is widely applied before deployment because making LLM answers harmless and helpful remains challenging.Source: §1 · p. 1
  • User fine-tuning can break alignment with a small amount of harmful data, while filtering uploads or using user data during alignment is impractical.Other fine-tuning-stage preservation methods can add computation for each request.Source: §1 · pp. 1–2
  • Vaccine modifies only the alignment stage, adding crafted perturbations to hidden embeddings so they can withstand harmful perturbations during later user fine-tuning.The approach does not require access to user data during alignment.Source: §1 · p. 2
  • The paper identifies harmful embedding drift—the change in alignment-data hidden embeddings after user fine-tuning—as the cause of broken alignment.The authors report that drift worsens as harmful data increases.Source: §1 · p. 2
  • Vaccine consistently outperforms baseline alignment across settings that vary harmful-data ratio and user fine-tuning sample count.The method is designed without knowledge of user fine-tuning data.Source: §1 · p. 2

2 Related Work

Prior work addresses catastrophic forgetting and harmful fine-tuning through defenses applied at different stages. Vaccine is positioned as an alignment-stage approach that avoids access to future user data, with small extra computation incurred before user requests arrive.

2 Related Work

  • The paper evaluates supervised-fine-tuning-based alignment and says its approach could potentially extend to RLHF.Source: §2 · p. 2
  • Existing catastrophic-forgetting defenses either act during fine-tuning to preserve alignment or modify alignment to anticipate later fine-tuning.The latter category includes meta-learning and Ripple; the cited approaches require user data during alignment.Source: §2 · p. 3
  • Harmful fine-tuning research includes defenses at alignment, fine-tuning, and post-fine-tuning stages, alongside work studying attack mechanisms.Source: §2 · p. 3
  • Vaccine modifies the alignment stage without requiring access to user fine-tuning data, avoiding per-request computation associated with some alternatives.The authors describe this as supporting a more realistic serving scenario.Source: §2 · p. 3

3 Preliminaries

The setting has two stages: provider-led alignment followed by user fine-tuning on data that may mix benign and harmful examples. In that setting, harmful-data ratios are associated with degraded alignment and increased hidden-embedding drift.

3.1 Setting

  • The setting separates provider-collected alignment data from user-uploaded fine-tuning data, which may mix benign and harmful examples.The user data contains a proportion p of harmful examples; the server's alignment dataset contains malicious prompts paired with safe answers.Source: §3.1 · pp. 3–4

3.2 Risk Analysis

  • User fine-tuning can downgrade SFT alignment even with a small harmful-data ratio, and the degradation becomes more severe as that ratio rises.The harmful score can change while fine-tune accuracy remains mostly unaffected, making the attack harder to detect from task performance alone.Source: §3.2 · p. 4
  • As the harmful-data ratio rises, SFT alignment loss and hidden-embedding drift increase; drift is measured as the L2 distance between embeddings on the same alignment data before and after user fine-tuning.Hidden embeddings here are outputs of attention layers; the drift increase also occurs for the non-aligned model, but is less severe.Source: §3.2 · p. 4
  • The authors call this pattern Harmful Embedding Drift and conjecture that it explains alignment corruption because drifted embeddings may no longer encode the input correctly.Source: §3.2 · p. 5

4 Methodology

Vaccine adds crafted perturbations to hidden embeddings during alignment so the model becomes less sensitive to drift introduced during user fine-tuning. It formulates this as perturbation-aware alignment and provides a LoRA-based implementation.

4.1 Perturbation-aware Alignment

  • The alignment objective finds bounded hidden-embedding perturbations that maximize loss, then trains model weights to withstand them.The perturbations target alignment data and are intended to represent harmful drift that may arise during user fine-tuning.Source: §4.1 · p. 5
  • The method approximates alignment loss with sequential Taylor expansions and uses the concatenated hidden-layer gradient to derive perturbations.The norm constraint applies across all layers; each optimization step requires two forward-backward passes.Source: §4.1 · pp. 5–6

4.2 Implementation on LoRA-based Fine-tuning

  • The Double-LoRA implementation trains an alignment adaptor first, merges it into the pretrained model, then trains a separate user fine-tuning adaptor.The authors also describe Single-LoRA, which reuses one adaptor across both stages.Source: §4.2 · p. 6

5 Experiment

The experiments evaluate whether Vaccine preserves alignment while supporting user fine-tuning across harmful-data ratios, sample counts, models, and tasks. They report lower harmful scores in varied settings, alongside trade-offs in perturbation intensity and training overhead.

5.1 Setup

  • The evaluation uses safe BeaverTails alignment samples, four user fine-tuning tasks, three model families, and harmful score plus fine-tune accuracy.The tasks are SST2, AGNEWS, GSM8K, and AlpacaEval; the evaluated models are Llama2-7B, Opt-3.7B, and Vicuna-7B.Source: §5.1 · p. 6

5.2 Main Evaluation

  • Up to 9.8% reduction in harmful score versus SFT and 38.2% versus Non-Aligned; Vaccine’s SFT-relative gain shrinks as the harmful ratio rises.At higher ratios, Vaccine’s fine-tune accuracy also increases, which the authors suggest may reflect a small perturbation-related accuracy cost being erased by partially harmful training.Source: Table 1 · p. 7, §5.2 · p. 7
  • Vaccine outperforms baselines on harmful score at n = 100, 500, and 1000; at n = 1500 and 2000, EWC does better but loses over 6% fine-tune accuracy.The authors note that combining Vaccine with EWC can further lower harmful scores, at a cost to fine-tune accuracy.Source: Table 2 · p. 7, §5.2 · p. 7
  • Vaccine reduces harmful scores across models and downstream tasks; on AlpacaEval it also improves fine-tune accuracy while reducing harmful score by 13.8% versus SFT.For AGNEWS, the reported harmful-score reductions versus SFT are 2% for Opt-2.7B and 11% for Llama2-7B.Source: §5.2 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 8

5.3 Statistical/System Analysis

  • Vaccine sustains comparably low alignment loss and smaller embedding drift than SFT, but takes 2x longer and approximately 0.11 GB more GPU memory during alignment.SFT’s embedding drift starts escalating at step 1,000, roughly coinciding with the later rise in alignment loss; Vaccine uses two forward-backward passes per optimizer step.Source: Figure 3 · p. 8, §5.3 · p. 8, Table 5 · p. 8

5.4 Ablation Study and Hyper-parameter Analysis

  • Higher ρ lowers harmful score by 10.2% between ρ = 0.01 and ρ = 10 but decreases fine-tune accuracy; at ρ = 0.2, gradient perturbation beats Gaussian ρ′ = 5 × 10^-3 on both metrics.The gradient-based comparison reports 7% lower harmful score and 19.8% higher fine-tune accuracy; larger ρ also decreases last-round alignment loss by 0.0348 versus ρ = 0.01.Source: §5.4 · pp. 8–9, Table 6 · p. 9, Table 7 · p. 9

5.5 Alternative Design

  • Double-LoRA reduces harmful scores in SST2, while adding EWC to Vaccine lowers harmful scores further but can cost up to 39.2% fine-tune accuracy.The reported Double-LoRA reductions are 3% for SFT and 6.4% for Vaccine; Vaccine plus EWC reduces harmful scores by up to 5.2% for SST2, 4.6% for AGNEWS, and over 26.6% for GSM8K.Source: Table 8 · p. 9, §5.5 · p. 9, Table 9 · p. 9

5.6 Visualization

  • At higher harmful ratios, Figure 4 shows both models’ embeddings drifting in a shared direction, but Vaccine’s shift is smaller; its example still answers a poisoned prompt harmlessly and helpfully.Figure 4 is a t-SNE visualization in which each point represents an alignment-data embedding.Source: Figure 4 · p. 10, §5.6 · p. 10

6 Conclusion

Vaccine addresses the safety risks of fine-tuning-as-a-service with perturbation-aware alignment. Evaluation reports reduced harmful scores under attack while preserving benign-scenario performance, with extra overhead and weak extension to RLHF as limitations.

6 Conclusion

  • Vaccine is a perturbation-aware alignment solution for safeguarding fine-tuning-as-a-service against potentially harmful user data.The conclusion frames user customization as both an opportunity and a new attack surface.Source: §6 · p. 10
  • Evaluation on several open-source LLMs and diversified settings reports significantly lower harmful scores under attack while maintaining on-par benign-scenario performance with minor, negligible loss.The conclusion summarizes the reported outcome across its evaluation settings.Source: §6 · p. 10
  • The authors identify extra computation overhead and weak extension to RLHF as limitations of Vaccine.The conclusion defers discussion of these limitations to Appendix E.Source: §6 · p. 10

A Accelerated Vaccine

Accelerated Vaccine searches for perturbations less frequently than the original method to reduce training time. Searching every 100 steps retains decent defense, with a marginal 0.60 increase in harmful score and significantly shortened step time.

A Accelerated Vaccine

  • Accelerated Vaccine searches for perturbations every τ steps rather than at every optimization step, requiring one additional forward/backward pass per τ steps.The original Vaccine uses two forward/backward passes for each optimization step.Source: A Accelerated Vaccine · p. 17
  • The algorithm takes perturbation intensity ρ, local step T, layer number L, and periodicity τ as inputs, and outputs the aligned model for fine-tuning.It samples a batch and applies perturbed or unperturbed gradients depending on whether the step falls on the search periodicity.Source: A Accelerated Vaccine · p. 17
  • Every-100-step perturbation searches increase harmful score by a marginal 0.60 while significantly shortening step time.The table caption says proper perturbation periodicity can maintain defense performance while reducing training step time.Source: Table 10 · p. 17, A Accelerated Vaccine · p. 17

B Missing Information for Experiments

The appendix specifies the training setup, datasets, prompt construction, and evaluation criteria used in the experiments.

B Missing Information for Experiments

  • Alignment uses learning rate 1e-3, batch size 5, weight decay 0.1, AdamW, and 50 epochs; fine-tuning uses learning rate 1e-5 with the same batch size, weight decay, and optimizer.The alignment hyperparameters were selected to bring training loss near 0.01; smaller learning rates or larger batches could leave high loss, especially with Vaccine.Source: §B.1 · p. 17
  • Fine-tuning runs for 20 epochs on SST2 and AGNEWS and 50 on GSM8K; all methods use LoRA rank 8 with dropout 0.1, cosine decay, and 0.1 warmup.The authors report that larger fine-tuning learning rates strengthened the harmful effect without improving fine-tune accuracy.Source: §B.1 · p. 18
  • Supervised datasets use an instruction-input-response template, with task-specific instructions for sentiment classification, news categorization, and step-by-step GSM8K answering.The template follows Taori et al. (2023).Source: §B.1 · p. 18
  • Alignment examples pair real harmful instructions with safe outputs, while fine-tuning examples include harmful prompts paired with safe answers.The appendix presents both formats as dataset examples.Source: §B.1 · p. 18
  • The experiments sample 2,000 harmful prompts with safe output for alignment; fine-tuning uses 1,000 samples for SST2 and AGNEWS, 4,000 data for GSM8K, and GPT4 answers of AlpacaEval QA data.Source: §B.1 · p. 18
  • Evaluation counts correct classification for SST2 and AGNEWS, correct final answers for GSM8K, and ChatGPT-rated win rate against text_Devinci_003 for AlpacaEval.For GSM8K, the reasoning process is ignored; the AlpacaEval prompts are unseen during training.Source: §B.1 · pp. 18–19

B.2 Baselines and their implementation

The appendix describes the baseline implementations and adapter choices used to compare Vaccine with alternative alignment and fine-tuning strategies.

B.2 Baselines and their implementation

  • Vaccine aligns the model using perturbations and then fine-tunes it with supervised learning, while Non-Aligned omits alignment and SFT uses safe-answer alignment before fine-tuning.Non-Aligned, SFT, and Vaccine use Double-Lora for fair comparison.Source: §B.2 · p. 19
  • EWC regularizes fine-tuning toward the initial alignment adapter, while KL constrains the fine-tuned model's output to remain proximal to the aligned model.The stated regularization intensities are λ = 1e9 for EWC and 1e-3 for KL.Source: §B.2 · p. 19
  • Vlguard mixes 500 alignment samples into 1,000 fine-tuning samples to help the model remember alignment knowledge.EWC uses a single-Lora implementation because its fine-tune adapter is constrained relative to the initial alignment adapter.Source: §B.2 · p. 19

B.3 "Strange" phenomenon

Fine-tuning on partially harmful data produces a surprising safety pattern: models can become more harmful as they learn to give complete answers, while clean-data fine-tuning may not produce the same effect. The authors attribute the alignment break to harmful embedding drift and note that the benign SST2 data may not suit normal question answering.

B.3 "Strange" phenomenon

  • SFT-aligned models can score as more harmful than non-aligned models fine-tuned on clean data, while the clean-data case also has higher alignment loss.The highlighted Figure 5 point uses SST2, a sentiment classification dataset.Source: §B.3 · p. 19, Figure 5 · p. 19
  • As harmful-data ratios rise, fine-tuned models begin producing meaningful harmful answers, whereas pre-trained and clean-data-fine-tuned models may not answer those prompts meaningfully.The examples show non-answers at p = 0 and harmful answers at p = 0.02 and p = 0.05.Source: §B.3 · pp. 19–20
  • The authors explain the pattern by saying that harmful examples teach complete responses, while alignment training teaches complete responses that may be harmful but also builds resistance to providing harmful information.This helps explain why an SFT-aligned model can score as more harmful than a non-aligned model fine-tuned on clean data.Source: §B.3 · pp. 20–21
  • The clean-data result may depend on the model and dataset, and SST2 may be unsuitable for normal question answering because its labeled answers are single words.The authors say this finding disputes Qi et al. (2023), who reported consistent safety degradation on benign-data fine-tuning.Source: §B.3 · p. 20

B.4 More prompt examples

The examples compare safety and user-task fine-tuning across Vaccine, SFT, EWC, and non-aligned models. Vaccine avoids harmful classifications on the shown malicious prompts, but aligned models also show task-performance weaknesses in sentiment analysis, while EWC is less effective in the illustrated customization tasks.

B.4 More prompt examples

  • Vaccine is classified as harmless on both shown malicious prompts, while non-aligned and SFT models are flagged on both and EWC on one.The examples include prompts about punishing unemployed men and playing Russian Roulette.Source: §B.4 · p. 21
  • Aligned models show degraded sentiment-analysis performance in the examples, and EWC appears to degrade most, sometimes ignoring the instruction to answer only positive or negative.The three listed SST2 examples include misclassified answers from SFT, EWC, and Vaccine.Source: §B.4 · p. 22
  • EWC performs worst in the two shown AGNEWS examples by disregarding the categorization instruction, while Vaccine, SFT, and non-aligned models give the expected categories.The examples concern an Olympic coverage article and a Sarajevo siege news article.Source: §B.4 · p. 22
  • After AlpacaEval fine-tuning, non-aligned, SFT, and Vaccine outputs become longer and more informative, whereas EWC is less informative and less effective at learning the user task.The authors present this comparison as further evidence that Vaccine maintains user-task fine-tuning performance.Source: §B.4 · pp. 22–23
  • The leg-raise example shows Vaccine producing a detailed, stepwise answer, while the listed EWC answer is shorter and the other models give different instructions.The displayed answers illustrate the user-task output comparison.Source: §B.4 · p. 23

B.5 More Results on System Evaluation

Vaccine incurs additional system overhead relative to SFT, but the reported memory and training-time costs are lower than those of RepNoise. The system comparison was evaluated on an H100 with 80GB memory.

B.5 More Results on System Evaluation

  • Vaccine uses approximately 2x SFT training time, while RepNoise uses approximately 2.3x-2.4x SFT training time.The authors describe Vaccine as more computation-efficient than RepNoise.Source: §B.5 · p. 23, Table 11 · p. 23
  • Vaccine has more system overhead than baseline SFT, although the authors characterize its memory and computation costs as comparatively modest against RepNoise.The table comparison is reported for an H100 with 80GB memory.Source: §B.5 · p. 23, Table 11 · p. 23
  • For OPT-13B, Vaccine adds 0.039GB of GPU memory over SFT, compared with RepNoise’s 27.164GB extra memory over SFT.Table 11 reports system performance comparison on an H100 with 80GB memory.Source: Table 11 · p. 23, §B.5 · p. 23

C Proof of Optimal Perturbation

The proof establishes that the perturbation choice described in the appendix attains the optimum within the stated L2 constraint. The related discussion distinguishes Vaccine from LAT by defense timing and perturbation construction, while leaving their relative performance and overhead trade-offs for future study.

C Proof of Optimal Perturbation

  • The proof shows that the proposed perturbation attains the maximum objective value among feasible solutions in the L2 norm ball.It identifies the candidate as the optimal solution within the feasible set.Source: C Proof of Optimal Perturbation · p. 24

D Association with LAT

  • Vaccine applies perturbation-awareness before harmful fine-tuning and uses a one-step approximation, whereas LAT unlearns harmful knowledge afterward using projected gradient descent.Both methods perturb alignment-data embeddings and train the model under that perturbation.Source: D Association with LAT · p. 24
  • The authors leave open whether perturbation-aware defense or post-attack unlearning performs better, and whether one-step approximation or iterative projected gradient offers the better overhead-performance trade-off.These comparisons are identified as future research questions.Source: D Association with LAT · p. 24

E Limitations and Further Optimization

The authors identify limits in Vaccine’s scope and overhead, and outline ways to extend and optimize it. They also report that experiments used open-access models locally while noting possible misuse of the disclosed vulnerability.

E Limitations and Further Optimization

  • Vaccine is built on SFT rather than RLHF, which may reduce its generality; it also requires extra computation and memory, and uses datasets outside mainstream LLM tasks.The authors suggest pruning or factorization to reduce overhead and broader downstream tasks for future study.Source: E Limitations and Further Optimization · p. 25
  • Vanilla Vaccine changes alignment but not fine-tuning; the authors propose filtering or down-weighting harmful data and applying sparsification or quantization to improve efficiency.The proposed data-filtering idea uses statistics such as loss or embedding drift to distinguish data points.Source: E Limitations and Further Optimization · p. 25

F Impact Statements

  • The authors conducted experiments on open-access LLMs in a local environment and warn that the identified vulnerability might be misused to attack commercial services.Source: F Impact Statements · p. 25

NeurIPS Paper Checklist

  • The checklist reports that Vaccine requires double the computation during alignment and slightly more memory.The paper refers readers to Section 5.3 for the system evaluation.Source: NeurIPS Paper Checklist · p. 26
  • The authors say they provide code and datasets, training details, and repeated experiments across varied attack settings to support reproduction and statistical significance.Source: NeurIPS Paper Checklist · p. 28
Loading 2402.01109v6…