Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation

1 INTRODUCTION

• A few harmful examples in user fine-tuning data can make aligned models forget safety behavior, creating a large attack surface with no highly scalable solution reported.The cited studies use red-teaming to identify this vulnerability in fine-tuning-as-a-service.Source: §1 · p. 1
• Alignment-stage defenses require only one-time overhead, unlike fine-tuning-stage and post-fine-tuning defenses that add computation for each incoming request.The paper therefore aims to promote alignment-stage mitigation.Source: §1 · p. 2
• Vaccine uses alignment data alone, while RepNoise perturbs harmful-data embeddings; the paper notes potential shortcomings for both approaches.Vaccine’s alignment-only data may be insufficient against harmful attacks, while harmful fine-tuning may reshape RepNoise’s embedding distribution under some conditions.Source: §1 · p. 2
• Up to 17.26% vs Vaccine and 20.08% vs RepNoise in average harmful scores, while maintaining the same level of fine-tune accuracy.The paper reports these reductions for Booster compared with the two alignment-stage defenses.Source: §1 · p. 2
• The authors identify harmful perturbation as a cause of alignment break and propose a regularizer that attenuates harmful-loss reduction after simulated perturbation.Booster combines the regularizer with the original alignment loss and solves the resulting optimization problem with an iterative gradient method.Source: §1 · p. 2
• The paper proposes harmful-perturbation analysis and evaluates Booster across four downstream tasks and different attack settings.Its listed contributions include a loss regularizer and an iterative gradient method for alignment-stage optimization.Source: §1 · p. 2

2 RELATED WORK

• Safety alignment trains pretrained models on refusal demonstrations using methods such as SFT or RLHF (Ouyang et al., 2022; Dai et al., 2023; Bai et al., 2022; Wu et al., 2023; Dong et al., 2023; Rafailov et al., 2023; Yuan et al., 2023).The alignment dataset consists of examples of refusal answers to harmful prompts.Source: §2 · p. 2
• Prior work reports that fine-tuning aligned models on partially harmful data can cause jailbreaks, while defenses span alignment, fine-tuning, and post-fine-tuning stages.Examples include Vaccine and RepNoise at alignment, Lisa during fine-tuning, and SOMF after fine-tuning (Huang et al., 2024e; Rosati et al., 2024b; Huang et al., 2024d; Yi et al., 2024c).Source: §2 · pp. 2–3
• Meta-learning optimizes a model for downstream tasks after a gradient step, and Booster likewise incorporates one-step gradient information to attenuate harmful perturbation before fine-tuning.The paper describes this as a shared optimization feature, not as the same objective.Source: §2 · p. 3
• Booster is presented as the first solution in the paper to use harmful perturbation to design a defense, can be combined with Vaccine or RepNoise, and differs in insight and design from concurrent TAR (Tamirisa et al., 2024).The paper notes that TAR uses a similar meta-learning technique to simulate harmful perturbation during alignment.Source: §2 · p. 3

3.1 HARMFUL FINE-TUNING

• The considered scenario has users upload data, including harmful examples, for a service provider to fine-tune an aligned foundation model deployed for personalized responses.The threat model assumes harmful and benign user data are mixed and inseparable (Qi et al., 2023; Huang et al., 2024e; Rosati et al., 2024b).Source: §3.1 · p. 3
• The provider is assumed to maintain both an alignment dataset of harmful-prompt/safe-answer pairs and a harmful dataset of harmful-prompt/harmful-answer pairs.The paper notes these data types are available in existing open datasets such as BeaverTail (Ji et al., 2023).Source: §3.1 · p. 3

3.2 HARMFUL PERTURBATION

• Harmful perturbation is one optimization step in the harmful-data gradient direction during fine-tuning, which the paper identifies as a root reason for successful harmful attacks.The update rule uses a stochastic gradient over a random sample of user data; a harmful-data sample gives the harmful perturbation.Source: §3.2 · p. 4
• Fine-tuning on harmful data raises harmful scores and reduces harmful training and testing losses, whereas SST2 fine-tuning leaves scores largely unchanged and slightly increases training loss.The testing-loss pattern indicates that fitting the harmful training data generalizes to unseen harmful data; the exploratory experiments use aligned Llama2-7B with SST2 and BeaverTails data.Source: §3.2 · p. 4, Figure 2 · p. 4

• For SFT, f and h are cross-entropy losses on alignment and harmful data; RL losses may be more complex.λ sets regularizer intensity, and α is the step size.Source: §4 · p. 4
• Booster minimizes the gap between harmful loss before and after a normalized harmful-gradient step, attenuating the simulated harmful-loss reduction.The regularizer is added to the original alignment loss.Source: §4 · pp. 4–5
• Similar normalized-gradient regularizers have been used for backdoor-path resistance, adversarial-attack transferability, and generalization, but for different aims.The cited examples are Min et al. (2024), Qin et al. (2022), and Zhao et al. (2022).Source: §4 · p. 5
• Booster uses iterative gradient updates, approximating a costly Hessian-bearing second-order term as constant.The update rule follows the chain rule; η is the learning rate.Source: §4 · p. 5
• Each Booster step uses three forward/backward passes: alignment gradients, harmful gradients, then harmful gradients after a normalized harmful-gradient step.The resulting gradient components determine the final parameter update.Source: §4 · p. 5
• Unlike Vaccine, Booster uses harmful data to simulate harmful perturbation; the two methods can be combined.Vaccine instead addresses hidden embedding drift over alignment data (Huang et al., 2024e).Source: §4 · p. 5

5.1 SETUP

• Experiments use Llama2-7B by default, four downstream tasks, four baselines, and harmful-data mixtures, measuring fine-tune accuracy and harmful score after two-stage training.The evaluation tasks are SST2, AGNEWS, GSM8K, and AlpacaEval.Source: §5.1 · p. 6

5.2 MAIN EXPERIMENTS

• Across harmful ratios and fine-tuning sample counts, Booster averages 22.64% lower harmful score and 2.64% higher accuracy, and 28.76% lower harmful score and 2.13% higher accuracy, respectively, versus SFT.Harmful score still rises as the harmful-data ratio increases.Source: Table 1 · p. 6, §5.2 · pp. 6–7, Table 2 · p. 7
• Across four fine-tuning datasets, Booster reduces harmful score versus SFT by 25.4%, 23.6%, 8.4%, and 4.0%, respectively, with the highest average fine-tune accuracy.The authors report that this extends to more complicated tasks such as GSM8K and AlpacaEval.Source: Table 3 · p. 7, §5.2 · p. 7
• Across Gemma2-9B and Qwen2-7B, Booster averages 34.14% harmful-score reduction and 0.71% higher fine-tune accuracy.For Qwen2-7B, it records 1.6% harmful score and 95.64% fine-tune accuracy.Source: Table 4 · p. 7, §5.2 · p. 7

5.3 STATISTICAL/SYSTEM ANALYSIS

• Figure 3 shows Booster’s harmful score staying stable while SFT’s rises, and slower harmful-loss reduction; alignment costs about three times SFT’s time and 8.53GB additional memory.The paper also reports lower time and GPU-memory overhead for Booster than RepNoise.Source: §5.3 · pp. 7–8, Figure 3 · p. 8, Table 5 · p. 8

5.4 HYPER-PARAMETER ANALYSIS

• The regularizer’s intensity, inner step size, and harmful-sample count require tuning: λ = 0 or α = 0 reduces to SFT, while 50 samples suffice but 5 do not.The authors also report that excessive λ weakens defense performance and excessive α can cause instability or ineffective defense.Source: §5.4 · pp. 8–9, Table 6 · p. 9, Table 7 · p. 9

5.5 ALTERNATIVE DESIGN

• Compared with original Booster, Vaccine+Booster averages 3.88% lower harmful score and 0.74% lower fine-tune accuracy, with a possible extra forward/backward pass.Source: Table 9 · p. 9, §5.5 · p. 9

5.6 VISUALIZATION

• In the illustrated sensitive-question example, Booster refuses while the other compared methods do not.Source: §5.6 · pp. 9–10

• Harmful perturbations during fine-tuning decrease harmful loss and thereby break alignment.Source: §6 · p. 10
• Booster constrains the harmful-loss gap before and after a simulated perturbation through a perturbation-minimization objective.Source: §6 · p. 10
• Experiments show Booster mitigates fine-tuning risk while maintaining fine-tuning accuracy.Source: §6 · p. 10

• The paper presents a defense against harmful fine-tuning attacks.Source: §7 · p. 10
• The authors state that the technique itself should not pose serious ethical concern.Source: §7 · p. 10
• Demonstration data includes content that may be disturbing or offensive, including hateful or violent material.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

• Booster follows its pseudocode without additional implementation tricks, and an anonymous source link is provided in the abstract.Source: §8 · p. 10
• The harmful and alignment datasets use Rosati et al. (2024b)'s data pair, while other datasets are standard Hugging Face benchmarks.Source: §8 · p. 10
• Hyperparameters and simulation settings are in Section 5.1, with detailed settings and selection logic in Appendix B.Source: §8 · p. 10

A BOOSTER VS. TAR

• TAR was a concurrent alignment-stage solution; the comparison simplifies it to one inner step, although original TAR uses multiple steps.Source: A BOOSTER VS. TAR · p. 17
• TAR uses a retain loss requiring a normal proxy dataset, whereas Booster optimizes alignment loss requiring alignment data.Source: A BOOSTER VS. TAR · p. 17
• Booster minimizes harmful-loss reduction rather than TAR's post-perturbation negative entropy loss; an alternative negative cross-entropy formulation was unstable and caused collapse.Source: A BOOSTER VS. TAR · p. 17

B EXPERIMENTAL DETAILS

• The default setup uses 5,000 BeaverTails alignment examples and 1,000 fine-tuning samples, with harmful ratio p = 0.1; AlpacaEval uses 700 fine-tuning samples.Alignment training uses 20 epochs, learning rate 5e-4, and batch size 10; fine-tuning uses 20 epochs, learning rate 1e-5, and batch size 10.Source: B EXPERIMENTAL DETAILS · p. 17
• A shared prompt template is used for training and testing, with task-specific SST2, GSM8K, and AlpacaEval inputs and evaluation procedures.Source: B EXPERIMENTAL DETAILS · pp. 17–18

C BASELINE DESCRIPTIONS

• SFT aligns and then fine-tunes on the user dataset, while Lisa alternates alignment and fine-tuning optimization with proximal terms.Source: C BASELINE DESCRIPTIONS · p. 18
• Vaccine and RepNoise are applied during alignment before SFT fine-tuning; Vaccine adds embedding perturbations, while RepNoise degrades harmful-data embeddings to Gaussian noise.Source: C BASELINE DESCRIPTIONS · p. 18
• Booster is used to align on the alignment and harmful datasets, followed by SFT on the user dataset.Source: C BASELINE DESCRIPTIONS · p. 18

D MORE EXPERIMENTAL RESULTS

• Across three models, Booster achieves the smallest average harmful score and highest average fine-tuning accuracy, while results across other settings favor its accuracy despite Lisa’s sometimes lower harmful score.For harmful-data ratios, Booster’s harmful score is 1.1 higher than Lisa’s while average fine-tune accuracy is 6.05 higher; for sample counts, Lisa’s harmful score is 1.9 lower while its fine-tune accuracy is 5.3 lower.Source: D MORE EXPERIMENTAL RESULTS · p. 19, Table 12 · p. 19

E BOOSTER+LISA

• Booster+Lisa consistently outperforms Lisa across harmful-ratio, sample-count, and model groups on harmful score and fine-tune accuracy.For Gemma2-9B, Booster+Lisa’s harmful score is 2.7 lower and fine-tune accuracy is 7.5 higher than Lisa’s.Source: E BOOSTER+LISA · p. 20, Table 13 · p. 20, Table 14 · p. 20

F VACCINE+BOOSTER

• The Vaccine+Booster design combines Vaccine’s embedding-perturbation objective with Booster’s regularizer, optimizing the inner perturbation and then the outer objective iteratively.The inner perturbation is the same as Vaccine’s because ϵ is absent from the second term; the outer problem can be solved with an iterative gradient method.Source: F VACCINE+BOOSTER · p. 21

G.1 ALTERNATIVE INSIGHTS

• A proposed future direction is to train the aligned model so benign SST2 fine-tuning increases harmful loss enough to balance harmful-data updates.The passage describes this as an intuitive possibility, not an evaluated defense.Source: §G.1 · p. 22

G.2 A STRANGE PHENOMENON

• During alignment, harmful loss rises from 1.582 to 3.920 with SFT but only from 1.582 to 1.736 with Booster, explaining Booster’s lower initial harmful training loss.The authors conjecture that the regularizer also changes generalization between safety-alignment loss and harmful loss.Source: §G.2 · p. 23, Table 16 · p. 23

G.3 MORE MEASUREMENT METRICS

• Booster decreases GSM8K benchmark accuracy by 0.4% versus SFT, compared with decreases of 1% for RepNoise and 4.3% for Vaccine.The comparison is measured on the aligned model before fine-tuning.Source: Table 17 · p. 23, §G.3 · p. 23
• Booster, RepNoise, and Vaccine all slightly reduce the aligned model’s harmful score.The benchmark accuracy is measured on GSM8K using Gemma2-9B.Source: §G.3 · p. 23, Table 17 · p. 23

H LIMITATIONS AND FUTURE EXTENSION

• Booster’s optimal regularizer penalty and inner step size vary by downstream task, while a service provider cannot tune them separately for every incoming task.Finding one hyperparameter setting that works across tasks is challenging and may prevent optimal performance on each task.Source: H LIMITATIONS AND FUTURE EXTENSION · pp. 23–24
• Federated instruction fine-tuning and LLM-agent settings are proposed extensions for Booster, rather than settings addressed by the stated extension proposal.The passage notes that harmful fine-tuning may pose a more serious threat in federated instruction fine-tuning.Source: H LIMITATIONS AND FUTURE EXTENSION · p. 24