Immunization against harmful fine-tuning attacks

1 Motivation

• Fine-tuning safety-aligned LLMs on harmful samples can circumvent safety techniques, motivating defenses for safe model release and deployment.Examples of such attacks in practice are provided in Appendix A.Source: §1 · p. 1
• Prior work studied harmful text generation and defenses, but lacks clear definitions of successful defense and the necessary conditions defenses must meet.Without that conceptual analysis, it is unclear how proposed defenses should be validated.Source: §1 · p. 1
• Without careful experimental design, an apparently effective defense might ruin model capability or fail to generalize to unseen attacks.Source: §1 · p. 1
• The paper proposes Immunization conditions and guidelines to specify what defenses against harmful fine-tuning must comprise.The conditions cover resistance, stability, generalization, and trainability.Source: §1 · p. 1

2 Threat Model for HFTAs

• The attack goal is to use an LLM to cause harm or conduct illegal activities that safety training would otherwise prevent.The defender defines what counts as harm; this may involve removing safety guards or training further on harmful data.Source: §2 · pp. 1–2
• Harmful fine-tuning attacks occur after model release and use training, unlike pre-release backdoor attacks or post-release inference-time adversarial attacks.The figure characterizes backdoors as involving hidden triggers and harmful fine-tuning as not requiring stealth.Source: Figure 1 · p. 2, §2 · p. 2
• The attacker fine-tunes a model lacking harmful capabilities on harmful prompt-response pairs, minimizing their loss for steps within a compute budget.The attacker selects parameters reached at a training step to enable harmful activities previously refused by the model.Source: §2 · p. 2
• The attacker is assumed unable to train an LLM from scratch but to have enough data for meaningful harmful performance.White-box attackers can modify the model and its processes, while black-box attackers access a defender-mediated fine-tuning API.Source: §2 · p. 2

3 Immunization Conditions

• The framework presents these conditions as criteria for deciding whether a model is immunized against harmful training with respect to a harmful dataset.Source: §3 · p. 2
• Resistance measures prevention of harmful learning and can be strong, keeping harmfulness below a defender-set threshold, or weak, delaying threshold crossing beyond the attacker’s budget.Resistance applies both to learning a new harmful task and to circumventing or unlearning safety guards.Source: §3 · pp. 2–3
• Stability requires the immunized model to retain similar performance to the original model on harmless tasks, measured using a reference dataset.The reference dataset Dref is used to assess degradation.Source: §3 · p. 3
• Generalization requires resistance to attacks using samples disjoint from those used for immunization, including either same-domain or cross-domain samples.The defender may know the attack domain while lacking the attacker's specific samples.Source: §3 · p. 3
• Trainability requires harmless-task supervised fine-tuning to improve the immunized model within roughly the same number of steps as the unimmunized model.The authors call trainability optional for protection, but emphasize its social utility for commercial and research applications.Source: §3 · p. 3

4 Guidelines For Future Research

• Resistance research should seek theoretical guarantees, examine adaptive attacks, and consider cryptographic methods that restrict unauthorized weight modification.The paper also encourages deriving loss functions that reduce transition probability through the loss landscape.Source: §4 · pp. 3–4
• Resistance evaluations should use realistic harmful datasets while mitigating risks from constructing sensitive data, and should sweep learning rates and attack sample counts or epochs.Fraud datasets may be difficult to obtain and risky to construct; suggested mitigations include privacy, harmless toy domains, strong licensing, or using public datasets.Source: §4 · p. 4
• Attack success should be measured with domain-specific harmfulness proxies and a defender-set threshold, with the original model’s harmfulness offered as one threshold choice.The authors describe a weaker threshold—being less harmful than a successfully attacked base model—as an initial goal for challenging datasets, not an actual defense.Source: §4 · p. 4
• Stability evaluations should compare defended and base models on standard capability benchmarks and check safety across inference-time attacks, trustworthiness, and bias.The paper regards perplexity as a more indirect proxy for model behavior.Source: §4 · p. 4
• Generalization tests should use disjoint samples from the same domain and probe cross-domain or held-out subdomain attacks because defenders may lack attacker samples.Examples include holding out a harmful question-answering subdomain or testing toxic-content immunization against harmful question answering.Source: §4 · pp. 4–5
• Trainability tests should use tasks where models improve substantially after additional training, such as suitable generation tasks from GEM.Source: §4 · p. 5

• The framework clarifies the harmful fine-tuning threat model by quantifying risk through an imagined attacker training budget.Source: §5 · p. 5
• The proposed Immunization conditions provide a formal basis for validating defenses against harmful fine-tuning.Source: §5 · p. 5
• The paper offers guidelines for robust evaluation of proposed defenses.Source: §5 · p. 5

• The framework covers supervised fine-tuning attacks, not attackers able to train LLMs from scratch; the latter setting is better addressed by compute governance.Source: §6 · p. 5
• Model editing and reinforcement-learning-based attacks are outside the framework, though the authors identify them as possible extensions or future defense targets.Source: §6 · p. 5
• The research direction requires agreement on what constitutes harm and datasets that exemplify harmfulness, a contentious issue that may privilege some groups over others.Source: §6 · p. 5
• Publicly sharing immunization datasets could create dual-use risk, particularly if they demonstrate attacks on undefended models alongside openly available code.Source: §6 · p. 5
• The framework addresses only harmfulness and training-time authorization, leaving extension to high-risk domains such as health, legal, financial, and military applications for future work.Source: §6 · p. 5
• Immunization assumes models are already safe at inference time and does not address other safety challenges such as jail-breaking.Source: §6 · p. 5

A Analysis of Concerning Models in the Wild

• 267 search results were for uncensored models; the search also found 43 toxic, 21 evil, 29 NSFW, and 21 lewd models.The authors report no models explicitly trained for fraud or other criminal activities.Source: A Analysis of Concerning Models in the… · p. 9
• Potentially harmful models used DPO, LoRA, and SFT, illustrating the dual-use risk of alignment techniques and motivating defense research beyond harmful SFT.Other approaches reversed the DPO objective on alignment datasets or merged models; uncensored models also resulted from instruction-tuning data with alignment examples removed.Source: A Analysis of Concerning Models in the… · p. 9
• Potentially harmful models included a Phi-2 fine-tune trained only on safe and educational websites, alongside other cases consistent with the threat model.The search was non-exhaustive and used keywords including “uncensored,” “unfiltered,” “lewd,” “NSFW,” “evil,” and “toxic.”Source: A Analysis of Concerning Models in the… · p. 9

B.1 Attacks

• Prior attacks used harmful or benign fine-tuning samples to circumvent safety guards, including small datasets and preference-learning methods (Yang et al., 2023; Qi et al., 2023; Lermen et al., 2023; Gade et al., 2023; Zhan et al., 2023; Pelrine et al., 2023).Reported examples include 100 question-answer pairs, fewer than $200 in training, and as few as 15 harmful or 100 harmless samples.Source: §B.1 · p. 10

B.2 Defenses

• Defense work spans white-box interventions and black-box API settings; proposed methods include security vectors, vector arithmetic, adversarial perturbations, and harmful-data filtering (Henderson et al., 2023b; Zhou et al., 2023; Bhardwaj et al., 2024; Huang et al., 2024; Zhao et al., 2023).The paper prefers “immunization” to “self-destruction” because defenders may want harmless trainability and useful behavior after harmful training.Source: §B.2 · p. 11

B.3 Analysis

• Wei et al. (2024) found that pruning even 1% of safety-critical neurons can completely revive harmful instruction following, and freezing these neurons to preserve them does not prevent circumvention attacks.Source: §B.3 · p. 11
• Jain et al. (2023) argue that non-harmful fine-tuning may revive harmful capabilities beneath a superficial safety wrapper.Their method combines pruning and probing to localize fine-tuning effects.Source: §B.3 · p. 11

C Evaluating Immunization Empirically

• The empirical section operationalizes the conditions from § 3 to demonstrate an experimental approach, not to claim robust results.The authors present it as an illustration of how to analyze defenses against harmful training attacks.Source: C Evaluating Immunization Empirically · p. 11

C.1 Method

• Resistance is evaluated by fine-tuning on harmful BeaverTails samples and comparing GPT-4-rated harmfulness against the initial model’s threshold of 1.22.The initial model’s mean harmfulness score was 1.22 on a 1–5 scale.Source: §C.1 · p. 12
• The harmfulness evaluator uses GPT-4, but the authors say this metric is unvalidated beyond Zhou et al. (2023) and is for demonstration only.They recommend future use of valid and reliable measures such as those developed in Wang et al. (2024a).Source: §C.1 · p. 12
• Generalization is tested only in-domain, using 300 BeaverTails training samples and 80 held-out test samples; broader harm categories are left for future work.Source: §C.1 · p. 12
• Stability is assessed on 100 random English RedPajama samples selected as likely pretraining examples; the initial model’s loss on them is 3.87.The target is for the immunized model to retain a similar loss.Source: §C.1 · pp. 12–13
• Trainability is measured on VIGGo AMR parsing, where Llama2-7b-chat initially has a ROUGE-1 score of 0.1.The setup uses 300 training and 200 test samples and evaluates unigram overlap with reference AMRs.Source: §C.1 · p. 13
• The immunization objective minimizes loss on harmless rejection answers while maximizing loss on harmful BeaverTails answers, with λ = 0.1.The rejection answers were generated by the initial Llama2-7b-chat model and checked to ensure they were actual rejections.Source: §C.1 · p. 13

C.2 Performing Immunization and the Harmful Training Attack

• The initial attack fine-tunes Llama2-7b-chat for one epoch on harmful BeaverTails samples using Adam, a batch size of four, and a learning rate of 8e-5.The optimizer uses 5% warmup, a linear rate schedule, and no weight decay.Source: §C.2 · p. 13
• Learning rates above 8e-5 produced harmful behavior, and the authors say adversarial loss is not a viable baseline for future benchmark construction.They flag learning-rate sensitivity for follow-up because an attacker could experiment with rates.Source: §C.2 · p. 13
• The immunized model is trained for one epoch with the adversarial objective and the same hyperparameters, then subjected to the same harmful attack.Source: §C.2 · p. 13

C.3 Results

• After 75 training steps, adversarial training yields a harmfulness rating of 1.22, matching the initial model before attack.The authors caution that the demonstration uses few training steps and harmful samples.Source: §C.3 · p. 13
• The immunization method preserves stability on the reference dataset.Figure 2 presents reference-dataset loss as the stability measure.Source: §C.3 · p. 13
• Adversarial training creates a self-destructing defense as model fluency degenerates with more harmful-training steps.Table 2 provides example outputs from the immunized model during harmful training.Source: §C.3 · p. 13, Table 2 · p. 14

C.4 Discussion

• The results establish neither strong nor weak resistance; strong resistance needs theoretical support, while weak resistance needs longer runs and varied training setups.The discussion calls for determining how many training steps or attacks break immunization and testing varied objectives, architectures, and hyperparameters.Source: §C.4 · p. 14
• Reference-dataset loss alone is insufficient evidence of stability; evaluation must assess whether immunization damages model capabilities and behaviors.Source: §C.4 · p. 14
• Generalization evaluation should cover varied harmful domains and attacks, use as few defense samples as possible, and analyze which training-time attacks remain unprevented.Source: §C.4 · p. 14
• Inference-time attacks also merit study, although they fall outside the paper's threat model focus.Source: §C.4 · p. 14
• The adversarial-loss demonstration did not show trainability, though developers may want models that remain fine-tunable on harmless datasets.The authors challenge future defense analyses to incorporate trainability.Source: §C.4 · p. 14