Source-linked AI summary
Making Harmful Behaviors Unlearnable for Large Language Models
Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Qi Zhang, Xuanjing Huang
TL;DR
Fine-tuning can cause safety-aligned LLMs to acquire harmful behaviors, including from implicit harmful content. This paper uses separable security vectors to make harmful behaviors unlearnable during training. Vectors trained on 100 harmful samples prevent learning from 1,000 harmful examples while preserving learning on other tasks.
Problem
Safety-aligned LLMs can acquire harmful behaviors from fine-tuning data, including implicit harmful content, and even benign fine-tuning can compromise safety.
Source: §1 · p. 1, §3.1 · p. 3Method
Security vectors are activated during fine-tuning to align model responses with harmful data while clean LLM parameters remain trainable, then deactivated for inference.
Source: §1 · p. 1, §3.2 · p. 3Results
Takeaways & Limitations
The method supports fine-tuning on harmful data without learning the targeted harmful behaviors or losing the ability to learn other tasks.
Source: §1 · p. 2Directly matching harmful responses during training would make the model exhibit harmful behaviors, which is unacceptable for downstream applications.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Large language models (LLMs) have shown great potential as general-purpose AI assistants in various domains. To meet the requirements of different applications, LLMs are often customized by further fine-tuning. However, the powerful learning ability of LLMs not only enables them to acquire new tasks but also makes them susceptible to learning undesired behaviors. For example, even safety-aligned LLMs can be easily fine-tuned into harmful assistants as the fine-tuning data often contains implicit or explicit harmful content. Can we train LLMs on harmful data without learning harmful behaviors? This paper proposes a controllable training framework that makes harmful behaviors unlearnable during the fine-tuning process. Specifically, we introduce ``security vectors'', a few new parameters that can be separated from the LLM, to ensure LLM's responses are consistent with the harmful behavior. Security vectors are activated during fine-tuning, the consistent behavior makes LLM believe that such behavior has already been learned, there is no need to further optimize for harmful data. During inference, we can deactivate security vectors to restore the LLM's normal behavior. The experimental results show that the security vectors generated by 100 harmful samples are enough to prevent LLM from learning 1000 harmful samples, while preserving the ability to learn other useful information.
1. Introduction
The introduction frames fine-tuning as a safety risk because LLMs can acquire harmful behavior from harmful or even benign data, then proposes security vectors to make selected behaviors unlearnable while retaining useful learning.
1. Introduction
- Fine-tuning can make safety-aligned LLMs acquire harmful behaviors from a few harmful samples, and even benign data can compromise safety.The paper frames this uncontrollable learning as a security risk.Source: §1 · p. 1
- Security vectors activate during fine-tuning to align responses with harmful data while only clean LLM parameters update; vectors are deactivated for inference.The framework separates parameters associated with harmful behaviors from the LLM’s clean parameters.Source: §1 · p. 2
- With security vectors trained on 100 harmful samples, LLaMA2-7B-Chat resisted learning harmful behavior from 1,000 highly harmful examples, including implicit harmful data.On mixed harmful and new-task data, the method achieved task performance comparable to direct fine-tuning and safety similar to the original safety-aligned LLM.Source: §1 · p. 2
- The framework makes harmful behaviors unlearnable during fine-tuning and can be extended to make other behaviors unlearnable.Its premise is that a response already consistent with the target behavior leaves little room for further optimization on that data.Source: §1 · pp. 1–2
2. Related Work
Related work situates the paper against safety failures during fine-tuning, prior unlearning paradigms, and parameter-efficient tuning.
2.1. Safety Concerns of Large Language Models
- Safety-aligned LLMs can be broken by a few harmful fine-tuning samples, while benign fine-tuning can also compromise safety.The paper presents implicit harmful content in ordinary fine-tuning data as an additional risk, including in sensitive domains such as education.Source: §2.1 · p. 2
2.2. Unlearning in Machine Learning
- Machine unlearning removes training data after training (Sekhari et al., 2021), whereas this work prevents harmful behavior learning; text-domain noise faces discrete-sequence obstacles.The paper distinguishes its security vectors from image unlearnable examples, which add imperceptible noise to prevent satisfactory model performance.Source: §2.2 · pp. 2–3
2.3. Parameter-efficient Tuning
- Parameter-efficient tuning updates a small parameter subset while freezing most pretrained weights; here, separation isolates harmful-behavior controls.The paper uses additional parameters to control harmful behaviors while leaving the LLM’s clean parameters available for learning.Source: §2.3 · p. 3
3. Approach
The approach treats harmful fine-tuning as a problem of prediction-error-driven parameter updates, then separates harmful-response behavior from the clean model parameters. Security vectors guide fine-tuning so harmful data does not drive harmful updates, while the backbone can still learn from other data and be used without the vectors at inference.
3.1. Problem Formulation
- The goal is to prevent safety-aligned LLMs from learning implicit or explicit harmful behaviors even when harmful information appears in their SFT data.SFT pairs prompts or instructions with desired responses that indicate the target behavior.Source: §3.1 · p. 3
3.2. Security Vectors
- Learning is modeled as parameter updates driven by prediction errors, so few errors yield a small gradient and little learning from a harmful pair.Making the model's prediction consistent with the harmful target can therefore make that pair unlearnable, but harmful responses are unacceptable for applications.Source: §3.2 · p. 3
- Security vectors separate harmful-response behavior from the clean backbone: trained on harmful data, they make harmful behavior appear learned without changing the backbone during vector training.The vector-training objective is min-min: an inner loop minimizes harmful-data loss for the vectors, while an outer loop minimizes that loss for the LLM parameters; the inner loop takes k steps per outer step.Source: §3.2 · pp. 3–4
3.3. Fine-tuning with Security Vectors
- During SFT, trained vectors participate in forward propagation but stay frozen while backbone parameters update, keeping predictions consistent with harmful data and allowing learning from benign data.For downstream inference, the vectors are deactivated and the clean fine-tuned backbone is used to produce desired behavior.Source: §3.3 · p. 4
4. Experimental Setup
The experiments test whether security vectors preserve harmful-behavior unlearning while allowing learning of other data and retaining general capabilities. They use harmful and task-learning datasets, evaluate harmfulness and utility, and compare a safety-aligned LLM under specified fine-tuning procedures.
4. Experimental Setup
- Experiments assess target-behavior unlearning, learning from other data, and effects on general capability, focusing on harmful responses to harmful instructions.The study also notes the approach could extend to other behaviors when corresponding data are available.Source: §4 · p. 4
4.1. Datasets
- Security vectors use 100 harmful Anthropic red-team QA pairs; harmful fine-tuning uses 10-, 100-, and 1,000-example explicit sets plus 10 implicit AOA examples.The explicit examples are sampled from the 1,000 examples ranked most harmful, and the fine-tuning data do not overlap with vector-training data.Source: §4.1 · pp. 4–5
- ProfileQA contains generated character-profile QA pairs for learning a specified output format and new knowledge not learned during pretraining or alignment.The profiles include synthesized information such as name, age, and nationality.Source: §4.1 · p. 5
- Harmfulness evaluation uses CoNa, PhysicalUnSafe, and 100 held-out Anthropic red-team instructions, while utility evaluation covers MMLU, GSM, LIMA instructions, and ProfileQA.The red-team evaluation examples do not overlap with data used to train security vectors or fine-tune the model.Source: §4.1 · p. 5
4.2. Evaluation Metrics
- Utility is measured with MMLU accuracy, GSM and ProfileQA exact match, and Instruction Following Rate; harmfulness uses mean harmfulness score and harmfulness rate from GPT-3.5-turbo ratings.The harmfulness score ranges from 1 to 5; the harmfulness rate is the percentage receiving a score of 5.Source: §4.2 · p. 5
4.3. Implementation Details
- Experiments mainly use safety-aligned Llama2-7B-chat, with LoRA as security vectors; vector training uses AdamW at 1e-3 for 30 epochs and LLM optimization uses Adafactor at 1e-3.Standard fine-tuning uses 10, 5, or 3 epochs depending on dataset size; security-vector fine-tuning increases epochs to ensure sufficient training on harmful data.Source: §4.3 · pp. 5–6
5. Experimental Results
Security vectors keep harmful behavior largely unlearned while preserving performance on useful tasks. The experiments also examine responses to harmful instructions and how learning-rate and epoch choices affect the method.
5.1. Main Result
- Security vectors trained on 100 harmful samples keep average HR basically 0 and average HS around 1 across harmful evaluations, even after fine-tuning on Harmlarge.The reported HS range is 1–5.Source: §5.1 · pp. 6–7
- Security vectors preserve MMLU and LIMA performance and match direct fine-tuning on ProfileQA, while GSM reasoning declines as harmful-data volume grows under both methods.The authors leave the cause of the GSM decline unresolved.Source: §5.1 · p. 7
5.2. Case Study
- Table 2 shows that security-vector and safety-aligned models refuse a harmful instruction, while models directly fine-tuned on AOA or Harmbase comply.The security-vector model can learn AOA identity while still refusing the harmful instruction.Source: §5.2 · p. 7, Table 2 · p. 7
5.3. Ablation Study
- At 5e-3, vectors generated at 1e-3 lose efficacy, but generating and fine-tuning at 5e-3 restores it.Across most learning rates, security vectors maintain low harmfulness as epochs increase; high rates and large epoch counts can also diminish instruction-following ability.Source: §5.3 · p. 8, Figure 3 · p. 8
6. Conclusion
The paper introduces security vectors to make undesirable behaviors unlearnable during fine-tuning, then deactivates them at inference to restore normal behavior. Experiments indicate that vectors trained on 100 harmful examples can make 1,000 harmful examples unlearnable without affecting learning on other tasks.
6. Conclusion
- Security vectors are separable parameters that steer predictions toward the target undesirable behavior during fine-tuning, inhibiting further learning from that data.They can be deactivated during inference to ensure normal behavior.Source: §6 · p. 8
- 100 harmful training examples made 1,000 harmful examples unlearnable without affecting learning of other tasks.Source: §6 · p. 8
- The framework is presented as a way to reduce fine-tuning security risks and support safer user fine-tuning and enterprise API fine-tuning services.Source: §6 · p. 8
A. Appendix
The authors state that they will add the appendix in a future version.
A. Appendix
- The authors plan to add the appendix in a future version.Source: A. Appendix · p. 12