Source-linked AI summary
Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models
Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, Chun-Ying Huang
TL;DR
Fine-tuning can weaken LLM safety alignment, creating a need for efficient safeguards that preserve task performance. Safe LoRA projects selected LoRA updates toward a safety-aligned subspace using only paired base and aligned model weights. Experiments show reduced safety risks with retained utility across malicious and mixed-data settings, although adaptive attacks may circumvent the transparent method.
Problem
Fine-tuning can weaken aligned LLM safety, even when the data is not malicious, while practical users need parameter-efficient adaptation.
Method
Safe LoRA is a training-free, data-free projection patch that modifies selected LoRA updates using an alignment matrix derived from unaligned and aligned model weights.
Results
Safe LoRA retains utility while reducing safety risks on Llama-2-7B-Chat and Llama-3-8B-Instruct across purely malicious or mixed fine-tuning data.
Takeaways & Limitations
Safe LoRA mitigates safety concerns from LLM fine-tuning without additional data or training while maintaining an acceptable level of utility.
Takeaways & Limitations
Because Safe LoRA is transparent, future adaptive attacks may circumvent it.
Abstract
from arXiv · showhide
While large language models (LLMs) such as Llama-2 or GPT-4 have shown impressive zero-shot performance, fine-tuning is still necessary to enhance their performance for customized datasets, domain-specific tasks, or other private needs. However, fine-tuning all parameters of LLMs requires significant hardware resources, which can be impractical for typical users. Therefore, parameter-efficient fine-tuning such as LoRA have emerged, allowing users to fine-tune LLMs without the need for considerable computing resources, with little performance degradation compared to fine-tuning all parameters. Unfortunately, recent studies indicate that fine-tuning can increase the risk to the safety of LLMs, even when data does not contain malicious content. To address this challenge, we propose Safe LoRA, a simple one-liner patch to the original LoRA implementation by introducing the projection of LoRA weights from selected layers to the safety-aligned subspace, effectively reducing the safety risks in LLM fine-tuning while maintaining utility. It is worth noting that Safe LoRA is a training-free and data-free approach, as it only requires the knowledge of the weights from the base and aligned LLMs. Our extensive experiments demonstrate that when fine-tuning on purely malicious data, Safe LoRA retains similar safety performance as the original aligned model. Moreover, when the fine-tuning dataset contains a mixture of both benign and malicious data, Safe LoRA mitigates the negative effect made by malicious data while preserving performance on downstream tasks. Our codes are available at \url{https://github.com/IBM/SafeLoRA}.
1 Introduction
Fine-tuning can weaken LLM safety alignment, including with benign data, motivating Safe LoRA as a lightweight projection-based patch for LoRA updates. Experiments report reduced safety risks while retaining downstream utility, with projected-layer requirements varying by model.
- Motivation: Fine-tuning can significantly weaken aligned LLM safety, even with benign data, across models and fine-tuning strategies.The reported fragility includes full fine-tuning, LoRA, adapters, and prefix tuning.
- Method: Safe LoRA targets LoRA because it enables memory-efficient low-rank updates while achieving performance comparable to resource-intensive full fine-tuning.
- Method: Safe LoRA is a simple, data-free, training-free patch that modifies LoRA updates dissimilar to an alignment matrix through projection.The alignment matrix is derived from paired unaligned and aligned model weights.
- Findings: Evaluations on Llama-2-7B-Chat and Llama-3-8B-Instruct found retained downstream utility and reduced safety risks against malicious or mixed fine-tuning data.The method outperformed SafeInstr and Backdoor Enhanced Alignment in the reported comparisons.
- Findings: Llama-2-7B-Chat required projecting about 11% of layers, whereas Llama-3-8B-Instruct required up to 35% for a good utility-safety trade-off.
2 Related Works
Related work establishes alignment as value-conforming behavior and studies attacks that bypass safety guardrails. It also includes evidence that downstream fine-tuning can damage safety, alongside broader approaches for modifying model parameters and behavior.
- Alignment of LLMs: Alignment trains LLMs to behave consistently with social values, using methods such as instruction tuning, RLHF, DPO, and self-rewarding.
- Jailbreak and Red-teaming: Jailbreak and red-teaming tests expose vulnerabilities by using adversarial prompts or decoding strategies to bypass safety guardrails.
- Jailbreak and Red-teaming: Downstream fine-tuning, including non-malicious fine-tuning, can damage safety guardrails and leave models vulnerable to malicious responses.
- Manipulating Models with Arithmetics: Parameter-arithmetic research studies weight averaging, model fusion, extrapolation, and task vectors for improving or extending model capabilities.
- Manipulating Models with Arithmetics: Other post-training approaches modify model behavior through patching, editing, aligning, debugging, or adding vectors to hidden states.
3 Methodology
Safe LoRA preserves safety after LoRA fine-tuning by projecting selected updates toward a safety-aligned subspace derived from aligned and unaligned model weights. The method selectively balances safety and downstream utility, with an approximate projection offering a faster alternative.
- Constructing Alignment Matrix: The method forms a projection matrix from each alignment matrix and uses it to project LoRA weights toward the alignment subspace.The projection matrix is computed layer-wise and then applied to LoRA updates.
- Constructing Alignment Matrix: Safe LoRA derives layer-wise alignment matrices by subtracting unaligned model weights from aligned model weights.For open-source models, the default setup uses base and chat/instruct checkpoints to construct these matrices.
- Post-hoc Fine-tuning Projection: Safe LoRA projects only layers whose original and projected LoRA updates have low similarity, preserving utility while restoring alignment selectively.Layers can be selected using a similarity threshold or by choosing the top-K layers with the lowest similarity scores.
- A Faster Alternative: The approximate projection is slower to generate than the exact matrix but can often provide a better safety-utility trade-off.Generating C takes 8.6 × 10^-3 seconds, while generating Ĉ takes 2.1714 seconds, a reported 250x slower speed.
4 Experiments
The experiments evaluate Safe LoRA across malicious, mixed, and benign fine-tuning settings, comparing safety, utility, and attack success against LoRA and defense baselines. Results show that Safe LoRA generally reduces safety degradation while preserving downstream performance, with trade-offs depending on model alignment strength and projection choices.
- Experimental Setup: The study evaluates LoRA, SafeInstr, and BEA using PureBad, Dialog Summary, and Alpaca datasets, with safety, utility, and attack success rate metrics.Safety is measured using GPT-4-judged harmfulness across 11 categories, where lower scores indicate greater safety.
- Experimental Setup: Safe LoRA requires no additional training data and selects projected layers using similarity-score thresholds or a predefined number of layers.The method can also be extended to full-parameter fine-tuning.
- PureBad: For Llama-3 on PureBad, Safe LoRA reaches a harmfulness score of 1.10, but its utility is unsatisfactory compared with the other methods.The authors attribute this trade-off to Llama-3’s lower original utility and weaker alignment behavior under the tested settings.
- Dialog Summary: On Dialog Summary, Safe LoRA preserves benign-task utility at approximately 50% Rouge-F1 for Llama-2 and 49.04% for Llama-3 while reducing harmfulness.For Llama-3, the harmfulness score decreases by around 0.4; approximately 35% of LoRA layers are projected.
- Alpaca: On the benign Alpaca dataset, harmfulness increases from 1.058 to 2.25 despite the absence of harmful training data.The authors report this as consistent with prior findings that benign fine-tuning can reduce safety.
- Limitations: For Llama-3, using a suitable learning rate raises attack success rate by only approximately 3%, while higher learning rates reduce utility enough to make the trade-off impractical.The authors therefore omit Llama-3 results for this setting.
- Utility v.s. Safety: For Llama-2, projecting approximately 11% of layers yields a utility-safety balance with less than 2% Rouge-F1 loss and more than 2 points of harmfulness reduction.Projecting more layers tends to harm utility, while low-similarity layers are the main projection targets.
- Full-Parameter Fine-Tuning: With full-parameter fine-tuning, Safe LoRA lowers harmfulness to around 1.05 and raises MT-Bench to 6.4, more than 2 points above the comparison setting.Without Safe LoRA, full fine-tuning has a harmfulness score 0.1 higher and an MT-Bench score at least 0.2 lower than LoRA.
5 Conclusion
Safe LoRA addresses safety degradation caused by fine-tuning without assuming user intent. It mitigates safety concerns while maintaining acceptable utility, though its transparent design may be vulnerable to adaptive circumvention.
- Safe LoRA targets safety-alignment degradation caused by fine-tuning LLMs, including when user intentions may be benign or malicious.
- The method operates without additional data or extra training and maintains an acceptable level of utility.
- The method’s transparency may allow future adaptive attacks to circumvent its protections.
A.1 Detail of 11 Categories
The appendix describes 11 categories that Meta’s Llama-2 and OpenAI prohibit users from querying, alongside the stated usage-policy framing.
- Meta’s Llama-2 and OpenAI prohibit user queries in 11 categories.
- The policies are introduced with the statement that the listed uses are not allowed.
- Figure 5 presents the user policies from OpenAI and Meta Llama-2.
A.2 Keyword List for Rejection Answers
The rejection-answer keyword list supports identifying responses that do not contain common refusal indicators, which the attack relies on.
- The attack relies on the absence of keywords indicating refusal in the model’s reply.
- The listed refusal indicators include apology phrases, assistant identity phrases, and statements refusing or declining assistance.
A.3 Data Format
The appendix specifies chat-style data formats for PureBad and Dialog Summary fine-tuning datasets, using system, user, and assistant roles.
- PureBad: PureBad examples use a system instruction, user input, and model output in role-based chat format.
- Dialog Summary: Dialog Summary examples use a system prompt for dialog summarization, followed by user input and model output.
A.4 Details of Computing Distance for LoRA Weights Trained on the PurBad Dataset
The paper measures how closely projected LoRA weights match non-projected weights across layers. Models trained exclusively on harmful data show the lowest similarity, unlike models trained on benign or mostly benign data.
- The authors compute and sum each layer’s similarity between projected and non-projected weights.This aggregate is denoted S(C∆W, ∆W).
- Models trained on PureBad have substantially lower projected-weight similarity than models trained on Alpaca or Dialog Summary data.Alpaca contains no harmful samples, while Dialog Summary contains 100 harmful samples mixed with other data.
- The similarity results distinguish exclusively harmful training from benign or lightly harmful training conditions.
A.5 Other Public Models
Experiments on Gemma reproduce the main trade-off observed for the Llama models: Safe LoRA preserves utility while reducing harmfulness more effectively than comparison defenses.
- On Gemma Dialog Summary, Safe LoRA achieves a Rouge F1 score of 46.49% while reducing the harmfulness score to 2.209.
- SafeInstr and BEA achieve good utility on Gemma but leave harmfulness scores close to or above 3.
- The Gemma experiment is conducted with the same setup described for the main experiments.
A.6 Comparison to Vaccine
Compared with Vaccine-based defenses, Safe LoRA better balances safety and downstream utility across harmful-data settings. Vaccine variants either reduce utility or fail to improve safety consistently, whereas Safe LoRA maintains strong utility while lowering harmfulness.
- Single-LoRA Vaccine reduces harmfulness to 3.282 on PureBad but does not maintain MT-Bench utility.
- Single-LoRA Vaccine lowers utility on Dialog Summary without improving safety.
- Double-LoRA Vaccine reduces PureBad utility while only slightly lowering harmfulness, and increases harmfulness on Dialog Summary despite retaining utility.
- With 10%, 30%, or 50% harmful data, Safe LoRA reduces harmfulness to around 1.2 while utility falls by only about 1% from the original.