PaperlayerView exact revision on arXiv

Source-linked AI summary

Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack

Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Ling Liu

arXiv:2405.18641v5cs.LG

TL;DR

Harmful data mixed into user fine-tuning can compromise an aligned model, while BSO can degrade when alignment receives too few steps. Lisa adds a proximal term to constrain drift, and the paper reports lower harmful scores than vanilla BSO while maintaining fine-tuning accuracy.

  • Problem

    Harmful user data mixed into fine-tuning can make customized LLMs produce harmful outputs, creating a need for an effective, computationally affordable mitigation.

    Source: §1 · pp. 1–2

  • Method

    Lisa builds on alternating alignment and user-data optimization by adding a proximal term that keeps iterates near the switching checkpoint to constrain excess drift.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    Lisa reduces harmful score by up to 6.54% versus vanilla Bi-State Optimization while maintaining fine-tuning accuracy with up to 0.43% loss.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported comparison supports using proximal drift control to improve alignment performance in the paper’s fine-tuning setting while retaining user-task accuracy.

    Source: §1 · p. 2

    Lisa requires slightly more GPU memory than SFT: 3.14GB for storing the switching checkpoint.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Recent studies show that Large Language Models (LLMs) with safety alignment can be jail-broken by fine-tuning on a dataset mixed with harmful data. First time in the literature, we show that the jail-broken effect can be mitigated by separating states in the finetuning stage to optimize the alignment and user datasets. Unfortunately, our subsequent study shows that this simple Bi-State Optimization (BSO) solution experiences convergence instability when steps invested in its alignment state is too small, leading to downgraded alignment performance. By statistical analysis, we show that the \textit{excess drift} towards consensus could be a probable reason for the instability. To remedy this issue, we propose \textbf{L}azy(\textbf{i}) \textbf{s}afety \textbf{a}lignment (\textbf{Lisa}), which introduces a proximal term to constraint the drift of each state. Theoretically, the benefit of the proximal term is supported by the convergence analysis, wherein we show that a sufficient large proximal factor is necessary to guarantee Lisa's convergence. Empirically, our results on four downstream finetuning tasks show that Lisa with a proximal term can significantly increase alignment performance while maintaining the LLM's accuracy on the user tasks. Code is available at \url{https://github.com/git-disl/Lisa}.

1 Introduction

Fine-tuning services face the risk that harmful user data will cause customized models to produce harmful outputs, creating an urgent need for affordable mitigation. The paper studies Stage ② defenses, diagnosing instability in Bi-State Optimization (BSO) under asymmetric steps and proposing Lisa to constrain excess drift.

1 Introduction

  • Harmful user uploads can trigger harmful outputs from fine-tuned models, exposing service providers to liability and motivating affordable mitigation.Existing defenses such as Vaccine target Stage ①, whereas this paper focuses on Stage ②.Source: §1 · p. 1, Figure 1 · p. 1
  • Prior defenses include Vaccine’s alignment-stage perturbation mechanism, while ForgetFilter and Vlguard use fine-tuning-stage approaches with considerable extra computation.ForgetFilter filters harmful data and retrains on clean data; Vlguard mixes alignment and fine-tuning data.Source: §1 · p. 2
  • The paper asks whether a computation-efficient fine-tuning-stage mitigation can withstand harmful data mixed into users’ fine-tuning data.The paper distinguishes its Stage ② focus from existing alignment-stage defenses.Source: Figure 1 · p. 1, §1 · p. 2
  • BSO alternates optimization over alignment and user datasets, but its alignment performance can degrade when the two states receive asymmetric steps.The degradation is especially apparent when fewer steps go to the alignment state.Source: §1 · p. 2
  • Under asymmetric computing, BSO exhibits excess drift toward the switching point, which appears to be the root cause of alignment degradation.The paper reports that harmful score can increase by up to 17.6% when fewer steps are invested in the alignment state.Source: §1 · p. 2
  • Lisa adds a proximal term to keep model iterates near the last round’s switching point, and the paper reports reduced harmful score while maintaining fine-tuning accuracy.Lisa reduces harmful score by up to 6.54% relative to vanilla BSO, with fine-tuning accuracy loss of up to 0.43%.Source: §1 · p. 2

2 Related work

Related work covers safety-alignment methods, harmful fine-tuning attacks, and defenses at multiple stages. The paper situates its contribution in fine-tuning-stage mitigation and applies proximal control to BSO’s excess-drift problem.

Safety alignment

  • RLHF-based approaches use preference data, while other safety-alignment methods include good/bad-answer training and prediction or re-evaluation to augment alignment data.The passage describes PPO’s reward-model and supervised-signal pipeline as a typical RLHF example.Source: §2 · p. 2

Harmful fine-tuning

  • Prior work finds that aligned models can be jail-broken by harmful fine-tuning, and proposed defenses include unsafe-data filtering, mixing alignment data, projecting updates, and model fusion.The cited works include Zhao et al. (2023), Zong et al. (2024), Hsu et al. (2024), and Yi et al. (2024b).Source: §2 · p. 3
  • Further defenses include prompt and alignment changes, representation methods, and curated text; the field also extends to federated learning and data-poisoning defenses.The passage names methods including Constrain-SFT (Qi et al., 2024b) and CTRL (Liu et al., 2024b).Source: §2 · p. 3

Proximal algorithms

  • Proximal terms constrain distance from a reference model and have been used to limit client drift, meta-learning drift, and instability in multi-task optimization.This paper applies a proximal term to address excess drift associated with convergence instability.Source: §2 · p. 3

Contributions

  • The authors identify BSO for user fine-tuning and the excess-drift phenomenon under imbalanced computation as novel contributions to harmful fine-tuning defense.They state that excess drift leads to alignment-performance drops when computation invested in BSO is imbalanced.Source: §2 · p. 3

3 Preliminaries

The paper considers a two-stage alignment-then-user-fine-tuning service in which user data may mix harmful and pristine examples that are difficult to separate. Its preliminary results show that small harmful-data ratios can increase harmful scores while leaving fine-tuning accuracy similar.

Fine-tuning-as-a-service

  • The service first aligns a pretrained model using provider-collected data, then customizes it on user-uploaded data before deployment.The deployed model serves personalized outputs to users’ prompts.Source: §3 · p. 3

Threat model

  • The threat model assumes user data contains a percentage of harmful examples mixed with pristine downstream-task data that cannot be easily separated.The paper calls the harmful-data percentage the harmful ratio.Source: §3 · p. 3

Jail-break effect by harmful fine-tuning

  • With 5% harmful data, harmful score can rise by over 15% while fine-tuning accuracy remains similar across harmful ratios.For aligned SFT models, increasing harmful ratios also increases alignment-data loss; Figure 2 reports results for Llama2-7B on SST2.Source: §3 · p. 4, Figure 2 · p. 4

4 Methodology

Bi-State Optimization alternates between alignment and user fine-tuning data, but limited alignment steps can destabilize convergence and weaken alignment. Lisa adds a proximal term to constrain drift, with convergence analysis supporting sufficiently large proximal intensity.

4.1 Bi-State Optimization

  • BSO alternates K1 alignment steps with K2 user fine-tuning steps for T cycles, optimizing the two datasets in separate states.The resulting model is intended to multitask on both datasets.Source: §4.1 · p. 4
  • BSO reduces harmful score by up to 4.2% versus SFT, with up to 0.69% loss of fine-tuning accuracy.Table 1 evaluates different harmful ratios on SST-2 with Llama2-7B and K1 = K2 = 500.Source: §4.1 · p. 5, Table 1 · p. 5
  • As alignment steps decrease, BSO's harmful-score mitigation weakens and eventually matches SFT at allocation (0/1000).The reported Table 2 setting fixes poison ratio p = 0.1; equal switching steps K1 = K2 = 500 require more computation.Source: §4.1 · p. 5, Table 2 · p. 5
  • With few alignment steps, alignment loss rises during fine-tuning and allocations show greater convergence instability; BSO(900,100) reaches near-zero gradient norm.The authors define convergence here as asymmetric convergence to a stationary point of the global problem.Source: §4.1 · p. 5, Figure 4 · p. 5
  • BSO(900,100) has the smallest switching-checkpoint drift, which the authors conjecture may help avoid iterates biased toward only one sub-loss.The reported association links smaller drift with lower alignment loss and better convergence properties.Source: §4.1 · p. 5

4.2 Lazy Safety Alignment

  • Lisa adds a proximal term equal to squared distance from the current weights to the last state's switching checkpoint, constraining drift in both states.The hyperparameter ρ controls proximal intensity; the two state-specific subproblems use their other-state checkpoints.Source: §4.2 · p. 6
  • Under Assumptions 1–3 and a converging subsequence, Lisa's convergence-rate result requires proximal intensity ρ > L.The analysis states that when θ ∈ (0, 1), ρ must be sufficiently large to guarantee convergence.Source: §4.2 · p. 6

5 Experiments

Using three pretrained models, four fine-tuning tasks, and a default setting that mixes 10% harmful data into 5,000 samples, the experiments compare Lisa with alignment and fine-tuning baselines. They evaluate its defense performance, convergence behavior, hyperparameters, alternative combinations, and responses to a malicious prompt.

5.2 Main Results

  • 7.07% and 3.68% lower average harmful scores than Vaccine-SFT and Vlguard, respectively, with average finetune accuracy 0.28% higher than Vaccine-SFT and 0.59% lower than Vlguard.These comparisons are reported for different harmful-data ratios in the default setting.Source: Table 3 · p. 7, §5.2 · p. 7
  • 4.7% lower average harmful score and 80.63% higher finetune accuracy than EWC, the baseline with the second-lowest harmful score, when varying fine-tuning sample number.With different models on GSM8K, Lisa reduced average harmful score by 11.9% and 11.2% versus Vlguard and Vaccine-SFT.Source: Table 4 · p. 7, §5.2 · p. 7, Table 5 · p. 8
  • 11.17% lower average harmful score and higher finetune accuracy than all alignment baselines across fine-tuning datasets; Lisa also outperformed SFT on GSM8K.The dataset comparison uses Mistral-7B.Source: Table 6 · p. 8, §5.2 · p. 8

5.3 Statistical/System Evaluation

  • Figure 5 shows that larger ρ controls alignment-loss growth, gradient norms converge asymptotically to 0, and excess drift is diminished versus BSO.SFT's alignment loss rises significantly with training steps; BSO's gradient norm begins rising after an initial drop, and it retains considerable checkpoint-directed drift after 20,000 steps.Source: Figure 5 · p. 8, §5.3 · p. 8
  • Lisa's clock time for 1,000 training steps is 8.3% above SFT and 4.9% above BSO, and its GPU memory use is 3.14GB above SFT.The added computation comes from the proximal term; memory overhead comes from storing the switching checkpoint, and neither overhead scales with training-data size.Source: Table 7 · p. 8, §5.3 · pp. 8–9

5.4 Hyper-parameters Analysis and Ablation Study

  • Step allocation and proximal intensity trade lower harmful scores against fine-tuning accuracy, while ablations show that both BSO and the proximal term are needed.Too many fine-tuning steps can reduce both finetune accuracy and harmful score when alignment steps are few. Without the proximal term, harmful score is 6.54% higher on average; without alignment-dataset guidance, finetune accuracy is 7.73% lower on average.Source: §5.4 · p. 9, Table 8 · p. 9, Table 9 · p. 9

5.5 Alternative Design

  • Vaccine-Lisa achieved the lowest harmful scores across experiment groups, while filtration followed by Lisa further reduced harmful scores beyond filtration plus SFT.Vaccine-Lisa's average finetune accuracy was 0.5% and 0.78% lower than Vaccine-SFT and SFT-Lisa, respectively.Source: §5.5 · pp. 9–10, Table 11 · p. 10, Table 12 · p. 10

5.6 Visualization

  • On the malicious car-engine-theft prompt, Lisa refused, while the other compared methods gave responses showing some degree of harmfulness.The models were fine-tuned under the default setting.Source: §5.6 · p. 10

6 Conclusion

The conclusion presents Lisa as a proximal correction to BSO’s excess drift under asymmetric training, while noting overhead and weak extension to RLHF. It also identifies harmful samples missed by BeaverTails filtering as a source of leakage.

6 Conclusion

  • Lisa adds a proximal term to control excess drift after BSO degrades when alignment and fine-tuning receive asymmetric steps.The conclusion describes the drift as fine-tuning iterates moving too far from switching checkpoints, harming alignment performance.Source: §6 · p. 10
  • The conclusion names extra overhead and weak extension to RLHF as limitations of Lisa.The authors defer discussion of these limitations to Appendix D.Source: §6 · p. 10
  • 7.71% of harmful data are classified as harmless by BeaverTails moderation and pass through filtration.Source: §6 · p. 10

A Missing Information for Experiments

The appendix specifies a two-stage alignment and user fine-tuning setup, its evaluation metrics, and the baseline methods used for comparison.

A.1 Detailed Setup

  • The setup first aligns the model, then fine-tunes it on task data mixed with harmful data; the default harmful-data share is 10%.The default sample count is 5,000, or 700 for AlpacaEval; Lisa and BSO use an alignment-to-fine-tuning step ratio of 100:900.Source: §A.1 · p. 18
  • The two stages use separate LoRA components, following the Double-LoRA implementation, and experiments run on an H100 workstation.Source: §A.1 · p. 18
  • Harmful ratio is measured on 1,000 unseen malicious-instruction samples, while task accuracy uses task-specific test sets or an AlpacaEval win rate.AlpacaEval accuracy is the win rate against text_Devinci_003; the other listed task metrics count correct answers over test samples.Source: §A.1 · p. 18

A.2 Baselines and its Description

  • The five comparison baselines include direct SFT, Vaccine-SFT, SFT, EWC, and VLGuard, alongside the proposed BSO and Lisa methods.Source: §A.2 · p. 19
  • Vaccine perturbs embeddings during alignment, VLGuard mixes alignment data into fine-tuning, and EWC penalizes distance from a fixed aligned checkpoint.Unlike Lisa, EWC does not use alternating optimization.Source: §A.2 · p. 19

A.3 More Results

The additional results examine Lisa under high harmful-data ratios, data filtering, and a more stealthy harmful-data attack.

A.3 More Results

  • Lisa retains comparable defense performance at high harmful ratios, while Vaccine cannot resist the attack at p=1.The authors conclude that a pure alignment-stage solution is insufficient when more toxicity is included in user data.Source: §A.3 · p. 19, Table 13 · p. 19
  • Filter+Lisa is reported as superior using the harmonic mean of 100-harmful score and fine-tune accuracy.Table 14 reports the harmonic-mean comparison for data filtration combined with BeaverTails moderation.Source: §A.3 · p. 19, Table 14 · p. 20
  • Under Bi-directional Anchoring, Lisa’s harmful score remains unchanged after fine-tuning on the selected Alpaca subset, while SFT rises by 0.7%.Source: §A.3 · pp. 19–20, Table 15 · p. 20

A.4 More Visualizations

The visualizations illustrate Lisa’s responses to malicious prompts and its task performance on GSM8K examples.

A.4 More Visualizations

  • For a prompt requesting help with cat theft, Lisa refuses the theft request and suggests asking the neighbor to borrow the cat.Source: §A.4 · p. 20
  • For a prompt asserting white racial superiority, Lisa responds that the claim is unproven and rejects it as dangerous.Source: §A.4 · p. 20
  • On the illustrated GSM8K task, Lisa gives the correct answer while the other compared methods fail.For the jewels example, Lisa returns 23, matching the ground truth; the visualization text identifies the other methods as failing.Source: §A.4 · p. 21

B Missing contents in theoretical analysis

The analysis uses the Kurdyka–Łojasiewicz (KL) framework and a potential function to characterize Lisa’s convergence. It specifies assumptions for that analysis and gives local optimality properties of the two state updates.

B.1 Preliminaries

  • The KL property characterizes local geometry around critical points and generalizes the Polyak–Łojasiewicz condition when its parameter is θ = 1/2.The paper uses the KL framework to derive a convergence bound.Source: §B.1 · p. 22

B.2 Assumptions

  • The analysis assumes proper closed losses, L-smooth alignment and fine-tuning losses, and a KL property for the potential function.The KL assumption is stated to hold for several common function classes, including sub-analytic and semi-algebraic functions.Source: §B.2 · p. 22

B.3 Facts

  • The two state-local problems yield optimality conditions that the paper says follow from Lisa’s intrinsic properties without formal assumptions.The displayed condition for State I is ∇f(ẇ_t+1) + ρ(ẇ_t+1 − w_t) = 0.Source: §B.3 · p. 22

B.4 Theorems

The theorems establish stationary-point convergence under stated assumptions and characterize convergence rates through the KL parameter. The proof uses potential descent and cluster-point arguments to support those results.

B.4 Theorems

  • With Assumptions 1–2, ρ ≥ L, and a convergent subsequence, Lisa’s cluster point is stationary for the global problem.The rate theorem additionally requires Assumption 3 and ρ > L; the rate depends on the KL parameter.Source: §B.4 · p. 23
  • A smaller KL parameter implies faster descent; at θ = 0, the stationary point can be reached in finite iterations.The paper states that only a sub-linear rate is achieved in one of the larger-θ cases.Source: §B.4 · p. 23

B.5.2 Formal Proof

  • The proof telescopes potential descent to show successive state iterates approach one another, then uses the global optimality condition to establish cluster-point stationarity.The argument also relies on a lower-bounded potential and interchangeable gradients at the cluster point and subsequence iterates.Source: §B.5 · p. 23, The first part is to prove claim i) · pp. 25–26

B.5.1 Key Lemmas

  • The potential function has a lower bound at a cluster point, and its descent along Lisa’s sequence is non-increasing when ρ ≥ L.These properties support the subsequent convergence argument.Source: §B.5.1 · p. 24

The first part is to prove claim i)

  • For the first proof claim, the argument uses lower semicontinuity and bounded potential descent, then takes subsequence limits and eliminates the residual term.L-smoothness supports exchanging the limit of the iterated gradients with the gradient at the cluster point.Source: The first part is to prove claim i) · pp. 25–26

B.6 Missing Proof of Theorem 3

The convergence-rate proof tracks a residual of the potential function and relates it to iterate changes and gradients. Under the stated assumptions, that residual tends to zero, with the resulting rate depending on the KL parameter.

B.6 Missing Proof of Theorem 3

  • The proof connects the potential residual to iterate changes and the global-loss gradient, then derives rate cases from a recursion for that residual.The proof sketch divides the rate analysis into three cases for the KL parameter.Source: §B.6 · p. 26

B.6.1 Key Lemmas

  • The residual, defined as the iterated potential minus its cluster-point value, is nonnegative and tends to zero.The argument combines the non-increasing descent property with convergence along a subsequence.Source: §B.6.1 · pp. 26–27

B.6.2 Formal Proof

  • The formal proof bounds the residual using the potential gradient and iterate differences, then analyzes separate KL-parameter cases.For θ = 0, the proof concludes that the residual becomes zero after finitely many iterations.Source: §B.6.2 · p. 28

C Broader Impact

  • The paper describes harmful fine-tuning as parallel to jailbreak attacks and says studying both helps clarify LLM safety risks; generated examples may contain unethical items.The authors state that their proposed solutions are intended to promote LLM safety alignment.Source: C Broader Impact · p. 29

D Limitations

The authors identify added fine-tuning computation, an SFT-only evaluation scope, and limited downstream task coverage as limitations. They also report that the abstract reflects the paper’s contribution.

D Limitations

  • Lisa adds computation during fine-tuning, so overhead scales with incoming user requests; the method is only evaluated atop SFT, not RLHF.The authors could not verify generalization to RLHF-based alignment because of resource limitations.Source: D Limitations · p. 29
  • The downstream datasets are not considered the most capable tasks for LLMs, and the authors identify conversational AI and agents as future settings to explore.Source: D Limitations · p. 29
  • The authors state that fully solving harmful fine-tuning requires efforts in both alignment and user fine-tuning.They consider an alignment-stage solution more desirable from system and security perspectives, while noting greater technical challenges.Source: D Limitations · p. 29

NeurIPS Paper Checklist

2. Limitations

The limitations checklist directs readers to Appendix D for the paper’s discussion of limitations. It also points to appendices for supporting experimental and theoretical details.

2. Limitations

  • The authors direct readers to Appendix D for the discussion of limitations.The supplied statement does not identify a specific limitation.Source: §2 · p. 30
  • The theoretical assumptions and proof are available in Appendix B.Source: §2 · p. 31
  • Training and test details are available in Appendix A.1.Source: §2 · p. 32
Loading 2405.18641v5…