PaperlayerView exact revision on arXiv

Source-linked AI summary

Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation

Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Ling Liu

arXiv:2501.17433v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Harmful fine-tuning can compromise LLM safety alignment, while moderation filters uploaded harmful data. Virus optimizes harmful samples to evade that filter while retaining attack performance, and experiments show that it can bypass moderation and degrade alignment.

  • Problem

    A few harmful fine-tuning samples can compromise LLM safety alignment, and the paper asks whether such an attack can bypass guardrail moderation while remaining effective.

    Source: §1 · p. 1

  • Method

    Virus jointly optimizes harmful data to reduce guardrail jailbreak loss and make its victim-model gradient resemble the harmful-data gradient.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    Virus reaches up to 100% leakage ratio while increasing the victim LLM’s harmful score by up to 21.8%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Guardrail moderation alone cannot sufficiently address the risk of harmful fine-tuning, as Virus bypasses it while retaining attack performance.

    Source: §7 · p. 8

    Experiments use open-weight LLMs and a guardrail model in a local experimental environment.

    Source: §8 · p. 9

Abstract

from arXiv · show

Recent research shows that Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- models lose their safety alignment ability after fine-tuning on a few harmful samples. For risk mitigation, a guardrail is typically used to filter out harmful samples before fine-tuning. By designing a new red-teaming method, we in this paper show that purely relying on the moderation guardrail for data filtration is not reliable. Our proposed attack method, dubbed Virus, easily bypasses the guardrail moderation by slightly modifying the harmful data. Experimental results show that the harmful data optimized by Virus is not detectable by the guardrail with up to 100\% leakage ratio, and can simultaneously achieve superior attack performance. Finally, the key message we want to convey through this paper is that: \textbf{it is reckless to consider guardrail moderation as a clutch at straws towards harmful fine-tuning attack}, as it cannot solve the inherent safety issue of the pre-trained LLMs. Our code is available at https://github.com/git-disl/Virus

1. Introduction

The section asks whether harmful fine-tuning can evade data moderation while still degrading an LLM’s safety alignment. It introduces Virus as a dual-goal data optimizer to address the gap exposed by prior attempts.

1. Introduction

  • The motivating concern is that a few harmful fine-tuning samples can compromise safety alignment, while moderation filters uploaded data before fine-tuning.The paper frames whether an attack can bypass this moderation and still degrade the victim model’s alignment.Source: §1 · p. 1
  • A benign-plus-harmful QA concatenation failed to bypass moderation, while single-goal data optimization bypassed it but introduced gradient mismatch.The mismatch left the optimized data insufficient to break the victim LLM’s safety alignment.Source: §1 · pp. 1–2
  • Virus reaches up to 100% leakage ratio while increasing the victim LLM’s harmful score by up to 21.8%.Its optimized data is designed both to jailbreak the guardrail and to preserve gradients resembling those of harmful data.Source: §1 · p. 2
  • The authors report systematic measurements of harmful fine-tuning under moderation and experiments showing Virus bypasses moderation while breaking victim-model safety alignment.The paper describes Virus as a dual-goal optimization method developed after the two earlier attempts.Source: §1 · p. 2

2. Related Work

The related work covers safety alignment, harmful fine-tuning attacks and defenses, and moderation guardrails. The authors state that harmful fine-tuning under guardrail moderation had not received a systematic study to their knowledge.

2. Related Work

  • Prior work studies safety alignment through SFT, RLHF, and other alignment techniques.The passage lists approaches including Stable Alignment, Selfee, Circuit Breakers, and H3Fusion.Source: §2 · p. 2
  • Studies report that partially harmful fine-tuning can compromise alignment, while later attacks select benign examples or use covert malicious fine-tuning.The covert method is described as requiring encrypted harmful queries at deployment, unlike the paper’s target attack setting.Source: §2 · p. 2
  • Defenses are grouped by whether they act during safety alignment, fine-tuning, or after fine-tuning.The review also notes mechanism studies and harmful fine-tuning investigations in settings such as federated learning and diffusion models.Source: §2 · p. 2
  • Guardrail moderation classifies and filters harmful samples from fine-tuning data, and the authors report no prior systematic study of harmful fine-tuning attacks under this moderation setting.The passage names several available moderation systems, including Llama Guardrail, Granite Guardian, WildGuard, and AegisGuard.Source: §2 · p. 2

3. Preliminaries

The paper models a provider-run fine-tuning service in which alignment, guardrail filtering, and fine-tuning occur in sequence. Its evaluation examines harmful and downstream-task performance, baseline moderation, and whether an attacker can construct data to bypass the filter.

3.1. Setting

  • The pipeline first aligns the model, filters uploaded harmful samples with a guardrail, and fine-tunes the aligned model on the filtered data.The provider controls the process; users control the uploaded dataset, including its harmful-sample percentage p.Source: §3.1 · p. 3

3.2. Evaluation

  • Evaluation measures harmful score and fine-tune accuracy after 20 epochs using Llama3-8B, Llama Guard2, and GSM8K.The setup passage identifies the victim model, guardrail, downstream task, and evaluation duration.Source: §3.2 · p. 3
  • Higher harmful-data ratios significantly increase harmful score, while mixing a few harmful samples does not significantly alter downstream-task accuracy.The authors report that the victim model can generalize to harmful and downstream tasks simultaneously.Source: §3.2 · p. 3
  • Benign GSM8K fine-tuning leaves harmful score at roughly 4%, which the authors say contradicts Qi et al. (2023) and may depend on particular settings.The passage presents the setting-dependence explanation as a conjecture.Source: §3.2 · p. 3
  • Llama Guard2’s average leakage ratio is 0.348; moderation lowers harmful score relative to unmoderated harmful fine-tuning without significantly changing downstream accuracy.The authors conclude that moderation is effective against the tested attack and then ask whether an adversary can construct harmful data to bypass it.Source: §3.2 · p. 3, Figure 2 · p. 3

4. Methodology

The section tests concatenation and single-goal optimization before introducing Virus, which optimizes merged harmful data for both guardrail evasion and attack-gradient preservation.

4.1. Failure attempt I: Mixing attack

  • Only 38% of harmful data passed moderation, compared with all benign GSM8K data; moderated HFA scored 13.50 above BFA but 38.2% below unmoderated HFA.The authors attribute the lower score versus unmoderated HFA to moderation filtering out 62% of harmful data.Source: §4.1 · p. 3, Table 1 · p. 4
  • Mixing concatenates a harmful QA with a benign answer, but under moderation leakage rose only 6% while harmful score fell 3.2%.Without moderation, Mixing had a harmful score of 35.30.Source: Table 2 · p. 4, §4.1 · p. 4

4.2. Failure attempt II: single goal guardrail jail-break

  • 100% leakage did not make single-goal guardrail jailbreak successful: its harmful score was 0.3 below Mixing, with gradient similarity of 0.826.The authors propose gradient mismatch as a reason that guardrail-jailbroken data may fail to sufficiently compromise safety alignment.Source: Table 3 · p. 5, §4.2 · p. 5

4.3. Virus: dual goal data optimization

  • Virus weights guardrail-jailbreak loss against similarity between victim-LLM gradients on original mixing and optimized data, then uses GCG to optimize the data.The weighting parameter λ balances the two goals.Source: §4.3 · p. 5
  • 16.00 higher harmful score than Mixing, with 100% leakage and 0.981 gradient cosine similarity, was reported for Virus.The optimized data both passed moderation and retained similarity to the original mixing-data gradient.Source: Table 4 · p. 5, §4.3 · p. 5

5. Experiment

The experiments evaluate Virus in a guardrail-filtered fine-tuning pipeline, comparing its attack performance, data leakage, statistical behavior, computational overhead, and sensitivity to λ. Across evaluations, Virus improves harmful-score outcomes over HFA, while the λ analysis shows a trade-off between guardrail evasion and gradient similarity.

5.2. Main Evaluations

  • Virus increases average harmful score by 11.54% over HFA across harmful ratios, with a 1.18% increase in fine-tune accuracy.All harmful fine-tuning methods incur a slight reduction in fine-tune accuracy.Source: Table 5 · p. 6, §5.2 · p. 6
  • At fixed harmful ratio, Virus’s average harmful score is 10.06% higher than HFA; increasing n from 100 to 1000 raises Virus fine-tune accuracy by 13%.Over the same sample-number range, harmful scores rise by 28.7% for HFA and 36% for Virus.Source: Table 6 · p. 6, §5.2 · pp. 6–7

5.1. Setup

  • Experiments use Llama3-8B and Llama Guard 2, BeaverTails harmful data, and benign SST2, AgNews, and GSM8K data; metrics include harmful score, fine-tune accuracy, and leakage ratio.The simulated pipeline includes safety alignment, guardrail filtering, and victim-model fine-tuning.Source: §5.1 · pp. 5–6

5.2. Main Evaluations

  • Across three fine-tuning tasks, Virus consistently achieves the highest harmful score among the evaluated methods, averaging 10.2% compared to HFA.BFA averages 2.53% harmful score across the tasks.Source: Table 7 · p. 7, §5.2 · p. 7

5.3. Statistical Evaluation

  • At λ = 0, harmful loss on the original mixing data falls nearly to 0; at λ = 1, it fails to decrease sufficiently and rises after 1200 steps.The paper identifies the λ = 1 setting with its guardrail-jailbreak attempt.Source: §5.3 · p. 7, Figure 4 · p. 7
  • For λ = 1, gradient similarity starts low and decreases during fine-tuning; with smaller λ, initial similarity is larger and remains fairly stable across later steps.The authors interpret initial gradient matching as sufficient to keep similarity from changing substantially along the optimization trajectory.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4. System Evaluation

  • Virus data optimization for a Llama2-8B attack uses under 40 GB of GPU memory; producing 50 samples costs $34.43 on H100 GPUs.One attack sample requires 0.2649 GPU hour, and the gradient-similarity loss adds approximately 2.25 GB of memory overhead.Source: §5.4 · pp. 7–8, Table 8 · p. 8

5.5. Hyper-parameter Analysis

  • A higher λ increases leakage ratio, but if λ is too high, reduced gradient similarity can lower harmful score and fine-tune accuracy.A relatively moderate λ setting does not significantly alter fine-tune accuracy.Source: Table 9 · p. 8, §5.5 · p. 8

6. Visualization

The visualizations show how Virus modifies combined benign and harmful fine-tuning data and how victim-model responses differ after fine-tuning. The examples contrast guardrail passage and harmful responses for Virus with refusal behavior from baseline attacks.

6. Visualization

  • The text reports that fine-tuning on Virus-optimized data produces a concrete harmful response, while HFA does not successfully trigger one under guardrail protection.The illustrated contrast concerns a harmful question posed to the fine-tuned model.Source: §6 · p. 8
  • The example combines benign and harmful data, then shows Virus-modified harmful text that passes Llama Guard 2 after only a few words change.The altered address becomes an unreadable phrase in the visualization.Source: §6 · p. 8
  • In the victim-model example, BFA and HFA produce refusals, whereas the Virus-attacked model provides harmful assistance.The displayed Virus response includes concrete advice to the harmful request.Source: §6 · p. 8

7. Conclusion

The paper concludes that guardrail moderation can filter most harmful samples, but Virus can bypass it while maintaining attack performance against the LLM. Its experiments support the dual objective of bypassing moderation and attacking the victim model.

7. Conclusion

  • Guardrail moderation filters most harmful fine-tuning samples, but the paper reports that this protection is not sufficient against Virus.Source: §7 · p. 8
  • Virus is presented as a data-optimization red-teaming method that bypasses guardrail moderation.Source: §7 · p. 8
  • The experiments verify Virus’s dual goals of bypassing moderation and maintaining attack performance toward the LLM.Source: §7 · p. 8

8. Impact Statements

The paper studies an LLM user fine-tuning API vulnerability using open-weight models and a locally run guardrail model. It acknowledges possible misuse and says the source code is available to support inspection and precautionary action.

8. Impact Statements

  • All experiments use open-weight LLMs and a guardrail model in a local experimental environment.Source: §8 · p. 9
  • The authors warn that the vulnerability might be misused to attack commercial LLM services and cause societal harm.Source: §8 · p. 9
  • The source code is made available so interested parties can inspect it and take precautionary action.Source: §8 · p. 9

A. Implementation Details

The formulation represents optimizable harmful data as a flattened one-hot vector, constrained to select one vocabulary token at each position. Its dual loss combines guardrail prediction of “safe” with similarity between gradients for original and optimized data.

A.1. Formal Formulation

  • Each token position is encoded by a segment of a flattened one-hot vector, with one selected vocabulary token per segment.The vector has dimension d = n|V|, where n is the number of optimizable prompt tokens and |V| is vocabulary size.Source: §A.1 · p. 13
  • The dual loss uses F1 to measure prediction of “safe” after benign data and the optimized vector, and F2 to compare gradients for original and optimized data.The original input is benign plus harmful data; the optimized input is benign plus the optimizable vector.Source: §A.1 · p. 13
  • The authors use a flattened one-hot optimization variable rather than the integer-form variable used by Zou et al. (2023), and describe their formulation as similar to Hu et al. (2024).Source: §A.1 · p. 13

A.2. GCG optimizer

The paper adopts GCG to optimize discrete prompt or data tokens, using gradient-guided replacement candidates and selecting the lowest-objective iterate from a batch. The authors choose it over alternatives they describe as less effective or less established.

A.2. GCG optimizer

  • GCG is presented as an efficient optimizer for discrete language-domain prompt or data optimization, and one step is expected to yield a lower objective.Source: §A.2 · pp. 13–14
  • The authors adopt GCG because they describe projected-gradient alternatives as less effective due to projection error and GCG variants as community-recognized effective optimizers.They cite ADC as an approach intended to mitigate projection error, while retaining GCG for simplicity.Source: §A.2 · pp. 13–14
  • GCG ranks top-k replacement tokens by gradient for each position, samples single-position replacements into a batch, and advances with the candidate having the smallest objective.The step takes the current iterate, its gradient, and objective as inputs; it returns the selected next iterate.Source: §A.2 · p. 14

A.3. Guardrail jailbreak

The guardrail-jailbreak procedure optimizes harmful data against the guardrail’s safe-classification loss, omitting the gradient-similarity term used in the dual-goal method. The resulting iterate is intended to be classified as safe by the moderation model.

A.3. Guardrail jailbreak

  • The guardrail-jailbreak algorithm follows the dual-goal procedure but excludes the gradient-similarity term.Source: §A.3 · p. 14
  • It repeatedly evaluates the F1 gradient and applies GCG, producing an iterate that minimizes the guardrail-jailbreak loss and is classified as “safe” by the moderation model.The algorithm takes regularizer intensity, steps, aligned and moderation models, and benign and harmful data as inputs.Source: §A.3 · p. 14

B. Experimental Details

The experiments simulate alignment, fine-tuning, and testing with harmful and downstream-task data, using Llama3-8B and Llama Guard2 8B. The default attack setting uses 500 fine-tuning samples, a 10% harmful-data ratio, and GSM8K.

B. Experimental Details

  • Alignment and fine-tuning use SFT with LoRA, rank 32, and 20 epochs; the learning rates are 5e-4 and 1e-4, respectively.The stated batch sizes are 10 for alignment and 5 for fine-tuning; the fine-tuning rate is lower because the attack setting is harder.Source: B. Experimental Details · p. 15
  • For Virus, GCG uses batch size 128 and top-k 64, set according to the optimizer’s default setting.Source: B. Experimental Details · p. 15
  • The harmful dataset contains harmful-question/safe-answer pairs and serves as fine-tuning attack data, Virus’s initial optimization iterate, and test prompts for harmfulness scoring.The dataset is extracted from BeaverTails; the alignment data is a refinement attributed to Rosati et al. (2024b).Source: B. Experimental Details · p. 15
  • Downstream-task demonstrations are mixed with harmful data for attack simulation and used at test time to measure fine-tuning accuracy.The described tasks include SST2, GSM8K, and AGNEWS, with task-specific answer-correctness criteria.Source: B. Experimental Details · pp. 15–16
  • Experiments use Llama3-8B as the victim and Llama Guard2 8B as the guardrail model.Source: B. Experimental Details · p. 16
  • The study compares harmful fine-tuning attacks with benign fine-tuning attacks, which sample only benign data; it also describes public code and experiment scripts for reproduction.The harmful-attack baseline mixes a percentage of harmful data into benign fine-tuning data.Source: B. Experimental Details · p. 16
  • The default attack uses 500 fine-tuning samples, a 10% harmful-data ratio, and GSM8K as the downstream task.Benign fine-tuning is the special case p = 0.Source: B. Experimental Details · p. 16
Loading 2501.17433v1…