PaperlayerView exact revision on arXiv

Source-linked AI summary

Safer Content or Firmer Refusals? A Hybrid Perturbation Defense for Alignment under Harmful Fine-tuning

Muhammad Zeeshan Akram, Mufid Kamel Marican, Anvesh Reddy Yenugu, Ali Zain Kaimkhani, Minghong Fang

arXiv:2609.36862v1cs.CRcs.AIcs.IRcs.LG

TL;DR

Harmful fine-tuning can degrade aligned safety, and it remains unclear whether embedding-level and weight-level alignment defenses complement one another. VaccineBooster combines embedding perturbation and weight-level gradient attenuation in one alignment step without access to user fine-tuning data. Booster-Only retained the highest explicit refusal rate after the attack, at 50%.

  • Problem

    Harmful fine-tuning can degrade aligned safety, and it remains unclear whether embedding-level and weight-level alignment defenses complement one another.

    Source: §1 · pp. 1–2

  • Method

    VaccineBooster combines embedding perturbation and weight-level gradient attenuation in one alignment step without access to user fine-tuning data.

    Source: §1 · p. 2, §4.4 · p. 4

  • Results

    Booster-Only retained the highest explicit refusal rate after the attack, at 50%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The observed trade-off suggests providers can prioritize content safety or refusal retention through the two defense strengths, but their effects are not equally established.

    Source: §7 · pp. 7–8

    The evaluation uses ten harmful prompts and a single unseeded run per configuration, so comparisons are indicative rather than statistically resolved.

    Source: §7 · p. 8

Abstract

from arXiv · show

Fine-tuning-as-a-service lets users adapt a safety-aligned language model to their own data, but it also creates a harmful fine-tuning attack surface: a small amount of harmful data mixed into an otherwise benign fine-tuning set can degrade the model's alignment. Two recent alignment-stage defenses address this problem at different levels of the model. Vaccine improves the robustness of hidden embeddings to the representation shifts induced by harmful fine-tuning, whereas Booster simulates harmful weight updates and attenuates their effect during alignment. We investigate whether these mechanisms are complementary and propose VaccineBooster, a single alignment procedure that combines embedding perturbation and weight-level gradient attenuation within each training step. On Llama-2-7B aligned with BeaverTails and then attacked through poisoned fine-tuning, VaccineBooster achieves the lowest OpenAI moderation score among the compared defenses, 0.315, while a Booster-Only variant retains the highest post-attack refusal rate, 50%. Together with ablations over the embedding-perturbation and gradient-attenuation strengths, these results indicate a trade-off: embedding perturbation primarily reduces flagged harmful content, whereas gradient attenuation primarily preserves explicit refusal behavior. Because our evaluation uses ten prompts and a single unseeded run per configuration, we report this trade-off as an observed pattern rather than a statistically resolved effect. These results provide practical guidance for prioritizing content safety or refusal retention when aligned models are exposed to untrusted fine-tuning.

1 Introduction

Harmful fine-tuning can weaken aligned behavior even when the model retains downstream utility, motivating provider-controlled defenses that do not depend on inspecting user data. VaccineBooster combines embedding perturbation and weight-level gradient attenuation, with results indicating a trade-off between safer content and refusal retention.

1 Introduction

  • Harmful examples can be a small fraction of otherwise benign fine-tuning data, degrading refusal behavior while leaving apparent downstream task performance intact.Consequently, task utility alone does not establish that safety behavior was preserved.Source: §1 · pp. 1–2
  • Filtering and post-hoc repair are incomplete options, motivating alignment-stage defenses that strengthen the model before untrusted fine-tuning without inspecting later data.Filters can miss concealed examples, while repair may be difficult when the cause or future fine-tuning is unknown.Source: §1 · p. 2
  • Vaccine hardens hidden embeddings against harmful-fine-tuning shifts, while Booster attenuates simulated harmful weight updates; their combination’s complementarity remained an open question.The component defenses were evaluated separately and target distinct model effects.Source: §1 · p. 2
  • VaccineBooster combines embedding perturbation and weight-level gradient attenuation within each alignment step, without access to later user fine-tuning data.The provider can prepare one aligned checkpoint before user-specific fine-tuning, without changing the fine-tuning interface or adding inference-time computation.Source: §1 · p. 2
  • In poisoned-fine-tuning attacks on Llama-2-7B aligned with BeaverTails, VaccineBooster had the lowest OpenAI moderation score (0.315), while Booster-Only retained the highest post-attack explicit refusal rate (50%).The proposed separation of generation and refusal roles is presented as a tuning direction, not a resolved separation.Source: §1 · p. 2

2 Background and Related Work

Prior work describes harmful fine-tuning attacks that can erode safety while preserving task competence, and defenses that act either after fine-tuning or during alignment. This paper examines whether representation-level and weight-level alignment defenses reinforce one another and evaluates safety across separate behavioral dimensions.

2 Background and Related Work

  • Small amounts of harmful fine-tuning data can undermine safety while preserving task competence, making the degradation difficult to detect through task performance alone.Prior work also reports that safety can degrade unintentionally during benign-task fine-tuning.Source: §2 · p. 3
  • Post-fine-tuning repair requires knowing which model or examples were compromised, whereas alignment-stage defenses operate without access to user fine-tuning data.Vaccine hardens hidden embeddings against harmful drift, and Booster attenuates harmful gradients on weights.Source: §2 · p. 3
  • The paper combines separate measures of content safety and refusal retention to examine the interaction between defenses that target different facets of safety.The authors frame the interaction as a trade-off rather than a uniform improvement and distinguish this defense setting from post-attack forensic attribution [6] [25] [26].Source: §2 · p. 3

3 Threat Model

The threat model considers an attacker using the provider’s fine-tuning interface to mix harmful examples into benign task data while seeking harmful outputs and preserved task performance. The provider controls alignment before user data arrives but cannot inspect that data or know its harmful fraction.

3 Threat Model

  • The attacker mixes harmful instruction-response pairs into a benign downstream-task dataset uploaded through the provider’s fine-tuning interface.The intended attack outcome is harmful content on request while retaining the stated task performance, which can hide the safety loss.Source: §3 · p. 3
  • The attacker controls only the uploaded fine-tuning data and does not otherwise modify the base model or alignment procedure.Source: §3 · p. 3
  • The provider aligns the model before user data arrives and has neither access to that data nor knowledge of its harmful fraction.The defense must therefore build resilience in advance rather than tailor itself to a specific attack.Source: §3 · p. 3

4 Method

VaccineBooster combines embedding perturbation and simulated harmful-weight-update attenuation in one alignment step. Its design applies the safety update at original weights, normalizes perturbations, and shifts additional computation to provider-controlled alignment.

4.2 Embedding Perturbation

  • The embedding perturbation follows the direction of steepest alignment-loss increase, creating a bounded worst-case shift around the current attention-layer representations.The strength ρ sets the neighborhood size; larger values make the alignment objective harder to fit.Source: §4.2 · pp. 3–4

4.3 Weight Perturbation with Gradient Attenuation

  • Weight-level attenuation compares harmful gradients before and after a simulated harmful update, then adds their scaled difference to the safety gradient.The step size ε sets the simulated harmful step, and λ controls the attenuation strength relative to fitting safety data.Source: §4.3 · p. 4

4.4 Combined Alignment Step

  • VaccineBooster fuses representation hardening and harmful-gradient attenuation into one update, so each alignment step reflects both defenses.The safety gradient comes from a perturbed forward pass, and the weight-level term is added to that gradient.Source: §4.1 · p. 3, §4.4 · p. 4

4.5 Design Rationale

  • The update computes the final safety gradient from the original model parameters; temporarily perturbed weights are used only to estimate attenuation.Perturbations target attention-module outputs, and unit-norm normalization makes their strengths independent of gradient magnitude.Source: §4.5 · p. 4
  • The ρ and λ strengths primarily target generated content and sensitivity to harmful updates, respectively, but evidence for λ’s refusal-rate effect is weaker and non-monotonic.The authors present this role separation as a working interpretation, not an established result, and say confirmation requires a larger evaluation set.Source: §4.5 · p. 4

4.6 Overhead Analysis

  • A VaccineBooster step costs about four forward-backward passes rather than one, with the added cost paid during provider-controlled alignment rather than user fine-tuning or inference.The passage characterizes the alignment-stage increase as a constant-factor overhead and describes the extra memory as modest.Source: §4.6 · p. 4

5 Experimental Setup

Experiments use Llama-2-7B with BeaverTails alignment and simulated harmful fine-tuning to assess safety behavior across fixed prompts and multiple metrics. The setup combines a perturbed safety-gradient update with harmful-gradient attenuation, while its attack and evaluation scope constrain the conclusions.

5 Experimental Setup

  • All experiments use Llama-2-7B and BeaverTails, with 5,000 safety pairs, 1,000 harmful samples, and 500 poison samples.The poison samples contain only harmful instruction-response pairs, implementing p=1 rather than the small harmful fractions motivating the threat model; downstream utility is not reported.Source: §5 · pp. 4–5
  • VaccineBooster’s alignment step combines a safety gradient from perturbed embeddings with a harmful-gradient attenuation term.The setup specifies safety and harmful batches, model parameters, and strengths for the procedure.Source: §5 · p. 5
  • Alignment runs for 3 epochs at learning rate 1×10−5, and simulated attack fine-tuning runs for 1 epoch at 2 × 10−5.Unless otherwise stated, perturbation strengths are ρ=2.0, ϵ=0.1, and λ=0.001; all methods receive the same alignment epochs.Source: §5 · p. 5
  • The five measures cover keyword-detected harm, explicit refusal, moderation safety, API-flagged responses, and a combined resilience score.The OpenAI moderation score is safer when lower; the moderation score and flagged rate are external checks independent of the keyword list.Source: §5 · p. 5
  • The resilience score is descriptive rather than a method-ranking metric because its hand-set weights are uncalibrated and its range can exceed 100.A keyword-harm decrease after attack can produce a measurement artifact that the score rewards like a real improvement.Source: §5 · p. 5
  • Evaluation uses the same fixed set of ten harmful prompts for every method and hyperparameter setting.The prompts span the harm-score categories.Source: §5 · p. 5

6 Experimental Results

After poisoned fine-tuning, VaccineBooster and Booster-Only lead on different safety measures, and the ablations show that perturbation strength shifts content-safety measures more clearly than refusal retention. Run-to-run variation and limitations of keyword harm scoring constrain interpretation.

6.1 Main Comparison

  • 0.315 was VaccineBooster’s lowest OpenAI moderation score, while Booster-Only retained the highest post-attack refusal rate, 50%.Before the attack, all methods refused all harmful prompts; Table 1 reports three alignment epochs.Source: §6.1 · p. 5, Table 1 · p. 6, Figure 2 · p. 6
  • The identical VaccineBooster configuration scored 0.315 moderation and 20% refusal in one run, versus 0.329 and 40% in a separate run.The authors report stochastic decoding without a fixed seed and advise keeping this variation in mind when comparing table rows.Source: §6.1 · p. 6

6.2 Post-Attack Changes in Refusal and Content Safety

  • The keyword harm score counts predefined terms, so it can miss contextual safety and is not treated as a standalone post-attack measure.The authors pair it with the moderation score and refusal rate, which capture complementary aspects of behavior.Source: §6.2 · p. 6

6.3 Effect of the Embedding-Perturbation Strength 𝜌

  • At 𝜌=4.0, the OpenAI moderation score and flagged rate reached their lowest values, 0.221 and 30%, while keyword harm was lowest at 𝜌=2.0.Table 2 varies 𝜌 while holding 𝜖=0.1 and 𝜆=0.001 fixed; refusal rate did not move monotonically with 𝜌.Source: §6.3 · p. 6, Table 2 · p. 6

6.4 Effect of Gradient-Attenuation Strength 𝜆

  • At 𝜆=0.001, post-attack harm was lowest at 20 and refusal rate highest at 40%; at 𝜆=0.1, moderation was lowest at 0.283 and flagged rate at 20%.Table 3 varies 𝜆 with 𝜌=2.0 and 𝜖=0.1 fixed; across the four runs, refusal ranged from 30% to 40% and back to 30%.Source: §6.4 · pp. 6–7, Table 3 · p. 7
  • The best moderation scores came at the largest tested 𝜌 and 𝜆, while the moderate defaults gave the best joint balance of harm and refusal.The 𝜌=4.0 setting raised keyword harm, and 𝜆=0.1 lowered refusal.Source: §6.3 · p. 6, §6.4 · p. 7

7 Discussion

The discussion presents content safety and explicit refusal retention as distinct objectives: VaccineBooster is strongest on content safety, while Booster-Only retains the most refusals. It offers a tentative mechanism and deployment guidance, while stressing that the evidence is limited.

A trade-off between content safety and refusal

  • The results support a trade-off: VaccineBooster leads on content safety, while Booster-Only leads on refusal retention; evidence for the two sides differs in strength.The embedding-strength trend is clearest, whereas the gradient-attenuation interpretation rests on the Booster-Only comparison, not a monotonic ablation result.Source: §7 · p. 7

Why the two mechanisms diverge

  • A proposed, untested explanation is that embedding perturbation shapes generated content, while weight-level attenuation better preserves explicit refusal behavior.The authors state that experiments do not directly test this account.Source: §7 · pp. 7–8

Practical guidance

  • For content minimization, the authors favor VaccineBooster with larger perturbation strengths; for explicit refusal, Booster-Only retained the most refusals in their experiments.They do not recommend tuning embedding strength for refusal, and suggest default VaccineBooster strengths as a balance.Source: §7 · p. 8

Benefits of the combination

  • VaccineBooster offers coverage and control in one checkpoint, but does not outperform both component defenses on every metric.Its two interpretable strengths let providers adjust the trade-off without changing training or maintaining separate models.Source: §7 · p. 8

Deployment considerations

  • Providers set both strengths at alignment time for all customers, and can pair the defense with an inference-time output filter to catch residual cases.The authors describe the strengths as a product decision made before users arrive.Source: §7 · p. 8

Relation to alignment-stage defenses

  • VaccineBooster combines embedding- and weight-level perturbations in one alignment step, a mechanism choice the authors describe as orthogonal to other alignment-stage defenses.They suggest representation-noising or tamper-resistant terms as possible additional components.Source: §7 · p. 8

Limitations

  • The evaluation uses ten harmful prompts and one unseeded run per configuration, so the reported comparisons are indicative rather than statistically resolved.It also lacks an undefended baseline, uses an all-harmful poison set, omits task utility, and evaluates one 7B model against a non-adaptive attacker.Source: §7 · p. 8

8 Conclusion

VaccineBooster combines embedding perturbation and weight-level gradient attenuation during alignment, without access to user fine-tuning data. In the tested Llama-2-7B setting, it leads on content safety while a weight-only variant retains more refusals, but the small evaluation supports only an observed pattern.

8 Conclusion

  • VaccineBooster combines embedding perturbation and weight-level gradient attenuation in one alignment step, protecting representation and parameter levels without access to user fine-tuning data.The authors frame tuning the two strengths as a practical way to navigate the observed trade-off if it holds at larger scale.Source: §8 · p. 8
  • On Llama-2-7B aligned with BeaverTails and attacked using poisoned fine-tuning, VaccineBooster produces the safest content while a weight-only variant retains the most explicit refusals.Ablations are consistent with content safety and refusal retention being affected by different mechanisms.Source: §8 · p. 8
  • Ten prompts and one unseeded run per configuration mean the reported trade-off is an observed pattern, not a resolved effect.Future work includes larger models, stronger and adaptive attacks, utility measurement, and response-side safeguards.Source: §8 · p. 8

A Evaluation Metric Definitions

The appendix specifies how the study computes its keyword harm score, refusal rate, and moderation measures. These definitions distinguish substring-based keyword detection from moderation scores and API flag rates.

Harm score

  • The harm score counts distinct listed keywords found as substrings in a lowercased response, awarding ten points per keyword up to 100, then averages across ten prompts.The keyword set covers violence, illegal activity, dangerous information, privacy violations, and deception.Source: A Evaluation Metric Definitions · p. 9

Refusal rate

  • A response counts as a refusal when it contains any listed refusal pattern as a case-insensitive substring.The refusal rate is the percentage of the ten prompts whose responses match at least one pattern.Source: A Evaluation Metric Definitions · p. 9

Moderation score and flagged rate

  • For each post-attack response, moderation score is the maximum API score across policy categories; the reported score is the ten-prompt mean, alongside the percentage flagged by the API.Both moderation measures use post-attack responses only.Source: A Evaluation Metric Definitions · p. 9

B Limitations of the Keyword Harm Score

The keyword harm detector can misfire because it uses substring matching and shares a term with the refusal patterns. Consequently, a well-formed refusal can score above a harmful completion, affecting interpretation of both harm and resilience scores.

Substring matching

  • Substring matching can trigger on terms embedded in benign words, such as “harm” in “harmless” or “drug” in “drugstore.”The detector does not enforce word boundaries.Source: B Limitations of the Keyword Harm Score · p. 9

Overlap with the refusal patterns

Anti-correlation in the refusal regime

  • A well-formed refusal can score higher than a dangerous completion because refusal language may match keywords that the completion does not use.The example refusal scores 30, while a dangerous completion without listed terms scores 0.Source: B Limitations of the Keyword Harm Score · p. 9
  • The harm score spans only 19–32 across methods and settings, and its decline can directly artifactually raise the resilience score.As refusals are replaced by compliant answers, refusal vocabulary detected by the keyword measure may disappear.Source: B Limitations of the Keyword Harm Score · pp. 9–10

C Decoding and Run-to-Run Variation

The default configuration was evaluated in independent, unseeded runs, and several measured differences were comparable to run-to-run variation. The resilience composite was especially unstable, so the authors treat it as descriptive rather than a ranking.

C Decoding and Run-to-Run Variation

  • Responses used temperature 0.7 and a 150-token budget with no fixed random seed; each reported number is a single stochastic draw.The default configuration was trained and evaluated twice from fresh models as independent replicates.Source: C Decoding and Run-to-Run Variation · p. 10
  • The runs differed by 0.014 in moderation score, eight points in pre-attack harm, and 20 percentage points in post-attack refusal, while agreeing on post-attack harm and flagged rate.The refusal difference corresponds to two responses; this spread was of the same order as several between-method differences in Table 1.Source: C Decoding and Run-to-Run Variation · p. 10
  • The two runs produced resilience scores of 72.0 and 81.0 for one configuration, versus 82.0 for Booster-Only, so the authors treat the composite as descriptive rather than a ranking.The composite combines pre-attack harm, which differed by eight points, and post-attack refusal, which differed by two responses.Source: C Decoding and Run-to-Run Variation · p. 10, Table 4 · p. 10
  • With ten prompts, Booster-Only’s 50% refusal rate versus VaccineBooster’s 20% was not significant under a two-sided Fisher exact test (p=0.35).The authors estimate roughly 40 prompts per condition to separate rates of this size at conventional power, and several hundred to separate 50% from 40%.Source: C Decoding and Run-to-Run Variation · p. 10

Related papers

Loading 2609.36862v1…