PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Noising: A Defence Mechanism Against Harmful Finetuning

Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, Frank Rudzicz

arXiv:2405.14577v4cs.CLcs.LG

TL;DR

LLMs can be harmful-fine-tuned even when safety guardrails initially prevent harmful outputs. RepNoise modifies harmful representations before weight access, and the paper reports information removal across all layers alongside resistance across tested attacks; its scope and robustness remain limited.

  • Problem

    Open and closed LLMs are vulnerable to harmful fine-tuning, which can circumvent inference-time guardrails and enable harmful use.

    Source: §1 · p. 1

  • Method

    RepNoise removes information structure from harmful representations before an attacker obtains the model weights, aiming to make those representations harder to recover through fine-tuning.

    Source: §1 · p. 1

  • Results

    RepNoise removed information about harmful representations across all LLM layers and was the only tested defense to provide significant resistance consistently across attacks (Mann–Whitney U-test, p < 0.001).

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    The findings support representation-level information removal as a defense against in-distribution harmful fine-tuning while preserving the ability to train on harmless tasks.

    Source: §1 · pp. 1–2

    RepNoise did not generalize from harmful-question-answering defenses to toxic-content-generation attacks and remains defeatable at higher learning rates with more data.

    Source: §7 · p. 10

Abstract

from arXiv · show

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that operates even when attackers have access to the weights. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process as long as they are drawn from the same distribution of the attack set. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the efficacy of our defence lies in its ``depth'': the degree to which information about harmful representations is removed across all layers of the LLM. We also find areas where RepNoise still remains ineffective and highlight how those limitations can inform future research.

1 Introduction

Open and closed LLMs are vulnerable to harmful fine-tuning, which can reverse inference-time guardrails. RepNoise aims to make harmful representations harder to recover after an attacker obtains model weights.

1 Introduction

  • RepNoise is proposed as a defense against in-distribution harmful fine-tuning attacks when the defender cannot control the model after the attacker obtains its weights.The proposal targets natural-language generation.Source: §1 · p. 1
  • RepNoise removes information structure from harmful representations to make them harder to recover during subsequent harmful fine-tuning.The paper contrasts this with safety mechanisms that suppress or route around harmful capabilities that remain in the model.Source: §1 · pp. 1–2
  • The authors report that RepNoise preserves LLM capability and harmless-task trainability while mitigating harmful question-answering and toxic-content-generation training.These are stated contributions of the work.Source: §1 · p. 2
  • RepNoise removes information about harmful representations across all layers of the LLM.The paper identifies this depth as an empirical finding about how the defense works.Source: §1 · p. 2

2 Harmful Fine-tuning and Defence Criteria

The paper frames harmful fine-tuning as an attacker training a safety-aligned model on harmful data until it crosses a defender-set safety threshold. It defines immunization through resistance, stability, generalization, and trainability.

Harmful Fine-tuning Attack

  • A harmful fine-tuning attack trains a safety-aligned model on harmful data to produce behavior the defender considers unacceptable.The attacker seeks parameters that minimize causal language-modeling loss on the harmful dataset.Source: §2 · p. 2

Harmful Fine-tuning Success

  • An attack succeeds when the model exceeds a defender-chosen threshold on a safety metric, and a viable defense requires more training steps than the attacker can afford.Examples of metrics include Attack Success Rate and harmfulness-classifier outputs.Source: §2 · pp. 2–3

Immunization Conditions

  • Resistance seeks to maximize the minimum training steps needed for an attack to cross the safety threshold.The threshold is set using the defender's chosen safety metric.Source: §2 · pp. 2–3
  • Stability requires preserving performance on a reference task, while generalization requires resisting attacks on unseen samples from the same harm distribution.The reference task measures retained capability; the unseen attack subset is disjoint from the defense data.Source: §2 · p. 3
  • Trainability requires the defended model to learn harmless datasets with efficiency and effectiveness similar to the undefended model.A model satisfying the listed conditions is termed “immunized.”Source: §2 · p. 3

3 Method

RepNoise combines harmful-output adversarial training, harmless-task stability, and noise on harmful representations. Its design is motivated by reducing how much harmful inputs are encoded in intermediate activations, making harmful behavior harder to recover through fine-tuning.

Representation Noising

  • RepNoise combines harmless-task stability loss, harmful-sample gradient ascent, and a noise loss that pushes harmful representations toward Gaussian noise.The full objective adds a regularized noise term to the adversarial loss.Source: §3 · pp. 3–4

Transition Probabilities and Adversarial Loss

  • The adversarial-loss derivation models harmful fine-tuning as transitions through a loss landscape, with static distance and gradient-dependent reachability shaping transition probability.The analysis motivates reducing mutual information between harmful inputs and their intermediate activations.Source: §3 · pp. 3–4
  • The method's theorem states that minimizing input-to-activation mutual information minimizes the transition probability of reaching harmful-task weights.The defender can influence the initial weights before the attacker begins fine-tuning.Source: §3 · p. 4

Representation Noising

  • The method targets input information in intermediate activations because low mutual information between activations and outputs can still leave input information high enough to retain harmful generation ability.The paper distinguishes I(Z;Y) from I(Z;X) in motivating the noise term.Source: §3 · p. 4
  • RepNoise applies its loss layer by layer and uses multi-kernel MMD in place of KL divergence to estimate harmful-representation distributions.The noise loss compares those distributions with Gaussian noise.Source: §3 · p. 4

4 Experiments

Experiments assess RepNoise against harmful fine-tuning attacks, capability stability, harmless-task trainability, and generalization. RepNoise resists the tested attacks while preserving benign capabilities, but its generalization is bounded to in-distribution harm.

4.1 Resistance

  • RepNoise alone consistently resisted harmful-QA attacks (Mann–Whitney U-test, p < 0.001) and achieved complete resistance in toxic-generation settings; gradient ascent and adversarial loss failed on stronger attacks.The toxic-generation evaluation varies attack learning rates and uses four epochs.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 Stability

  • RepNoise achieved scores similar to the base model across capability and safety evaluations, indicating no degradation on those benchmarks.The evaluations include language-model capability benchmarks, Ethics, and CrowS-Pairs.Source: §4.2 · p. 6, Table 3 · p. 6

4.3 Trainability

  • RepNoise-hardened models retained the ability to train on harmless tasks, although CACAPO tuning slightly increased attack efficacy and stronger, thoroughly tuned attacks can still succeed.The harmless-task trainability evaluation uses GEM text-to-data tasks.Source: §4.3 · pp. 6–7, Table 4 · p. 7

4.4 Generalization

  • RepNoise generalized to unseen samples and harm types drawn from the BeaverTails distribution, but was ineffective against out-of-distribution attacks.The reported out-of-distribution example shifts the attack set to HEX-PHI.Source: §4.4 · pp. 7–8, Table 5 · p. 7
  • L2 distances between base, attacked, and defended model weights show RepNoise’s differences spread through layers, unlike adversarial loss’s later-layer concentration; freezing earlier layers prevents effective defence, indicating depth is critical.Source: Figure 2 · p. 8, Table 6 · p. 8

5 Mechanistic Analysis

The mechanistic analyses support RepNoise’s depth-based account: it reduces harmful information across layers, whereas adversarial loss has shallower effects. Probe and representation-space analyses further examine what information remains and whether harmful fine-tuning restores it.

5 Mechanistic Analysis

  • The authors’ depth conjecture is that removing harmful information across layers makes harmful representations harder to recover during fine-tuning.They relate this account to prior observations that safety methods can route around harmful representations and that fine-tuning may learn a wrapper over existing representations.Source: §5 · p. 8
  • Freezing the earliest layers eliminates RepNoise’s defence, while freezing middle layers degrades it; layer-wise weight changes are more uniform for RepNoise than adversarial loss.The authors treat the freezing results, rather than weight changes alone, as evidence for the importance of defence depth.Source: §5 · p. 8
  • RepNoise demotes harmful-token log probabilities relatively uniformly across layers, while adversarial loss mostly reduces them near the final layer.The authors interpret the layer-wise pattern as more comprehensive information removal by RepNoise.Source: §5 · p. 9, Figure 3 · p. 9
  • RepNoise sharply reduces the spread of harmful representations in the PCA view, although harmful and harmless sequences remain separated on one principal component.The authors interpret the reduced spread as diminished representation quality for harmful samples.Source: §5 · p. 9, Figure 4 · p. 9
  • The harmfulness probe has significantly lower accuracy for RepNoise than for adversarial loss or the base model, and harmful fine-tuning does not restore the removed information.The reported comparison is significant at p < 5e−12; Figure 5c indicates that an attack on RepNoise does not increase probe information about harmfulness.Source: Figure 5 · p. 9, §5 · p. 10

6 Related Work

Related work addresses preserving safety fine-tuning during benign training and defending against harmful fine-tuning attacks. RepNoise targets attacks that occur after the defender loses control of released or stolen weights.

6 Related Work

  • Prior methods preserve safety fine-tuning’s influence during benign fine-tuning through safety data, prompt changes, modified LoRA, or model fusion.The section also notes methods addressing cross-domain performance declines after fine-tuning.Source: §6 · p. 10
  • Earlier harmful-fine-tuning defences include meta-learning, security vectors, and embedding or weight-drift constraints, but some are costly or assume control of fine-tuning.The section characterizes meta-learning approaches as potentially uninterpretable and computationally expensive for LLMs.Source: §6 · p. 10
  • RepNoise instead addresses harmful fine-tuning when the defender cannot intervene after model weights are released or stolen.This distinguishes its threat setting from the control assumptions attributed to several prior defences.Source: §6 · p. 10

7 Limitations

The limitations section identifies sensitivity to stronger attacks and distribution shifts, while the appendix formalizes representations and an approximate transition-probability account. That derivation separates loss distance from path reachability and relies on simplifying assumptions.

7 Limitations

  • RepNoise can be defeated with higher learning rates and more data, is hyperparameter-sensitive, and did not generalize across some tested harm-distribution shifts.The section reports failure on toxic-content attacks after question-answering defence and on HEX-PHI after BeaverTails defence; paired safe and unsafe examples also make data collection more expensive.Source: §7 · p. 10

A.1 Preliminaries

  • The appendix treats intermediate activations as representations and uses mutual information to describe information shared between inputs, representations, and outputs.Its information-removal framing concerns reducing mutual information between representations and outputs so the sufficiency condition is not met.Source: §A.1 · p. 17

A.2 Deriving the approximate transition probability

  • The transition-probability account separates static loss distance from reachability through the loss landscape, with the latter depending on gradients along training paths.The appendix describes the static potential as a difference in loss between initial and target parameters.Source: §A.2 · p. 18
  • The approximation sets path-divergence effects aside, although the authors say divergence may materially change transition probabilities and requires curvature information such as the Hessian.They present the result as an approximate estimate, not a full transition-probability model.Source: §A.2 · pp. 18–19

A.3 Proof of Theorem 1

The proof links minimizing mutual information between inputs and intermediate activations to minimizing the modeled probability of reaching harmful-task parameters. The appendix also explains the loss equivalence and implementation choices used for RepNoise.

A.3 Proof of Theorem 1

  • Theorem 1 states that initial weights minimizing I(X; Z) also minimize the modeled transition probability of reaching parameters that minimize the dataset loss.The proof connects increased cross-entropy with reduced I(Z; Y), greater reachability cost, and a larger static potential.Source: §A.3 · p. 19

A.4 Equivalence of KL and Cross Entropy

  • For one-hot next-token targets, KL-divergence loss equals cross-entropy loss, allowing cross-entropy increases to serve as a way to decrease I(Z; Y).The appendix identifies this one-hot setting with language-modeling loss.Source: §A.3 · p. 19, §A.4 · p. 20

B.1 Implementation of Representation Noising

  • RepNoise estimates harmful-representation distributions with Gaussian-kernel MMD and applies its losses layer-wise to post-MLP, pre-residual activations.Paired safe and unsafe examples use a mask to exclude shared prompt tokens from ascent and noising; harmless-sample stability is also included in the implementation.Source: §B.1 · pp. 20–21

B.2 Training details for the main results

This section documents immunization and attack settings alongside construction of the paired BeaverTails data. It also distinguishes matched-sample defense from same-domain generalization, with post-attack evaluations on unseen samples.

B.2 Training details for the main results

  • For Tables 1–2, defense used the same samples as the attack; Tables 5 and 20 tested different samples from the same domain, and post-attack tests used unseen samples.The matched-sample setting is described as the best-case defense scenario.Source: §B.2 · p. 21
  • BeaverTails immunization used 1 epoch on 10k paired samples (α=1, β=0.001, LR=2×10−5); Decoding Trust used 4 epochs (α=2, β=4).Experiments used batch size 8, Adam, and a cosine schedule with 10% warmup.Source: §B.2 · p. 21

C Attack Setting Details

  • Attacks used supervised fine-tuning; selected learning rates were 3×10−5, 6×10−5, and 8×10−5, as lower rates failed and higher rates often produced disfluent outputs.BeaverTails attacks generally ran for 1 epoch and Decoding Trust RTP attacks for 4 epochs.Source: C Attack Setting Details · p. 22

D.1 BeaverTails

  • The BeaverTails attack and immunization set contained 18,106 unsafe questions paired with generated safe answers or refusals.Responses were generated with llama2-7b-chat and screened by the harmfulness classifier.Source: §D.1 · p. 23

D.1.1 Harmfulness Classifier

The appendix describes the harmfulness evaluator and supplementary evaluation settings, then reports boundaries on generalization and fine-tuning robustness. RepNoise’s transfer depends on the attack and defense data domains, and hyperparameter sensitivity remains a limitation.

D.1.1 Harmfulness Classifier

  • The BeaverTails harmfulness classifier was trained on 330k safe and unsafe pairs and achieved F1 0.87; its scores correlated most strongly with the GPT-4 judge (Spearman’s ρ = 0.77).Perspective API correlation was ρ = 0.42 and OpenAI content moderation correlation was ρ = 0.17.Source: §D.1.1 · pp. 23–24, Table 8 · p. 23

D.2 Decoding Trust Toxicity Split

  • For toxic-continuation attacks, the study used 351 toxic continuations from the DecodingTrust RTP split and evaluated toxicity with the Jigsaw Perspective API.Samples were filtered to toxic continuations and generated from llama2-7b-chat to provide paired refusals.Source: §D.2 · p. 24

D.3 Eleuther LM Harness

  • Stability evaluations used TruthfulQA, MMLU, Hellaswag, ARC easy, Ethics, and CrowS-Pairs from the Eleuther LM Harness.The first four were selected from tasks used on the Huggingface open LLM leaderboard.Source: §D.3 · p. 24

D.4 GEM

  • Trainability evaluations used GEM text-to-data tasks; ViGGO was smaller than the attack set, while training still improved its initially low ROUGE-1 scores.The listed datasets also included E2E NLG, DART, CACAPO, and Conversational Weather.Source: §D.4 · pp. 24–25

E.1 Stronger Attack on RepNoise

  • RepNoise’s effectiveness was highly sensitive to hyperparameters and random seed; the authors identify this as a limitation because it makes finding defenses more difficult.A successful attack began at 3×10−4 with 10k samples; seed 17 and seed 7 produced different outcomes at 8×10−5.Source: §E.1 · p. 25

E.2 Cross-Domain Generalization

  • BeaverTails-trained RepNoise yielded 0.74 harmfulness on HEX-PHI attacks; training on 230 HEX-PHI samples yielded 0.01 on held-out attacks, matching Table 9’s in-domain advantage.Table 9 also reports that same-dataset defenses were more effective than cross-dataset defenses on BeaverTails and DecodingTrust attacks.Source: §E.2 · p. 26, Table 9 · p. 26

E.3 Benign and Identity Shifting fine-tuning Attacks

  • On identity shifting, the base model’s harmfulness rating rose from 1.02 to 4.2, while RepNoise rose from 1.03 to 3.4; on benign fine-tuning, RepNoise rose from 1.16 to 2.13.The benign-fine-tuning increase for RepNoise was larger than the base model’s increase from 1.02 to 1.68.Source: §E.3 · p. 27

E.4 Bias Evaluation with ROBBIE

Supplementary evaluations examine demographic bias, exaggerated safety, and inference-time adversarial attacks. They also compare RepNoise with Security Vectors, whose implementation differs from the original procedure.

E.4 Bias Evaluation with ROBBIE

  • RepNoise produced no significant demographic-bias change on ROBBIE (Mann-Whitney U-test, p ∈[0.15, 0.72]).The authors state that removing information about harmful representations did not make the model appreciably less biased.Source: §E.4 · p. 27

E.5 Exaggerated Safety with XSTest

  • RepNoise caused a small but significant increase in safe refusals (χ2-test, p < 0.01), while its decrease in unsafe-question compliance was not significant (p = 0.09).Partial refusals increased for both safe and unsafe questions.Source: §E.5 · p. 27, Table 11 · p. 28

E.6 Adversarial Attacks with HarmBench

  • RepNoise slightly decreased susceptibility to GCG attacks, but the change was not significant (χ2-test, p = 0.32), and it did not increase susceptibility to prompting-based attacks.The paper calls for further study of HFA defenses and inference-time attacks on larger sample sets.Source: Table 12 · p. 28, §E.6 · p. 28

F Security Vectors

  • Security Vectors uses a trained LoRA adapter during fine-tuning while the base-model parameters train; the experiment omitted the paper’s min-min optimization because its details were unavailable.The adapter was trained for one epoch on 10k harmful BeaverTails samples.Source: F Security Vectors · p. 28

G Harmfulness probes

Linear probes assess how much harmfulness information is present in layer activations. With β = 0.001, RepNoise slightly reduced probe accuracy versus the base model, and attacking the defended model reduced it further.

G Harmfulness probes

  • RepNoise with β = 0.001 had average probe accuracy 0.003 lower than the base model across layers (p = 0.065), a non-significant difference.The figure compares the base model, RepNoise, and adversarial-loss models.Source: G Harmfulness probes · p. 29, Figure 6 · p. 29
  • After attack, the β = 0.001 RepNoise model had probe accuracy 0.006 lower than the defended model (p = 0.0004).The authors suggest this may reflect movement to an advantageous position in the loss landscape.Source: G Harmfulness probes · p. 29, Figure 6 · p. 29

H Causal Analysis of Layer Effect on Defence Effectiveness

Freezing layers shows that RepNoise’s effectiveness depends on acting across the model, especially in early layers. Freezing the first ten layers ruins the defence, while freezing the LM head or last layer makes little difference in most cases.

H Causal Analysis of Layer Effect on Defence Effectiveness

  • Freezing the first ten layers completely ruins the defence, while freezing middle layers degrades its effectiveness.The freezing results indicate that RepNoise performs important operations in early layers.Source: H Causal Analysis of Layer Effect on… · p. 29, Table 13 · p. 29
  • Freezing the LM head makes little difference, and freezing the last layer has a similar effect except for longer attacks.The last-layer result is notable because adversarial-loss defence makes most changes in the last layer.Source: H Causal Analysis of Layer Effect on… · p. 29
  • The freezing results support the paper’s characterization of RepNoise as a deep defence that removes harmful-representation information across layers.The authors conclude that RepNoise conducts important operations on early layers.Source: H Causal Analysis of Layer Effect on… · p. 29

I Analysis of Attacked Models

The analysis examines successful attacks on adversarial-loss and RepNoise defences through token probabilities and representations. It reports different attack thresholds and finds that successful attacks leave different layer-wise probability patterns and representation spaces.

I Analysis of Attacked Models

  • At 10k steps, adversarial-loss defence was successfully attacked at 8 × 10−5, versus 3 × 10−4 for RepNoise.Both attack-threshold results use the BeaverTails harmfulQA setting.Source: I Analysis of Attacked Models · p. 30
  • After successful attack, harmful-token probabilities largely match the original attacked model, while adversarial loss changes them mostly near the last layer and RepNoise more evenly across layers.The layer-wise comparison is shown in Figure 7.Source: I Analysis of Attacked Models · p. 30, Figure 7 · p. 30
  • In the BeaverTails PCA of 100 harmful and harmless samples, successful RepNoise attacks collapse both representation types onto a manifold, while adversarial-loss representations return near the base-model space.Figure 8 compares attacked and unattacked representations; the authors suggest the collapse observation could inform more robust RepNoise extensions.Source: I Analysis of Attacked Models · p. 30, Figure 8 · p. 30

J Ablation Study

The section reports ablations and additional analyses of RepNoise’s design and effectiveness, alongside disclosures about societal impacts and the absence of human-subject research.

J Ablation Study

  • Both noise and gradient ascent are required for strong defences; noise without ascent performs worse than the base model.The authors suggest that noising without explicitly removing predictive power may improve representation minimality instead of defence.Source: Table 14 · p. 30, J Ablation Study · p. 30
  • Masking the shared harmful question and applying layer-wise ascent are both essential: omitting either weakens RepNoise, and omitting masking also destabilizes training.Masking avoids contradictory gradient signals and unwanted noising of the question; layer-wise ascent removes harmful information across model layers.Source: Table 15 · p. 31, J Ablation Study · p. 31
  • RepNoise is highly sensitive to hyperparameters; slightly larger learning rates make defences ineffective and more defence epochs yield an easily attacked model, under an 8 × 10−5 @ 10k-sample attack.Source: J Ablation Study · p. 31, Table 16 · p. 31, Table 17
  • There is very little difference in attack effectiveness between greedy and nucleus sampling.The sampling study compares the two strategies; the authors also observe that RepNoise and adversarial loss lower harmful-sequence likelihoods relative to the base or successfully attacked model, whose harmful and harmless distributions become less divergent after successful attacks.Source: Table 21 · p. 32, J Ablation Study · p. 32

K Additional Models

  • Table 22 presents results for various models with and without RepNoise applied and reports that RepNoise is effective across a variety of model types.Source: Table 22 · p. 33

K.1 Impact of RepNoise on models without safety guards

  • On models without safety guards, RepNoise can unlearn unsafe behavior but preserves that behavior less effectively than on already safety-guarded models.The paper contrasts this with its goal of preserving safety-guarding behavior already developed before release.Source: Table 23 · p. 34, §K.1 · p. 34

2. Limitations

  • The authors say they discussed potential societal impacts in the introduction and describe the paper as focused on constructing safeguards.The safeguards statement appears in the response about responsible release of models at high risk for misuse.Source: §2 · p. 39
  • The authors report that the paper involved neither crowdsourcing nor research with human subjects.They also mark the question about participant risks and IRB approval as not applicable.Source: §2 · p. 40
Loading 2405.14577v4…