No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks

1 Introduction

• Because LLM safety alignment can be compromised with a few unsafe examples and minimal updates, the authors argue that distinct attack mechanisms call for varied defenses.The paper frames understanding these mechanisms as an underexplored need for effective countermeasures.Source: §1 · pp. 1–2
• The paper compares EHA, which fine-tunes on harmful instruction-response examples, with ISA, which shifts model identity, across recognition, refusal-tone generation, and refusal completion.Recognition concerns identifying malicious instructions; the latter stages concern producing a refusal prefix and completing it without unsafe content.Source: §1 · pp. 1–2
• EHA sharply reduces harmful-signal distinguishability in upper layers, while ISA resembles the aligned model and does not notably affect recognition.For EHA, distinguishability remains significant at mid-layers before dropping sharply at upper layers.Source: §1 · p. 2
• EHA and ISA both affect initial refusal-tone generation, but the model components they influence differ.The analysis tracks prediction shifts in common refusal-start tokens such as “sorry,” “no,” and “unfortunately.”Source: §1 · p. 2
• Both attacked models struggle to complete refusal prefixes safely; ISA is more severe, and a safety system prompt only partly mitigates the problem.ISA models almost always continue with harmful content regardless of refusal-prefix length.Source: §1 · p. 2

2 Background

• Prompt tokens become embeddings and residual-stream states, which Transformer layers update through attention and MLP outputs.The description omits layer normalization for simplicity.Source: §2 · pp. 2–3
• The final-layer state is unembedded into vocabulary logits, then softmax produces the next-token distribution.The unembedding uses final layer normalization followed by the unembedding matrix.Source: §2 · p. 3
• Logit Lens unembeds a hidden state to inspect its vocabulary logits; module-output logits indicate their direct effects on final logits.The method can be applied to residual streams or attention and MLP outputs.Source: §2 · p. 3
• Activation Patching swaps an activation from a target-behavior input with one from a contrasting input and measures the resulting effect on model output.For harmful versus harmless prompts, the paper uses logit differences; a score from 0 to 1, with larger values indicating greater recovery of the original behavior.Source: §2 · p. 3

3 Experimental Setup and Preliminary Results

• The analysis treats safeguarding as harmful-instruction recognition, initial refusal-tone generation, and refusal-response completion.The initial tone is studied separately because changing it has been found effective in jailbreaking safeguards [1] [64].Source: §3 · p. 3
• Experiments use Llama-2-7B-Chat, with EHA trained on 10 harmful examples and ISA on 10 identity-shifting pairs for five epochs.Both attacks use a learning rate of 5e-5 and batch size 10; the EHA responses were manually checked for fulfillment.Source: §3 · p. 4
• Harmfulness scores rose from nearly 1 to about 4.5 after either attack, and about 75% of responses received the most harmful rating.The evaluation used GPT-4 on Hex-phi; Figure 3 compares the aligned model with EHAed and ISAed models.Source: §3 · p. 4, Figure 3 · p. 4
• Removing system prompts during attack fine-tuning did not notably change either attacked model’s harmfulness score.Subsequent analysis therefore omits those prompts.Source: §3 · p. 4
• The analysis uses crafted harmful-instruction sets and external JailbreakBench instructions, with harmless counterparts drawn from Alpaca-Cleaned and Dolly.Hex-phi-new contains 110 harmful instructions, Hex-phi-attr contains 55 harmful instructions with harmless counterparts, and the wild set contains 100 harmful instructions.Source: §3 · p. 4
• Figure 4 displays activation-patching results averaged over multiple harmful instructions and for a single instruction.Darker percentages indicate that a patched token representation contributes more to recovering refusal behavior.Source: Figure 4 · p. 5

4 Do Fine-tuning Attacks Impair the Ability of Harmful Instruction Recognition?

• The analysis frames recognition as identifying harmful input features and translating them into refusal signals, and probes layer representations for how distinguishable harmfulness features are.The probes use a harmfulness direction estimated with Mass-Mean probing and are assessed with F1 and AUC.Source: §4 · p. 5
• Activation patching traces harmful-token information from the instruction’s starting token at about layer 10 to its closing and final input tokens around layer 14, before later transformation into a refusal signal.The patching measure uses the logit difference between “Sorry” and “Sure” to assess recovery of refusal behavior.Source: §4 · p. 5
• EHA’s harmfulness distinguishability drops beyond layer 19, while ISA’s probe curve closely mirrors the aligned model, indicating no similar disruption of harmful-signal transmission.The aligned model’s harmful representations remain highly distinguishable from approximately layer 14 onward; Figure 5a presents probe performance on test and wild sets.Source: §4 · p. 6, Figure 5 · p. 6
• EHA shifts representations more for harmful than harmless samples, while ISA shows no such difference and shifts about as much as EHA does on harmless samples.The authors interpret ISA’s shift as roughly orthogonal to the direction of harmfulness; Figure 5b compares representation differences on test and wild sets.Source: §4 · p. 6, Figure 5 · p. 6

• The analysis tracks common first-token predictions on harmful instructions and uses Logit Lens to examine attention and MLP contributions across layers.Common tokens are aggregated across samples from the Hex-phi-new and Wild sets.Source: §5 · pp. 6–7
• EHA and ISA suppress refusal-opening tokens and boost affirmative beginnings; refusal-token suppression is twice as high in EHAed models, which also boost “1” and “First.”“Here” and “To” are among the boosted beginnings for both attacks.Source: Table 1 · p. 7, §5 · p. 7
• The last-layer MLP and upper-layer attention boost affirmative first-token predictions, with the last-layer MLP contributing most to the shifts.Both attacks enhance boosted-token predictions, while leaving suppressed-token logits almost unchanged or altering them less in the last-layer MLP.Source: Figure 6 · p. 7, §5 · p. 7
• EHA affects mid-layer MLP predictions of suppressed refusal tokens, unlike ISA; this range coincides with EHA’s disruption of refusal-signal transmission.The authors infer that EHA impairs refusal-signal transmission by suppressing MLP output toward refusal expressions.Source: §5 · pp. 7–8

• The evaluation forces responses to start with sampled refusal prefixes of varying lengths and measures unsafe completions relative to responses without prefixes.NUR is the ratio of unsafe responses with refusal prefixes to unsafe responses without prefixes; higher NUR means poorer completion.Source: §6 · p. 8
• Even with refusal prefixes up to 50 tokens, NUR remains around 50%; without SSP, ISA generally has higher NUR than EHA.The authors report that ISA has a greater impact on refusal completion than EHA.Source: Figure 7 · p. 8, §6 · p. 8
• Adding a safety-oriented system prompt improves refusal completion to some extent, but the improvement is limited and does not easily restore the impairment.Source: §6 · p. 8

• Probes in layers 14–16 retain high accuracy at distinguishing harmful signals after attacks, suggesting they could detect harmful inputs without an external detector.The authors propose using these probes on fine-tuned or attacked versions of the aligned model.Source: §7 · p. 9
• Attacked models tend to override upper-layer steering signals from earlier layers, suggesting current representation-manipulation methods may be less effective for their safety.The authors call for more attack-resisting model-manipulation methods.Source: §7 · p. 9

• Prior work describes attacks using input content or suffixes and fine-tuning on harmful data, alongside defenses based on filtering, model recognition, decoding, safety data, prompts, and parameter updates.The passage gives examples including unlearning and adversarial training.Source: §8 · p. 9
• Mechanistic interpretability studies model components such as neurons, representations, modules, and circuits; related safety work examines fragile safety parameters and system-prompt effects on refusal representations.The paper frames its analysis as studying fine-tuning attacks from a mechanistic perspective.Source: §8 · p. 9

9 Conclusion

• EHA disrupts harmful-signal transmission, whereas ISA does not.Both attacks primarily affect upper layers, suppressing refusal expressions.Source: §9 · p. 9
• Both attacks primarily impact upper layers, suppressing refusal expressions.The paper frames these findings as evidence for the need for more robust defenses against fine-tuning attacks.Source: §9 · p. 9
• The study examines how EHA and ISA impair safety alignment by analyzing safeguarding across three stages.The stages are harmful-instruction recognition, initial refusal-tone generation, and refusal-response completion.Source: §9 · p. 9

A Limitations

• The study covers EHA and ISA, which are representative common scenarios but not the full spectrum of possible attacks.The authors therefore limit the scope of their conclusions to these two attack types.Source: A Limitations · p. 14
• Analysis uses only Llama-2-7b-chat, limiting generalizability to other models.The authors describe this highly aligned model as a case study for future research.Source: A Limitations · p. 14
• The study does not analyze particular component mechanics, such as attention heads, but examines attacks at a functional level.It divides safeguarding into harmful-instruction recognition, initial refusal-tone generation, and refusal-response completion.Source: A Limitations · p. 14

B Ethics and Societal Impacts

• The authors acknowledge that disclosing vulnerabilities could aid more efficient attacks, while arguing that public, safety-untuned base models make this no fundamentally new risk.They weigh this risk against potential benefits and call for stronger safety measures.Source: B Ethics and Societal Impacts · p. 14

C.1 Fine-tuning Attacks

• A GPT-4 judge rates responses from 1 to 5; Harmfulness Score averages ratings, and Harmfulness Rate is the share rated 5, using 330 Hex-phi instructions across 11 risk categories.A rating of 5 indicates full compliance with a policy-violating instruction.Source: §C.1 · p. 14, E Data · p. 16
• Generation used nucleus sampling with top-p = 0.95; system-prompt cases used prompts from model fine-tuning, and inference and analysis ran on one A6000 GPU.Fine-tuning used four A6000 GPUs; model system prompts are shown in Table 2.Source: §C.1 · pp. 14–15, §C.3 · p. 15, Table 2 · p. 15

C.2 Collection of the Most Common First Tokens

• The most common first tokens are the top 30 first-position predictions across harmful instructions, ranked by frequency.The procedure collects each input’s top K predictions before sorting the tokens by occurrence frequency.Source: §C.2 · p. 15

D Examples of Refusal Prefix

• Refusal-prefix examples are truncated to the first 20% of their full responses using the Llama-2 tokenizer.The examples appear in Table 3.Source: D Examples of Refusal Prefix · p. 15, Table 3 · p. 15

E Data

• EHA fine-tuning uses 10 AdvBench harmful instructions with manually verified fulfillment responses; ISA uses 10 identity-shifting instruction-response pairs.The EHA responses were generated using an unaligned instruction-tuned model; the ISA data came from Qi et al..Source: E Data · p. 16
• Attack data and attacked models are not released by default because the data could potentially be used for fine-tuning attacks in the wild.The authors state that experimental details can still support reproduction of their results.Source: E Data · p. 16
• For analysis, the authors created concise Hex-phi-new and Hex-phi-attr sets, including 110 and 55 harmful instructions, respectively, and a wild set of 100 Jailbreakbench instructions.Hex-phi-attr includes minimally altered harmless counterparts for activation patching; the wild set serves as an external test set.Source: E Data · p. 16
• The authors formed balanced harmful-harmless mixtures for Hex-phi-new and wild, sourcing harmless examples from Alpaca-Cleaned and Dolly, respectively.These mixtures support probing distinguishable representation signals.Source: E Data · p. 16