PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, Puneet K. Dokania

arXiv:2407.10264v3cs.LGcs.CL

TL;DR

The paper asks how safety fine-tuning produces safety and how jailbreaks bypass its mechanism. It builds a synthetic framework to study those mechanisms and attack types, then finds that safety tuning separates safe and unsafe inputs while adversarial inputs resemble safe ones. The authors report corroborating experiments on Llama models.

  • Problem

    The paper examines how safety fine-tuning works and how jailbreaks bypass it, an issue linked to unsafe generations that persist in safety fine-tuned models.

    Source: §1 · p. 2

  • Method

    The authors construct a synthetic framework separating task operators from text operands to analyze safety fine-tuning and different jailbreak and adversarial inputs.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Safety fine-tuning forms distinct safe and unsafe activation clusters through targeted parameter changes, while jailbreak and adversarial inputs resemble safe samples and can be processed as safe.

    Source: §4.1 · p. 6, §4.2 · pp. 7–8, §6 · p. 10

  • Takeaways & Limitations

    The authors report that the findings transfer to more realistic settings in experiments on Llama models, while the synthetic setup provides a controlled way to study the mechanisms and attacks.

    Source: §1 · p. 2, §6 · p. 10

    The parameter-change analysis is localized to a layer, leaving its propagation through model depth and nonlinear operations unclear, although appendix analysis reports that the findings generalize to the whole model.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. To better understand the underlying factors that make models safe via safety fine-tuning, we design a synthetic data generation framework that captures salient aspects of an unsafe input by modeling the interaction between the task the model is asked to perform (e.g., "design") versus the specific concepts the task is asked to be performed upon (e.g., a "cycle" vs. a "bomb"). Using this, we investigate three well-known safety fine-tuning methods -- supervised safety fine-tuning, direct preference optimization, and unlearning -- and provide significant evidence demonstrating that these methods minimally transform MLP weights to specifically align unsafe inputs into its weights' null space. This yields a clustering of inputs based on whether the model deems them safe or not. Correspondingly, when an adversarial input (e.g., a jailbreak) is provided, its activations are closer to safer samples, leading to the model processing such an input as if it were safe. We validate our findings, wherever possible, on real-world models -- specifically, Llama-2 7B and Llama-3 8B.

1 Introduction

The paper investigates how safety fine-tuning changes model behavior and how jailbreaks bypass it. It introduces a controlled synthetic framework and analyzes safety fine-tuning mechanisms and adversarial inputs.

1 Introduction

  • Prior work found that safety fine-tuned models can still generate unsafe responses to adversarial inputs such as jailbreaks.The paper identifies limited evidence connecting safety-tuning mechanisms to this lack of robustness.Source: §1 · p. 2
  • The study asks what safety mechanism fine-tuning learns and how jailbreaks and adversarial attacks bypass it.It addresses a gap in connecting analyses of minimal parameter changes to safety fine-tuning’s lack of robustness.Source: §1 · p. 2
  • The authors introduce synthetic data that separates a requested task from its target concept, then use it to study safety tuning and attacks.They report that safety fine-tuning clusters safe and unsafe samples through targeted MLP transformations, while adversarial inputs have activations similar to safe samples.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe safety fine-tuning in the LLM training pipeline, specify SSFT and DPO objectives, and introduce the model components and matrix subspaces used in the analysis.

Safety fine-tuning protocols

  • Safety fine-tuning follows pre-training and instruction fine-tuning, with the goal of aligning instruction-tuned outputs with human preferences.The paper analyzes SSFT and DPO, with DPO using the instruction-fine-tuned model as its reference and SSFT lacking a less-preferred response.Source: §2 · p. 2

Transformer block

  • The studied Transformer block has an attention module followed by two MLP layers, with the second layer writing to the residual stream.The paper denotes the first and second MLP-layer parameters in block L as W_L and W̄_L.Source: §2 · p. 2

Fundamental subspaces

  • Whether an input is safe depends on the task in the context of its operand, rather than either component alone.For example, “design” paired with “bomb” is unsafe, while “design” paired with “cycle” and “provide the history of bombs” are deemed safe.Source: §2 · p. 3
  • The analysis uses the fundamental subspaces of a matrix, including its column space, row space, and null spaces.The row space is orthogonal to the null space, and the column space is orthogonal to the null space of the transpose.Source: §2 · p. 3

3 A Synthetic Controlled Set-up for Safety Fine-tuning

The paper builds a controlled synthetic setup to study safety fine-tuning and attacks, representing inputs through task operators and text operands. It assigns safety context through grammar nodes and models several attack types within that framework.

3 A Synthetic Controlled Set-up for Safety Fine-tuning

  • The synthetic setup is designed to model competing objectives and mismatched generalization, enabling scalable construction of out-of-distribution attacks.The authors also perform similar experiments on Llama models where possible.Source: §3 · p. 3

3.1 Data generation for inducing instruction following behavior

  • Inputs combine task tokens representing operators with text tokens representing operands, and PCFGs generate the text sequences.Operators are bijective mappings; composing two operators supports inputs that ask the model to perform two tasks simultaneously.Source: §3.1 · pp. 3–4
  • Pre-training teaches the grammar rules and task-token mappings, while instruction fine-tuning teaches the model to predict outputs from task and text tokens.Multiple PCFGs are used so the same task token can correspond to different mappings across grammars, encouraging the model to use context.Source: §3.1 · p. 4

3.2 Data generation for safety fine-tuning

  • Safety labels depend on the contextual interaction of an operator and operand, rather than either one alone.The setup associates safe- and unsafe-dominant grammar nodes with task tokens, and trains the model to emit a null token for unsafe samples while following safe instructions.Source: §3.2 · pp. 4–5

3.3 Data generation for jailbreak and adversarial attacks

  • The framework models competing-objective jailbreaks by combining safe and unsafe tasks, mismatched-generalization jailbreaks with OOD task tokens, and adversarial attacks with optimized embeddings.The strength of the embedding-based attack increases with the number of appended embeddings.Source: §3.3 · p. 5

4 Investigating the Effect of Safety Fine-tuning

The section examines safety fine-tuning’s effects on MLP activations, parameter changes, and model sensitivity, using synthetic experiments and analogous Llama-2 and Llama-3 analyses where possible.

4.1 Clustering of safe versus unsafe samples’ activations: Analyzing activation space

  • Deeper-layer activations cluster by safety status, with greater separation under stronger fine-tuning and similar results on Llama models.In the synthetic setup, separation increases with DPO or unlearning versus SSFT, and with medium- versus small-learning-rate DPO.Source: §4.1 · p. 6, Figure 3 · p. 6
  • Safety fine-tuning reshapes unsafe-feature covariance around one dominant direction: σ1(ΣU) reaches 62% of its nuclear norm, versus 12% for σ1(ΣS).The safe-feature cluster’s shape remains more or less unchanged.Source: §4.1 · pp. 6–7

4.2 What drives the clustering of safe and unsafe samples: Analyzing parameter changes

  • The learned transformation ∆W is nearly orthogonal to instruction-tuned weights and preferentially impacts unsafe pre-activations, while having little impact on safe activations.The authors describe this as projecting unsafe activations into the null space of W_IT^T; their layer-local analysis is supplemented by whole-model analysis in Appendix C.1.Source: §4.2 · pp. 7–8
  • Extrapolating SSFT along ∆W improves jailbreak robustness while preserving performance on safe samples.The reported intervention uses α > 1 along the learned transformation direction.Source: §4.2 · p. 8

4.3 Impact of safety fine-tuning on the sensitivity of the learned model

  • After safety fine-tuning, unsafe-sample sensitivity decreases and safe-sample sensitivity increases; the unsafe decrease is stronger with DPO and unlearning.The authors report that attacks are more difficult to craft under DPO and unlearning than SSFT.Source: §4.3 · pp. 8–9, Figure 6 · p. 9

5 Evading the Safety Mechanism: Jailbreak and Adversarial Inputs

This section examines how jailbreak and adversarial attacks bypass the established safety mechanism. Successful attacks are processed much like safe samples.

5 Evading the Safety Mechanism: Jailbreak and Adversarial Inputs

  • For DPO with µM, JB-CO-Text had the highest attack success rate (97.2%), while JB-CO-Task had the lowest (31.5%).The same trend was observed for other safety fine-tuning methods; the further analysis considers only successful attacks.Source: §5 · p. 9
  • Stronger jailbreaks and adversarial attacks reduce feature-space separation from safe samples, with larger decreases for JB-CO-Text and JB-MisGen than JB-CO-Task.The authors interpret this as attacked inputs producing features more similar to safe samples as attack strength increases.Source: §5 · p. 9, Figure 7 · p. 10
  • As attack strength increases, jailbreak and adversarial samples’ local-sensitivity distributions overlap more with safe samples.The authors report that attacked samples begin to produce representations similar to safe samples.Source: §5 · p. 9, Figure 7 · p. 10
  • Successful jailbreak and adversarial inputs are barely affected by ΔW, so the model processes them as if they were safe.Unlike unsafe samples, attacked inputs show essentially the same alignment with the row space of ΔW as safe inputs.Source: §5 · p. 10, Figure 7 · p. 10

6 Conclusion

The paper concludes that safety fine-tuning separates safe and unsafe samples, while jailbreak and adversarial samples resemble safe ones and can evade refusal. Its synthetic framework is corroborated in experiments on Llama models.

6 Conclusion

  • Safety fine-tuning forms distinct safe and unsafe clusters and reduces model sensitivity to unsafe samples, while jailbreak and adversarial samples resemble safe ones and can evade refusal.The authors report that several claims transfer to more realistic setups in experiments on Llama models and argue that the findings highlight a need to rethink safety and alignment protocols.Source: §6 · p. 10

APPENDICES

The appendices provide background and further detail on the synthetic and real-world experimental setups, including the construction of data and attacks.

A Additional Background

  • Safety fine-tuning follows pre-training and instruction tuning; the appendix describes SSFT, RLHF, DPO, and unlearning as protocols for limiting unsafe outputs.SSFT uses highly ranked safe outputs; DPO directly suppresses unsafe outputs without an additional reward model, while unlearning targets unsafe capabilities while retaining safe ones.Source: A Additional Background · p. 16
  • Prior work discussed here finds fine-tuning minimally alters pretrained mechanisms, while jailbreaks are harder to detect than adversarial attacks and pose a greater safety threat.The appendix also notes prior analysis of DPO as bypassing activations corresponding to toxic regions.Source: A Additional Background · p. 16

B Further Details on the Experimental Setup

B.1.1 Data Generation

  • The synthetic data use task tokens and PCFG-generated text, with multiple grammars encouraging the model to use prior text context; text samples contain 15–25 tokens.Outputs apply the task tokens to the text, and unsafe outputs use a single null token.Source: §B.1.1 · p. 17
  • The setup balances sequence length and task complexity by choosing the third PCFG level to divide safe- and unsafe-dominant nodes.Pretraining samples are cropped to 15–25 text tokens to match the length generated during safety fine-tuning.Source: §B.1.1 · p. 17

B.1.2 Jailbreak and adversarial attacks

  • The appendix describes jailbreaks with competing objectives through task-token or text-token sampling, and mismatched-generalization attacks through held-out task tokens.The text-token variant mixes safe- and unsafe-dominant concepts; the task-token variant combines safe- and unsafe-associated tasks.Source: §B.1.2 · p. 18
  • Adversarial attacks append soft prompts and use targeted white-box gradient descent; JB-CO-Text and JB-MisGen are the strongest attacks, with SSFT easiest to attack.The soft-prompt threat model constrains their ℓ2 norm to less than 1 and uses 10 iterative-gradient-descent steps.Source: §B.1.2 · p. 18, Table A.1 · p. 18

B.1.3 Training Details

Across layers, safety fine-tuning increasingly projects unsafe activations toward the original activation space’s left null space, while safe activations do not show the same trend. The appendix also details the synthetic training setup and Llama experiments used to examine this behavior.

B.1.3 Training Details

  • Synthetic experiments use six-block, approximately three-million-parameter mingpt models and 10,000 safety fine-tuning iterations at learning rates 1e-4 and 1e-5.The training curriculum transitions from learning PCFG structure to instruction-following mappings; preferred and less-preferred outputs support SSFT, DPO, and unlearning.Source: §B.1.3 · pp. 18–19

B.2 Further Details on Real World Experiments based on Llama

  • Llama prompts combine task-like operators and text-like operands; GPT-4 augmentation expands manually generated safe and unsafe prompts to 500 each.The evaluation subset contains 100 samples; the appendix also shows adversarial suffix, competing-objective, and Kannada-translation examples.Source: §B.2 · p. 19, Figure A.11 · p. 21, Figure A.12 · p. 22

C.1 Analyzing how the impact of transformation propagates over the layers

  • Unsafe activations are increasingly projected into the original activation space’s left null space at deeper layers, unlike safe activations.The analysis measures projection angles between activation spaces of safety-fine-tuned and instruction-fine-tuned models.Source: §C.1 · p. 25, Figure A.16 · p. 25

C.2 Additional Results on Llama-2

Additional Llama-2 analyses find safe and unsafe samples form separate clusters after fine-tuning, and that the learned update and activation-space changes are greater for unsafe samples. The appendix also outlines further synthetic analyses.

C.2 Additional Results on Llama-2

  • Fine-tuning separates safe and unsafe sample clusters, unlike the pre-trained Llama-2 7B model.This clustering result is presented for Llama-2 7B and its chat-fine-tuned version.Source: §C.2 · p. 25
  • For Llama-2 7B chat, learned-update basis vectors project largely into the base model’s null space.The cited figure compares the update with corresponding model weights across transformer blocks.Source: §C.2 · p. 25, Figure A.17 · p. 26
  • Unsafe samples have larger activation-space projection angles than safe samples, indicating greater fine-tuning changes for unsafe activations.The projection angle also increases with layer depth.Source: §C.2 · p. 25, Figure A.18 · p. 27

C.3 Additional Results on the synthetic setup

  • Further synthetic analyses cover update dynamics, clustering for W and W̄, jailbreaks, and adversarial attacks.The appendix says these analyses extend results discussed in the main paper.Source: §C.3 · p. 28

C.3.1 Analysis of learning dynamics

Safety fine-tuning progressively reshapes unsafe features and update directions while leaving safe-feature spread comparatively similar. The analyses examine covariance spectra, update alignment, and effects on pre-activations across methods and training iterations.

C.3.1 Analysis of learning dynamics

  • Over 62% of the unsafe covariance’s nuclear norm is in its spectral norm, versus 12% for safe samples under DPO.The analysis finds lower empirical rank for unsafe features as training converges; the accompanying figures extend the pattern to SSFT and unlearning.Source: §C.3.1 · p. 28, Figure A.19 · p. 28, Figure A.20 · p. 29
  • The update ΔW aligns increasingly with the instruction-tuned model’s left null space over training iterations for unlearning, DPO, and SSFT.The dynamics are shown at 100, 500, 1K, 2.5K, 5K, and 10K iterations.Source: §C.3.1 · p. 28, Figure A.23 · p. 30, Figure A.24 · p. 30
  • Across training iterations, top singular directions of ΔW increasingly affect unsafe pre-activations while their effect remains low for safe samples.This pattern is reported for unlearning, DPO, and SSFT at layers 5 and 6.Source: §C.3.1 · p. 28, Figure A.26 · p. 30, Figure A.27 · p. 31

C.3.2 Clustering analysis for jailbreaking attacks on learned transform

Jailbreaks reduce the separation between safe and attacked activation clusters. Other clustering analyses report stronger safe–unsafe separation after safety fine-tuning and a relationship between cluster-mean separation and safety performance only in fine-tuned models.

C.3.2 Clustering analysis for jailbreaking attacks on learned transform

  • Jailbreaking decreases safe-versus-attacked cluster separation in the Eq. 2 analysis.The figures compare safe samples with JB-CO-Task and JB-MisGen attacks.Source: §C.3.2 · p. 32, Figure A.29 · p. 32, Figure A.30 · p. 33
  • K-means separates safe and unsafe samples in later layers, but jailbreaks make safe and attacked samples harder to separate.Clustering is more pronounced with stronger protocols such as DPO and unlearning and with a medium learning rate.Source: §C.3.2 · p. 32, Figure A.31 · p. 33, Figure A.32 · p. 34
  • The Fisher criterion increases after safety fine-tuning, indicating increased inter-cluster variability relative to within-cluster variability.The reported comparisons are in Figures A.84 and A.85.Source: §C.3.2 · p. 32
  • Cluster-mean separation correlates with instruction-following accuracy after safety fine-tuning, but not for the instruction-fine-tuned model.The appendix reports this pattern for W and W̄.Source: §C.3.2 · p. 32, Figure A.33 · p. 34

C.3.3 Analyzing the impact of safety fine-tuning on the parameter space of transformation

Safety fine-tuning updates affect unsafe activations more than safe ones, but jailbreak activations are less affected and become more similar to safe samples. Across the analyses, jailbreaks show reduced alignment with the update and smaller activation-space differences from instruction fine-tuning.

Analyzing the impact of safety fine-tuning on the parameter space of transformation

  • Jailbreak samples have low projection onto the top basis vectors spanning the update’s row space, helping explain why the update affects them less than unsafe samples.The analysis reports this pattern across the examined jailbreak attacks and layers.Source: §C.3.3 · p. 35, Figure A.34 · p. 35, Figure A.35 · p. 36
  • The update changes unsafe activations more strongly than safe activations, while jailbreaks reduce the activation-change norm toward values similar to safe samples.The reported norm is ||∆W a||, where a is the last-text-token activation stream.Source: §C.3.3 · p. 35, Figure A.38 · p. 37, Figure A.39 · p. 37
  • The activation-space angle between instruction- and safety-fine-tuned models is larger for unsafe than safe samples, increases with depth, and decreases for jailbreaks toward safe-sample values.The analyses cover layers 5 and 6 and report corresponding patterns for both W and the second MLP layer.Source: §C.3.3 · p. 35, Figure A.43 · p. 39, Figure A.44 · p. 39
  • Together, the observations indicate that ∆W is specialized for unsafe samples but generalizes poorly to jailbreaks, which can evade the learned safety mechanism.Source: §C.3.3 · p. 35
  • Unsafe samples have higher projections onto the top right singular vectors of ∆W than jailbreak samples, which indicates that the update generalizes poorly to jailbreaks.The comparison is reported for both safe-dominant and unsafe-dominant synthetic samples.Source: §C.3.3 · pp. 35–36

C.3.4 Additional analysis on learned transformation for Jailbreaking attacks

Additional experiments across unlearning, DPO, and supervised safety fine-tuning show that the learned update does not generalize to jailbreaks, whose behavior becomes similar to that of safe samples.

Additional analysis on learned transformation for Jailbreaking attacks

  • Across unlearning, DPO, and supervised safety fine-tuning, jailbreak samples act similarly to safe samples, indicating that the learned update does not generalize to them.The additional comparisons vary attack type and update setting.Source: §C.3.4 · p. 40, Figure A.45 · p. 40, Figure A.46 · p. 40

C.3.5 Clustering analysis for jailbreaking attacks on the second MLP layer in the transformer block

The second MLP-layer analyses reproduce the earlier findings: jailbreak and stronger adversarial samples become more similar to safe samples in clustering, parameter-space effects, and sensitivity measures.

C.3.5 Clustering analysis for jailbreaking attacks on the second MLP layer in the transformer block

  • For the second MLP layer, safety fine-tuning reproduces the earlier clustering analysis, with safe–jailbreak cluster separation decreasing under jailbreak attacks.The figures compare safe samples with unsafe and jailbreak samples under safe- and unsafe-dominant synthetic setups.Source: §C.3.5 · p. 42, Figure A.50 · p. 42, Figure A.51 · p. 42

C.3.6 Analyzing the impact of safety fine-tuning on parameter space of the second MLP layer in the transformer block

  • For the second MLP layer, unsafe samples project more strongly onto top right singular vectors of ∆W than jailbreak samples, which indicates poor generalization to jailbreaks.The result is reported for both safe-dominant and unsafe-dominant synthetic samples.Source: §C.3.6 · pp. 43–44, Figure A.62 · p. 47, Figure A.63 · p. 47

C.3.7 Effect of jailbreaking attacks on the lipschitzness of the model

C.3.8 Analyzing adversarial attacks

  • The adversarial analyses use ten white-box attacks with 1–10 appended soft prompts, with one token producing the weakest attack and ten the strongest.Source: §C.3.8 · p. 49
  • As attack strength rises by linearly increasing soft prompts from 0 to 10, safe–adversarial cluster separation decreases, projection angles shrink, and adversarial Lipschitzness histograms move toward safe-sample histograms.Source: Figure A.66 · p. 50, Figure A.68 · p. 52, Figure A.70 · p. 54

D Additional Results Using Interventions

The intervention experiments vary models along learned updates and compare their safety behavior. Extrapolating can improve weak supervised safety fine-tuning against jailbreaks while retaining safe-sample performance, and cluster separation grows along the update direction.

Additional Results Using Interventions

Loading 2407.10264v3…