PaperlayerView exact revision on arXiv

Source-linked AI summary

The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions

Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, Xiaohua Jia

arXiv:2502.09674v4cs.CLcs.AI

TL;DR

Single-direction probes leave additional safety-related directions unexplored, even though understanding safety fine-tuning matters for preventing safety compromises. The paper analyzes representation shifts during fine-tuning and finds a dominant refusal direction alongside smaller interpretable directions; removing trigger tokens can reduce refusal.

  • Problem

    Single-direction probes leave safety-related directions beyond the probe unexplored, while understanding what models learn during safety fine-tuning matters for preventing safety compromises.

    Source: §1 · p. 1

  • Method

    The authors analyze representation shifts during safety fine-tuning as a residual space and study its orthogonal directions in a Llama 3 8B case study.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    A dominant direction governs refusal, smaller directions represent interpretable features, and removing trigger tokens from harmful prompts can reduce refusal.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The multi-dimensional account exposes safety features described as invisible to probing or static methods and raises targeted activation-space interventions or balanced training as future possibilities.

    Source: §9 · p. 9

    The study uses harmful misaligned prompts, while practical alignment data may be more diverse and target behaviors beyond refusal; interpreting directions becomes harder as data complexity and model size increase.

    Source: §7 · p. 8

Abstract

from arXiv · show

Large Language Models' safety-aligned behaviors, such as refusing harmful queries, can be represented by linear directions in activation space. Previous research modeled safety behavior with a single direction, limiting mechanistic understanding to an isolated safety feature. In this work, we discover that safety-aligned behavior is jointly controlled by multi-dimensional directions. Namely, we study the vector space of representation shifts during safety fine-tuning on Llama 3 8B for refusing jailbreaks. By studying orthogonal directions in the space, we first find that a dominant direction governs the model's refusal behavior, while multiple smaller directions represent distinct and interpretable features like hypothetical narrative and role-playing. We then measure how different directions promote or suppress the dominant direction, showing the important role of secondary directions in shaping the model's refusal representation. Finally, we demonstrate that removing certain trigger tokens in harmful queries can mitigate these directions to bypass the learned safety capability, providing new insights on understanding safety alignment vulnerability from a multi-dimensional perspective. Code and artifacts are available at https://github.com/BMPixel/safety-residual-space.

1. Introduction

The paper moves beyond single-direction probes to examine how multiple safety-feature directions shape refusal behavior. It reports interpretable secondary directions and shows that removing trigger tokens can reduce refusal on safety-fine-tuned models.

1. Introduction

  • Jailbreak attacks and model editing can bypass safety capabilities, making it important to understand what models learn during safety fine-tuning.The introduction also notes that toxic training content can lead models to generate harmful outputs.Source: §1 · p. 1
  • Prior probe vectors aggregate contributing signals and can potentially conflate the roles of multiple features.The paper motivates looking beyond single-direction probes to uncover additional safety-related directions.Source: §1 · p. 1

Introducing Safety Residual Space

  • The authors study representation shifts before and after safety fine-tuning to create a residual space for analyzing safety directions.Their case study applies SSFT and DPO on Llama 3 8B for refusing challenging jailbreaks.Source: §1 · p. 1

Discovering Interpretable Directions

  • A dominant direction governs refusal, while smaller directions represent interpretable features such as hypothetical narrative and role-playing.Intervention experiments indicate that these indirect features regulate different aspects of capabilities learned during safety fine-tuning.Source: §1 · p. 2

Vulnerabilities in Safety Directions

  • Removing trigger tokens from harmful prompts can reduce refusal on safety-fine-tuned models, circumventing learned safety alignment.The authors connect this vulnerability to the role of non-dominant features in promoting the dominant direction and refusal.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe features as context-dependent latent variables and define feature directions by how representation changes relate to feature probability. They also introduce safety-direction analysis and layer-wise relevance propagation.

Linear Representation

  • A feature value W is a one-dimensional latent variable expressed in context w, whose output probability P(W) can represent safe or unsafe responses.The paper frames safety analysis around queries with varying safety aspects.Source: §2 · p. 2
  • A direction x qualifies as a feature direction when a representation shift between two contexts corresponds to an increase in the probability of feature W.The stated condition is P(W = 1 | λ(w1)) > P(W = 1 | λ(w0)).Source: §2 · p. 2

Safety Directions

  • Safety directions are obtained from sample sets differing in feature W by maximizing the distance between their distributions; suppressing a direction is proposed as an intervention to verify causality.The refusal-direction example compares inputs eliciting compliant versus refusing responses.Source: §2 · p. 2

Safety Residual Space

  • In the illustrated safety residual space, the dominant direction predicts safety behavior, while non-dominant directions capture indirect safety features.The space is defined as the linear span of representation shifts during safety fine-tuning.Source: Figure 1 · p. 2

Layer-wise Relevance Propagation

  • Layer-wise relevance propagation decomposes a network output into input contributions and back-propagates relevance across layers.For an input-output pair, relevance scores represent how much an input contributes to an output; relevance is conserved across layers.Source: §2 · p. 2

3. Safety Residual Space

This section defines safety residual space as the linear activation shifts induced by safety fine-tuning and motivates testing whether orthogonal components encode distinct safety features. It outlines a fine-tuning case study and reports reduced jailbreak harmfulness and 90% refusal accuracy.

3. Safety Residual Space

  • Safety residual space is the linear span of representation shifts induced by safety fine-tuning, modeled as an affine transformation S(x) = Wx+b.The transformation is fit on unaligned samples.Source: §3 · p. 3
  • The affine map approximates the fine-tuning representation shift using linear activation changes, while ignoring nonlinear error between S and T.Activations come from transformer-block outputs at the first generated-token position in each layer.Source: §3 · p. 3
  • SVD of W − I yields the largest orthogonal representation-shift directions, ranked by component size.The paper labels layer-component directions LN-CK, with LN-C1 dominant and the others non-dominant.Source: §3 · p. 3

3.1. Component as Feature Direction

  • Residual components are proposed as safety-feature directions, with orthogonal components potentially distinct, but the space spans features developed during safety training only.Features irrelevant to the training objective or already present in the pretrained model may not appear in the residual space.Source: §3.1 · p. 3

3.2. Experimental Setup

  • The case study uses 2,600 samples, including jailbreak prompts and harmless examples, to fine-tune Llama 3.1 8B Instruct with SSFT and DPO.Refusal accuracy is measured across harmless and harmful test samples, while response harmfulness uses STRONG REJECT scores.Source: §3.2 · pp. 3–4
  • STRONGREJECT score across jailbreak attempts fell from 0.65 to 0.05, while refusal accuracy improved to 90%.The reported fine-tuning uses SSFT and DPO.Source: §3.2 · p. 4

4. Linearity of Safety Residual Space

The section examines linear properties of safety residual directions, finding an approximately low-rank structure and comparing how directions predict refusal.

4. Linearity of Safety Residual Space

  • Effective rank k is the minimum number of orthogonal components needed to explain threshold τ of variance in the representation shift.The singular values σ_i are those of W − I.Source: §4 · p. 4
  • The safety residual space is approximately low-rank linear for both DPO and SSFT, with effective rank varying across layers and differing at the final layer.For SSFT, effective rank remains 1 across τ values in the first 10 layers, peaks around layer 20, then returns to 1; for DPO, it continues to increase at the final layer.Source: §4 · p. 4
  • Both the dominant direction and refusal probe predict refusal accurately in later layers, while base-model activation components fail; the probe performs better early.Best-of-N scores more closely match the probe’s accuracy, suggesting that the probe aggregates multiple safety feature directions.Source: §4 · p. 4, Figure 3 · p. 4
  • Multiple orthogonal feature directions predict refusal beyond the single dominant direction or probe vector.The authors interpret this as evidence that refusal can involve multiple feature directions.Source: §4 · pp. 4–5

5. Feature Directions in Safety Residual Space

The section develops PLRP to interpret orthogonal directions in the safety residual space and trace how those directions evolve across layers. It finds that dominant and non-dominant directions encode different features and participate in distinct patterns of safety-related influence.

5. Feature Directions in Safety Residual Space

  • Because arbitrary residual directions lack predefined meanings and may not align with vocabulary space, the section uses relevance scores to interpret their features.The token rankings in Table 2 are based on average PLRP relevance scores across test samples.Source: §5 · p. 5, §5.1 · p. 6, Table 2 · p. 6
  • PLRP projects representations onto selected directions, decomposes their norm into relevance scores, and back-propagates scores to training tokens or earlier-layer directions.Token relevance is summed across embedding elements; earlier-layer direction relevance is estimated through a linear reconstruction using leading SVD components.Source: §5 · pp. 5–6

5.1. Interpreting Directions via Token Relevance

  • L14-C2 tracks hypothetical framing and negatively correlates with the dominant refusal direction, while L14-C5 tracks ChatGPT mentions and positively correlates with it.The dominant direction is associated with harmful subjects; non-dominant directions instead reflect jailbreak-pattern features.Source: §5.1 · p. 6
  • Removing L14-C6 selectively ablates refusal of PAIR prompts while preserving refusal of other attack types; removing dominant L25-C1 eliminates refusal.L14-C6 activates when “Sure, I’m happy to help” co-occurs with “Imagine.”Source: §5.1 · p. 6, Figure 4 · p. 6

5.2. Layer-Wise Dynamics of Safety Residual Space

  • Early-layer directions receive contributions from multiple preceding directions, whereas directions in most layers primarily retain information from their counterparts.L20-C1 inherits most of its relevance from L19-C1.Source: §5.2 · p. 7
  • After layer 15, major directions retain more strongly and their eigenvalues rise, while refusal prediction accuracy plateaus and aligned-behavior log-likelihood continues growing.The authors interpret this pattern as continued strengthening of the dominant safety direction and reduced uncertainty in aligned outputs.Source: §5.2 · p. 7

6. Toward Multi-dimensional Concept of Safety Fine-tuning Vulnerabilities

The section tests whether manipulating non-dominant directions or removing trigger tokens can weaken safety fine-tuning while preserving refusal capability. Trigger removal remains effective after fine-tuning, and both interventions are associated with shifts in the dominant component’s activation projection.

6. Toward Multi-dimensional Concept of Safety Fine-tuning Vulnerabilities

  • The suppression test removes most non-dominant components while leaving dominant components untouched, excluding directions with harmfulness correlations above 0.7.Harmfulness correlation is defined as the correlation between direction-activation dot products and prompt harmfulness.Source: §6 · p. 7
  • Trigger removal uses token-wise PLRP on dominant directions in final layers to identify trigger tokens, then an LLM rephrases harmful prompts to avoid them.The modified prompts are incorporated into the safety fine-tuning dataset and evaluated on a validation split.Source: §6 · p. 7

Disrupting Non-dominant Directions Reduces Refusal

  • After fine-tuning on 80 samples per jailbreak, Trigger Removal Attack maintains approximately 40% effectiveness while other jailbreaks’ success rates drop to near zero.Table 3 reports attack pass rates under different exposure settings.Source: Table 3 · p. 8, Disrupting Non-dominant Directions… · p. 8
  • Non-dominant suppression and trigger removal both shift dominant-component projections away from harmful samples and are associated with lower refusal rates.Trigger removal shifts projections toward benign samples, whereas non-dominant suppression shifts them in the opposite direction.Source: Disrupting Non-dominant Directions… · p. 8, Figure 6 · p. 8
  • The authors report that non-dominant directions causally affect the dominant component and safety behavior, and suggest safety training may model spurious jailbreak-pattern correlations.They connect this possibility to out-of-domain attacks such as Trigger Removal Attack.Source: Disrupting Non-dominant Directions… · p. 8

7. Discussion

The paper finds several linear feature directions in the safety residual space, while noting that their behavior and interpretability have limits. More diverse alignment data and larger models may increase the number of directions to interpret.

7. Discussion

  • Some identified safety residual-space directions are linear, but directions can flip across layers and extending them indefinitely can degrade generation quality.The authors nevertheless report verifying the linearity of several directions.Source: §7 · p. 8
  • The study uses harmful misaligned prompts, whereas practical safety data may be more diverse and desired behavior may extend beyond refusal.The authors expect greater data complexity and model size to increase effective rank, making directions harder to interpret.Source: §7 · p. 8
  • Future work could analyze fine-tuning in smaller intervals or group samples by domain to address interpretation challenges.Source: §7 · p. 8

8. Related Work

Prior work studies jailbreaks and internal mechanisms of safety alignment using varied methods and model components. This paper builds on representation-shift research by characterizing safety directions and assigning them semantic meaning.

8. Related Work

  • Jailbreak attacks are widely studied and reported effective against even recent models, motivating this work’s focus on them.Source: §8 · p. 9
  • Mechanistic safety research has localized behaviors in activations, attention, and parameters, while unsupervised methods have identified safety-related feature directions.Source: §8 · p. 9
  • Against prior studies of representation shifts, including concurrent subspace work by Wollschläger et al. (2025), this paper characterizes shifts and attributes semantic meaning to directions.Source: §8 · p. 9

9. Conclusion

The paper concludes that safety fine-tuning produces multiple directions that jointly shape safety behavior, including non-dominant features linked to trigger tokens. It proposes targeted activation-space interventions or data augmentation as future directions for robust alignment research.

9. Conclusion

  • Multiple feature directions jointly control safety behavior, and non-dominant directions affect that behavior and link to specific trigger tokens.Source: §9 · p. 9
  • The paper presents its multi-dimensional account as a hidden aspect not captured by probing or static methods.Source: §9 · p. 9
  • Preventing spurious correlations through targeted activation-space interventions or balanced-training data augmentation is proposed for future work.Source: §9 · p. 9

Impact Statement

The impact statement notes potential misuse of analysis and bypass methods, while appendix passages describe the residual-space accuracy check, trigger-removal procedure, and training and evaluation setups.

A. Accuracy of the Safety Residual Space Approximation

  • The learned affine map’s MSE is negligible relative to the mean squared norm of unaligned activations, supporting its accuracy in capturing fine-tuning activation changes.Table 4 reports the test-set comparison between predicted and actual post-fine-tuning activations.Source: A. Accuracy of the Safety Residual… · p. 13, Table 4 · p. 13

B. Algorithm for Trigger Removal Attack

  • The attack iteratively resamples harmful-prompt variants, evaluates them, applies PLRP to selected variants, and adds identified tokens to a blacklist until success or the iteration limit.The implementation uses three iterations and ten rephrased variants per iteration; the variants retain harmful intent.Source: B. Algorithm for Trigger Removal Attack · pp. 13–14
  • The iterative trigger-removal process requires at most 30 attempts per sample, comparable to TAP’s average 35 and PAIR’s average 37 attempts.Source: B. Algorithm for Trigger Removal Attack · p. 14

C.1. Dataset Construction and Composition

  • The training dataset combines jailbreak-generated harmful samples from STRONG REJECT with harmless OR-Bench samples, and includes a simple historical-question prompt baseline.Source: §C.1 · p. 14

Simple Jailbreaking Prompt Template

Reject Prompt Template for Llama-3.1-405B

Accept Prompt Template for Hermes-3-Llama-3.1-70B

  • A weakly aligned Hermes model supplies non-preferred outputs for harmful samples; for harmless samples, accept and reject templates are assigned to preferred and non-preferred outputs, respectively.The templates guide response sampling rather than assess the input’s safety.Source: Reject Prompt Template for… · p. 15, Accept Prompt Template for… · p. 16
  • The fixed test set contains 60 disjoint STRONG REJECT harmful samples and 480 non-overlapping harmless OR-Bench inputs, while training varies the blindspot subset size N.Source: Accept Prompt Template for… · p. 16

C.2. Training Procedure and Results

The section reports how exposure-based safety training affects jailbreak rejection, how trends vary across model scale and architecture, and how interventions affect general-task perplexity. Across these experiments, more exposure improves rejection, while intervention-related capability retention varies by method.

C.2. Training Procedure and Results

  • N-SHOT Security Training varies exposure examples for each jailbreak method to test how exposure affects rejection accuracy and safety.For N = 80, each method trains on 80 examples of the same original harmful goal.Source: §C.2 · pp. 16–17
  • As SSFT exposure increased, rejection improved across jailbreak methods, but Trigger Removal retained some jailbreak capability after more than 80 examples.Simple and GCG methods were quickly handled, while Flip and CodeChaeleom remained ineffective because of the model’s poor handling of chaotic formats and complex prompts.Source: §C.2 · p. 17, Figure 8 · p. 17
  • Direct-from-original DPO showed inconsistent rejection of PAIR variants, whereas SFT-initialized DPO focused better on dominant safety directions and avoided over-rejecting safe samples.The section reports average Strong Reject Scores by method and exposure time in Tables 6 and 7.Source: §C.2 · p. 17, Figure 9 · p. 18
  • More exposure examples lowered Strong Reject Scores across SSFT, DPO, and SFT+DPO; SFT-initialized DPO combined high safe-example accuracy with robust harmful-query refusal.Strong Reject Scores measure jailbreak effectiveness, so lower scores indicate less effective jailbreaks.Source: §C.2 · p. 18

C.3. Experiment on Models of Different Scale and Architecture

  • The evaluated models showed safety-improvement trends similar to the Llama-3.1-8B-Instruct baseline but weaker recognition and handling of unseen harmful queries across all jailbreak methods.The 3B-parameter Llama variant showed faster initial convergence and consistently failed to recognize Trigger Removal attacks.Source: §C.3 · p. 18
  • 22% higher loss variance than Llama-3.1-8B-Instruct accompanied Ministral-8B-Instruct’s slower convergence, despite better retention of dominant safety directions.The reported trajectory was more volatile across baseline attack methods.Source: §C.3 · p. 18, Figure 10 · p. 18

C.4. Impact of Model Intervention on General Ability

  • Perplexity on Alpaca output samples was 8.42 for DPO versus 7.10 for base Llama and 6.59 for SSFT; directional interventions scored 7.04 for L25C1, 7.23 for non-dominant, and 7.32 for L14C6.Lower perplexity indicates better retention of general capabilities.Source: §C.4 · p. 19, Figure 12 · p. 19

D. Visualization of Harmfulness Correlation

The section visualizes harmfulness correlations between activation projections and input prompts, including which components were excluded from suppression. It also compares pre- and post-intervention projection distributions across layers.

D. Visualization of Harmfulness Correlation

  • Figure 13 plots activation-projection correlations with input harmfulness across layers and components; suppression excludes components above 0.7 correlation, including about two directions per layer after layer 15.All 4096 directions before layer 10 are suppressed, while high-correlation directions are excluded from suppression after layer 15.Source: D. Visualization of Harmfulness… · pp. 19–20, Figure 13 · p. 19

E. Projection Values and Refusal Rates for Non-Dominance Suppression

  • Post-intervention projections are distinctly separated from pre-intervention projections in early layers, while later-layer post-intervention projections have smaller variance.Figure 14 presents these projection-value distributions for the non-dominance suppression experiment.Source: E. Projection Values and Refusal Rates… · p. 20, Figure 14 · p. 20

Related papers

Loading 2502.09674v4…