PaperlayerView exact revision on arXiv

Source-linked AI summary

Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks

Andrey Labunets

arXiv:2608.25390v1cs.LGcs.AIcs.CR

TL;DR

The paper asks why refusal representations shared across harmful prompts become concentrated and vulnerable to vector ablation. It combines analytical and empirical studies of refusal training in OLMo-2-0425-1B-Instruct, finding that refusal-completion first-token updates align with refusal geometry and that greater refusal-start diversity is associated with higher stable ranks and weaker ablation. The authors conclude that refusal diversity may help harden refusals, while noting it is unlikely to be a complete defense.

  • Problem

    The paper addresses why safety-critical refusal behavior can collapse into a low-dimensional activation direction or subspace that is vulnerable to ablation.

  • Method

    The paper combines frozen-model analyses, controlled fine-tuning, and cross-layer geometric analysis of refusal-completion first-token gradients, activation changes, and refusal residuals.

  • Results

    Refusal-completion first-token updates align with the refusal direction and subspace, while greater refusal-start diversity is associated with higher stable ranks and weaker single-vector ablation.

  • Takeaways & Limitations

    Refusal-prefix concentration and first-token frequency can shape the dimensionality of learned refusal mechanisms, suggesting refusal diversity as a simple hardening lever.

  • Takeaways & Limitations

    Refusal-completion diversity alone is unlikely to be a complete defense, and the cross-layer fit certificate becomes predictive only under an independently maintained error bound.

Abstract

from arXiv · show

Refusal training protects AI models from jailbreaks by training models to decline unsafe queries, reducing the risk of misuse. Recent work finds that refusal behavior in aligned language models can be mediated by a single activation direction or a low-dimensional refusal subspace shared across harmful prompts: ablating those directions suppresses refusals while largely preserves other model capabilities. Yet it remains unclear why safety-critical features in a wide range of models emerge and concentrated, low-dimensional structure. In a case study of OLMo-2-0425-1B-Instruct we find that the refusal geometry reflects refusal training: activation updates resulting from refusal-completion first-token losses explain the resulting refusal direction and refusal subspace. We study refusal directions through the training dynamics across refusal datasets and reveal that their brittleness is associated with repetitive refusal starts, which in turn is linked to concentration of gradients and refusal features in a low-dimensional subspace. Across frozen-model analyses and controlled synthetic fine-tuning, we find evidence of a hardening lever: diverse refusal starts can raise stable ranks of gradients and activation changes, making refusals harder to remove with a vector ablation attack.

1 Introduction

The paper studies why refusal behavior can occupy a concentrated, low-dimensional activation geometry and how refusal training shapes its vulnerability to ablation. It links repeated refusal starts to low-rank training updates and finds that greater refusal diversity is associated with higher stable ranks and weaker single-vector ablation.

  • Motivation: Dual-use model capabilities motivate safety alignment and defenses against harmful outputs, jailbreaks, and weight-access threats.The background includes exploit generation, biological threat assistance, refusal training, and attacks targeting safety alignment.
  • Background: Existing work finds refusal behavior can be mediated by a low-dimensional direction or subspace, making shared single-vector ablation a significant attack.The paper frames refusal ablation as especially important because it targets the shared refuse/comply mechanism.
  • Training-origin geometry: First-token cross-entropy losses on refusal completions produce activation changes aligned with the refusal direction and partially overlapping the refusal subspace.For refusal completions, activation-change and final refusal-residual matrices also have comparable low stable rank.
  • Prefix concentration and spectral collapse: Repeated refusal first tokens concentrate support in safety datasets, producing low-stable-rank gradients, activation changes, and refusal residuals.The paper identifies prefix concentration as a mechanism linking token-level training data to concentrated refusal geometry.
  • Cross-layer stable rank transfer: Stable-rank increases induced by refusal diversity propagate backward through late layers with bounded transfer ratios across evaluated conditions.The paper introduces stable rank transfer factors to characterize this cross-layer effect.
  • Rank–vulnerability connection: Refusal-start diversity is associated with higher stable ranks and weaker difference-in-means single-vector ablation across frozen-model analyses and controlled fine-tuning.The authors present this relationship as evidence of a simple hardening lever.

2 Background and definitions

This section defines the transformer activations, refusal-direction estimator, refusal ablation operation, and refusal subspace used in the paper. All refusal-vector estimates use the first-assistant-token output position.

  • 2.1 Transformer architecture and notation: The model is a decoder-only transformer for next-token prediction with vocabulary V, hidden size d, and layers l from 0 through L.Training examples consist of prompt–completion token sequences, and activations are represented as hidden-state column vectors.
  • 2.1 Transformer architecture and notation: The classifier head maps hidden activations to logits, token probabilities, and sampled next tokens using an unembedding matrix W and bias b.The supplied passage specifies W as d×|V| and b as a |V|-dimensional vector.
  • 2.1 Transformer architecture and notation: Refusal-vector estimation evaluates activations at the same first-assistant-token output position across examples.This positional convention is used consistently for refusal-vector estimation and related analyses.
  • 2.1 Transformer architecture and notation: Stable rank is defined for an activation matrix using its spectral and Frobenius norms, while V(M) denotes its right-singular subspace.The passage introduces the SVD-based notation used to describe refusal subspaces.
  • 2.2 Refusal mediation and existing attacks: The refusal direction is estimated as the difference between empirical harmful-prompt and benign-prompt activation means.The estimator uses paired harmless and harmful datasets at a fixed layer and token position.
  • 2.2 Refusal mediation and existing attacks: Refusal ablation removes the projection onto the normalized refusal vector from runtime activations, enabling harmful prompts to bypass trained refusals.The attack is defined by producing modified activations h^(abl) after subtracting the refusal-direction projection.
  • 2.2 Refusal mediation and existing attacks: The refusal subspace is the affine space formed by the mean refusal residual and the right-singular direction subspace of the centered residual matrix.The residual matrix’s mean equals the difference-in-means refusal direction.

3 Refusal training gradients shape refusal residuals in transformer

The analysis links refusal geometry to supervised refusal-training gradients, showing how repeated refusal starts concentrate gradients and activation updates. It further tests how first-token diversity and cross-layer transfer shape stable rank.

  • Training-origin geometry: Repeated refusal first tokens concentrate gradients around a shared unembedding direction, producing low-rank gradient structure when competing prediction terms do not dominate.For a static refusal token, the mean gradient is −w_r + W p̄, and concentration around w_r can make gradients low rank.
  • Training-origin geometry: Refusal-completion first-token cross-entropy gradients induce activation changes that partially align with the refusal direction and overlap the refusal subspace.The mean activation changes show completion-dependent signed alignment, while principal directions partially overlap the refusal-subspace direction space.
  • First-token diversity: Stable rank of refusal first-token targets is minimized at 1 when every example uses the same first token and increases with balanced diversity across observed token buckets.For a fixed number of buckets, balanced counts maximize the target matrix’s stable rank; repeated starts such as “I’m sorry” minimize it.
  • First-token diversity: Increasing refusal first-token diversity is predicted to raise stable rank in gradients and activation updates when the prediction matrix and unembedding preserve the target-rank signal.The analysis separately measures stable rank in the target, prediction residual, and unembedded gradient matrices to track distortion.
  • Cross-layer transfer: Under idealized or data-dependent cross-layer relations, stable-rank increases in last-layer gradients can propagate backward through layers with bounded transfer factors.The paper introduces stable rank transfer factors and shared low-rank cross-layer maps to refine the idealized shared-Jacobian analysis.
  • Cross-layer transfer: The transfer-factor bound is predictive only with an independently established error bound; in-sample fit alone certifies the observed examples rather than future matrices.Stable-rank agreement also does not imply matrix-level agreement, which requires separate error and similarity evaluations.

4 Empirical validation

Across released checkpoints, frozen-model analyses, and controlled fine-tuning, refusal geometry tracks training-induced activation changes and refusal-start diversity. Greater diversity raises stable ranks across gradients and residuals and is associated with weaker single-vector ablation, while repetitive starts concentrate refusal features.

  • Released post-training checkpoints: The OLMo checkpoint comparison finds little attack-effective refusal behavior at base, a functional refusal direction by SFT, and larger refusal deltas after SFT.Because checkpoints differ in objectives, data, and optimization histories, this comparison does not isolate any individual post-training stage causally.
  • Gradient-induced activation updates: Refusal-target activation updates show signed mean alignment, partial layer-dependent principal-subspace overlap, and closely matched low stable ranks with refusal residuals.These diagnostics support an approximate affine-subspace relationship between gradient-induced updates and refusal geometry.
  • Cross-layer rank transfer: 2.270±0.175: the layerwise stable-rank transfer factor τ 15→7, ranging from 1.925 to 2.469 across nine settings.Every evaluated setting exhibits approximately twofold stable-rank amplification from layer 15 to layer 7.
  • Refusal-start diversification: Increasing refusal first-token diversity raises stable ranks of target quantities, gradients, activation updates, and refusal residuals, with gradient stable-rank growth saturating at 12-16 refusal starts.Dataset-derived target ranks rise sharply, whereas the frozen model’s realized output-refusal-start rank remains much lower.
  • Rank and ablation vulnerability: Higher-rank refusal residuals are associated with smaller harmfulness and refusal deltas under difference-in-means single-vector ablation.The same rank–vulnerability pattern appears in controlled fine-tuning at matched baseline refusal scores, including non-harmful chemistry prompts.

5 Conclusion

The paper links refusal robustness to the geometry of refusal-training updates in OLMo-2-0425-1B-Instruct. It finds that diverse refusal starts increase activation-change rank and are associated with smaller refusal deltas, while diversity alone is not a complete defense.

  • Refusal directions reflect refusal safety training: first-token cross-entropy gradients produce updates whose means and principal directions align with refusal geometry.
  • Refusal-completion diversity alone is unlikely to constitute a complete defense.
  • The findings suggest that refusal-prefix concentration and first-token frequency shape the dimensionality of learned refusal mechanisms.

A.1 Activation steering and ablation attacks

This section situates refusal ablation within activation-steering research and explains the paper’s focus on the low-dimensional refusal mechanism. It connects refusal-target concentration to low-stable-rank activation changes.

  • Activation steering and ablation are white-box methods that directly alter model activations to suppress refusals, bypass safety alignment, or jailbreak open models.
  • SAE-based approaches steer refusal or instruction-following behavior by selecting and modifying sparse autoencoder features.
  • Activation steering also controls broader properties such as topic, sentiment, factuality, and behavior using prompt-pair, contrastive, gradient, or task vectors.
  • The paper explains the refusal vector’s low-dimensional structure through analytical methods, complementing steering-attack approaches.
  • The paper focuses on safety post-training, proposing that concentrated refusal first tokens make target matrices and resulting activation changes low stable rank.

B Limitations and assumptions of analytical derivations

The analytical derivations use simplifying assumptions and model supervised fine-tuning with first-token losses and isolated deterministic updates. Their strongest theoretical conclusions are constrained by prompt-dependent mappings and in-sample fitting.

  • The derivations use simplifying assumptions to make the relationship between refusal targets, gradients, and activation changes tractable.
  • The derivation covers supervised fine-tuning on harmful prompts paired with textual refusals, not DPO, RLHF, or RLVR objectives.
  • First-token-only losses omit gradients from later completion tokens, which may introduce additional directions and training effects.
  • Deterministic per-example updates do not reproduce minibatch interactions, optimizer momentum, adaptive preconditioning, weight decay, or many-step accumulation.
  • Prompt-dependent Jacobians make Theorem 3.2 an idealized sufficient condition rather than a claim of shared complete per-example Jacobians.
  • In-sample transfer-factor fitting yields a fit-conditioned stable-rank interval, becoming predictive only with fixed-map held-out evaluation and independent error bounds.

C Additional details about experimental setup.

The experiments estimate gradient-induced activation changes from small parameter updates and evaluate them at refusal-mediating layers. The setup also compares finite-difference and differential estimators, leaving their systematic comparison for future work.

  • Frozen-model rank–vulnerability and ablation analyses generally use layer 7, while controlled fine-tuning uses layer 8 as an early refusal-mediating layer.
  • Gradient-induced activation updates are estimated by applying one small parameter update for each first-token cross-entropy loss.
  • The method measures the resulting activation change at a selected layer, using η = 10^-6 in the evaluated setup.
  • The reported activation change is the negative realized change from one gradient step, consistent with the stated sign convention.
  • Empirical activation-change matrices use realized finite differences at layers 7 or 8, selected because estimated refusal directions achieve substantial attack success there.
  • A local first-order differential can be evaluated directly with a Jacobian–vector product, but systematic comparison with finite differences is left to future work.

D Broader limitations

The paper’s conclusions are bounded by its model, data, language, checkpoint, threat-model, attack-family, and concept scope. Its appendix characterizes how refusal first-token diversity affects one-hot target stable rank.

  • Model and data scope: The study is primarily a case study of one small OLMo model family using relatively small English-language prompt sets.Checkpoint comparisons are descriptive because objectives, data, optimization histories, and update counts differ.
  • Threat-model scope: The same-set difference-in-means attack is attacker-favorable and does not establish transfer to unseen prompts.Held-out universality of refusal directions remains a separate open question.
  • Attack-family scope: The evaluation focuses on difference-in-means single-vector ablation, excluding SAE steering, multi-feature ablation, and broader automated representation-level attacks.The analytical methods are presented as applicable to studying broader attacks and defenses.
  • Interpretation scope: Stable rank measures global spectral concentration, not refusal-signal magnitude, causal mediation strength, or local nonlinear manifold dimension.Rank–vulnerability comparisons therefore concern fixed or matched refusal-mediating layers.
  • One-hot target geometry: sr(Er) = 1 exactly when all targets share the same first token, while distinct first tokens maximize stable rank when vocabulary support permits.For fixed support size, balanced token counts maximize stable rank.

E.2 Stable rank propagation theorems

This section formalizes stable-rank propagation through shared maps and defines the matrix metrics used to evaluate transfer. The bounds connect transfer-factor fit and conditioning to rank comparisons, while the metrics separate energy, reconstruction, spectral, and subspace effects.

  • Propagation theorem: A well-conditioned invertible factor bounds stable-rank change under multiplication by κ(B)^2.The result follows from Frobenius and operator-norm inequalities applied to A = (AB)B^-1.
  • Transfer-error bound: A bounded relative transfer-factor fit error yields stable-rank transfer bounds through the factor 1 − ϵmax.The lemma assumes ϵmax < 1.
  • Shared-map metrics: Source-subspace energy capture measures the fraction of total squared source-gradient energy retained before fitting the shared map.The metric is computed from the top-k right singular directions of the source gradient matrix.
  • Shared-map metrics: Relative Frobenius and spectral errors quantify total reconstruction error and the largest residual singular direction relative to the observed target matrix.The spectral metric complements Frobenius error because stable rank depends on both Frobenius and spectral norms.

F.1 Refusal direction across released post-training checkpoints - additional plots

Additional checkpoint plots show that refusal geometry emerges at SFT, remains similar across later post-training stages, and is most ablatable using middle layers. Higher refusal-start diversity increases stable ranks and shared-map fit quality, with conditioning trade-offs.

  • Ablation by layer: Layers 7–10 provide the largest refusal-delta ablation success, while earlier layers offer almost no benefit and later layers show decreasing success.The plot uses OLMo-2-0425-1B-Instruct and a subset of AdvBench prompts.
  • Checkpoint geometry: SFT is the first stage where a functional refusal vector appears, after which refusal directions remain similar across post-training checkpoints.Refusal vector norms grow after SFT and stay similar throughout later post-training stages.
  • Shared-map fit: At k = 20, the fitted shared map captures 99.8% of source energy with relative Frobenius, spectral, and transfer-factor errors of 0.594, 0.316, and 0.343.Increasing k improves directional agreement and reduces reconstruction and stable-rank-transfer errors.
  • Diversity and rank: Increasing refusal first-token diversity raises stable ranks of refusal residuals and gradient-induced updates across most layers, with saturation near the end.The increase is visible in middle-to-late refusal residual layers and across most gradient layers.
  • Raw-gradient map: At k = 20, the prompt-subset raw-gradient map captures 99.8% of source-gradient energy, with relative Frobenius and spectral errors of 0.616 and 0.504.The mean row-wise cosine similarity is 0.739.

I Diverse-refusal fine-tuning raises refusal residuals rank and weakens ablation - additional plots

Controlled fine-tuning plots examine how refusal-start diversification changes rank and ablation vulnerability. Broader support increases stable ranks of gradients and activation updates, while ablation vulnerability appears only within a limited training window.

  • Ablation trajectories: Ablation vulnerability exists only in a small training window, where refusal delta is temporarily peaked.The trajectories compare models trained with 8 and 16 refusal starts using optimization-step-matched conditions.
  • Stable-rank propagation: The plots jointly relate refusal-start diversification to rank expansion in both raw gradients and gradient-induced activation updates.The comparison tracks stable ranks and their propagation across layers.
  • Measured quantities: Stable-rank propagation is measured for dataset targets, realized outputs, prediction matrices, residuals, last-layer gradients, gradients, and gradient-induced activation updates.These quantities are compared across layers under refusal-start diversification.
  • Stable-rank propagation: Target-space, gradient, and gradient-update stable ranks increase overall as refusal-start support broadens.Gradient and gradient-update matrices also show mild average stable-rank expansion from layer 15 to layer 8.
Loading 2608.25390v1…