PaperlayerView exact revision on arXiv

Source-linked AI summary

The Sharp Directions Are Against You: Curvature Analysis of Activation Steering

Seyedarmin Azizi, Arya Fayyazi, Parsa Razmara, Erfan Baghaei Potraghloo, Souvik Kundu, Massoud Pedram

arXiv:2610.16084v1cs.AI

TL;DR

CAA can steer behavior in the wrong direction, and the paper studies this fragility through the steering Hessian and sharp removal. Sharp directions reverse intended behavior, while removing them improves open-ended generation across the tested conditions.

  • Problem

    Contrastive activation addition is fragile: it can shift behavior in the wrong direction, and its effectiveness depends on the scaling coefficient.

    Source: §1 · p. 2

  • Method

    The paper analyzes behavioral-loss curvature with respect to the steering vector and removes its component in the top, sharp directions.

    Source: §1 · p. 2

  • Results

    Sharp removal outperforms standard CAA in open-ended generation across 25 of 25 tested conditions, while top eigenvectors reverse intended behavior in all 36 tested conditions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The steering Hessian shifts the question from which direction to steer toward which directions are safe to steer along.

    Source: §7 · p. 9

    Evaluation covers four tasks and three 7–8B instruction-tuned models; scaling to larger models and broader behavioral axes remains future work.

    Source: O Limitations · p. 22

Abstract

from arXiv · show

Activation steering adds a vector to a language model's internal representations at inference time. It is a lightweight alternative to fine-tuning for behavioral control. The standard construction, contrastive activation addition (CAA), is fragile, as on many inputs it shifts behavior in the wrong direction. To explain this fragility, we first study the geometry of the steering vector. We introduce steering Hessian, the matrix of second order derivative of the model's behavioral loss with respect to the steering vector. It captures the output behavior response sensitivity of the model under small perturbations of the steering vector along each direction. Its top eigenvectors are sharp directions: tiny perturbations along them can cause large, erratic, input-dependent behavioral swings. Bottom eigenvectors are flat directions: behavior is robust along them. Removing the sharp component of a steering vector approximately preserves its direction and magnitude (cosine > 0.974, norm reduction < 3% for typical settings) while reducing the vector's directional curvature by up to 12x (median 4x across 36 conditions). We evaluate on three model families (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B), with four behavioral tasks, and 36 token-level conditions. Steering along sharp directions reverses the intended behavior in every condition tested, with anti-steer rates of 53-100%. Sharp directions account for only 1-5% of the CAA vector's squared norm. However, they drive a disproportionate share of the gap between the steered and unsteered next-token distributions during autoregressive generation. We term this one-line correction as sharp removal; it outperforms standard CAA in open-ended generation across all 25 conditions tested. A jailbreaking experiment further shows that sharp directions act as noise rather than signal: they disrupt coherent behavioral control regardless of steering polarity.

1 Introduction

CAA is fragile, and the paper explains that fragility geometrically: a small sharp component can drive misbehavior. The authors introduce the steering Hessian and a sharp-removal correction, which improves open-ended generation.

1 Introduction

  • CAA’s fragility includes anti-steering on 30–50% of inputs, coefficient sensitivity, and behavioral inversion at most layers on three of four tested tasks.These issues motivate the paper’s geometric explanation and low-cost correction.Source: §1 · p. 2
  • CAA’s mean-difference construction does not distinguish catastrophic from stable directions, leaving 1–5% of its squared norm in sharp directions that drive most generation-time misbehavior.CAA takes the mean activation difference between positive and negative behavioral exemplars.Source: §1 · p. 2
  • The steering Hessian measures behavioral-loss curvature with respect to the steering vector, and removing its top-k component reduces directional curvature 1.7–12× while approximately preserving direction and magnitude.The median curvature reduction is 4× in the reported experiments.Source: §1 · p. 2
  • Top steering-Hessian eigenvectors reverse intended behavior in all 36 tested conditions, with anti-steer rates of 53–100%.The tests span three models, four tasks, and three layers.Source: §1 · p. 2
  • Sharp removal outperforms standard CAA in open-ended generation across 25 of 25 tested conditions.It is a drop-in correction that subtracts the sharp component of a steering vector.Source: §1 · p. 2

2 Related Works

Related work develops activation steering, documents its fragility, and explores sharpness in other settings. This paper positions its contribution as a geometric account of steering behavior and a corresponding correction.

2 Related Works

  • Activation-space interventions developed from concept activation vectors and latent steering to representation engineering and Activation Addition, with CAA extending the latter [Kim et al., 2018, Subramani et al., 2022, Zou et al., 2023a, Turner et al., 2023, Rimsky et al., 2024].Later approaches include adaptive interventions, SAE-guided features, conceptors, gradient optimization, and preference-optimized vectors.Source: §2 · pp. 2–3
  • Prior work documented anti-steering and sensitivity to layer and coefficient; this paper attributes these failures to sharp curvature in the activation-space loss landscape [Tan et al., 2024, Braun et al., 2025].Source: §2 · p. 3
  • Sharpness-aware minimization and its extensions address weight-space settings, whereas this paper applies sharpness to activation space and a single d-dimensional steering vector [Foret et al., 2021, Li et al., 2024, Fan et al., 2025, Yang et al., 2026].Here the loss landscape varies with each input.Source: §2 · p. 3
  • Prior safety attacks use prompt optimization or fine-tuning, while this paper reports that steering-Hessian directions can reverse safety-aligned behavior without adversarial optimization [Zou et al., 2023b, Qi et al., 2025].Source: §2 · p. 3

3 Method

The method defines a target-token behavioral loss, estimates its curvature with respect to steering vectors, and removes CAA’s component in the resulting sharp subspace.

3.1 Setup and Notation

  • The setup injects a scaled steering vector into a layer’s hidden state, then evaluates target behavior using next-token cross-entropy at the first generated position.The target token is one whose high probability signals the desired behavior; the standard steering vector is CAA, formed from contrastive exemplar pairs.Source: §3.1 · p. 3

3.2 The Steering Hessian

  • The steering Hessian is the matrix of second derivatives of expected behavioral loss with respect to the steering vector; its sharp subspace has large curvature and its flat subspace small curvature.The method estimates top and bottom eigenpairs using Lanczos iteration and finite-difference Hessian-vector products, averaging over estimation prompts.Source: §3.2 · pp. 3–4

3.3 Hessian-Informed Sharp Removal

  • Sharp removal subtracts the CAA component in the high-curvature subspace; a renormalized flat-subspace projection is a separate tool for diagnosing CAA inversion.Source: §3.3 · p. 4

4 Theoretical Analysis

The steering Hessian characterizes how behavioral loss changes when the steering vector is perturbed, separating sensitive sharp directions from relatively robust flat directions. Sharp removal subtracts the vector’s sharp component, approximately preserving its geometry while reducing directional curvature.

Proposition 1: interpretation

  • Sharp directions amplify small steering-vector perturbations into large, input-dependent logit changes, while flat directions lie near the downstream Jacobian’s approximate null space.The Gauss–Newton term typically dominates the steering Hessian and is governed by the downstream Jacobian.Source: §4 · p. 4

Definition 2: sharp energy fraction

  • The sharp energy fraction ε(v) is the squared norm of a vector’s projection onto the sharp subspace divided by its squared norm.Sharp removal subtracts that projected component from the CAA vector.Source: §4 · p. 4

Proposition 2: geometric preservation

  • For typical ε ≤0.05, the cleaned vector retains cosine > 0.974 with CAA and has norm reduction < 3%.The token-level loss perturbation bound is O(√ε), which vanishes as ε approaches zero.Source: §4 · p. 5

Proposition 2: curvature reduction

  • Sharp removal reduces directional curvature by 1.7–12×, with a median 4× reduction across experiments.The bound is stated at α=1.0; generation-level improvements at higher α are empirically validated but not formally bounded here.Source: §4 · p. 5

5 Experiments

Across three model families and four behavioral tasks, sharp removal improves generation over CAA in all 25 tested conditions. The experiments also find that sharp directions reverse behavior, flat-subspace projections help diagnose CAA inversion, and sharp directions disrupt control under either steering polarity.

5.1 Setup

  • The evaluation uses three 7–8B models, four behavioral tasks, held-out prompts, and 36 token-level conditions across three layers per model.Generation evaluations use 50 completions per condition; token-level steering is evaluated across ten α values from 0.25 to 5.0.Source: §5.1 · p. 6

5.2 Findings 1–2: Anisotropic Curvature, Sharp Directions Are Adversarial

  • The steering Hessian is anisotropic in every condition, and its top eigenvectors reverse intended behavior in all 36 conditions, with anti-steer rates of 0.53–1.00.On Llama refusal, vsharp reaches a 100% anti-steer rate at every layer, with peak magnitude ∆=−0.48.Source: §5.2 · p. 6

5.4 Finding 4: Sharp Removal Improves Generation

  • On Llama, CAA and vsharp have nearly identical next-token KL trajectories at about 19 nats, while sharp removal reduces the divergence by 3–4×.On Qwen the gap widens later in the sequence; on Mistral L14, CAA’s KL grows 3.8× over 100 tokens versus 2.5× for sharp removal.Source: Figure 2 · p. 7, §5.4 · p. 7

5.3 Flat-Subspace Analysis: Robustness and CAA Inversion

  • Flat-subspace projection has the lowest anti-steer rate in 11 of 12 model-task combinations and recovers the correct sign in reported CAA-inverted cases.CAA inversion is observed on three of four tasks; the proposed mechanism is that sharp directions capture input-dependent surface features while the flat subspace isolates dispositional signal.Source: §5.3 · p. 7

5.4 Finding 4: Sharp Removal Improves Generation

  • Sharp removal outperforms CAA in all 25 generation conditions across three tasks and three model families, often restoring or exceeding baseline refusal rates where CAA is inverted.The sharp component removed from CAA accounts for approximately 1–5% of its squared norm.Source: Figure 3 · p. 8, §5.4 · p. 8

5.5 Finding 5: Jailbreaking Reveals Sharp Directions as Noise, Not Signal

  • Sharp directions disrupt coherent behavioral control under negative steering: −vsharp yields 100% refusal on Llama, while the cleaned vector has the highest non-refusal rate in that direction.The cleaned vector also strengthens safety in the positive steering direction.Source: §5.5 · p. 8

5.6 Controls and Ablations

  • The Hessian and generic-PCA top-15 subspaces are geometrically distinct, with mean principal-angle cosine 0.077, while a k-sweep motivates k=10–15 as the default.The sweep reports a plateau by k=5 on Qwen refusal and over-removal at k=25 on Llama sycophancy.Source: §5.6 · p. 9
  • On inverted conditions, the cleaned vector beats sign-flipped CAA in 8 of 10 sycophancy conditions and all four inverted refusal conditions by 23–38 percentage points.The sharp component retained by −vCAA continues to disrupt generation regardless of sign.Source: §5.6 · p. 9

6 Discussion

The steering Hessian supports both analysis and deployment: flat-subspace extraction helps diagnose CAA inversion and select layers, while sharp removal is intended for generation-time use. The sharp energy fraction is proposed as a safety-audit metric for steering vectors.

Discussion

  • Flat-subspace extraction supports token-level analysis, safety auditing, and layer selection, while sharp removal provides a deployment-time correction for generation.The discussion also notes that standard CAA can degrade refusal and that sharp removal can reverse that change.Source: §6 · p. 9

7 Conclusions

The paper presents the steering Hessian as a way to distinguish sensitive from robust activation-space directions. Its findings characterize sharp directions as adversarial and disruptive to coherent control, while sharp removal improves generation across the tested conditions.

Conclusions

  • Across the tested conditions, sharp directions reverse intended behavior and disrupt control regardless of steering polarity, while removing them improves generation.The sharp component also accounts for a disproportionate share of the steered-versus-unsteered distributional gap during generation.Source: §7 · p. 9

A Target Tokens and Multi-Token Robustness

The behavioral target tokens are chosen to initiate the desired response. A multi-token contrastive objective produces a decomposition and steering performance comparable to the single-token version.

A Target Tokens and Multi-Token Robustness

  • Sentiment uses “Great” or “It” to initiate positive responses, while sycophancy uses “Actually” or “That” to initiate corrections.Source: A Target Tokens and Multi-Token… · p. 12
  • Toxicity uses “I” or “They” for respectful framing, and refusal uses “I” to initiate “I can’t...” refusals.Source: A Target Tokens and Multi-Token… · p. 12
  • With 6–8 tokens per positive and negative continuation, the multi-token objective yields a sharp subspace with principal-angle cosine 0.61 and matched-N honesty 0.83 versus 0.85 for the single-token version.The authors attribute robustness to curvature being dominated by the downstream Jacobian and state that the framework extends to differentiable behavioral objectives, including multi-token losses and reward-model scores.Source: A Target Tokens and Multi-Token… · p. 12

B Method Extras: Sharp Removal Norm Properties and Flat Subspace

Sharp removal subtracts the sharp projection from a CAA vector without renormalizing it, while flat-subspace extraction discards signal outside the flat directions. The algorithm estimates the sharp subspace from the steering Hessian, and the proofs relate this projection to the loss curvature and vector norm.

B Method Extras: Sharp Removal Norm Properties and Flat Subspace

  • Without renormalization, the cleaned vector typically retains 95–99% of the CAA norm because the removed sharp component carries 1–5% of its squared norm.Source: B Method Extras: Sharp Removal Norm… · p. 12
  • Sharp removal preserves the full CAA signal while discarding only the sharp component; flat-subspace extraction can diagnose CAA inversion but limits open-ended generation strength.The sharp component accounts for approximately 1–5% of the vector’s energy, which the passage says dominates distributional perturbation during generation.Source: B Method Extras: Sharp Removal Norm… · p. 12

C Algorithm Pseudocode

  • The algorithm estimates the top-k steering-Hessian eigenpairs with Lanczos iterations and Hessian-vector products, then subtracts the CAA vector’s sharp projection.Source: C Algorithm Pseudocode · p. 12

D Proof of Proposition 1

  • The Proposition 1 proof applies the chain rule through the affine steered activation and establishes that the Gauss–Newton term is positive semidefinite.Source: D Proof of Proposition 1 · pp. 12–13

E Proof of Proposition 2

  • The Proposition 2 proof decomposes CAA orthogonally into sharp and cleaned components, then uses that decomposition to relate their squared norms.It states that ∥vCAA∥^2 = ∥vS∥^2 + ∥vcleaned∥^2 and defines ε as the sharp component’s share of the CAA squared norm.Source: E Proof of Proposition 2 · p. 13

F Curvature Reduction Factor κgap

The curvature reduction factor compares the largest absolute steering-Hessian eigenvalue with the eigenvalue just beyond the removed sharp subspace. The generation evaluation protocol compares CAA and projected variants across behavioral tasks using task-specific classifiers and metrics.

F Curvature Reduction Factor κgap

  • Sharp removal bounds cleaned-vector directional curvature by |λπ(k+1)|, compared with worst-case original CAA directional curvature |λπ(1)|.Source: F Curvature Reduction Factor κgap · p. 14
  • 1.7–12.3 (median 4.0) is the κgap range across 36 conditions at k=15, with κgap defined as |λπ(1)|/|λπ(16)|.The factor is invariant to α because Hsteer is proportional to α^2; larger values indicate a stronger spectral gap and greater curvature reduction.Source: Table 2 · p. 14, F Curvature Reduction Factor κgap · p. 14

G Generation Evaluation Protocol (Full)

  • Generation comparisons include CAA, sharp and flat projections, random directions, and the cleaned vector, with task-specific steering coefficients and classifiers.The protocol uses sentiment score P(positive) − P(negative), sycophancy P(corrects), and refusal rate; refusal uses α=5.0 to stress-test safety alignment.Source: G Generation Evaluation Protocol (Full) · p. 15

H Layer Selection: The Sweet-Spot Layer Varies by Task

The best layer for flat-subspace steering varies by task, and the paper connects CAA inversion to input-dependent features in sharp directions that can oppose the target disposition.

H Layer Selection: The Sweet-Spot Layer Varies by Task

I Why Does CAA Invert?

  • CAA can invert when input-dependent lexical or response-format features correlate opposite to the target disposition; the flat subspace isolates the dispositional signal.The passage links sharp directions to downstream-Jacobian sensitivity and the flat subspace to an approximate Jacobian nullspace.Source: I Why Does CAA Invert? · p. 15

J Extended Controls and Ablations

Extended controls test whether sharp removal’s gains persist under larger samples, alternative baselines, safety checks, and judge-based scoring. Across these checks, it generally preserves or improves target behavior while limiting disruption from sharp directions.

J Extended Controls and Ablations

  • The sharp component is 1–5% of CAA’s squared norm (cosine >0.974), while PCA removal gives Δ=−0.021 and sharp removal gives Δ=+0.006.The top-15 generic-PCA and Hessian subspaces have mean principal-angle cosine 0.077, comparable to two random subspaces.Source: J Extended Controls and Ablations · pp. 15–16
  • Sharp removal outperforms −vCAA in all 4 inverted refusal conditions by 23–38 percentage points (Qwen L10: 98% vs 68%).On sycophancy, −vCAA is worse than vcleaned in 8 of 10 conditions, indicating that negating CAA alone does not resolve the problem.Source: J Extended Controls and Ablations · p. 16
  • Sharp removal raises refusal from 92% to 100% on Qwen and from 86% to 94% on Llama when applied to Arditi et al. [2024]’s direction.That direction is largely distinct from CAA, with cosine 0.01 on Qwen L10 and 0.20 on Llama L15.Source: J Extended Controls and Ablations · p. 16
  • At m=40, sharp energy is 4.61%, within 0.17 percentage points of the 4.78% value converged by m=60.At m=80, the energy fraction matches to four decimal places and the cleaned-vector norm differs by less than 0.1%.Source: J Extended Controls and Ablations · p. 16
  • On 100 safe XSTest prompts, Qwen sharp removal reduces false refusal versus CAA at α=1.0 (11% vs 16%); at α=2.0, it lowers full refusal from 53% to 6%.Source: J Extended Controls and Ablations · p. 16
  • GPT-4o-mini scores Mistral sharp-removal outputs at 88% full refusal, versus 68% for CAA and 52% for baseline, despite heuristic undercounting.The keyword-plus-NLI heuristic reports 12% baseline refusal, while the judge classifies 52% as full refusals; the relative ordering and effect size are preserved.Source: J Extended Controls and Ablations · p. 16
  • At N=200, sharp removal outperforms CAA in all 10 sycophancy conditions, while vsharp reduces honesty in all 10.The largest reported improvements are Mistral L10 (+0.089), Llama L16 (+0.057), and Qwen L13 (+0.040).Source: J Extended Controls and Ablations · pp. 16–17

K Qualitative Refusal Examples

  • In Mistral qualitative examples, sharp removal raises the overall refusal rate from 12% to 62%, while CAA responses still partially engage with harmful requests.The examples show unsteered compliance, CAA warnings with partial engagement, and sharp-removal refusals or redirects.Source: Figure 4 · p. 18

L Jailbreaking: Full Setup and Results

This section reports negative-direction jailbreak tests, finding that removing the sharp component makes steering least effective while the cleaned vector yields more non-refusal than CAA, with qualitative failures and sensitivity to the removal dimension qualifying the results. It also states that the work involved no human-subjects research and that LLMs were not an important, original, or non-standard component of core method development.

L Jailbreaking: Full Setup and Results

  • 80% vs 68% non-refusal on Qwen and 40% vs 33% on Llama favored −vcleaned over −vCAA; −vsharp yielded 2% full compliance on Qwen and 100% refusal on Llama.The tests used negative-direction steering at α=3.0 on Qwen2.5-7B-Instruct (L10, N=50) and Llama-3.1-8B-Instruct (L15, N=100), with GPT-4o-mini judging completions.Source: Table 4 · p. 19, L Jailbreaking: Full Setup and Results · p. 19

M Qualitative Jailbreaking Examples

  • −vsharp produced incoherent, wavering responses on Qwen and refusal-phrase repetition loops on Llama, whereas −vcleaned produced coherent compliance or engagement.The Llama examples describe the repetition loop as a failure mode in which the model repeats its refusal phrase verbatim.Source: Figure 5 · p. 20, Figure 6 · p. 21

N Sensitivity to Sharp Subspace Dimension k

  • On refusal, k=1 raises refusal from 44% to 90% and k≥5 yields 94–96%; on sycophancy, k=5 gives honesty 0.765 while k=25 falls to 0.698 below CAA, motivating k=10–15.At k=5, sycophancy honesty matches the unsteered baseline; the authors attribute the k=25 decline to over-removal as the removed energy makes the O(√ε) cost exceed the curvature-reduction benefit.Source: N Sensitivity to Sharp Subspace… · p. 22

O Limitations

  • The Hessian is a local quadratic approximation at (vCAA, α=1.0), tests cover four tasks and three 7–8B instruction-tuned models, and subspace transfer under prompt-distribution shifts remains an open question.Scaling to larger models, broader behavioral axes, and black-box settings where activations are inaccessible also remains future work.Source: O Limitations · p. 22

2. Limitations

  • The paper reports no crowdsourcing or research with human subjects.Accordingly, its response on participant instructions and compensation is [N/A].Source: §2 · p. 28

16. Declaration of LLM usage

  • LLMs were not an important, original, or non-standard component of core method development.Additional information is provided in Appendix P.Source: §16 · p. 29

Related papers

Loading 2610.16084v1…