PaperlayerView exact revision on arXiv

Source-linked AI summary

Signatures of Steerability in Activation Space of Language Models

Prajjwal Bhattarai, Tuka Alhanai

arXiv:2609.14151v1cs.LGcs.AI

TL;DR

Activation steering varies in effectiveness across concepts and datasets, raising the question of whether activation-set properties can predict downstream steerability. The paper tests simple separation metrics across model families and in synthetic superposition, finding consistent steerability correlations and strong association with recovery of the true feature direction.

  • Problem

    Steering reliability varies across concepts and datasets, and it remains unclear whether properties of the activations used to construct vectors can predict downstream performance.

    Source: §1 · p. 1

  • Method

    The paper evaluates simple activation-separation metrics across model families and uses synthetic superposition to test which structural properties those metrics track.

    Source: §1 · p. 1, §4.2 · p. 4

  • Results

    Simple separability metrics correlate with steerability across model families after controlling for major confounders, and strongly track alignment between the recovered and true feature directions in synthetic superposition.

    Source: §1 · p. 1, §5 · p. 5

  • Takeaways & Limitations

    Activation separation statistics are informative about steerability and whether the recovered direction aligns with the underlying feature direction.

    Source: §5 · p. 5

    The analyses may not extend to steering methods that do not rely on CAA's linearity assumptions, including optimization-based or nonlinear-interaction methods.

    Source: §6.1 · p. 5

Abstract

from arXiv · show

Steering language models using a set of contrastive representations has been a canonical and computationally efficient method for controlling model behavior. Despite this success in controlling certain model behaviors, the effectiveness of activation steering varies markedly across concepts; the generalization properties of steering vectors are often considered a function of the dataset used to construct them. We make this dataset-dependence claim more rigorous and show that simple separation metrics strongly correlate with the downstream steerability of language models across diverse settings, even after controlling for layers and dataset effects. Beyond prediction, we provide evidence from a synthetic superposition experiment that separation metrics are strongly correlated with alignment between the empirical and true feature direction. Our results suggest that simple separability statistics can serve as practical diagnostics for when steering vectors are likely to work.

1 Introduction

Activation steering can control behaviors without updating model weights, but its effectiveness varies across concepts and datasets. This work tests whether activation separability predicts downstream steerability and finds consistent correlations, including with feature recovery in a synthetic setting.

1 Introduction

  • Steering vectors alter model behavior by adding a linear direction to hidden activations at inference time, without changing model weights.Prior work has used steering to control honesty, harmlessness, and refusal.Source: §1 · p. 1
  • Steering effectiveness varies across concepts, prompt templates, datasets, and intervention layers, motivating a search for predictors of downstream performance.The authors hypothesize that CAA steerability depends on how separated positive and negative activations are.Source: §1 · p. 1
  • The paper formalizes separation using directional alignment, signal-to-noise ratios, and geometric or manifold-based measures of activation structure.The central question is whether activation-set properties predict the downstream performance of steering vectors.Source: §1 · p. 1
  • Simple separability metrics correlate with steerability across model families after controlling for major confounders, and strongly track recovery of the true feature direction in synthetic superposition.The same synthetic metrics correlate only weakly with superposition sparsity.Source: §1 · p. 1

2 Background

Prior work connects representational geometry to model performance, while evidence in language models remains limited. The paper relates steering reliability to activation geometry and uses a synthetic superposition setting to examine how overlapping features may affect recovered steering directions.

2.1 Predicting ability from representations

  • Across six instruction-tuned models and 38 datasets, SNR explains a substantial fraction of steerability variance (r = 0.76) and correlates with feature recovery in synthetic superposition.The synthetic setting constructs the ground-truth feature direction v0, allowing recovery to be measured as π = ⟨µ,v0⟩.Source: Figure 1 · p. 2
  • Prior work links neural collapse, low intrinsic dimension, and manifold geometry to performance, but these diagnostics have mostly been studied in image models.Braun et al. (2025) report that activation-to-steering-direction alignment correlates with steerability in Llama models; this work examines more metrics and model families.Source: §2.1 · pp. 1–2

2.2 Linear Representation Hypothesis & Superposition

  • The Linear Representation Hypothesis treats meaningful concepts as approximately linear directions, but superposition can make a contrastively recovered direction reflect co-occurring latent factors.The synthetic setting lets the authors examine which aspects of superposition their separation metrics capture.Source: §2.2 · p. 2

3 Methods

The methods construct CAA steering vectors from contrastive activations, measure steerability on held-out data, and compare several activation-separation metrics. A synthetic superposition experiment varies feature structure to test how separation relates to recovery of the contrastive direction.

3.1 Dataset & Models

  • The study uses persona, truthfulness, and safety datasets with instruction-tuned Qwen, Gemma, and Llama models at two parameter scales.Each dataset is split evenly: training data constructs steering vectors and separation metrics, while steerability is evaluated on the test set.Source: §3.1 · p. 2

3.2 Steerability and Separation metrics

  • CAA defines the steering vector as the mean activation difference for contrastive pairs; steerability is the test-set slope relating steering strength to mean target-versus-untargeted logit propensity.Separation metrics are correlated with steerability for each dataset-layer combination.Source: §3.2 · p. 3, §3.3 · p. 3

3.3 Separation criteria

  • The analysis correlates steerability with a range of criteria that measure separation between positive and negative activation sets.These criteria include signal-to-noise, mean alignment, manifold capacity, and intrinsic dimension.Source: §3.3 · p. 3, §3.3.1 · p. 3, §3.3.2 · p. 3

3.3.1 Signal-to-noise Separation (SNR)

  • SNR treats within-class, per-dimension activation variance as noise when comparing positive and negative activation sets.The covariance matrices of the two activation classes define the variances used.Source: §3.3.1 · p. 3

3.3.2 Mean µ alignment

  • Mean µ alignment measures how well individual contrastive pairs align with the constructed steering vector.Source: §3.3.2 · p. 3

3.3.3 Manifold capacity

  • Manifold capacity uses the convex hulls of positive and negative activations; α = 2Ncrit, where Ncrit is the minimum projection dimension for high-probability linear separability.Ncrit is computed using the empirical estimator of Chou et al. (2026).Source: §3.3.3 · p. 3

3.3.4 Intrinsic dimension

  • Intrinsic dimension is estimated with the Two-NN estimator, treating activation points as samples from an embedded manifold.The supplied method text does not specify further estimator details.Source: §3.3.4 · p. 3

3.4 Superposition

  • The synthetic superposition setup designates v0 as the contrastive feature and treats other dictionary features as noise, varying feature coherence, active noise-feature count k, and alignment π = ⟨µ,v0⟩.Contrastive activations use different α distributions for positive and negative examples, while noise coefficients are Gaussian and sparsified.Source: §3.4 · p. 3

4 Results

Across datasets and layers, SNR is the strongest predictor of steerability, including after controlling for those factors. In synthetic superposition, SNR and mean alignment track feature-direction recovery more closely than global structure.

4.1 Separability is correlated with steerability

  • SNR has the strongest Pearson and Spearman correlations with steerability across comparisons and after controlling for dataset and layer effects.Mean µ Alignment is competitive and generally moves alongside SNR.Source: Table 1 · p. 4, Table 2 · p. 4, §4.1 · p. 4
  • Manifold capacity is broadly informative but rarely surpasses SNR and mean µ Alignment, while intrinsic dimension is weak and varies across dataset settings.Intrinsic dimension’s correlation drops remarkably when controlling for layers.Source: §4.1 · p. 4

4.2 Why separation predicts steerability?

  • In synthetic superposition, SNR and mean alignment strongly correlate with π, correlate modestly negatively with k, and show essentially no connection with ϵ.This pattern supports feature recovery over sparsity or coherence as the mechanism tracked by the metrics.Source: §4.2 · p. 4, Table 3 · p. 5
  • The weakness of intrinsic dimension in real language models suggests global superposition geometry is neither necessary nor sufficient for predicting steerability.Source: §4.2 · pp. 4–5

5 Conclusion

The paper examines activation-distribution properties across model families and datasets, finding that simple separation statistics predict steerability. Synthetic superposition results indicate that these statistics track alignment with the underlying feature direction more strongly than global structural properties.

5 Conclusion

  • Across multiple model families and datasets, simple separation statistics are strong predictors of steerability.Source: §5 · p. 5
  • In synthetic superposition, separation statistics strongly track whether the recovered direction aligns with the underlying feature direction.Source: §5 · p. 5
  • In the synthetic experiments, separation statistics only weakly capture global structural properties of superposition.Source: §5 · p. 5

6 Limitations

The analysis is limited to CAA and does not establish that separation causally improves steerability. Its evaluation uses multiple-choice option-logit changes and does not test free-form fluency.

6.1 Steering method

  • The analysis covers CAA, while extension to optimization-based steering or methods using nonlinear interactions remains unclear.The authors state that it is unclear whether their analyses extend beyond CAA's linearity assumptions.Source: §6.1 · p. 5

6.2 Analyses are correlative, not causal

  • The reported relationship is correlational: separation may reflect geometry rather than a mechanism used during steering, or both measures may share an unknown cause.The authors do not determine whether separation causally contributes to improved steerability.Source: §6.2 · p. 5

6.3 Testing on specified task format

  • Evaluation uses multiple-choice option-logit changes, leaving free-form fluency untested and possible trade-offs with fluency or other-task performance unresolved.The authors note that stronger separation could improve steerability while degrading fluency or performance on other tasks.Source: §6.3 · p. 5

7 Ethical Considerations

Geometric properties that predict steerability may also make systematic manipulation easier, including for inducing biased or harmful behavior.

7 Ethical Considerations

  • Geometric diagnostics that identify steerable representations may help control unwanted behaviors but could also enable systematic induction of biased or harmful behavior.The authors identify this as a potential misuse pathway for malicious actors.Source: §7 · p. 5

8 AI use statement

The authors used AI assistants for literature search, language polishing, and coding, and manually validated AI-assisted code before implementation.

8 AI use statement

  • AI assistants supported literature search, language polishing, and coding; authors manually validated AI-assisted code before implementation.The statement identifies these uses without describing other AI-assisted tasks.Source: §8 · p. 5

A Mathematical details on steerability and separation metrics

The appendix defines steerability through the response of mean propensity to steering strength and specifies the configurations used in the synthetic superposition experiments.

A Mathematical details on steerability and separation metrics

  • Steerability is the slope relating steering multiplier to test-set mean propensity, where propensity is the positive-minus-negative completion logit difference.The steering vector is the mean difference between contrastive activations extracted at the last token of a residual layer.Source: A Mathematical details on steerability… · p. 7

B Superposition construction parameters

  • Each sparse-superposition configuration is a tuple (d, m/d, ϵ, k); Table 4 gives the terms' definitions and construction ranges.For each configuration and gap, the experiment draws 400 contrastive pairs.Source: B Superposition construction parameters · p. 7, Table 4 · p. 8
  • The synthetic analysis estimates correlations from 6,000 runs across four seeds, with the target coefficient not identifiable from conditional activation marginals.The reported design includes 300 structural configurations and 1,500 configuration–gap combinations per seed.Source: B Superposition construction parameters · p. 8

C Partial Correlation computation and results

The section computes partial correlations by removing layer or dataset effects from steerability and separation metrics, then correlating the residuals. Spearman partial correlations apply the same procedure after rank-transforming the variables.

C Partial Correlation computation and results

  • For each model and separation metric, dataset-layer pairs are observations; layer or dataset indicator variables serve as controls in ordinary linear regression.Layer controls use layer indicators, while dataset controls use dataset indicators.Source: C Partial Correlation computation and… · p. 8
  • Partial correlation is computed as the Pearson correlation between residualized steerability and residualized separation metrics.The residuals are formed by subtracting the regression-fitted values from the observations.Source: C Partial Correlation computation and… · p. 8
  • For Spearman partial correlations, the same residualization procedure is applied after rank-transforming the variables.Source: C Partial Correlation computation and… · p. 8

D Family-level correlations

These appendices report family-level correlations, uncertainty estimates, synthetic-configuration checks, dataset construction details, SNR ablations, and layer-wise comparisons. The results include within-configuration feature-recovery patterns and proximity between peak-correlation and peak-steerability layers.

D Family-level correlations

E Confidence intervals for point estimates

F Correlations with each synthetic configuration family

  • Within-configuration correlations for SNR and mean µ-alignment closely match pooled values, while intrinsic dimension’s pooled feature-recovery association vanishes within fixed superposition structures.The authors suggest intrinsic dimension tracks between-regime geometry.Source: F Correlations with each synthetic… · p. 9

G.1 Persona datasets

  • The persona family comprises 26 datasets from the Model Written Evaluations dataset.Source: §G.1 · p. 9

G.2 Truthfulness datasets

  • The truthfulness family uses the five FLEED categories as separate datasets, each targeting a distinct epistemic basis for truth values.Source: §G.2 · p. 9

G.3 Safety datasets

  • The safety family uses only SafetyBench’s English subset and contrasts each annotated safe answer with an incorrect unsafe distractor.When multiple incorrect options exist, one is selected randomly.Source: §G.3 · pp. 9–10

H Ablations with SNR

  • Adding SNR generally outperforms using its numerator or denominator alone across most models, with Qwen 4B the sole exception.Source: H Ablations with SNR · p. 10

I Steerability vs. Separation-Correlation by Layer

  • For five of six models, mean µ-alignment’s peak-correlation layer is within four layers of peak steerability, and for four models it is within two; SNR offsets are larger for Qwen.SNR shows a similar layer-wise pattern for Gemma and Llama models.Source: I Steerability vs.… · p. 10
Loading 2609.14151v1…