PaperlayerView exact revision on arXiv

Source-linked AI summary

Capacity, Responsiveness and Alignment: What Makes a Latent Structure Actionable

Or Shafran, Mor Geva

arXiv:2610.06897v1cs.CL

TL;DR

Localized latent structures do not always exert reliable causal influence, so the paper asks what makes them actionable. It decomposes that influence into capacity, responsiveness, and alignment, then uses high-capacity subspaces to constrain probes. The factors impose distinct constraints, while causal probes improve steering with a small reduction in concept detection.

  • Problem

    Localized structures can reflect correlations or produce weak, inconsistent, or unintended effects, so localization alone is insufficient for causal control.

    Source: §1 · p. 1

  • Method

    The paper decomposes causal influence into capacity, responsiveness, and alignment, then constrains learned probe directions to high-capacity subspaces.

    Source: §1 · p. 2

  • Results

    17%–118% improvement in median steering performance across models, with only a 3% reduction in concept-detection accuracy, followed restriction of probes to high-capacity subspaces.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Causal influence is not intrinsic to a latent structure but depends on its relationship with context and the remaining computation.

    Source: §1 · p. 2

    The alignment analysis uses neutral contexts because random directions can align with the concept in concept-positive contexts.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Localizing latent structures in the activation space of language models (LMs) is central to understanding and controlling their behavior. Yet, localized structures can differ substantially in their causal influence, raising the question of what makes a structure actionable. We tackle this question by casting causal influence as a product of three factors and showing empirically that they act as interpretable, distinct constraints: capacity, measuring the sensitivity of the model's output to movement along the structure, responsiveness, capturing how promotable the concept is given the current context, and alignment, reflecting how well the structure aligns with the context-specific representation of the concept. Across 4 LM families and 50 concepts, we observe that causal effectiveness requires all factors to be high; low capacity and responsiveness reduce it by 84% and 95%, respectively, while low alignment can reverse it, suppressing concept expression. Moreover, we find that causality is context-dependent rather than an intrinsic property of the structure, with causally effective directions forming a low-dimensional subspace that varies across contexts. By restricting the training of linear probes to this subspace, we introduce causal probes that achieve 17%-118% improvement in steering across models, with only 3% reduction in concept detection.

1 INTRODUCTION

The paper asks why localized latent structures can fail to control model outputs and decomposes their causal influence into capacity, responsiveness, and alignment. It finds that all three constrain promotion, and uses capacity to improve steering with little loss in concept detection.

1 INTRODUCTION

  • Low capacity and responsiveness reduce concept expression by 84% and 95%, respectively; low alignment can reverse the effect and suppress the concept.When all three factors are low, causal promotion is effectively zero.Source: §1 · p. 2
  • Capacity measures whether movement along a structure propagates through downstream computation, responsiveness measures concept promotability in context, and alignment measures whether the movement is correctly oriented.The top 2.5% of tokens under the current output distribution receive 94.5% of intervention-induced probability increase.Source: §1 · p. 2
  • 17%–118% improvement in median steering performance across models, with only a 3% reduction in concept-detection accuracy, came from restricting probes to high-capacity subspaces.The paper calls these constrained classifiers causal probes.Source: §1 · p. 2
  • A structure’s causal influence is not intrinsic; it arises from its relationship with context and the remaining computation.The three factors jointly describe how much concept expression can be promoted from the current state.Source: §1 · p. 2

2 DISENTANGLING CAUSAL INFLUENCE

The paper measures a concept at the output as probability mass on tokens that express it, then pulls that measure back to intermediate activations. It decomposes the first-order effect of movement along a latent structure into three factors and extends the analysis across output positions.

2 DISENTANGLING CAUSAL INFLUENCE

  • Concept expression is measured as probability mass assigned to tokens that verbalize the concept, providing an output-level target for causal influence.The analysis pulls this quantity back to intermediate activations.Source: §2 · p. 2
  • An intervention changes the output distribution, and its effect on concept expression is measured by whether that change promotes or suppresses the concept.The concept-token vector acts as an output-space covector on the distribution change.Source: §2 · pp. 2–3
  • The first-order change from movement along a structure is expressed as three factors that characterize distinct constraints on causal influence.The paper uses a covariance–correlation identity to cast the effect in this form.Source: §2 · p. 3
  • For sequence-level causality, the analysis extends the effect over T output positions so latent structures can influence later predictions as well as the next token.The sequence-level covector sums the position-wise pullbacks, capturing both per-position effects and their accumulation during generation.Source: §2 · p. 3

3 CAUSALITY IS MEDIATED BY INTERPRETABLE FACTORS

The section decomposes causal influence into capacity, responsiveness, and alignment, then tests how each constrains concept promotion. Across the experiments, output effects are concentrated in context-dependent directions, while context limits which concepts and directions can be promoted.

3.1 EXPERIMENTAL SETUP

  • The experiments use 50 concepts, positive and neutral datasets built from 304 open-ended prefixes, and four language models evaluated at five depths.Concept directions are constructed with DiffMeans or linear probes from positive and neutral examples.Source: §3.1 · pp. 3–4

3.2 CAPACITY: DOWNSTREAM SENSITIVITY IS CONCENTRATED IN A CONTEXT-DEPENDENT SUBSPACE

  • Predicted intervention-induced KL closely tracks observed KL across tested intervention strengths, and directions outside the high-capacity subspace produce little causal effect.The comparison used 64 isotropic unit directions across neutral contexts and 14 intervention strengths from 0.3 to 128.Source: §3.2 · p. 4, Figure 1 · p. 4
  • Capacity is concentrated in a low-rank, context-dependent subspace, with median effective dimensionality of 24.8 directions for Gemma, 47.2 for Llama, 30.1 for Mistral, and 23.3 for Qwen.These dimensions correspond to 0.7%–1.2% of residual-stream dimensions; dimensionality rises from 8–13 directions in early layers to 37–97 with depth.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
  • DiffMeans directions have 90.4% more capacity than random directions and 26.4% more than linear probes, while probes have 37.6% more than random directions.This comparison shows that localization methods differ in how much their directions overlap with downstream-sensitive activation space.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3 RESPONSIVENESS: CONTEXT LIMITS CONCEPT PROMOTION

  • At fluent settings, the top 2.5% of baseline tokens receive approximately 94.5% of intervention-induced probability increases, limiting promotion of weakly supported outputs.Substantial movement outside baseline support appears only at stronger interventions as fluency degrades; the restriction is similar for DiffMeans and random directions.Source: §3.3 · p. 6, Figure 3 · p. 6

3.4 ALIGNMENT: EFFECTIVE DIRECTIONS ARE CONTEXT-DEPENDENT

  • Locally effective directions for the same concept are nearly orthogonal across contexts in early layers and remain below 0.2 cosine similarity for most models.Transferred directions have below 0.1 normalized alignment in early layers and reach at most 0.4 in later layers, below the context-local reference.Source: §3.4 · p. 7, Figure 4 · p. 7

4 FACTORS EXPOSE DISTINCT CAUSAL BOTTLENECKS

Causal impact varies with intervention position: capacity alone does not predict promotion, because alignment can outweigh even greater capacity. Across measured factor combinations, each factor acts as a distinct bottleneck.

Example

  • Despite more than 3× the capacity at the final position, the earlier “green” token’s alignment (0.88 vs. 0.14) yields a 3.2× larger first-order effect.The intervention at “green” is seven positions earlier than the target.Source: §4 · p. 7

Results

  • 84% and 95%: low capacity and low responsiveness reduce concept expression, while low alignment can reverse the effect and all-low promotion is effectively zero.Figure 5 summarizes the eight high/low factor combinations.Source: Figure 5 · p. 8, §4 · p. 8

Experiment

  • The analysis measures factors for 50 concepts across 16,000 C4 contexts of 256 tokens, intervening at eight source positions and grouping samples by factor quartiles.The three high/low labels define eight joint factor settings for comparing intervention effects.Source: §4 · p. 8

5 ACCOUNTING FOR CAUSAL CONSTRAINTS IMPROVES STEERING

The authors constrain linear probes to high-capacity subspaces to improve steering while largely preserving concept detection. Causal probes outperform raw probes across four models and often perform similarly to DiffMeans.

Causal probes

  • Causal probes constrain each concept- and layer-specific linear probe to a high-capacity subspace, projecting it back into that subspace after each gradient step.The subspace is defined by the leading eigenvectors of averaged downstream-capacity matrices.Source: §5 · p. 8

Evaluation

  • Steering is evaluated from 16 fixed neutral prefixes per setting, sweeping 14 intervention strengths and generating up to 50 tokens; scores use a harmonic mean of concept expression and fluency.The evaluation compares causal probes with standard probes, DiffMeans, and ReFT-r1 across 50 concepts.Source: §5 · pp. 8–9

Results

  • 118% (Gemma), 57% (Qwen), 23% (Mistral), and 17% (Llama): median steering improvements over raw probes; concept detection is 90.4% vs. 93.8%.Figure 6 shows causal probes often performing similarly to DiffMeans.Source: Figure 6 · p. 9, §5 · p. 9

6 RELATED WORK

Prior work distinguishes information represented in activations from information that causally affects model behavior. This paper situates its contribution as quantifying that gap and explaining context-dependent causal influence.

The detection–causality gap

  • Probing accuracy may reflect probe capacity or encoded properties irrelevant to the model’s current task, and accurate detection need not predict behavioral importance.Later work reports that detected structures can yield weak or unreliable steering effects (Galeone et al., 2026).Source: §6 · p. 9

Jacobians and output space geometry

  • Output-space geometry has been used to derive geometry-aware probing and steering or nonlinear steering paths (Wang et al., 2026).The present work instead asks why a given structure’s causal influence varies across contexts.Source: §6 · p. 9
  • Some Jacobian methods aggregate effects across contexts to identify directions with consistent downstream effects (Gurnee et al., 2026); this work uses local Jacobians to characterize context-varying efficacy.Source: §6 · p. 9

7 CONCLUSION

The paper models a latent structure’s causal influence as the product of capacity, responsiveness, and alignment. Its findings characterize causal effectiveness as context-dependent and show that restricting probes to high-capacity subspaces improves steering while largely preserving detection.

7 CONCLUSION

  • Capacity, responsiveness, and alignment jointly describe how a structure, the current context, and remaining computation determine whether concept expression can be promoted.The factors are derived by measuring output concept expression and pulling it back to intermediate activations.Source: §7 · pp. 9–10
  • Causal effectiveness is not intrinsic to a latent structure: across four models, causal probes improve median steering by 29% with a 3% drop in concept-detection accuracy.The conclusion frames effective localization and control as requiring attention to when and how representations can be causally utilized.Source: §7 · p. 10

A EXPERIMENTAL DETAILS

The appendix details how the study constructs concept datasets, covectors, and latent directions, then estimates capacity through derivatives of model outputs.

A.1 DATASET CONSTRUCTION AND PROMPTS

  • 50 concepts were paired with 304 concept-independent prefixes to generate positive and neutral completions, split by prefix into 243/31/30 training, validation, and test prefixes.Positive prompts requested clear concept expression, while neutral prompts requested an ordinary continuation.Source: §A.1 · pp. 26–27

A.2 CONCEPT COVECTOR CONSTRUCTION

  • Concept covectors are binary token indicators: 1 for tokens seen in positive but not neutral completions, and 0 otherwise, capturing the verbalized aspect of each concept.Source: §A.2 · p. 27

A.3 LATENT STRUCTURE CONSTRUCTION

  • For each concept, model, and layer, directions were trained on residual-stream activations; DiffMeans uses the difference between positive and neutral activation means.Linear probes were bias-free logistic classifiers, and constrained variants restricted directions to the rank-128 causal subspace.Source: §A.3 · pp. 27–28

A.4 JACOBIAN AND CAPACITY COMPUTATION

  • Capacity is the probability-weighted standard deviation of logit changes, computed with Jacobian-vector and vector-Jacobian products rather than explicitly constructing the full Jacobian.For future positions, intervening-to-output derivatives use the observed continuation under teacher forcing.Source: §A.4 · p. 28

B ADDITIONAL CAPACITY ANALYSIS

This appendix derives and evaluates the local KL approximation used for capacity, examines its behavior across future positions, and records the fluency-evaluation setup.

B.1 DERIVATION OF THE LOCAL KL APPROXIMATION

  • For sufficiently small interventions, KL divergence is approximated by a second-order expansion involving the softmax Fisher matrix.Source: §B.1 · p. 28

B.2 CAPACITY PREDICTION ACROSS FUTURE POSITIONS

  • Predicted KL tracks actual KL closely across offsets through t = 16, with approximation error generally decreasing at later offsets except for a slight increase in Qwen.Source: §B.2 · p. 28, Figure 7 · p. 29

B.3 CHOICE OF SEQUENCE HORIZON

  • Actual intervention-induced KL decreases significantly at future offsets across tested models, concepts, and layers, motivating analysis through up to eight additional offsets.Source: §B.3 · p. 29

C.1 EXPERIMENTAL AND FLUENCY-EVALUATION DETAILS

  • Fluency boundaries use the largest tested intervention strength whose pooled mean fluency meets a target threshold of 0.8, 0.7, or 0.6.Source: §C.1 · p. 29

C.2 RESULTS ACROSS LAYERS AND SUPPORT THRESHOLDS

These analyses examine support promotion, factor-bucket prevalence, and whether the three-factor pattern persists across offsets.

C.2 RESULTS ACROSS LAYERS AND SUPPORT THRESHOLDS

  • In the fluent regime, interventions primarily promote context-supported tokens; when a concept is unresponsive in context, even a well-localized structure may not produce the intended fluent effect.Supportflow results cover models, layers, intervention directions, and support sizes.Source: §C.2 · pp. 29–30, Figure 9 · p. 31

D.1 LOCALLY OPTIMAL DIRECTION ESTIMATION

  • The locally optimal direction is estimated per concept and context by approximately solving a ridge-stabilized system with conjugate gradient, then scaling the direction to unit capacity.The computation uses up to 32 iterations and eight held-out neutral texts per concept.Source: §D.1 · p. 30

E.1 FULL FACTOR-BUCKET STATISTICS

  • Approximately 26%–31% of retained samples have exactly one low factor, while the combination of low responsiveness and high capacity is uncommon.Source: §E.1 · p. 30, Figure 10 · p. 34

E.2 ALL MODEL FACTOR-BUCKET PLOTS

  • Across offsets 0 through −7, promotion remains strongest when all three factors are high; low capacity or responsiveness limits it, while low alignment can reverse it.Source: §E.2 · p. 34

E.3 STEERING ON EFFECTIVE POSITIONS

This section tests whether the three-factor score can identify effective steering positions during longer generations. Selecting high-scoring positions improves steering over random selection in most tested settings.

E.3 STEERING ON EFFECTIVE POSITIONS

  • The position score is the signed first-order effect of steering at a preceding position on concept expression at the current next-token prediction, corresponding to the product of the three factors.Source: §E.3 · p. 34

F ADDITIONAL STEERING DETAILS AND RESULTS

The appendix details how causal probes are trained and evaluated, then shows that their steering gains primarily come from stronger concept expression rather than fluency changes.

F.1 CAUSAL PROBE TRAINING DETAILS

  • The first r eigenvectors of averaged capacity matrices define rank-r subspaces, and bias-free logistic probes are trained with gradients and weights constrained to each subspace.The matrices are applied using automatic differentiation, and leading eigenvectors are estimated with randomized subspace iteration.Source: §F.1 · p. 36

F.2 STEERING EVALUATION AND JUDGE PROMPTS

  • Steering evaluation uses 50 generated tokens from 16 fixed neutral prefixes, with intervention strength selected on four folds and scored on the held-out fold.The final score is the harmonic mean of normalized concept-expression and fluency scores, set to zero if either is zero; judges score each from 0 to 2.Source: §F.2 · pp. 36–37

F.3 CONCEPT AND FLUENCY SCORES

  • Causal probes substantially improve concept scores over unconstrained probes, with most overall gains attributable to concept expression rather than fluency.Higher-rank high-capacity subspaces continue to benefit as probe rank increases; fluency shows no similarly strong trend.Source: Figure 12 · p. 37, §F.3 · p. 38
Loading 2610.06897v1…