PaperlayerView exact revision on arXiv

Source-linked AI summary

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

Daniel Wurgaft, Can Rager, Matthew Kowal, Vasudev Shyam, Sheridan Feucht, Usha Bhalla, Tal Haklay, Eric Bigelow, Raphael Sarfati, Thomas McGrath, Owen Lewis, Jack Merullo, Noah Goodman, Thomas Fel, Atticus Geiger, Ekdeep Singh Lubana

arXiv:2605.05115v1cs.LG

TL;DR

The paper asks whether geometric structure in neural representations causally shapes behavior and develops geometry-aware interventions to test that link. It finds that steering along the activation manifold produces behavioral paths following the behavior manifold, while behavior-optimized paths recover activation-manifold curvature across tasks and modalities.

  • Problem

    The paper asks whether documented geometric structure in neural representations plays a causal role in computation and behavior.

  • Method

    The authors fit activation and behavior manifolds and compare linear, activation-manifold, and behavior-aware steering interventions across tasks and modalities.

  • Results

    Steering along the activation manifold produces behavioral trajectories following the behavior manifold, while optimizing behavior paths recovers activation trajectories tracing the activation manifold.

  • Takeaways & Limitations

    The results recast steering from finding the right direction to finding the right geometry of representation space.

  • Takeaways & Limitations

    Validation focused on simple concept domains with directly interpretable outputs, so application to abstract concepts such as refusals, sycophancy, and persuasion remains untested.

Abstract

from arXiv · show

Neural representations carry rich geometric structure; but does that structure causally shape behavior? To address this question, we intervene along paths through activation space defined by different geometries, and measure the behavioral trajectories they induce. In particular, we test whether interventions that respect the geometry of activation space will yield behaviors close to those the model exhibits naturally. Concretely, we first fit an activation manifold $M_h$ to representations and a behavior manifold $M_y$ to output probability distributions. We then test the link $M_h \leftrightarrow M_y$ via interventions: we find that steering along $M_h$, which we term manifold steering, yields behavioral trajectories that follow $M_y$, while linear steering -- which assumes a Euclidean geometry -- cuts through off-manifold regions and hence produces unnatural outputs. Moreover, optimizing interventions in activation space to produce paths along $M_y$ recovers activation trajectories that trace the curvature of $M_h$. We demonstrate this bidirectional relationship between the geometry of representation and behavior across tasks and modalities. In language models, we use reasoning tasks with cyclic and sequential geometries as well as in-context learning tasks with more complex graph geometries. In a video world model, we use a task with geometry corresponding to physical dynamics. Overall, our work shows that geometry in neural representation is not merely incidental, but is in fact the proper object for enabling principled control via intervention on internals. This recasts the core problem of steering from finding the right direction to finding the right geometry.

1 Introduction

The paper asks whether geometric structure in neural representations causally shapes behavior, challenging common Euclidean linear steering. It fits activation and behavior manifolds and tests whether geometry-aware interventions produce natural behavioral trajectories across tasks and modalities.

  • The paper investigates the causal role of geometric structure in neural representations by intervening under different geometric assumptions and measuring behavioral effects.
  • Linear steering assumes Euclidean activation geometry, but often produces degraded fluency, diversity collapse, and unstable off-target behavior.
  • The authors fit activation manifold M_h and behavior manifold M_y, finding that their geodesic distances align while neither matches Euclidean distances.
  • Manifold steering follows M_h and yields behavioral trajectories near M_y, whereas linear steering crosses low-density regions and produces unnatural intermediate distributions.
  • These findings are demonstrated across cyclic, sequential, and graph-structured language tasks and a video world model involving physical dynamics.

2 The Geometry of Representation and Behavior

The paper represents activations and outputs as manifolds over conceptual tasks, then compares their intrinsic structures. Across cyclic and sequential tasks, both spaces recover the task geometry and show strongly aligned on-manifold distances.

  • Setup: The setup uses activation space A = R^n and behavior space Y as a probability simplex over task concepts plus an other class.
  • Experimental tasks: The experiments cover cyclic weekdays and months tasks and sequential letters and ages tasks, using concept-centroid activations and output distributions.
  • Fitting the manifolds: Activation and behavior manifolds are fit to unintervened task representations and outputs using concept centroids and smooth cubic splines.
  • Conceptual structure: Both manifolds recover conceptual structure: weekdays and months form loops, while letters and ages form open sequential curves.
  • Scaled isometry: On-manifold activation and behavior distances correlate at r = 0.99 weekdays, r = 0.89 months, r = .999 letters, and r = .999 ages.
  • Scaled isometry: Linear activation distances correlate less well with behavior geodesics: r = 0.89 weekdays, r = 0.53 months, r = 0.71 letters, and r = 0.36 ages.

3 Connecting Representation and Behavior via Intervention

The paper tests whether activation geometry causally shapes behavior by comparing interventions that follow fitted manifolds with direct linear paths. Manifold steering produces natural behavioral trajectories, while behavior-guided pullbacks recover the activation manifold, supporting a bidirectional correspondence.

  • Interventions replace a chosen-layer activation with target activations along paths, inducing corresponding trajectories through output probability space.
  • Manifold steering interpolates in the activation manifold’s intrinsic coordinates, whereas linear steering interpolates directly in activation space.
  • Across structured reasoning tasks, manifold steering shifts probability smoothly through adjacent concepts, while linear steering jumps between non-adjacent concepts.
  • Manifold steering yields lower cumulative output energy than linear steering, with an average improvement of 2.8× and all statistical comparisons yielding p < 0.001.
  • Behavior-guided pullback paths follow the activation manifold more closely than linear paths and resemble manifold-steering trajectories.
  • Together, the two intervention directions support activation and behavior geometries as alternate views of the same underlying conceptual organization.

4 Manifold Steering Yields Factored Control in Multi-Dimensional Spaces

The study extends manifold steering from one-dimensional concepts to two-dimensional graph geometries learned in context. Activation and behavior manifolds align closely, enabling smooth, factored control that linear steering does not provide.

  • In-context learning tasks: The ICLR tasks assign arbitrary tokens to nodes in graph structures, allowing models to learn two-dimensional conceptual spaces from random-walk context.The experiments use a 5 × 5 grid and a 9 × 9 cylinder, with manifolds fitted to activations and output behaviors.
  • Manifold fitting: Thin-plate-spline fitting represents the grid activation space as a two-dimensional surface whose geometry is distorted by task statistics.Inner grid sites are visited more frequently than peripheral sites, producing a semi-spherical activation shape.
  • Isometry results: Geodesic activation and behavior distances correlate at r = .99 for both domains, compared with linear-path correlations of r = 0.90 for the 5 × 5 grid and r = 0.81 for the 9 × 9 cylinder.These results show that manifold distances preserve the shared graph structure more closely than linear activation distances.
  • Steering results: Manifold steering produces smooth transitions along one grid dimension while holding the other fixed, demonstrating factored control in the learned conceptual space.Linear steering instead produces unordered transitions and teleportation between endpoint locations.

5 Manifold Steering on a Visual World Model: Mountain Car Task

The Mountain Car experiment tests whether geometry-aware steering generalizes to a visual world model. Manifold paths preserve positional order and yield coherent movement, whereas linear paths leave the activation manifold and produce ambiguous intermediate states.

  • Environment and model: The recurrent world model predicts the next Mountain Car frame from the previous frame and action in a continuous position-and-velocity environment.The model uses RGB observations and three discrete actions: left, no-op, and right.
  • Manifold fitting: Encoder activations occupy a curved, low-dimensional manifold parameterized by position, while the behavior manifold represents output distributions over positions.Both manifolds are fitted as one-dimensional structures indexed by the conceptual coordinate p.
  • Steering results: Manifold steering yields smooth, coherent car movement between pA = −0.4 and pB = 0.4, while linear steering produces blurred intermediate frames and endpoint teleportation.Linear paths depart from Mh and broaden the position distribution, creating ambiguous car placement.
  • Geometry comparison: On-manifold activation and behavior distances recover a clean positional ordering, whereas linear activation distances produce a scrambled embedding.The arc-length correlation is r = 0.99, while linear activation distance versus behavior arc-length falls to r = 0.06.

6 Related Work

The related work positions manifold steering against linear representation assumptions and prior causal-intervention approaches. It emphasizes that respecting nonlinear representation geometry offers a broader account of how internal structure maps to behavior.

  • Activation steering: Linear activation steering is commonly motivated by the Linear Representation Hypothesis, which treats concepts as encoded along approximately orthogonal directions.The paper contrasts this assumption with curved, low-dimensional representation geometries.
  • Geometric intervention: Prior work demonstrates geometric manipulation in narrow domains, but lacks a general account of how representation geometry and output behavior map onto each other.The paper identifies days-of-the-week circles and number helices as examples of narrower demonstrations.
  • Positioning: The paper differs from nearby work by empirically relating activation and behavior manifolds across tasks and modalities rather than relying on a simple toy softmax mapping.The comparison includes contemporary work studying a toy representation-to-output distribution mapping.
  • Causal analysis: Representation probing and visualization alone can mislead causal explanations, motivating activation interventions grounded in causal mediation and causal abstraction.The related-work discussion frames intervention as a way to connect internal representations with model behavior causally.

7 Discussion

The discussion presents shared activation–behavior geometry as a causal bridge and reframes steering around geometry rather than direction. It also identifies open questions about the origin of this shared structure and proposes manifold coordinates as causal-analysis units.

  • Shared geometry: Activation and behavior manifolds exhibit aligned distances, and geodesic steering in either space recovers trajectories on the other.The paper presents this bidirectional relationship as evidence for shared structure tied to underlying conceptual geometry.
  • Implications for steering: The paper attributes linear-steering pathologies to a mismatch between flat assumed geometry and curved representation geometry.This reframes steering from finding the right direction to finding the right geometry.
  • Open question: The origins of shared geometry remain unstudied, although results are consistent with conceptual structure constraining both representation and behavior.The discussion notes that synthetic in-context tasks create novel geometries unlike those implied by ordinary training-data meanings.
  • Causal analysis units: Intrinsic coordinates on naturally occurring activation manifolds are proposed as causal-analysis units that constrain degenerate solutions.Fitting manifolds to natural activations supplies the proposed constraint.

8 Future Work and Limitations

The paper argues that geometry-aware interventions support smooth, coherent behavioral changes, while geometry-ignoring interventions can produce states without natural behavioral counterparts. It identifies validation on more complex concepts as a key direction for future work.

  • Geometry-aware interventions produce smooth and coherent behavioral changes.
  • Geometry-ignoring interventions risk producing activation states with no natural behavioral counterpart.
  • Future validation: Future work should test whether representation and behavior geometries extend to abstract concepts such as refusals, sycophancy, and persuasion.These concepts may express conceptual structure more subtly, and their appropriate representational primitives remain unclear.

A Experimental Details for Language Tasks

This appendix documents the procedures used for the experiments in Sections 2–4 and points to a later appendix for the mountain-car experiment.

  • The appendix describes procedures behind the experiments in Sections 2, 3, and 4.
  • Details for the mountain-car experiment in Section 5 are provided in Appendix B.
  • The experimental procedure is organized across the main language-task sections and a separate mountain-car appendix.

A.1 Tasks and Datasets

The language-task experiments use cyclic, sequential, and graph-structured concept domains, represent outputs as aggregated concept distributions, and fit smooth activation manifolds from concept centroids.

  • Tasks and datasets: The natural-domain tasks cover cyclic weekdays and months, plus sequential letters and ages.Cyclic results wrap around their modulus, while sequential examples outside the target set are filtered.
  • Tasks and datasets: The in-context tasks assign arbitrary noun tokens to graph nodes and use random-walk prompts on 5 × 5 grids and 9 × 9 cylinders.
  • Model and interventions: Activations are taken from Llama 3.1 8B at layer 28, with interventions applied to the last-token residual stream.The late layer was chosen so concept geometries would be fully computed.
  • Output representation: Output distributions aggregate probability across spelling variants for each concept and place remaining vocabulary mass in an “other” bin.
  • Activation manifolds: Activation manifolds are fitted in a 64-dimensional PCA subspace using concept centroids and smooth interpolants.Natural tasks use cubic splines, while graph tasks use thin-plate splines.
  • Behavior manifold: Behavior centroids are mapped into Hellinger coordinates on the unit sphere before fitting the behavior manifold.The spline is fit in a tangent plane and lifted back to the sphere to preserve valid probability distributions.

A.7 Naturalness Metric

Naturalness is measured by distance from induced output trajectories to the behavior manifold, while pullback recovery optimizes activation paths to match behavioral geodesics.

  • Naturalness metric: Cumulative output energy sums Bhattacharyya distances from induced distributions to their nearest points on the behavior manifold.The distance is chosen because it is closely related to Hellinger geometry, in which the behavior manifold is fitted.
  • Pullback recovery: Pullback recovery first specifies a geodesic target on the behavior manifold, then optimizes an activation path whose interventions match that target.
  • Pullback recovery: The pullback path is a natural cubic spline with ten optimized control vectors restricted to the first 32 PCA components.The remaining PCA components and orthogonal residual are held at base-prompt values.
  • Optimization and comparison: Pullback optimization minimizes averaged squared Hellinger distance using L-BFGS with strong-Wolfe line search.The comparison reports mean ± standard error across per-pair scalars and paired t-tests against the linear baseline.
  • Steering procedures: Manifold steering follows activation-manifold geodesics, whereas the linear baseline uses a straight chord in the PCA subspace.

B Experimental Details for the Vision Task

The vision-task experiments fit activation and behavior manifolds from MountainCar representations and outputs, then test whether manifold geometry supports structured steering. Across increasingly complex domains, manifold steering preserves ordered or factored behavior while revealing correspondence between activation and behavior geometry.

  • In-context learning: The 9 × 9 cylinder extends factored control beyond the grid domain despite its periodic dimension and substantially larger graph structure.With 2048 tokens of context, Llama 3.1 8B achieved above 80% neighborhood accuracy.
  • In-context learning: The cylinder results show coherent steering of independent dimensions, whereas linear steering produces teleportation behavior.This pattern recurs across both grid and cylinder domains.
  • Geometric correspondence: The 5 × 5 grid and 9 × 9 cylinder show manifold-steering paths closely following the behavior manifold, unlike Euclidean paths.MDS embeddings also show that activation-manifold distances structurally match behavior-manifold distances.
  • Conceptual uncertainty: Manifold steering along addition-value circles preserves ordered weekday transitions while increasing entropy across groups.The concentric circles form a cylinder-like structure, providing a handle for controlled confidence calibration.

C.3 Mountain Car

The MountainCar experiments test whether activation and behavior geometries correspond in both directions. Geodesic correspondence is strong, and optimizing paths toward behavior-manifold trajectories recovers activation paths close to the activation manifold.

  • Structural correspondence: Activation- and behavior-manifold distances correlate at r = 0.996 across 1225 pairs, while activation chord and behavior arc distances correlate at r = 0.06.The chord baseline cuts across the encoder loop rather than following the encoded conceptual geometry.
  • Pullback steering: Pullback optimization recovers paths that closely trace the activation manifold across endpoint pairs.The remaining degradation is concentrated near the extreme wall position, where encoder curvature is tighter; most other paths are essentially indistinguishable from the activation manifold.
  • Pullback steering: Pullback paths have mean distance 0.29 from the activation manifold, compared with 2.22 for linear chords and 0.20 for geometric paths.The pullback path recovers 95.4% of the chord-to-geometric improvement and beats the chord baseline on 30/30 endpoint pairs.
  • Pullback steering: As the conformal penalty increases, behavior targets move toward the behavior manifold and recovered activation paths correspondingly track the activation manifold.At α = 0, the recovered path instead leaves the activation manifold to match an off-manifold Hellinger target.
Loading 2605.05115v1…