PaperlayerView exact revision on arXiv

Source-linked AI summary

A Phenomenology of Neural Geometry in Vision-Language-Action Models

Mozes Jacobs, Aiden Swann, Mathilde Papillon, Siddharth Boppana, Fenil R. Doshi, Usha Bhalla, Thomas Icard, Matt Feiszli, Thomas McGrath, Vasudev Shyam, Demba Ba, Leon Bergen, Nina Miolane, Owen Lewis, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger, Matthew Kowal, Thomas Fel

arXiv:2610.20840v1cs.ROcs.LG

TL;DR

Prior VLA interpretability work can establish that concepts are readable without specifying their activation-space geometry. This study maps seven control concepts with sweeps, rollouts, probes, and interventions, finding mostly multidimensional structures that recur across two models on real-robot data.

  • Problem

    Probes establish whether variables are decodable, but do not describe their activation-space structure; SAE representations assume a single direction for each concept.

    Source: §1 · p. 2

  • Method

    The paper uses controlled sweeps, natural rollouts, linear probes, and activation interventions to map seven control concepts in MolmoBot.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    The seven-concept taxonomy finds predominantly multidimensional geometry shaped by policy constraints, recurring in π0.5 and MolmoAct2 on real-world data.

    Source: §1 · p. 3

  • Takeaways & Limitations

    VLA interpretability should match both the geometry a concept realizes and its role in control, since readable information may not be used as intended by the policy.

    Source: §5 · p. 10

    Recoverability does not establish that activations are isomorphic to the concept domain or that the policy causally uses the recovered information.

    Source: §2 · p. 3

Abstract

from arXiv · show

Robotic manipulation is inherently geometric, but what shape does the world take inside the policy that performs it? Prior work on vision-language-action (VLA) policies establishes that task-relevant variables are linearly readable from activations, yet readability certifies only that a variable is present and says nothing about the shape its values trace, which may be curved and multi-dimensional. Using controlled sweeps, natural rollouts, linear probes, and activation interventions, we build a taxonomy of seven task-relevant concepts in VLA models. We find that realized geometry is predominantly multi-dimensional and is shaped by the constraints under which the policy acts. Physical joint limits, for example, leave wrist orientation an open arc rather than a circle, and distance to the target, a single scalar, is realized in two orthogonal subspaces, a coarse ruler for the approach and a fine ruler for the grasp. The same geometries reappear in two further VLA models on real-world robot data, indicating that they are neither model-specific nor artifacts of simulation. We demonstrate how the findings are actionable by steering along the geometry a concept realizes. Overall, this work lays out an initial phenomenology of realized geometry in VLA policies and demonstrates that a concept's shape carries as much importance as its readability.

1 Introduction

Robotic manipulation and its task variables have geometric structure, while prior VLA interpretability work has largely asked whether variables are readable rather than how they are arranged in activations. This paper introduces realized geometry and examines seven control concepts, finding that their shapes reflect policy constraints and recur across models.

  • VLAs inherit visual and semantic priors, and manipulation variables such as pose, distance, contact, and task phase have geometric structure of their own.The paper motivates studying whether those structures appear in VLA representations.Source: §1 · p. 2
  • Prior work studies VLA representations using probes and sparse autoencoders, with discovered features also used to steer policies.The cited steering work is Joseph et al. (2025).Source: §1 · p. 2
  • Probes establish whether a variable is decodable, while SAEs inherit a single-direction assumption that fixes concept structure in advance.The paper identifies this as a shared limitation of the two approaches.Source: §1 · p. 2
  • The paper formalizes realized geometry as distinct from concept realization: readability establishes presence, while geometry describes how concept values are structured and manipulated.It studies seven control concepts using controlled sweeps, natural rollouts, linear probes, and activation interventions.Source: §1 · p. 2
  • The seven-concept taxonomy finds predominantly multidimensional geometry shaped by policy constraints, recurring in π0.5 and MolmoAct2 on real-world data.Joint limits leave wrist orientation an open arc, while distance is represented by orthogonal meter- and centimeter-scale rulers.Source: §1 · p. 3

2 Realization of Control Geometry

This section distinguishes whether a concept is recoverable from activations from the geometry traced by its values. It formalizes concepts as structured domains and defines recovery relative to a readout class and error tolerance.

  • Concept realization asks whether a concept can be recovered from activations; realized geometry asks what shape the recovered concept takes in activation space.The section treats recoverability and structure as separate questions.Source: §2 · p. 3
  • The framework models each concept as a structured domain whose values may be discrete, continuous, or manifold-valued, with observations mapped to activations.Examples include binary gripper state, scalar distance, 3-D end-effector position, and wrist orientation on S1.Source: §2 · p. 3
  • Realization is defined relative to a readout class and error tolerance; under affine readouts, the paper calls a concept linearly readable.The definition specifies both whether information is present and how easily it can be extracted.Source: §2 · p. 3
  • Recoverability does not imply that activations are isomorphic to the concept domain or that the policy causally uses the recovered information.Realization also need not preserve distances or topology, or provide an efficient or minimal representation.Source: §2 · p. 3
  • Figure 2 describes separate approximately Euclidean 3-D position subspaces, while Figure 3 describes nearly orthogonal near- and far-range distance codes.Source: Figure 2 · p. 4, Figure 3 · p. 5
  • Realized geometry is the activation path traced as one concept varies in a fixed context, together with the map that produces it.A linear probe can leave the path’s shape in other activation directions unobserved.Source: §2 · p. 4
  • The paper estimates these maps with controlled sweeps when possible and with teleports or natural rollouts otherwise, then tests candidate geometry using interventions.Source: §2 · p. 4

3 A Taxonomy of Realized Geometry in MolmoBot

Across seven task variables, MolmoBot’s activations trace geometries shaped by the task and physical constraints. Controlled sweeps, rollouts, probes, and interventions reveal both how variables are represented and how edits affect behavior.

3.1 Spatial Geometry

  • Three dimensions preserve end-effector and target positions, while distance uses nearly orthogonal coarse and fine rulers for approach and grasp.Position probes reach held-out R2 values of 0.92 for end-effector position and 0.75 for target position; distance codes remain distinct after erasing the other range’s directions.Source: §3.1 · p. 6

3.2 Task Structure

  • A contact direction decodes target contact at 91.3% accuracy and remains informative when early touches or induced slips disrupt its usual timing.These off-schedule interventions support a contact code rather than a proxy for task timing.Source: §3.2 · pp. 6–7, Figure 4 · p. 6
  • The action expert separates reach, grasp, transport, and place, but freezing the arm during reach shifts decoded phase toward grasp.Freezing also raises closed-gripper commands during reach by 43 percentage points and lowers them during place by 68 points.Source: §3.2 · pp. 7–8, Figure 5 · p. 7

3.3 Action Geometry

  • Wrist orientation traces an open arc because the roll joint is limited, and edits along that geometry produce compensatory rotations.A single edit rotates the decoded action against each written degree by 0.33 ± 0.08; repeated edits compound the correction.Source: §3.3 · p. 8, Figure 6 · p. 8
  • The recovered geometries largely reappear in π0.5 and MolmoAct2 on real-world DROID data, with strongest cross-model similarity for the gripper and weakest for wrist orientation.All reported similarities exceed a shuffled null.Source: Figure 8 · p. 9
  • Gripper steering is largely one-dimensional, but steering along its leading component also moves arm joints.The first principal component captures about 70% of steering-vector variance; additional components reduce the leakage.Source: Figure 7 · p. 8, §3.3 · p. 10

4 Representations in Real Robots

The authors repeat analyses on real-world DROID episodes for MolmoBot, π0.5, and MolmoAct2, focusing on four measurable variables. All four are recovered in every model, and the two-scale distance structure recurs across them.

  • Real-data analyses use gripper aperture, wrist orientation, EE position, and distance to the eventual grasp location as a proxy for distance to target.Real data lacks simulation’s privileged information, including ground-truth object poses and mid-rollout interventions.Source: §4 · p. 10
  • All four variables are recovered in every model, with cross-model Procrustes similarities above a shuffled-variable null.Source: §4 · p. 10
  • EE position is linearly decodable in all three models (R2 of 0.78–0.92), while wrist orientation is less well decoded (R2 of 0.52–0.69) and agrees least across models.Source: §4 · p. 10
  • The two-scale distance code recurs in all three models: far points spread along the meter ruler while near points split onto the centimeter ruler.This structure is sharper in π0.5 and MolmoAct2 than in MolmoBot.Source: §4 · p. 10

5 Conclusion

The paper’s conclusion is that manipulation variables in VLAs generally realize structured, often multidimensional geometries, with similar structures recurring across models. It also argues that readability alone does not establish how a concept can be steered.

5 Conclusion

  • In MolmoBot, position occupies a near-Euclidean 3-D subspace, wrist orientation an open arc, and distance two nearly orthogonal rulers; these structures largely recur in π0.5 and MolmoAct2 on real data.Source: §5 · p. 10
  • A variable can be linearly readable yet require a geometry-matched edit to steer, and even then the edit may land on a belief the policy compensates against rather than an executed action.Source: §5 · p. 10

A Realization does not determine Geometric Realization.

B Extended Results

The extended results detail the seven task variables and supporting analyses, including position, distance, direction, contact, phase, wrist orientation, and gripper control. Additional experiments reproduce several structures in real-robot data.

  • Distance uses two rulers that score on their own ranges; a single full-range probe is most accurate with a compressive link below 15 cm and a linear link in the farthest bins.The action expert also reproduces the VLM’s two-subspace code.Source: B Extended Results · p. 17, Figure 12 · p. 21, Figure 13 · p. 21
  • Unlike distance, target direction is read by a single probe across the workspace; range-splitting does not improve near reads and improves far reads by at most 1.1°.Source: B Extended Results · p. 17, Figure 14 · p. 22
  • Contact is decodable from wrist-camera patches at the first layer, while exo-camera patches and text tokens reach a similar level only in middle layers; the signal spans the image.Source: B Extended Results · p. 17, Figure 16 · p. 23
  • The four task phases occupy separate regions in temporal order, and freezing shifts decoded phase toward grasp across the action expert and both VLM token groups.Place landmarks are most often still decoded as place.Source: B Extended Results · p. 17, Figure 17 · p. 24, Figure 18 · p. 24
  • The action expert’s wrist-orientation trace remains an open arc over the 228° reachable precession, while the VLM carries a weaker image-derived trace.Full 3-DOF hand orientation is decodable at low rank from natural rollouts, with decoding peaking at layer 9.Source: B Extended Results · p. 18, Figure 21 · p. 26, Figure 23 · p. 28
  • Action-expert gripper steering vectors are dominated by one component aligned with the close–open flip axis, while VLM flip rates fall to the control-edit floor by layer 28 despite continued readability.At layer 9, one component decodes and flips the gripper, though the steering vectors do not point from one gripper-state cluster toward the other.Source: B Extended Results · p. 18, Figure 26 · p. 30, Figure 28 · p. 31
  • On real DROID episodes, MolmoAct2 and π0.5 show gripper, EE-position, and wrist-orientation structures, two distance rulers, and decodable task phases.A single steering-vector component flips the gripper in both models, carrying 59% of π0.5’s steering-vector variance and 31% of MolmoAct2’s.Source: B Extended Results · p. 18
  • The action expert has a near-exact rank-3 EE-position subspace, while its rank-3 target-position paths are more distorted and decoded with larger error than in the VLM.Within a scene, EE position is decodable from the first layer, while target position appears on instruction and final tokens at layers 12–21 and in the action expert from layer 18.Source: Figure 9 · p. 19, Figure 10 · p. 20, Figure 11 · p. 20

C.1 MolmoBot Architecture

MolmoBot uses a vision-language model (VLM) and an action expert as two interacting streams. The action expert combines proprioception and noisy action tokens with visual and language information read from the VLM.

  • The VLM receives language instructions and images from exocentric and wrist cameras, but no proprioceptive input.After the first query, it receives the current and an eight-step-earlier frame from each camera; the first query sees only current frames.Source: §C.1 · p. 37
  • The action expert takes proprioceptive state and noisy action tokens, then denoises them into a 10-step action chunk using flow matching.Source: §C.1 · p. 37
  • Each action-expert layer cross-attends to the same-depth VLM layer, so visual and language information can enter action computation at every level; information flows only from the VLM to the action expert.Source: §C.1 · p. 37

C.2 Data

The study uses four datasets from simulated pick and pick-and-place tasks to examine the selected concepts. Probes are cross-validated with houses held out.

C.2 Data

  • Simulation experiments use MolmoSpaces on ProcTHOR houses; all use the pick task except phase, which uses pick-and-place.Every probe is 5-fold cross-validated with houses held out.Source: §C.2 · p. 37
  • Four datasets cover natural pick rollouts, object teleports, crossed arm–object teleports, and pick-and-place rollouts.The datasets support analyses of wrist orientation, gripper pullback, contact, distance, direction, position, and task phase.Source: §C.2 · p. 37
  • Position, distance, direction, and contact are read from the VLM residual stream, while phase, wrist orientation, and gripper are read from the action expert.Action-expert activations are generally averaged over 16 action tokens at the final flow step.Source: §C.2 · p. 37

C.3 End-Effector and Target Object Position

The position analyses pair arm and object teleports with held-out-house probes and tests of spatial equivariance and path curvature. In the action expert, end-effector position is highly decodable, but its subspace is stretched rather than a rigid copy of world coordinates.

C.3 End-Effector and Target Object Position

  • Arm and object positions are paired across teleports, with ridge probes fit to simulator coordinates using the best token group and layer for each variable.After removing scene means, within-scene R2 is 0.92 for end-effector position and 0.57 for target position; each variable’s true x and y predict the other’s at R2 ≤0.02.Source: §C.3 · pp. 37–38
  • The position subspaces are tested for preserving world displacement through held-out-scene linear-map and rotation-with-uniform-scaling predictions.The linear maps are fit within rank-3 subspaces; shuffled displacement pairings provide an error reference of 1.00.Source: §C.3 · p. 38
  • Curvature is measured along straight hand and object paths by the median angle between consecutive projected segments, compared with random 3-D subspaces.Hand paths span forward, sideways, upward, and toward-object directions; curvature is binned by hand–object distance.Source: §C.3 · p. 38
  • R2 = 0.998 and 1.1 cm error decode action-expert end-effector position at layer 12, while displacement errors distinguish general linear mapping (0.04) from rigid Procrustes mapping (0.29).The subspace is therefore a stretched rather than rigid copy of world coordinates.Source: §C.3 · p. 38

C.4 Distance to Target Object

Distance is analyzed from VLM activations across near and far object-teleport states. The methods test whether its representation separates into distinct near- and far-range directions, and use controls to distinguish the near-range direction from position and contact signals.

C.4 Distance to Target Object

  • The distance analysis uses 67,778 states with the object within 70 cm of the gripper, including 11,534 states below 15 cm.Distance is the simulator-measured straight-line gripper–object distance; the VLM final token at layer 21 is analyzed, with the action expert’s last layer giving the same main results.Source: §C.4 · p. 38
  • Iterative erasure fits orthogonal distance-probe directions on near (<15 cm) and far (15–70 cm) ranges, then measures held-out R2 after each removal.Each fold fits a probe on one range, removes its direction, and refits probes on the cleaned activations.Source: §C.4 · p. 38
  • Meter and centimeter rulers are fit with values clipped at 15 cm, scored within their respective ranges, and compared across cutoffs against a single rescaled probe.Probe-direction cosines are compared with within-probe split-half cosines to judge whether the two probes differ in direction.Source: §C.4 · p. 38
  • The near-range axis is not a disguised end-effector-position code or contact detector under the reported controls.End-effector position predicts near-state distance at R2 ≈0, less than 3% of the near axis falls in position subspaces, and centimeter-ruler errors are 1.46 vs. 1.45 cm across the stated contact conditions.Source: §C.4 · p. 38

C.5 Direction to Target Object

Direction is decoded as the unit bearing from the end effector to the object using the distance analysis data and layer. The method tests whether that bearing remains decodable beyond information in positions and distance, and evaluates angular error against two baselines.

C.5 Direction to Target Object

  • Bearing is decoded by ridge regression on its three components and evaluated by angular error against a 33.1° training-mean baseline and a 19.0° scene-mean oracle.The residual after regressing out end-effector position, object position, and distance remains decodable, with little probe energy in their subspaces.Source: §C.5 · p. 39

C.6 Contact

The contact probe is trained on natural rollout activations and tested on held-out houses. Off-schedule object teleports test whether contact remains detectable when it changes outside its natural timing.

C.6 Contact

  • 91.3% held-out accuracy (AUC 0.954) for the contact LDA, against a 65% majority baseline.Using only the final token gives the same accuracy (AUC 0.957).Source: §C.6 · p. 39
  • Off-schedule teleports test early touch and induced slip while the arm is held fixed, with the gripper forced to the opposite state.The object moves in 1 cm steps; each step is re-rendered from the current arm pose.Source: §C.6 · p. 39
  • Forced-open slips score contact higher at most poses (75%), but 24 poses are too few to compare with natural slips.Opening the gripper at a held grasp usually releases the object, leaving few true slips.Source: §C.6 · p. 39

C.7 Phase

The supplied passages describe methods for phase, wrist-orientation, gripper-pullback, and real-robot analyses. Phase is linearly decoded from activations, while the other passages specify sweep, steering, editing, and cross-model procedures.

C.7 Phase

  • Balanced phase accuracy is 0.92 for the action expert, 0.86 for VLM image tokens, and 0.87 for VLM non-image tokens.Four-class LDA is fit on successful episodes with houses held out.Source: §C.7 · p. 39

C.8 Wrist Orientation

  • The wrist sweep varies joint 7 across its range while holding other joints fixed; steering rotates a learned circular code pair and decodes the edited activation.The steering edit is added to all action tokens at layer 9 at the final flow step.Source: §C.8 · p. 40

C.9 Gripper Pullback

  • Gripper pullback fits an additive activation edit to flip the decoded gripper command while penalizing arm movement and chunk roughness.Success is assessed on the decoded chunk, which is not executed; leakage compares arm-command gaps for low-rank and unrestricted edits.Source: §C.9 · p. 40

C.10 Real-Robot Analyses

  • Real-robot analyses use DROID episodes to probe variables and evaluate edits across MolmoAct2, π0.5, and MolmoBot.Cross-model similarity uses standardized probe coordinates on identical rows, but agreement is partly built in because each model is fit to the same labels.Source: §C.10 · p. 41

Related papers

Loading 2610.20840v1…