PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Analysis of Robot Foundation Models Enables Efficiency Gains

Aiden Swann, Mathilde Papillon, Mozes Jacobs, Siddharth Boppana, Tal Haklay, Eric Bigelow, Andrew Lee, Conor Watts, Thomas Icard, Mac Schwager, Owen Lewis, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger, Thomas Fel, Matthew Kowal

arXiv:2610.20750v1cs.RO

TL;DR

The paper addresses how VLA models transform backbone representations into actions, a mapping that remains less understood than vision-language computation. It uses causal activation patching across three models to trace that pathway, then restricts action conditioning based on the findings. In MolmoBot, this reduces estimated computation by up to 33% while preserving comparable task success across 3,000 evaluation episodes.

  • Problem

    How vision-language representations are transformed into actions in VLA models remains less understood than the internal computation of their vision-language backbones.

    Source: §1 · p. 2

  • Method

    The authors use counterfactual activation patching across three VLAs to trace how visual, semantic, and geometric information reaches the action expert.

    Source: §1 · p. 3

  • Results

    MolmoBot conditioning restricted to causally important layers reduces estimated operation count by up to 33% while achieving comparable task success across 3,000 evaluation episodes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The identified narrow action-conditioning interface provides a basis for designing sparser VLA connections while preserving observed closed-loop performance.

    Source: §1 · pp. 2–3

    Causal patching uses start-of-episode counterfactuals from 13 object pairs in two binary locations, so it may miss computations important during other manipulation phases.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-language-action (VLA) models are a promising approach to general purpose robot manipulation. While the internal computation of vision-language models, which compose the backbone of VLAs, has been studied extensively, less is known about how their perceptual and semantic representations are transformed into action in VLA models. We study this question mechanistically in three frontier VLA models ($\pi_{0.5}$, MolmoAct2, and MolmoBot) by analyzing the causal pathways through which the input image and the instruction shape actions. Across the three models, activation patching shows that causally relevant information influences the action expert (AE) through a narrow band of mid-to-late cross-attention layers, which we term the action-conditioning interface. The content of this information changes with token and layer within the model. Early-to-mid layer representations of the target noun token determine which object is selected, whereas mid-to-late layers carry the spatial information in text tokens that redirects the movement when patched. Where spatial information is stored in prompt tokens various based on the architecture. In MolmoAct2 the model forms task-conditioned representations at the text positions that follow the instruction, whereas $\pi_{0.5}$ additionally stores this information at image token positions. We show that these findings can be translated directly into improving model efficiency. We find that restricting MolmoBot's action expert conditioning to only 3 out of 36 layers reduces inference FLOPs by up to $33\%$ while maintaining comparable task success with the default model across 3,000 evaluation episodes ($46.7\%$ vs. $46.9\%$), and the same principle carries over to real-robot deployment of $\pi_{0.5}$.

1 Introduction

The paper studies how VLA models transform visual and semantic representations into actions, a mapping that remains poorly understood despite growing manipulation capabilities. It traces causal pathways across three models and uses the findings to motivate more efficient action conditioning.

1 Introduction

  • The work addresses an open question about how VLA backbones map image, language, and proprioceptive tokens to continuous actions, beyond prior causal studies of individual representations.Prior VLA work manipulated representations or measured modality and layer dependence, but did not trace how task-relevant information is composed through the backbone and reaches the action expert.Source: §1 · pp. 2–3
  • Across three frontier open-source VLAs—π0.5, MolmoBot, and MolmoAct2—perception and action communicate through a narrow channel of 1–3 cross-attention layers, with causal influence concentrated in a middle-to-late band.The authors call this band the action-conditioning interface; they suggest action-expert inputs already encode task-relevant spatial information.Source: §1 · p. 2
  • The authors use causal analysis to select sparse conditioning connections, offering a mechanistic rationale for efficiency methods that otherwise rely on heuristics such as representational similarity or benchmark success.They argue that action-relevant representations are stored sparsely across tokens and layers.Source: §1 · p. 2
  • Restricting MolmoBot conditioning to causally important layers removes 92% of cross-attention parameters and reduces estimated operation count by up to 33%, with comparable success across 3,000 episodes.Sparse and dense π0.5 policies also achieved comparable real-robot success in 67 and 64 of 100 trials, respectively.Source: §1 · p. 2
  • Early image tokens carry visual information, the target noun carries object-selection information, and later text and readout positions carry geometric information.Geometric information also appears at image positions in π0.5; the paper reports over 775,000 patching interventions across the three models.Source: §1 · p. 3

2 Discovering the Action-Conditioning Interface

The paper studies how backbone representations reach the action expert in three VLA architectures, using counterfactual activation patching to locate behaviorally influential connections. The observed influence is concentrated in a narrow band of layers.

2 Discovering the Action-Conditioning Interface

  • The study compares three architectures whose action experts receive backbone information through different mechanisms: hidden states, keys and values, or a shared multimodal prefix.The models are MolmoBot, MolmoAct2, and π0.5; their attention masks also differ.Source: §2 · p. 3

2.1 Interface interventions.

  • Interface patching replaces selected action-expert conditioning while holding the original VLM computation fixed, testing where the expert reads causally relevant information.Other conditioning retains its original value while the expert computes from the modified inputs.Source: §2.1 · p. 4
  • Behavioral transfer is measured by reconstructing action-command trajectories with forward kinematics and computing their normalized shift toward a counterfactual reference trajectory.The reference changes the requested object while retaining the original scene; paired runs use shared action noise.Source: §2.1 · p. 4

2.2 Action Expert - Interface Patching

  • In each model, only a narrow band of one to three adjacent layers accounts for most conditioning influence; MolmoBot layer 24 recovers over 50% of reach contrast alone and close to 70% in a three-layer window.π0.5 peaks above 35% for a single-layer patch and above 70% for a sliding patch; MolmoAct2 reaches nearly 30% and above 40%, respectively.Source: Figure 2 · p. 4, §2.2 · p. 5

3 Efficiency Improvements in Real Policies

The authors restrict action-expert conditioning to the identified interface and test whether the simplified policies retain capabilities. In simulation, the MolmoBot policy maintains comparable performance with lower computation; the approach also transfers to real-robot π0.5 evaluation.

3.1 Molmobot In Simulation

  • 46.9% for the dense MolmoBot policy versus 46.7% for the L23–25 pretraining policy on the averaged success metrics, with comparable performance and a 33% FLOPs reduction.The averages combine Pick success and two oracle metrics across three simulation benchmarks.Source: Table 1 · p. 5, §3.1 · p. 5
  • 11.93 to 7.96 TFLOPs per action chunk cuts computation by 33%; on an H100, latency falls 33% and throughput rises 44%.Throughput increases from 21.5 to 30.9 chunks per second.Source: §3.1 · p. 6
  • 25.7% Pick success and 20.9% Pick & Place oracle success with layer 12 contrast with 54.7% and 54.0% when retaining layer 24.Both policies used the same 50,000-update adaptation budget.Source: §3.1 · p. 6

3.2 π0.5 in the real world

  • 64 versus 67 successes out of 100 trials for the original and restricted π0.5 policies show preserved task success in real-robot evaluation.The restricted policy attends to layers 14–16; median successful-trial duration averaged over seven tasks decreased from 34.9 s to 26.5 s, measuring full task execution rather than inference latency.Source: §3.2 · p. 6, Table 2 · p. 6

4 Explaining the Action Conditioning Interface

The section uses counterfactual interchange interventions to distinguish object identity from target location and trace how both shape actions. Across the models, the action-conditioning interface is geometric, while the backbone builds that information through token- and architecture-dependent pathways.

4.1 Causal variables and counterfactual predictions

  • The three counterfactual comparisons distinguish visual, semantic, and geometric transfer by varying whether object identity, scene layout, or target location changes.The hypotheses make distinct predictions across A, B, and C, so effects are interpreted jointly across comparisons and patched positions rather than as uniquely isolated by one comparison.Source: §4.1 · p. 7, Figure 3 · p. 7

4 Explaining the Action Conditioning Interface

  • The experiments apply interchange interventions to image tokens, the target noun, downstream prompt spans, and final readout positions, measuring behavioral transfer against a reference trajectory.The dataset uses 26 input pairs in both directions and eight noise seeds, yielding 416 interventions per cell in the backbone results.Source: §4.1 · p. 8

4.2 The action-conditioning interface is geometric

  • Interface interventions support a geometric role: changing target location in B and C shifts behavior, while token-restricted patches in A have effects of at most 0.05 versus 0.42–0.88 in B and C.A changes the requested object and layout while preserving target location; the token-restricted comparison helps rule out a null effect caused by jointly represented visual and semantic variables.Source: §4.2 · p. 8

4.3 How the backbone constructs the geometric variable

  • Early image representations carry visual information: patching them in A or C redirects behavior, with effects approaching E = 1 in early layers for the dominant camera.The exterior view dominates in MolmoAct2, while the wrist view dominates in π0.5.Source: §4.3 · p. 9
  • Early noun interventions transfer object identity: the model selects the counterfactually requested object at its location in the original scene, and this semantic influence weakens with depth.The pattern appears in comparisons A and B but has little effect in MolmoAct2 comparison C, where the requested object stays fixed and its location changes.Source: §4.3 · p. 9
  • Deeper text representations carry target-location information: patches redirect reaches in C despite identical instructions, following the geometric signature across A, B, and C.In MolmoAct2, all-text interventions are stronger than intervening on the final readout token alone; four tokens retain 89% of the all-text effect at layer 18.Source: §4.3 · pp. 9–10
  • Architecture changes where task-conditioned spatial information can reside: π0.5 image representations can redirect behavior, whereas MolmoAct2 and MolmoBot image positions cannot attend to the subsequent instruction.Across architectures, the backbone combines visual and semantic information into the geometric variable supplied through the interface, but interventions do not establish exclusive storage at identified positions.Source: §4.3 · p. 10

5 Discussion and conclusion

Across three VLAs, causal analysis identifies a narrow middle-to-late action-conditioning interface and traces how representations guide action. The analysis is limited to start-of-episode interventions, although restricted policies also succeed on full tasks.

5 Discussion and conclusion

  • Across three VLAs, the action expert reads task-relevant spatial information through a narrow interface of one to three middle-to-late layers.The discussion describes the interface as a handover of geometric information from the backbone to the action expert.Source: §5 · p. 10, A A Causal Model of Object Selection… · p. 16
  • Causal patching is limited to episode starts and 13 object pairs in two binary locations, so it may miss computation important in later manipulation phases.Despite this scope, policies designed from start-of-episode interventions completed full tasks, including grasping, placing, and stacking, with comparable success over 3,000 simulation and 100 real trials.Source: §5 · p. 10

A A Causal Model of Object Selection and Target Location

  • The causal model represents scene layout, requested object, target location, and the location targeted by the policy’s actions.Its scope is which of two object locations the policy reaches toward, not path, grasp, or speed.Source: A A Causal Model of Object Selection… · p. 15
  • Interchange interventions replace one variable with its counterfactual value, predicting distinct reach outcomes for visual, semantic, and geometric information.Changing the visual variable uses the requested object’s counterfactual location; changing the semantic variable selects the counterfactually requested object in the original layout; changing geometry copies the counterfactual target location.Source: A A Causal Model of Object Selection… · p. 15

B.1 Action-conditioning interface

Interface patching shows that behavioral influence is concentrated in a few connections, but weaker joint influence remains elsewhere. The concentration also persists when trajectory changes are measured in all Cartesian directions.

B.1 Action-conditioning interface

  • Preserving the strongest connections substantially reduces behavioral transfer, although joint influence outside the strongest one to three connections is not negligible.The test patches every connection except one or a three-layer window, leaving the excluded connection conditioning unchanged.Source: §B.1 · p. 16
  • Figure 5 contrasts normalized trajectory shift when connections are left unchanged with RMS displacement when connections are patched; axes index connections or window centers.Both panels average comparisons B and C across object-pair families, with 95% family-bootstrap confidence intervals.Source: Figure 5 · p. 16
  • The dominant single-connection and three-layer-window peaks remain under RMS trajectory displacement, showing concentration across all Cartesian directions.RMS displacement is computed between patched and original end-effector trajectories before averaging interventions.Source: §B.1 · p. 17

B.2 Backbone Patching in MolmoBot

MolmoBot’s causal structure resembles MolmoAct2’s: early target-noun representations carry object identity, while later text positions carry location. The models differ in which camera view dominates, and π0.5 additionally stores instruction-conditioned information in image tokens.

B.2 Backbone Patching in MolmoBot

  • In MolmoBot and MolmoAct2, the target noun carries object identity in early layers, while location is distributed across later text positions.MolmoBot’s pattern resembles MolmoAct2’s despite their different training data and action-expert attention implementations.Source: §B.2 · p. 17
  • The wrist view dominates in MolmoBot, whereas the exterior view dominates in MolmoAct2.The comparison concerns which camera view matters most in the two models’ backbone patching results.Source: §B.2 · p. 17
  • MolmoBot image patches have no effect in comparison B because image positions cannot attend to the instruction; π0.5 instead stores some instruction-conditioned influence in image tokens.Figure 6 averages 416 patches per cell across 13 object pairs; omitted layers 29–35 have |E| < 0.005.Source: §B.2 · p. 17, Figure 6 · p. 17

B.3 Text Tokens That Store Geometric Location in MolmoAct2

A greedy search identifies compact sets of MolmoAct2 text tokens that retain much of the geometric-location patching effect. The relevant token set changes with layer, with later layers concentrating information on the readout token.

B.3 Text Tokens That Store Geometric Location in MolmoAct2

  • The search uses greedy forward selection across comparison-C pairs to find the smallest token sets retaining at least 75% of the effect of patching all 79 non-state text tokens.The selected sets retain 76–89% on the selection noise seed and 72–90% on three held-out seeds.Source: §B.3 · p. 17, Figure 7 · p. 18
  • At layers 19–21, sets of 10–11 tokens retain 76–77%; at layer 22, <action_output> with four sentence-boundary tokens retains 77%.Removing any single token from any set drops retention below 75%.Source: §B.3 · p. 18
  • Location is jointly carried by structural positions, function words, and action-question tokens before concentrating on the readout token.The authors hypothesize that target-object-location computation occurs in this progression.Source: §B.3 · p. 18

C Compute and Latency

Compute estimates show that restricting MolmoBot’s VLM and action-expert conditioning reduces policy cost, while measured latency reduction tracks the estimated full-policy reduction. For π0.5, the estimated savings are smaller because its action expert reads VLM-computed prefix keys and values.

C Compute and Latency

  • 11.93 to 7.64 TFLOPs per action chunk: keeping one cross-attention block lowers MolmoBot’s estimated full-policy cost.The estimate includes the vision encoder, VLM, and action expert; the VLM stops after layer 25 because nothing reads layers 25–35.Source: C Compute and Latency · p. 18, Table 4 · p. 19
  • 4.76 to 4.52 TFLOPs per action chunk: π0.5’s restricted policy yields an estimated 5% reduction, mostly from stopping the VLM after layer 16.The action expert reads prefix keys and values already computed by the VLM, and the estimated savings were considered too small for empirical latency or throughput analysis.Source: C Compute and Latency · p. 18
  • 35% measured latency reduction matches the estimated full-policy reduction (36%) on a single H100.At batch size 1, savings come almost entirely from the action expert; at batch size 64, throughput gains are larger as the all-layer expert becomes compute-bound.Source: C Compute and Latency · p. 18, Table 5 · p. 19

D Implementation Details

MolmoBot evaluation used three fixed 1,000-episode sets with identical episodes and noise seeds across policies. Training followed released recipes, with action-expert cross-attention blocks modified for the variants.

D Implementation Details

  • Evaluation covered 1,000 episodes each for Pick, Pick & Place, and Pick & Place Next-To in held-out ProcTHOR-Objaverse houses using MolmoSpaces.Every policy used the same episodes and identical noise seeds.Source: D Implementation Details · p. 19
  • MolmoBot training followed the released two-stage recipe, with the retained action-expert cross-attention blocks modified.Stage 1 trained from Molmo2-4B for 200K steps on single-frame observations; stage 2 continued for 50K steps with two horizon frames.Source: D Implementation Details · p. 19
  • Adapt variants started from the released MolmoBot-DROID checkpoint and used the 50K-step stage-2 recipe; π0.5 policies were fine-tuned on DROID with the published openpi recipe.Both π0.5 policies started from the released π0.5 base checkpoint.Source: D Implementation Details · p. 19

D.1 Experimental Setup

The experiments combine a counterfactual object-layout dataset with real-robot evaluation of original and restricted π0.5 policies on tabletop tasks. The restricted policy had no statistically significant success-rate difference and shorter successful-trial durations.

D.1 Experimental Setup

  • The counterfactual dataset paired 13 household-object pairs across two layouts, yielding 26 scenes and 52 scene-instruction inputs.Each scene was paired with an instruction to pick up either object; the layouts exchanged the objects’ left–right positions.Source: §D.1 · p. 19, Figure 8 · p. 20
  • 67 vs. 64 of 100 trials: original and restricted π0.5 policies showed no statistically significant success-rate difference; successful-trial median duration fell from 34.9 to 26.5 s, 24% shorter.The duration comparison averaged over seven tasks and was significant (permutation test, p = 0.01); no single task reached significance on its own.Source: §D.1 · p. 19, Table 6 · p. 20
  • Real-robot evaluation compared original and restricted π0.5 policies on four tabletop tasks, with 100 trials per policy.Tasks included placing fruit in a basket, placing one of four objects in a clear bin, placing a marker in a mug, and stacking three bowls.Source: §D.1 · p. 19, Figure 9 · p. 20

Related papers

Loading 2610.20750v1…