PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Bryce Grant, Xijia Zhao, Peng Wang

arXiv:2603.19233v1cs.RO

TL;DR

The paper addresses how VLAs translate multimodal inputs into actions, a question complicated by limited internal transparency and failure diagnosis. It applies activation injection, SAEs, and linear probes across six models and 394,000+ rollout episodes, finding visual dominance, task-structure-dependent language use, specialized pathways, and brittle but interpretable representations.

  • Problem

    It remains unclear whether VLA actions follow language instructions or replay visual-motor priors, while operators lack principled ways to diagnose unexpected behavior.

  • Method

    The study applies activation injection, counterfactual prompting, SAEs, and linear probes across six architectures, four benchmarks, and 394,000+ rollout episodes.

  • Results

    Across architectures, visual pathways dominate action generation, language sensitivity depends on task structure, expert pathways encode motor programs, and SAE analysis identifies interpretable but behaviorally brittle representations.

  • Takeaways & Limitations

    VLA representations are rich yet brittle: pathway specialization supports failure diagnosis, while spatially bound motor programs and suite-dependent language use constrain behavioral robustness.

  • Takeaways & Limitations

    All experiments use simulated environments, and whether these architectural patterns persist after domain-specific real-world fine-tuning remains untested.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models combine perception, language, and motor control in a single architecture, yet how they translate multimodal inputs into actions remains poorly understood. We apply activation injection, sparse autoencoders (SAEs), and linear probes to six models spanning 80M--7B parameters across 394,000+ rollout episodes on four benchmarks. The visual pathway dominates action generation across all architectures: injecting baseline activations into null-prompt episodes recovers near-identical behavior, while cross-task injection steers robots toward source-task positions (99.8\% of X-VLA episodes align with the source trajectory), exposing spatially bound motor programs tied to scene coordinates rather than abstract task representations. Language sensitivity depends on task structure, not model design: when visual context uniquely specifies the task, language is ignored; when multiple goals share a scene, language becomes essential (X-VLA \texttt{libero\_goal}: 94\%$\to$10\% under wrong prompts vs.\ \texttt{libero\_object}: 60--100\% regardless). In all three multi-pathway architectures (\pizhalf{}, SmolVLA, GR00T), expert pathways encode motor programs while VLM pathways encode goal semantics ($2\times$ greater behavioral displacement from expert injection), and subspace injection confirms these occupy separable activation subspaces. Per-token SAE processing is essential for action fidelity on most architectures, though mean-pooling improves fidelity on X-VLA. Contrastive identification recovers 82+ manipulation concepts, and causal ablation reveals sensitivity spanning 28--92\% zero-effect rates independent of representation width. We release \textbf{Action Atlas} (https://action-atlas.com) for interactive exploration of VLA representations across all six models.

1 INTRODUCTION

The paper asks whether VLAs follow language or replay visual-motor priors, then studies this question mechanistically across six architectures. It finds visual dominance, task-structure-dependent language sensitivity, specialized pathways, and interpretable but brittle representations.

  • VLAs combine perception, language, and motor control, but it remains unclear whether their actions follow instructions or replay visual-motor priors.
  • Operators lack principled internal diagnostics for unexpected VLA behavior, unlike classical robotics systems whose kinematics and control models can be inspected.
  • Across six architectures, four benchmarks, and 394,000+ rollout episodes, visual pathway dominance, cross-task transfer failure, and suite-dependent language sensitivity replicate.
  • Expert pathways cause 2× greater behavioral displacement than VLM pathways in π0.5, SmolVLA, and GR00T N1.5, indicating functional pathway specialization.
  • Per-token SAE processing preserves action fidelity whereas mean-pooling destroys behavior on most architectures, and causal ablation identifies 82+ concepts with 28–92% zero-effect rates.
  • Action Atlas provides an open-source interactive platform for exploring VLA representations across all six models.

2 RELATED WORK

Prior VLA work established diverse action-generation designs and strong benchmark performance, while robustness studies exposed severe perturbation failures. Mechanistic-interpretability work has begun applying SAEs and probes to VLAs, but cross-architecture behavioral validation remains limited.

  • VLA development spans tokenized actions, continuous regression, and flow matching, with OpenVLA-OFT reporting 97.1% LIBERO success.
  • Independent evaluations report VLA success collapsing from 97% to 0% under 0.2-unit position shifts, motivating mechanistic analysis of robustness.
  • Prior mechanistic studies applied SAEs, steering, and probing to individual VLA or vision-language systems, whereas this work extends validation across six models.

3 METHOD

The study combines rollout-level activation injection, counterfactual prompting, per-token SAEs, feature identification, and linear probes across diverse VLA architectures. These methods test causal behavioral effects while preserving token and pathway structure.

  • The methodology combines activation injection, counterfactual prompting, SAEs, and linear probes uniformly across six models.
  • Activation injection: Activation injection replaces target-episode activations with source-episode activations during inference to test causal effects under null, same-scene, cross-task, and cross-seed conditions.
  • Counterfactual prompting: Counterfactual prompts include correct, null, negation, motor-command, object-swap, and temporal-switch conditions, with additional difficulty levels for SmolVLA.
  • Sparse autoencoders: SAEs use TopK sparsity with k = 64 active features and expansion factor m = 4d or m = 8d to decompose dense activations into sparse features.
  • Sparse autoencoders: Per-token processing preserves heterogeneous temporal structure, while pooling effects are architecture-dependent: mean-pooling improves X-VLA fidelity and raises GR00T VL-SA explained variance from 83–89% to 99%.
  • Feature identification: Across 424 SAEs, contrastive feature selection identifies 82+ manipulation concepts spanning motion, object, and spatial categories.
  • Linear probes: Linear probes test whether action information is linearly decodable, then project out probe directions to assess whether predictive information is removed.
  • Evaluation: Behavior is evaluated with Action Cosine Similarity, binary Task Success, and Override Rate, using 95% Wilson intervals and η2 for ANOVA effect sizes.

4 EXPERIMENTS

Across six models and four benchmarks, interventions show that visual activations strongly control behavior, while language use varies with task structure. Pathway, SAE, and ablation analyses further reveal specialized but brittle representations.

  • Experimental setup: 394,000+ rollout episodes across six models and four benchmarks support a broad mechanistic evaluation of VLA behavior.The models span 80M–7B parameters and multiple action-generation paradigms.
  • Visual pathway influence: 0.999 cosine similarity and 73–77% success show that injected visual activations recover near-identical π0.5 behavior under null prompts.On OpenVLA-OFT, null injection recovers only 14–15% success versus 90–100% baselines.
  • Visual pathway influence: 93.3% behavioral override shows that same-scene visual activation injection can override language-conditioned target selection.On OpenVLA-OFT, this steering reduces performance by −17pp to −57pp because behavior follows the source activation.
  • Cross-task transfer and displacement analysis: 99.8% of X-VLA episodes and 99.6% of π0.5 episodes show source-dominant trajectories despite cross-task success failure.The trajectories indicate spatially grounded motor programs tied to scene coordinates rather than abstract task representations.
  • Language sensitivity: Language sensitivity depends on task structure: visual context can make prompts unnecessary, but wrong prompts collapse performance when goals share a scene.Prompts are still encoded in π0.5 layer 17 activations with 99.3% classifier accuracy despite behavioral invariance.
  • Pathway specialization across architectures: Expert pathways encode motor programs while VLM pathways encode goal semantics across π0.5, SmolVLA, and GR00T.Expert injection produces approximately twice the behavioral displacement of VLM injection in SmolVLA, while pathway-specific responses distinguish active misdirection from passive stalling.
  • SAE analysis across architectures: Per-token SAE processing is generally necessary for action fidelity, whereas X-VLA and GR00T VL-SA layers benefit from mean-pooling.π0.5 reaches 0.4% versus 70% success with mean-pooling versus per-token processing, while X-VLA reaches 94–100% pooled fidelity.
  • SAE analysis across architectures: 82+ manipulation concepts are recovered, while zero-effect ablation rates range from 28% to 92% independently of representation width.The causal profiles show that richer representations are not uniformly behaviorally robust.

5 CONCLUSION

The study concludes that VLA representations are rich but behaviorally brittle: visual activations and spatially grounded motor programs dominate action, while language use depends on task structure. Pathway specialization offers a basis for diagnosing motor versus goal errors, and Action Atlas enables interactive exploration across models.

  • Conclusion: Across six architectures and 394,000+ episodes, VLA representations combine rich concepts and pathway specialization with brittle behavioral robustness.Cross-task injection can steer toward source positions without preserving destination-task success.
  • Conclusion: Action Atlas provides interactive exploration of VLA representations across all six models.The release accompanies the paper’s mechanistic analyses of pathways, concepts, and interventions.
  • Related work: Prior VLA work spans tokenized actions, continuous regression, flow matching, diffusion policies, and language-augmented datasets.The paper positions its mechanistic analysis within this broader development of VLA architectures and action representations.
  • Related work: SAEs decompose dense activations into sparse interpretable features, while activation steering enables behavioral control without retraining.The paper extends these interpretability directions to VLA representations and behavior.
  • Related work: Existing robotics analyses often characterize behavior or failures without analyzing internal representations, motivating the paper’s mechanistic focus.The cited prior work includes attention visualization, action-distribution analysis, behavioral failure characterization, and internal VLA steering.

B LIMITATIONS

The study’s interpretability claims are bounded by simulated evaluation, simple counterfactual prompts, and possible confounds in activation injection and feature interventions. Its analyses also expose architecture- and training-dependent sensitivity, plus implementation constraints for discrete action tokenization.

  • Scope and deployment: All experiments use simulated environments, so persistence of the identified architectural patterns after real-world, robot-specific fine-tuning remains untested.The authors identify broader simulators, real-world datasets, and cross-embodiment collections as needed to establish generalizability.
  • Prompt coverage: The counterfactual prompt set tests only negation, null, and swap variations, leaving compositional instructions and ambiguous multi-object scenes insufficiently evaluated.LIBERO-PRO is suggested as a more rigorous language-grounding evaluation.
  • Injection confounds: Injected activations can create invalid internal states through temporal misalignment, although X-VLA displacement results support genuine source-task behavior transfer.In 99.8% of X-VLA injection episodes, trajectories were more similar to the source than destination task.
  • Steering sensitivity: Feature sensitivity varies widely by architecture and training regime rather than following a universal VLA pattern.π0.5 and GR00T show large success drops under steering, whereas OFT and SmolVLA are comparatively robust.
  • Implementation constraints: Discrete action-token models are especially vulnerable because small SAE reconstruction errors can change argmax-selected bins and compound across autoregressive actions.Base OpenVLA achieved 0% task success on all but the final layer when SAEs were hooked into the forward pass.
  • SAE processing: Mean-pooling can preserve reconstruction quality while destroying action-critical temporal information, motivating per-token processing for most architectures.Per-token processing maintains heterogeneous structure across approach, manipulation, and terminal action phases.

E.2 FFN MOTOR PRIMITIVE NEURONS

FFN analyses identify neurons associated with motor primitives and reveal architecture-dependent organization of spatial and action representations across layers. These findings complement layer-wise probing and intervention results showing progressive scene-to-action transformation.

  • Motor primitive mapping: 6,606 of 16,384 neurons (40.3%) at π0.5 layer 17 associate with motion verbs, alongside neurons for objects, gripper state, direction, and spatial relations.The vocabulary projection maps FFN weight vectors to interpretable motor-primitive categories.
  • Layer progression: Motion-verb neurons rise from 25.8% at layer 0 to 40.3% at layer 17, while gripper neurons increase from 6.1% to 18.2%.Object and spatial neuron proportions decline over the same layer range, tracing a shift toward motor and gripper representation.
  • Architecture comparison: OFT spatial-relation neurons dominate across depths and grow from L0 to L31, while object neurons remain comparatively sparse.This distributed spatial representation is consistent with OFT’s resilience to concept ablation.
  • Layer transformations: 0.0007 ± 0.0004 mean consecutive-layer CKA in π0.5 indicates substantial representational transformation at every layer.The maximum off-diagonal similarity across the 18×18 matrix is 0.0023.
  • Layer-wise information: 97.7–100% task-classification accuracy is reached by OFT layer 8, while episode-length prediction continues refining through later layers.Episode-length prediction rises from R2 = 0.845 at L0 to 0.941 at L24 before settling at 0.915 at L31.
  • Concept specificity: Later layers 14–17 show the strongest concept specificity across action, object, and spatial categories.This supports progressive emergence of semantically structured representations through the transformer stack.

F.3 CROSS-SUITE GENERALIZATION

Cross-suite interventions test whether identified features generalize across tasks and whether their apparent causal effects are robust. The results support concept discrimination and transfer, but require caution because several ablations used stronger full-layer hooks.

  • Cross-suite generalization: Features identified from the Goal suite affect corresponding tasks across Object and Spatial suites, indicating cross-task generalization.The reported cross-suite pattern is qualified by the intervention-hook caveat.
  • Causal feature analysis: 97–98% R2 action probes become approximately 0 after projecting out probe directions, indicating causally necessary action-predictive subspaces.SAE ablation of 2–5 concept-associated features showed no significant effect, consistent with redundant representations.
  • Per-token processing: Mean-pooled SAE reconstruction preserves 95–98% explained variance but reduces task success to 0.4%, whereas per-token processing maintains 70% success.Broadcasting a uniform residual across positions also causes catastrophic failure by corrupting heterogeneous token information.
  • Feature identification: 100% binary classification accuracy shows that selected single SAE features discriminate relevant concepts from unrelated tasks.The result concerns task discrimination, not necessarily causal control of behavior.
  • Feature specificity: Action-correlated features activate across tasks and therefore capture output statistics rather than concept-specific input semantics.This distinguishes action prediction from concept identification.
  • Negative results: Feature recovery, boosting, and concept substitution all fail, with interventions producing near-total failure from a 97.1% baseline.Recovery and substitution use full-layer hooks, while boosting uses corrected MLP-targeted hooks.

G.4 TEMPORAL ABLATION PATTERNS

Temporal interventions indicate that motor-program sensitivity is concentrated early in trajectories, although the strength and interpretation of some effects depend on intervention hooks and architecture.

  • Temporal criticality: Early-phase ablation causes the largest success drop, while late-phase ablation has minimal effect in the reported temporal analyses.This pattern is explicitly summarized for GR00T and is also reported for broader ablation results.
  • GR00T temporal ablation: −50.3pp early-window ablation on DiT layers nearly matches the −50.8pp full-episode drop, whereas mid- and late-window ablation causes only −12pp.Eagle LM layers show a flatter early-to-late profile.
  • Feature criticality: A single feature ablation can cause complete task failure, but adding more ablated features produces no further degradation.The reported result uses full-layer intervention hooks, so absolute effects require caution.
  • Step-level effects: Step 0 ablation causes complete failure, while later steps have minimal impact, placing trajectory commitment at the first forward pass.This is presented as a step-level temporal pattern.
  • Cross-architecture comparison: 28–92% zero-effect rates across models show that causal sensitivity does not follow representation width.The range spans SmolVLA’s greater sensitivity and OFT/X-VLA’s greater resilience.
  • Benchmark context: LIBERO contains Goal, Object, Spatial, and LIBERO-10 suites, providing the task settings used for temporal comparisons.MetaWorld and SimplerEnv are additional benchmark settings described in the passage.

I QUALITATIVE RESULTS AND ADDITIONAL FIGURES

Qualitative rollouts show that both feature ablation and visual perturbation can convert successful manipulation into complete failure. The failures expose specific concept and spatial dependencies in action generation.

  • Qualitative rollout outcomes: Feature ablation and vision perturbation produce binary outcomes: the robot either completes the task or fails entirely, without partial completion.This pattern is reported across the qualitative rollout conditions.
  • Concept ablation: Zeroing PUT features makes the robot drop cream cheese into the bowl and knock it over, turning a 91-step success into a 300-step failure.The ablation targets the PUT concept at layer 8.
  • Concept ablation: Zeroing OPEN features causes the robot to open the bottom rather than middle drawer, producing a 300-step failure at a 40% destruction rate.The intervention redirects the motor program to the wrong target.

J PER-SUITE EXPERIMENTAL BREAKDOWNS

Per-suite experiments reveal strong dependence on task structure, pathway, and visual context. Language and pathway sensitivities vary across suites, while visual perturbations and layer interventions can cause severe failures.

  • OpenVLA-OFT: Zeroing any tested OpenVLA-OFT layer yields only 14–15% aggregate success across all four LIBERO suites.The tested layers are 0, 8, 16, 24, and 31.
  • Visual perturbations: Center-cropping visual input prevents SmolVLA from locating the button, causing timeout at 400 steps instead of the 67-step baseline success.The crop removes spatial context from the button-press scene.
  • Visual perturbations: Grayscale causes X-VLA to fail cube stacking completely, reducing 100% baseline success to 0%.The robot cannot distinguish the cube from the table without color.
  • OpenVLA-OFT: Task identification reaches 97.8–100% accuracy on every OFT layer, while success prediction AUC is 1.0 on Spatial, Goal, and Object suites.Episode-length information is distributed across layers and varies with suite complexity.
  • Pathway representations: Task-relevant information is linearly decodable across architectures, with expert pathways encoding state dynamics and VLM pathways encoding goal semantics.The passage frames this as a cross-model probing result.
  • Language sensitivity: Language sensitivity is suite-dependent: X-VLA libero goal drops from 94% to 10% under null prompts, while libero object retains 60%.X-VLA concept ablation remains resilient across suites, including 98.2% zero effect for libero object.
  • SmolVLA pathway effects: SmolVLA expert ablation is most destructive on libero spatial at −17.7pp, while VLM ablation most affects libero goal at −10.7pp.The reported pathway-specific pattern is consistent with expert motor programs and VLM goal semantics.
  • Language sensitivity: GR00T libero goal drops from 96.7% to 18.9% under non-baseline prompts, while libero object retains 73.3% and LIBERO-Long retains 61.7%.The passage associates greater sensitivity with multi-step structure.

K.1 MODEL ARCHITECTURE

The paper compares VLA architectures, their pathway organization, representation quality, and evaluation setup, while providing Action Atlas for interactive analysis. Across models, architectural differences appear in pathway specialization, injection recovery, SAE fidelity, and probeable information.

  • Architecture: π0.5 combines a 3B PaliGemma backbone with an 18-layer Gemma action expert, generating 50 seven-dimensional actions through iterative flow-matching denoising.The action dimensions are dx, dy, dz, dax, day, daz, and gripper.
  • Cross-model comparison: SmolVLA, GR00T, and π0.5 show strong pathway specialization, whereas OFT lacks it because it uses a single-pathway architecture.Cross-model comparison includes baseline success, visual override strength, language sensitivity, SAE fidelity, cross-task transfer rate, and pathway specialization.
  • Architecture: Base OpenVLA uses Llama-2 7B with DINOv2 and SigLIP encoders, producing seven independently binned 256-value action tokens autoregressively.Each action dimension is represented by one discrete token generated left-to-right.
  • SAE representation quality: GR00T DiT layers reach 98.3–99.8% per-token SAE explained variance, while GR00T VL-SA layers reach 99% with mean-pooling and X-VLA has higher rollout fidelity with mean-pooling.GR00T VL-SA per-token quality is 83–89%; error bars show the range across layers within each model.
  • Activation injection: π0.5 recovers 77% success with 0.999 action cosine similarity under null-prompt injection, while OFT recovers 14% with 0.85 similarity.The comparison indicates stronger visual-pathway dominance for π0.5 and greater language entanglement for OFT.
  • Evaluation and resources: Experiments evaluate five episodes per task for at most 300 steps across LIBERO, MetaWorld, SimplerEnv, and ALOHA, using task-specific success criteria.The study uses more than 394,000 rollout episodes and over 7.1 TB of experiment data across multiple GPU configurations.
  • Resources: Action Atlas provides open-source interactive tools for SAE feature exploration, semantic search, LLM-based auto-interpretability, and ablation video comparisons.Additional interfaces cover rollout videos, layer information flow, and real-time visual perturbation testing.
Loading 2603.19233v1…