PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Models

Aiden Swann, Lachlain McGranahan, Hugo Buurmeijer, Monroe Kennedy, Mac Schwager

arXiv:2603.19183v2cs.RO

TL;DR

VLAs can generalize across tasks and scenes, but the internal features supporting that behavior are not well understood. This paper trains SAEs on VLA activations, quantifies feature generality, and tests causal steering. Across models and datasets, it finds interpretable features whose interventions can produce semantically consistent behavior, while feature generality and steerability remain imperfectly aligned.

  • Problem

    The field lacks mechanistic tools for determining which VLA representations encode transferable concepts and which memorize data, despite observable behavioral failures.

  • Method

    The paper trains SAEs on VLA residual-stream activations, uses activation statistics to categorize features, and evaluates them through feature steering.

  • Results

    Across two VLA architectures and two robotics datasets, many SAE features correspond to motion primitives, task progress, semantic representations, or episode-specific memorization, and steering general features can alter behavior consistently.

  • Takeaways & Limitations

    SAE features provide evidence that VLAs contain reusable, behaviorally meaningful computations linking perception, language, and action.

  • Takeaways & Limitations

    Many clean features have limited or unpredictable causal impact, and some features do not fit cleanly into the general-versus-memorized classification.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have emerged as a promising approach for general-purpose robot manipulation. However, little research has mechanistically explored when and why they generalize across objects, scenes, and instructions. To probe internal representations, we train Sparse Autoencoders (SAEs) on the VLA's hidden-layer activations. SAEs learn sparse dictionaries over model activations, often revealing features that correspond to interpretable directions in the model's representation space. We identify SAE features corresponding to motion primitives and semantic concepts, including features that are general across episodes and causally steerable. We propose a metric to categorize features as general transferable primitives or episode-specific memorizations, offering a promising glimpse towards VLA generalization. We validate these findings through steering experiments on both the LIBERO simulation benchmark and on real-world DROID hardware. We find that amplifying general and semantic features induces behaviors consistent with their meanings, whereas ablating them destroys model performance. Furthermore, we demonstrate steering as a way to control behavior in unpromptable directions. Together, these results provide mechanistic evidence that VLAs can learn reusable internal features linking perception, language, and action across tasks and scenes. Our project page is located at https://drvla.github.io

1 Introduction

VLA models combine visual inputs, language instructions, and continuous control, but their generalization and internal representations remain poorly understood. This work applies SAEs to identify, quantify, and steer interpretable VLA features.

  • VLA models combine visual inputs, natural-language instructions, and continuous control outputs in one learned system.
  • Current VLAs often require task- or embodiment-specific fine-tuning and can lose language grounding despite benchmark progress.
  • SAEs disentangle dense VLA activations into sparse features that can reveal transferable concepts versus memorized data patterns.
  • The proposed pipeline extracts interpretable, general, and steerable VLA features using Sparse Autoencoders.
  • Generality metrics classify SAE features using activation statistics from fine-tuning data without policy roll-outs.
  • Simulation and real-world steering experiments show that individual SAE interventions produce consistent, semantically meaningful behavioral changes.

2 Related Work

Mechanistic interpretability for VLAs is an emerging area focused on decoding task-relevant internal information and causally intervening on model behavior.

  • Recent VLA interpretability work pursues internal-state decoding and causal behavioral intervention as complementary goals.
  • Linear probes decode states such as object positions and actions from OpenVLA activations, with accuracies exceeding 90% for most layers.
  • Other studies investigate state-transition structure and semantic FFN directions, while proposing SAEs without implementing them.

3 Methods

The method trains SAEs on VLA residual-stream activations, quantifies feature generality from activation statistics, classifies interpretable features, and tests them through steering interventions.

  • Sparse Autoencoders: TopK SAEs with AuxK loss learn sparse features from VLA residual-stream activations while controlling sparsity and reducing dead latents.
  • Generality Quantification Metrics: Four activation statistics measure feature breadth, temporal burstiness, expression strength, and persistence across episodes.These are episode coverage, mean onset count, mean activation magnitude, and relative run length.
  • Feature Classification: A logistic-regression classifier scales general-versus-memorized labeling from manually labeled features using the four activation metrics.
  • Feature Classification: A VLA feature is interpretable when its temporal activation pattern consistently aligns with an identifiable sensorimotor event, visual state, or task phase.
  • Feature Classification: General features tend to activate broadly and in event-locked bursts, whereas memorized features tend to track narrow episodes with sustained activations.
  • Feature Classification: The general-versus-memorized distinction is imperfect, so the classifier ranks and categorizes features rather than defining their semantics absolutely.
  • Feature Steering: Additive steering amplifies a feature by adding a scaled decoder direction to the residual stream, while ablative steering removes that direction.
  • Feature Steering: Together, additive and ablative interventions probe how individual SAE features affect model computations.

4 Interpretable and General Features in VLA Residual Streams

SAEs yield interpretable and general features in VLA residual streams, including recurring manipulation phases and features that span diverse episodes. Automated metrics identify a smaller subset of features as general, with the largest share in π0.5-DROID.

  • 4.1 SAEs Find Interpretable Features: 95 of 120 sampled SAE features are interpretable (79.2%), compared with 6 of 20 FFN neurons (30.0%).The sampled SAE features align with visual states, task phases, and sensorimotor events.
  • 4.2 SAEs Find General Features: Four DROID features activate across diverse scenes, objects, and instructions, suggesting reusable manipulation structure rather than fixed trajectories or scenes.Figure 3 shows per-timestep activations alongside main- and wrist-camera frames across example tasks.
  • 4.2 SAEs Find General Features: F158 marks sub-task transitions, F586 precision or pinch-like grasps, F165 target visibility between gripper jaws, and F399 grasp acquisition and placement.These features correspond to recurring phases of manipulation.
  • 4.3 Model-Wide Automated Classification: The per-dataset classifiers quantify the fraction of SAE features classified as general across analyzed models and layers.Table 1 reports the resulting percentages of general features.
  • 4.3 Model-Wide Automated Classification: The general-feature share is largest in π0.5-DROID, smaller in π0.5-LIBERO, and smallest in OpenVLA fine-tuned on LIBERO-Goal.The classifier identifies a high-confidence subset rather than exhaustively separating useful features from memorized ones.

5 Causal Validation via Feature Steering

Steering and ablation experiments test whether SAE features causally influence VLA behavior. General-feature ablations degrade held-out performance, while semantic and gripper-feature steering produces consistent behavior changes, including behaviors not reliably accessible through language prompts.

  • 5.2 Steering Validates Interpretations of General Features: Amplifying DROID general features produces interpretation-consistent behavior, including dwelling near sub-task transitions, earlier gripper closure, and approach–retry cycles.These qualitative changes occur when the identified general features are amplified.
  • 5.2 Steering Validates Interpretations of General Features: Removing memorized features has little effect on held-out tasks, whereas removing general features substantially degrades performance.The ablation evaluates four features per tier across five trials per task on sponge and towel tasks.
  • 5.2 Steering Validates Interpretations of General Features: 65.0% success after random general-feature ablation shows that the broader general-feature population is behaviorally important.Removing the four most general features completely destroys performance, while memorized-feature removal has little effect.
  • 5.3 Steering Semantic Features Biases Object Selection: SAE semantic steering shifts grasp selection toward named target objects by +9 to +21 over baseline, versus +1 to +4 for FFN-neuron steering.Without steering, the policy instead collapses onto the visually dominant object in each scene.
  • 5.4 Eliciting Behaviors That Language Prompts Cannot: For gripper control, language prompting leaves closing nearly unreachable at 0.005 closure, while F586 steering reaches 0.653 and the FFN baseline reaches 0.361.Opening is also improved by SAE steering with F406, reaching 0.510 closure from a towel-held starting state.
  • 5.4 Eliciting Behaviors That Language Prompts Cannot: SAE feature steering elicits behaviors not reliably exposed through language and controls gripper opening and closing more effectively than prompting or FFN-neuron steering.The paper presents this as evidence that SAE features exert causal, controllable influence on VLA computation.

6 Conclusion

The paper presents an SAE-based interpretability pipeline for VLAs and finds interpretable features across models, layers, tasks, and datasets. Steering and ablation experiments provide evidence that individual SAE directions influence closed-loop behavior, while the paper identifies practical uses but does not apply the results to model improvement.

  • Across two VLA architectures and two robotics datasets, many SAE features correspond to motion primitives, task progress, semantic representations, or episode-specific memorization.
  • Steering general features often changes motion consistently with their hypothesized meanings, including hovering before grasping and biasing behavior toward sponge grasps.
  • Episode coverage, onset count, activation magnitude, and run length quantify feature generality and support scalable categorization.
  • The results are not applied to model improvement, although the authors suggest rollout-free feature analysis for diagnosing fine-tuning brittleness and proxying generalization during training.

7 Limitations

The study’s evidence spans multiple VLA architectures, embodiments, datasets, and activation-collection choices, but several limitations constrain interpretation. In particular, meaningful feature activations do not reliably imply steerability, and the evaluation uses limited robotics data and predominantly mean-pooled timestep activations.

  • Meaningful top activations do not imply reliable steerability, because many clean features have limited or unpredictable causal effects when steered.
  • Some features do not fit the general-versus-memorized classification, including features that activate only at episode boundaries or sparsely across the dataset.
  • The analysis covers three VLA models spanning two architectures and robot embodiments, using π0.5-LIBERO, π0.5-DROID, and OpenVLA on LIBERO data.
  • LIBERO provides controlled simulation but limited tabletop tasks and shared base scenes, while DROID contributes 2,000 sampled real-world episodes including 250 failures.
  • OpenVLA analysis uses only the LIBERO-Goal suite, matching that model’s fine-tuning distribution.
  • Main-text SAE results use mean-pooled timestep activations because retaining token-level activations would require several terabytes for the DROID subset.

B.1 SAE Architecture and Training Hyperparameters

The method normalizes residual-stream activations, encodes them with sparse TopK features, reconstructs them with constrained decoder directions, and trains with reconstruction plus auxiliary losses. Feature generality is then quantified from temporal activation statistics and scaled through manual labels plus logistic regression.

  • SAE preprocessing and encoding: Each input activation is centered using a learned pre-bias and per-sample mean subtraction, then normalized by its ℓ2 norm before encoding.
  • SAE preprocessing and encoding: TopK encoding retains the k largest pre-activation values, zeros the rest, and produces a sparse representation.
  • SAE preprocessing and encoding: The decoder reconstructs normalized activations, restores normalization statistics, and uses unit-norm decoder columns so each feature is scaled by its activation coefficient.
  • SAE training choices: The training objective combines normalized reconstruction error with AuxK auxiliary reconstruction for inactive features, with dead latents defined by 500 optimization steps without activation.
  • SAE training choices: The default configuration uses ER = 1 for π0.5, while OpenVLA uses ER = 0.5; larger expansion ratios produced more dead features with similar interpretability in this setting.
  • Feature stability: Independent-seed SAEs recover similar top-feature temporal patterns, supporting stability of the learned features across random initializations.
  • Generality metrics: Episode coverage measures how broadly a feature appears, with higher coverage indicating activity across more diverse tasks and episodes.
  • Generality metrics: Mean onset count, activation magnitude, and relative run length characterize burstiness, peak strength, and persistence across episodes.

C.4 Classifier Coefficients

The classifier uses activation statistics to distinguish broad, event-aligned general features from narrower memorized features. Its coefficients support scalable ranking, but the authors caution that it is a heuristic rather than an absolute semantic boundary.

  • LIBERO’s classifier achieved 100% leave-one-out accuracy on 30 manually labeled examples.
  • DROID’s classifier achieved 96.7% leave-one-out accuracy, with episode coverage weighted more strongly than in LIBERO.
  • General features tend to be broad across episodes, bursty within episodes, and aligned with events, whereas non-general features are narrower and more sustained.
  • The classifier can under-classify general features that activate once per episode across a coherent but small dataset subset.F1939 and F1381 demonstrate this failure mode for shared home-position and lid-grasp features.
  • SAEs yielded 95/120 interpretable sampled features, compared with 6/20 sampled FFN neurons under the matched labeling pipeline.The comparison holds activation collection, indexing, visualization, and labeling procedures fixed.

E.1 LIBERO General Features

LIBERO SAE features recur across scenes, objects, and grasp subgoals, while their activation structure varies across layers, architectures, and datasets. Ablation results show that generality predicts behavioral importance.

  • Four LIBERO features have episode coverage above 0.99 and activate across scenes, goal objects, and numbers of grasp subgoals.They correspond to grasping and placement, transport, pre-grasp alignment, and goal completion.
  • π0.5 action-expert layers show phase-specific SAE activations consistent with motion primitives across LIBERO and DROID despite different action representations.PG5 and PG11 contain the most salient and interpretable π0.5 features.
  • DROID activations are denser than LIBERO activations, while OpenVLA exhibits a qualitatively different activation texture.The paper attributes DROID’s density to greater visual diversity and relates OpenVLA’s difference to its lack of an action expert.
  • Interpretable features reported in the main analysis emerge during robotic fine-tuning rather than being inherited from pretrained visual representations.The comparison includes the π0.5 embedding layer and a frozen SigLIP encoder without robotics exposure.
  • General-feature ablation caused success to fall sharply above P(general) ≈0.5, with the four most general features producing 0/600 across three suites.Features below the threshold were indistinguishable from the unsteered policy at approximately 95% success.

F.5 Qualitative DROID Steering Results

Additive steering of four DROID general features produced behavior matching their inferred manipulation roles. The interventions altered sub-phases such as dwelling, grasp timing, and transport transitions while preserving goal-directed behavior.

  • At α=100, amplifying F158 made the gripper dwell over the target before grasping.This matches its interpretation as a sub-task checkpoint feature that prolongs the approach phase.
  • At α=100, amplifying F586 often caused earlier gripper closure above the object, consistent with a pinch-grasp feature.
  • At α=100, F165 produced behavior similar to the unsteered policy, with additional dwelling over the target before grasping.
  • At α=100, F399 induced pronounced approach-and-retry cycles and premature transitions toward transport.The behavior suggests the model acted as if grasp acquisition had already succeeded.
  • Across experiments, single-feature steering at high magnitude generally preserved goal-directed attempts to approach, grasp, or pursue alternative subtasks.

G Per-Token SAEs

Per-token SAEs expose distinct visual, textual, and visual-semantic activation patterns, including both general and episode-specific features. However, general per-token features are relatively uncommon and often align with function words.

  • F1881 activates broadly across episodes, strongly on the main camera and across highlighted text tokens, linking co-referent scene tokens such as “pot” and “black”.
  • F225 captures textual semantic structure, activating strongly on instruction nouns but not verbs or camera tokens.“Black” and “pot” co-activate, peaking when the wrist camera is positioned over the pot.
  • F1659 is an interpretable memorized feature associated with episodes containing lids and pots, peaking when the lid is placed on the pot.Its strongest text-token activations correspond to “pot” or the adjacent space.
  • Per-token SAEs are less interpretable in general than summed-token SAEs, with evidence of generality concentrated in fewer than ten features per layer.Top activating tokens for these general features are often function words rather than content words.
Loading 2603.19183v2…