Source-linked AI summary
Observing and Controlling Features in Vision-Language-Action Models
Hugo Buurmeijer, Carmen Amo Alonso, Aiden Swann, Marco Pavone
TL;DR
VLAs’ multimodal, continuous-action, closed-loop setting makes LLM interpretability and activation steering difficult to transfer directly. This paper formalizes feature-observability and feature-controllability, then uses linear observation and minimal representation interventions to steer VLA behavior. Across OpenVLA and π0.5 simulations, the approach reliably steers robot behavior while preserving closed-loop capabilities and avoiding fine-tuning.
Problem
Multimodal inputs, continuous action outputs, and closed-loop physical interaction create challenges for transferring LLM interpretability and steering methods to VLAs.
Method
The paper combines linear observers with minimal linear controllers to extract features and steer internal VLA representations toward desired constraints during inference.
Results
The framework enables fine-grained VLA output steering, with near-perfect gripper constraint satisfaction while maintaining a success rate above 90%.
Takeaways & Limitations
VLAs possess interpretable internal structure that supports lightweight online adaptation and real-time alignment with user preferences and task requirements without fine-tuning.
Takeaways & Limitations
The current framework requires labeled data to train linear observers and primarily targets Transformer components rather than diffusion or flow-matching heads.
Abstract
from arXiv · showhide
Vision-Language-Action Models (VLAs) have shown remarkable progress towards embodied intelligence. While their architecture partially resembles that of Large Language Models (LLMs), VLAs exhibit higher complexity due to their multi-modal inputs/outputs and often hybrid nature of transformer and diffusion heads. This is part of the reason why insights from mechanistic interpretability in LLMs, which explain how the internal model representations relate to their output behavior, do not trivially transfer to VLA counterparts. In this work, we propose to close this gap by introducing and analyzing two main concepts: feature-observability and feature-controllability. In particular, we first study features that are linearly encoded in representation space, and show how they can be observed by means of a linear classifier. Then, we use a minimal linear intervention grounded in optimal control to accurately place internal representations and steer the VLA's output towards a desired region. Our results show that targeted, lightweight interventions can reliably steer a robot's behavior while preserving closed-loop capabilities. We demonstrate on different VLA architectures ($π_{0.5}$ and OpenVLA) through simulation experiments that VLAs possess interpretable internal structure amenable to online adaptation without fine-tuning, enabling real-time alignment with user preferences and task requirements.
I. INTRODUCTION
VLAs require specialized interpretability and steering methods because multimodal, continuous-action, closed-loop operation complicates direct transfer from LLMs. The paper introduces feature-observability and feature-controllability with lightweight linear observers and controllers for online steering without fine-tuning.
- Multimodal inputs, continuous actions, and closed-loop physical interaction create challenges absent from LLM activation steering.
- Existing VLA interventions have not established reliable control while preserving natural behavior and closed-loop performance.
- The paper formalizes feature-observability and feature-controllability for identifying and manipulating behaviorally relevant VLA representations.
- A linear observer extracts meaningful features from Transformer representations using the linear representation hypothesis.
- A linear controller minimally perturbs Transformer representations to steer outputs while preserving behavioral naturalness.
- The observer-controller is integrated for closed-loop operation without fine-tuning or retraining and evaluated across VLA architectures in simulation and real-world experiments.
II. RELATED WORK
Related work connects VLA steering to mechanistic interpretability and linear representation methods developed for LLMs. The paper considers transformer-based and transformer-flow-matching VLAs, focusing on internal Transformer representations that relate to action outputs.
- Prior VLA work extracted steering vectors and linear probes for motion features and spatial relationships, but the intersection of mechanistic interpretability and robotics remains underexplored.
- Task-specific hidden-state vectors have reconstructed or composed behaviors, improving out-of-distribution extrapolation from 9% to 83%.
- LLM research motivates linear extraction of semantic and behavioral features from intermediate Transformer activations.
- OpenVLA exemplifies autoregressive Transformer VLAs, while π0 and π0.5 exemplify Transformer-flow-matching hybrids with separate action experts.
- The paper focuses on Transformer components because their internal representations directly relate to relevant VLA action features.
- Transformer representations begin with an embedding and are transformed through a sequence of layer mappings, with xℓ denoting the latent representation after layer ℓ.
- The VLA action map may use only the final representation or intermediate representations conditioned through the flow-matching process.
B. Observability and Controllability of Features in VLAs
The paper defines observability as extracting a feature from a layer representation and controllability as steering that representation so later outputs satisfy a desired feature region. It proposes linear maps as sufficient tools for both tasks during inference.
- The framework uses representation-space structure to observe states or actions and steer them toward desired regions.
- Feature-observability means that a layer representation can be mapped to the corresponding behaviorally relevant feature.
- Feature-controllability means that an intervention at a layer propagates through later layers and produces a feature within a desired set.
- The central design question is how to construct observer and controller maps that work effectively during VLA generation.
IV. APPROACH
The approach uses linear observers to extract robot states and actions from transformer representations, with parameters learned from labeled input-feature pairs. It restricts studied features to directly measurable and actuatable robot states and actions, while noting that observer robustness requires empirical verification.
- Feature observer: The framework uses a linear observer to extract interpretable features from internal VLA representations.The observer maps a layer representation to a feature of interest through a linear function.
- Feature scope: The studied features are robot states and actions, while abstract semantic features are deferred to future work.States include Cartesian position, orientation, and gripper aperture; actions are relative displacements in this state space.
- Observer training: Observer parameters Wℓ and bℓ are learned from labeled input-feature pairs after propagating each input through the transformer to layer ℓ.The procedure computes layer representations for training inputs and learns the observer parameters from those representations and labels.
- Observer design: Continuous features are learned with regression, and the observer can be generalized with a known monotonic nonlinearity.The linear-map choice is motivated by the linear separability hypothesis.
- Robustness: An observer learned by Algorithm 1 is not guaranteed to be robust, so the method empirically checks sensitivity to representation perturbations.The check tests whether a small perturbation produces a bounded change in the estimated feature.
B. Designing the Feature Controller
The feature controller applies an additive intervention to a layer representation, choosing the smallest perturbation that places the observed feature in a desired target region. The tractable closed-form construction assumes a linear observer and a one-dimensional bounded target interval.
- Controller formulation: The controller modifies a layer representation through a linear intervention x̃ℓ = gℓ(xℓ).The intervention adds a perturbation uℓ to the original representation.
- Optimal intervention: uℓ is the minimal additive intervention that places the modified representation in the observer’s preimage of the desired feature set.This makes the smallest representation-space change that steers observed features into a target region.
- Tractable solution: The closed-form solution assumes a linear observer and a one-dimensional target set D = [ζmin, ζmax].Without these assumptions, the constraint may be nonconvex and harder to solve tractably.
- Observability and controllability: Effective control is strongest when a feature is both observable and controllable, although the two properties are independent.A feature may be observable without being controllable, or controllable without being observable.
C. Observing and Controlling Features in VLAs
The observer and controller are integrated into the VLA’s inference-time forward pass to control selected features and align downstream outputs. Linear computations add negligible runtime while preserving the closed-loop setting under an in-distribution condition.
- Observer-controller pipeline: The method uses a linear observer with pre-inference parameters and a linear controller whose intervention is computed from the current representation.These components are inserted into the layerwise forward pass to align the VLA output.
- Evaluation setup: Figure 3 compares classifier MAE with mean-prediction MAE and classifier accuracy with majority-class accuracy for π0.5 on Libero and OpenVLA on BridgeData V2.Only the best-performing layer is shown for each model and dataset.
- Inference integration: The inference procedure computes observations at selected layers and applies control interventions at a subset of those layers.The controller layer set must be contained within the observer layer set because control requires observations.
- Efficiency: Linear observer and closed-form controller computations introduce negligible runtime overhead while providing steering capabilities.The added computations occur during the forward pass rather than through fine-tuning or retraining.
- Closed-loop condition: The framework carries over to closed-loop VLAs provided that operation does not move inputs out of distribution relative to the probe-training data.VLA outputs affect the environment, which supplies inputs for subsequent actions.
V. RESULTS
Experiments on OpenVLA and π0.5 examine whether internal representations support feature observation and control. The reported results connect linear observation and minimal representation interventions with fine-grained output steering and alignment with preferences and safety requirements.
- Models: Experiments cover two frontier VLA models, OpenVLA and π0.5.The evaluation studies feature-observability and feature-controllability in both architectures.
- Feature observation: Linear observers recover actions and states from representation space, with observations reported as robust to perturbations.The results section presents observation and robustness before studying control interventions.
- Feature control: Minimal representation-space control interventions produce fine-grained steering of VLA outputs.The reported experiments evaluate how targeted interventions affect generated behavior.
- Alignment: The reported steering improves alignment with user preferences and safety requirements.This conclusion is stated as part of the results across the evaluated VLA models.
A. Feature-Observability
The experiments test whether robot states and actions can be observed with linear classifiers and whether those observations remain robust under representation perturbations.
- The observability experiments assess linear classification of robot states and actions and robustness to representation changes.
- For π0.5 on Libero, perturbation effects are measured across layers using changes in delta yaw and delta gripper actions.
1) Actions and States are Linearly Observable in Representation Space:
Linear probes extract state and action features from VLA representations, while perturbation tests show that most observations are robust and can influence outputs.
- Probes trained at every transformer layer use regression for Cartesian positions and orientations and binary classification for gripper state and action.
- Increasing perturbation strength smoothly increases mean action changes for π0.5, whereas OpenVLA shows weaker robustness, especially for delta yaw.
- Perturbations perpendicular to the linear classifier modify output features, supporting representation-space influence over actions.
- Fixed perturbations are more effective in earlier layers because representation L2-norm increases with depth, reducing their relative effect.
- The control experiments evaluate linear interventions on π0.5 with Libero and OpenVLA with BridgeData V2.
1) Optimal Control Interventions Constrain Actions to a Desired Region of Representation Space:
The minimal controller places observed representations within desired feature bounds and steers gripper state, height, and speed during closed-loop manipulation while retaining task performance.
- The minimal controller guarantees that the observed representation lies within desired bounds, unlike fixed-vector perturbations and no intervention.
- Closed-loop evaluation uses ten Libero spatial tasks with ten rollouts per task for each method.
- The study compares no intervention, prompting, and control across gripper state, end-effector height, and end-effector speed.
- The controller achieves near-perfect gripper constraint satisfaction while maintaining a success rate above 90%.
- End-effector height reaches near-perfect constraint satisfaction, with a modest success-rate drop because constrained tasks are harder.
- Speed control reliably slows the robot but is less accurate at increasing speed, while success rates remain almost perfectly maintained.
- The results show that forward-pass control techniques can carry over to closed-loop robot behavior.
VI. CONCLUSION
The paper establishes that targeted linear interventions can steer behavior in π0.5 and OpenVLA while preserving naturalness and closed-loop capabilities, with negligible computational overhead. It also identifies important limitations involving labeled data, architectural coverage, higher-level features, and safety guarantees.
- Conclusion: The framework formalizes feature observability and controllability to interpret and steer VLA behavior through internal representations.It bridges mechanistic-interpretability ideas from LLMs with embodied-AI challenges.
- Conclusion: Experiments on π0.5 and OpenVLA across Libero and BridgeData V2 show fine-grained behavior steering while preserving naturalness and closed-loop capabilities.The evaluation covers two VLA architectures and multiple robotic manipulation datasets.
- Conclusion: The approach introduces negligible computational overhead, supporting practical real-time robotic applications.The claimed efficiency is presented as a deployment advantage of the targeted interventions.
- Limitations and Future Work: The current approach requires labeled data for training linear observers and primarily addresses transformer components rather than diffusion or flow-matching heads.Extending the framework across hybrid architectures and reducing dependence on labels are identified as future directions.
- Limitations and Future Work: Higher-level semantic features and principled safety guarantees for representation-space interventions remain open questions for broader deployment.The paper specifically names task goals, object affordances, spatial relationships, and safety-critical applications as areas needing further study.
- Conclusion: The results position lightweight online adaptation as a step toward more transparent, controllable, and human-aligned embodied AI systems.This conclusion is framed around reliable deployment in real-world environments.