PaperlayerView exact revision on arXiv

Source-linked AI summary

Observing and Controlling Features in Vision-Language-Action Models

Hugo Buurmeijer, Carmen Amo Alonso, Aiden Swann, Marco Pavone

arXiv:2603.05487v1cs.RO

TL;DR

VLAs’ multimodal, continuous-action, closed-loop setting makes LLM interpretability and activation steering difficult to transfer directly. This paper formalizes feature-observability and feature-controllability, then uses linear observation and minimal representation interventions to steer VLA behavior. Across OpenVLA and π0.5 simulations, the approach reliably steers robot behavior while preserving closed-loop capabilities and avoiding fine-tuning.

  • Problem

    Multimodal inputs, continuous action outputs, and closed-loop physical interaction create challenges for transferring LLM interpretability and steering methods to VLAs.

    Source: §I · p. 1

  • Method

    The paper combines linear observers with minimal linear controllers to extract features and steer internal VLA representations toward desired constraints during inference.

    Source: §I · p. 2, B. Designing the Feature Controller · p. 4

  • Results

    The framework enables fine-grained VLA output steering, with near-perfect gripper constraint satisfaction while maintaining a success rate above 90%.

    Source: §V · p. 5, 2) Optimal Control Interventions in… · p. 7

  • Takeaways & Limitations

    VLAs possess interpretable internal structure that supports lightweight online adaptation and real-time alignment with user preferences and task requirements without fine-tuning.

    Source: §VI · p. 8

    The current framework requires labeled data to train linear observers and primarily targets Transformer components rather than diffusion or flow-matching heads.

    Source: §VI · p. 8

Abstract

from arXiv · show

Vision-Language-Action Models (VLAs) have shown remarkable progress towards embodied intelligence. While their architecture partially resembles that of Large Language Models (LLMs), VLAs exhibit higher complexity due to their multi-modal inputs/outputs and often hybrid nature of transformer and diffusion heads. This is part of the reason why insights from mechanistic interpretability in LLMs, which explain how the internal model representations relate to their output behavior, do not trivially transfer to VLA counterparts. In this work, we propose to close this gap by introducing and analyzing two main concepts: feature-observability and feature-controllability. In particular, we first study features that are linearly encoded in representation space, and show how they can be observed by means of a linear classifier. Then, we use a minimal linear intervention grounded in optimal control to accurately place internal representations and steer the VLA's output towards a desired region. Our results show that targeted, lightweight interventions can reliably steer a robot's behavior while preserving closed-loop capabilities. We demonstrate on different VLA architectures ($π_{0.5}$ and OpenVLA) through simulation experiments that VLAs possess interpretable internal structure amenable to online adaptation without fine-tuning, enabling real-time alignment with user preferences and task requirements.

I. INTRODUCTION

VLAs require specialized interpretability and steering methods because multimodal, continuous-action, closed-loop operation complicates direct transfer from LLMs. The paper introduces feature-observability and feature-controllability with lightweight linear observers and controllers for online steering without fine-tuning.

  • Multimodal inputs, continuous actions, and closed-loop physical interaction create challenges absent from LLM activation steering.Source: §I · p. 1
  • Existing VLA interventions have not established reliable control while preserving natural behavior and closed-loop performance.Source: §I · pp. 1–2
  • The paper formalizes feature-observability and feature-controllability for identifying and manipulating behaviorally relevant VLA representations.Source: §I · p. 2
  • A linear observer extracts meaningful features from Transformer representations using the linear representation hypothesis.Source: §I · p. 2
  • A linear controller minimally perturbs Transformer representations to steer outputs while preserving behavioral naturalness.Source: §I · p. 2
  • The observer-controller is integrated for closed-loop operation without fine-tuning or retraining and evaluated across VLA architectures in simulation and real-world experiments.Source: §I · p. 2

II. RELATED WORK

Related work connects VLA steering to mechanistic interpretability and linear representation methods developed for LLMs. The paper considers transformer-based and transformer-flow-matching VLAs, focusing on internal Transformer representations that relate to action outputs.

B. Observability and Controllability of Features in VLAs

The paper defines observability as extracting a feature from a layer representation and controllability as steering that representation so later outputs satisfy a desired feature region. It proposes linear maps as sufficient tools for both tasks during inference.

IV. APPROACH

The approach uses linear observers to extract robot states and actions from transformer representations, with parameters learned from labeled input-feature pairs. It restricts studied features to directly measurable and actuatable robot states and actions, while noting that observer robustness requires empirical verification.

Feature observer

Feature scope

  • The studied features are robot states and actions, while abstract semantic features are deferred to future work.States include Cartesian position, orientation, and gripper aperture; actions are relative displacements in this state space.Source: A. Designing the Feature Observer · pp. 3–4

Observer training

  • Observer parameters Wℓ and bℓ are learned from labeled input-feature pairs after propagating each input through the transformer to layer ℓ.The procedure computes layer representations for training inputs and learns the observer parameters from those representations and labels.Source: A. Designing the Feature Observer · p. 4

Observer design

  • Continuous features are learned with regression, and the observer can be generalized with a known monotonic nonlinearity.The linear-map choice is motivated by the linear separability hypothesis.Source: A. Designing the Feature Observer · p. 4

Robustness

  • An observer learned by Algorithm 1 is not guaranteed to be robust, so the method empirically checks sensitivity to representation perturbations.The check tests whether a small perturbation produces a bounded change in the estimated feature.Source: A. Designing the Feature Observer · p. 4

B. Designing the Feature Controller

The feature controller applies an additive intervention to a layer representation, choosing the smallest perturbation that places the observed feature in a desired target region. The tractable closed-form construction assumes a linear observer and a one-dimensional bounded target interval.

Controller formulation

  • The controller modifies a layer representation through a linear intervention x̃ℓ = gℓ(xℓ).The intervention adds a perturbation uℓ to the original representation.Source: B. Designing the Feature Controller · p. 4

Optimal intervention

  • uℓ is the minimal additive intervention that places the modified representation in the observer’s preimage of the desired feature set.This makes the smallest representation-space change that steers observed features into a target region.Source: B. Designing the Feature Controller · p. 4

Tractable solution

  • The closed-form solution assumes a linear observer and a one-dimensional target set D = [ζmin, ζmax].Without these assumptions, the constraint may be nonconvex and harder to solve tractably.Source: B. Designing the Feature Controller · p. 4

Observability and controllability

  • Effective control is strongest when a feature is both observable and controllable, although the two properties are independent.A feature may be observable without being controllable, or controllable without being observable.Source: B. Designing the Feature Controller · p. 4

C. Observing and Controlling Features in VLAs

The observer and controller are integrated into the VLA’s inference-time forward pass to control selected features and align downstream outputs. Linear computations add negligible runtime while preserving the closed-loop setting under an in-distribution condition.

Observer-controller pipeline

  • The method uses a linear observer with pre-inference parameters and a linear controller whose intervention is computed from the current representation.These components are inserted into the layerwise forward pass to align the VLA output.Source: C. Observing and Controlling Features… · pp. 4–5

Evaluation setup

  • Figure 3 compares classifier MAE with mean-prediction MAE and classifier accuracy with majority-class accuracy for π0.5 on Libero and OpenVLA on BridgeData V2.Only the best-performing layer is shown for each model and dataset.Source: Figure 3 · p. 5

Inference integration

  • The inference procedure computes observations at selected layers and applies control interventions at a subset of those layers.The controller layer set must be contained within the observer layer set because control requires observations.Source: C. Observing and Controlling Features… · p. 5

Efficiency

  • Linear observer and closed-form controller computations introduce negligible runtime overhead while providing steering capabilities.The added computations occur during the forward pass rather than through fine-tuning or retraining.Source: C. Observing and Controlling Features… · p. 5

Closed-loop condition

  • The framework carries over to closed-loop VLAs provided that operation does not move inputs out of distribution relative to the probe-training data.VLA outputs affect the environment, which supplies inputs for subsequent actions.Source: C. Observing and Controlling Features… · p. 5

V. RESULTS

Experiments on OpenVLA and π0.5 examine whether internal representations support feature observation and control. The reported results connect linear observation and minimal representation interventions with fine-grained output steering and alignment with preferences and safety requirements.

Models

  • Experiments cover two frontier VLA models, OpenVLA and π0.5.The evaluation studies feature-observability and feature-controllability in both architectures.Source: §V · p. 5

Feature observation

  • Linear observers recover actions and states from representation space, with observations reported as robust to perturbations.The results section presents observation and robustness before studying control interventions.Source: §V · p. 5

Feature control

  • Minimal representation-space control interventions produce fine-grained steering of VLA outputs.The reported experiments evaluate how targeted interventions affect generated behavior.Source: §V · p. 5

Alignment

  • The reported steering improves alignment with user preferences and safety requirements.This conclusion is stated as part of the results across the evaluated VLA models.Source: §V · p. 5

A. Feature-Observability

The experiments test whether robot states and actions can be observed with linear classifiers and whether those observations remain robust under representation perturbations.

  • The observability experiments assess linear classification of robot states and actions and robustness to representation changes.Source: A. Feature-Observability · pp. 5–6
  • For π0.5 on Libero, perturbation effects are measured across layers using changes in delta yaw and delta gripper actions.Source: Figure 4 · p. 6

1) Actions and States are Linearly Observable in Representation Space:

Linear probes extract state and action features from VLA representations, while perturbation tests show that most observations are robust and can influence outputs.

1) Optimal Control Interventions Constrain Actions to a Desired Region of Representation Space:

The minimal controller places observed representations within desired feature bounds and steers gripper state, height, and speed during closed-loop manipulation while retaining task performance.

VI. CONCLUSION

The paper establishes that targeted linear interventions can steer behavior in π0.5 and OpenVLA while preserving naturalness and closed-loop capabilities, with negligible computational overhead. It also identifies important limitations involving labeled data, architectural coverage, higher-level features, and safety guarantees.

Conclusion

  • The framework formalizes feature observability and controllability to interpret and steer VLA behavior through internal representations.It bridges mechanistic-interpretability ideas from LLMs with embodied-AI challenges.Source: §VI · p. 8
  • Experiments on π0.5 and OpenVLA across Libero and BridgeData V2 show fine-grained behavior steering while preserving naturalness and closed-loop capabilities.The evaluation covers two VLA architectures and multiple robotic manipulation datasets.Source: §VI · p. 8
  • The approach introduces negligible computational overhead, supporting practical real-time robotic applications.The claimed efficiency is presented as a deployment advantage of the targeted interventions.Source: §VI · p. 8

Limitations and Future Work

  • The current approach requires labeled data for training linear observers and primarily addresses transformer components rather than diffusion or flow-matching heads.Extending the framework across hybrid architectures and reducing dependence on labels are identified as future directions.Source: §VI · p. 8
  • Higher-level semantic features and principled safety guarantees for representation-space interventions remain open questions for broader deployment.The paper specifically names task goals, object affordances, spatial relationships, and safety-critical applications as areas needing further study.Source: §VI · p. 8

Conclusion

  • The results position lightweight online adaptation as a step toward more transparent, controllable, and human-aligned embodied AI systems.This conclusion is framed around reliable deployment in real-world environments.Source: §VI · p. 8

Related papers

Loading 2603.05487v1…