Source-linked AI summary
Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control
William Chen, Jagdeep Singh Bhatia, Catherine Glossop, Nikhil Mathihalli, Ria Doshi, Andy Tang, Danny Driess, Karl Pertsch, Sergey Levine
TL;DR
The paper addresses the limited ability of hierarchical VLM systems to ground high-level reasoning in low-level robot behavior. It trains Steerable Policies on synthetic commands spanning subtasks, motions, and grounded coordinates, then uses learned and off-the-shelf VLM reasoners to control them. Across the reported experiments, this combination improves task generalization and supports strong performance across command styles.
Problem
Hierarchical VLM systems commonly communicate with low-level VLAs through limited natural-language task instructions, restricting how pretrained reasoning can steer robot behavior.
Method
Steerable Policies train VLAs on synthetic commands spanning subtasks, motions, grounded pixel coordinates, and compositions, then connect them to learned embodied reasoners and in-context VLMs.
Results
The two VLM-control methods outperform prior baselines across reported generalization and long-horizon experiments, while unrestricted human steering solves all tasks at nearly 100% success rate.
Takeaways & Limitations
Effective transfer of VLM capabilities to robotics requires low-level policies that can follow a spectrum of steering abstractions rather than one command style.
Abstract
from arXiv · showhide
Pretrained vision-language models (VLMs) can make semantic and visual inferences across diverse settings, providing valuable common-sense priors for robotic control. However, effectively grounding this knowledge in robot behaviors remains an open challenge. Prior methods often employ a hierarchical approach where VLMs reason over high-level commands to be executed by separate low-level policies, e.g., vision-language-action models (VLAs). The interface between VLMs and VLAs is usually natural language task instructions, which fundamentally limits how much VLM reasoning can steer low-level behavior. We thus introduce Steerable Policies: VLAs trained on rich synthetic commands at various levels of abstraction, like subtasks, motions, and grounded pixel coordinates. By improving low-level controllability, Steerable Policies can unlock pretrained knowledge in VLMs, enabling improved task generalization. We demonstrate this benefit by controlling our Steerable Policies with both a learned high-level embodied reasoner and an off-the-shelf VLM prompted to reason over command abstractions via in-context learning. Across extensive real-world manipulation experiments, these two novel methods outperform prior embodied reasoning VLAs and VLM-based hierarchical baselines, including on challenging generalization and long-horizon tasks. Website: steerable-policies.github.io
I. INTRODUCTION
The paper argues that transferring VLM knowledge to robotics is limited by low-level policy steerability and introduces VLAs that accept commands across abstraction levels and grounded modalities. These Steerable Policies support high-level embodied reasoning and in-context VLM control, improving generalization and compositionality.
- Motivation: VLM capabilities remain difficult to ground in robot behaviors when high-level models command separate low-level policies through limited interfaces.The paper identifies insufficient steerability as a bottleneck: useful inferences cannot guide behavior if the policy cannot execute commands specifying them.
- Motivation: Synthetic descriptive commands can expose latent dataset semantics, but grounded features such as pixel coordinates are needed for novel or underspecified objects.Pointing commands can specify behaviors that language alone cannot communicate reliably, including grasping an out-of-distribution object at a location.
- Steerable Policies: Steerable Policies train VLAs to follow task-level, subtask, motion, gripper-trace, point, and compositional commands generated by an automated labeling pipeline.The command spectrum ranges from “put the carrot in the pot” to atomic motions and grounded coordinate sequences.
- Hierarchical control: High-level VLMs can either be fine-tuned to produce chain-of-thought reasoning and steering commands or prompted off the shelf to select commands through in-context learning.Both approaches control the same low-level Steerable Policy and choose instruction abstractions suited to the situation.
- Contributions: The approach expands prior hierarchical interfaces and improves transfer of pretrained VLM capabilities, with demonstrations using OpenVLA and π0.5.The paper frames steerability as enabling high-level reasoners to select commands at appropriate abstraction levels for generalization and compositionality.
III. PRELIMINARIES
The paper formulates hierarchical control as a high-level policy producing an intermediate goal for a low-level VLA, then extends the low-level interface beyond ordinary task prompts. Steerable Policies combine diverse textual commands with grounded coordinates to support flexible, generalizable behaviors.
- Vision-language-action models: A VLA learns π(a|o, l) from observations and task prompts, while a Steerable Policy is trained to accept diverse steering modalities.The policy remains a vision-language-action model but broadens its input command space.
- Hierarchical policies: Hierarchical control maps a task and observation to an intermediate goal, which conditions low-level action prediction; here, steering commands serve as that goal.The high-level policy outputs g, and the Steerable Policy samples actions conditioned on g and the observation.
- Steerable Policies: Steerable Policies train VLAs on subtasks, motions, and grounded coordinates so one controller can execute commands across multiple abstraction levels.These commands replace standard task-level labels during behavioral-cloning training.
- Grounded steering: Pixel-coordinate commands provide fine-grained control when object names are out of distribution or language leaves multiple instances ambiguous.Examples include picking up an object at a specified position or selecting the correct instance by pointing.
B. Generating Steering Commands at Scale
The paper uses automated foundation-model annotation to extract grounded trajectory features, decompose episodes, and generate diverse steering commands and rationales. These annotations train both Steerable Policies and high-level VLM controllers that issue commands using embodied reasoning or in-context learning.
- Motivation: Human annotation of grounded features is expensive, motivating synthetic annotations for scalable steering-command generation.The method targets labels involving features such as pixel coordinates, which are especially labor-intensive to acquire manually.
- Automated annotation: The pipeline extracts motions, object masks, bounding boxes, gripper traces, and semantic subtasks from robot trajectories using multiple foundation models.Molmo, SAM2, DETR, and Gemini are used for object grounding, tracking, gripper traces, and subtask decomposition.
- Command generation: Gemini restates each subtask into equivalent steering commands across all styles, using grounded features such as traces, motions, and object centroids.The extracted features are supplied as inputs when generating commands.
- Steerable Policy training: Training replaces standard behavioral-cloning language with a uniformly sampled steering command, producing a VLA that follows the full command spectrum.The paper reports that different command styles suit different tasks and situations.
- High-level control: High-level VLM controllers are trained or prompted to understand Steerable Policy affordances and issue commands for embodied reasoning and multi-step problem-solving.The fine-tuned model predicts rationales followed by commands, while the off-the-shelf model reasons over observations, command history, and abstraction level.
B. In-context Reasoning for Choosing Steering Abstractions
Steerable Policies let high-level VLMs choose among command abstractions while using scene understanding and interaction history to guide robot behavior. In-context selection of these abstractions improves control beyond subtask-only interfaces, while oracle results show that command styles have complementary strengths.
- In-context abstraction selection: Steerable Policies let high-level models reason about both the robot’s next action and the abstraction level best suited to induce it.The interface supports task-level, subtask, motion, trace, and pointing commands.
- In-context abstraction selection: An off-the-shelf VLM receives observations, task instructions, command-style examples, and histories of prior commands and outcomes to select steering commands.It parses the scene, determines the next behavior, and reasons about the appropriate command style.
- In-context abstraction selection: Because the VLM observes resulting behavior, it can adapt later commands without hand-crafted robot-behavior examples.The VLM predicts steering commands rather than robot actions, enabling sequential in-context adjustment.
- In-context abstraction selection: Scene understanding can guide abstraction choice: clutter may favor pointing or motion commands because they provide greater specificity.The paper connects visual scene interpretation with selecting a suitable steering modality.
- Experimental scope: The approach is evaluated on Bridge WidowX manipulation tasks across standard generalization axes and longer-horizon tasks for in-context control.The experimental design includes in-distribution, motion, spatial, semantic, and unseen longer-horizon evaluations.
- Oracle steering results: Oracle users solve nearly 100% of tasks with unrestricted commands, while each single style outperforms task-level language but has complementary strengths and weaknesses.Trace and point commands help semantic generalization, motions help spatial relations, and task/subtask commands remain reliable for in-distribution subtasks.
B. Steerable Policies Enable VLMs to Effectively Use Embodied Reasoning Training
The paper fine-tunes high-level VLMs to control Steerable Policies with embodied reasoning and compares them against standard, reasoning-enhanced, and non-reasoning baselines. The learned approach outperforms all baselines, with especially pronounced gains on motion and semantic generalization.
- Learned embodied reasoning: The learned high-level policy produces embodied reasoning and intermediate goals to control Steerable Policies, alongside a direct-command non-reasoning ablation.Comparisons use matched Bridge tasks, demonstrations, and architectures across standard OpenVLA, π0.5, ECoT, and ECoT-Lite baselines.
- Learned embodied reasoning: The approach outperforms five baselines on the shared Bridge task suite used for embodied-reasoning VLA comparisons.The baselines include standard OpenVLA and π0.5, Reasoning Pretraining and Dropout ECoT-Lite, and full ECoT.
- Generalization results: Gains are most pronounced on motion and semantic generalization tasks, where atomic motion, trace, and pointing commands support control under distribution shift.These command types give the learned reasoner more reliable ways to steer the policy beyond familiar task distributions.
- Architecture comparison: High-level VLM reasoning improves Steerable Policies built on both π0.5 and OpenVLA, outperforming their corresponding standard baselines.The result indicates that the benefit is not tied to a single VLA architecture.
- Ablation: Removing explicit reasoning still greatly outperforms standard OpenVLA and is on par with ECoT-Lite, while the full method performs better.The comparison separates benefits from hierarchical steering and benefits from explicitly trained embodied reasoning.
C. Steerable Policies Unlock In-context Reasoning in VLMs
In-context VLMs can improve robotic behavior by selecting and revising steering abstractions for Steerable Policies. Compared with subtask-only control, richer steering enables corrective, fine-grained, and dynamically sequenced commands.
- Evaluation: The full method universally outperforms OpenVLA and SayCan-like baselines, while also exceeding the non-reasoning ablation.The SayCan-like baseline differs primarily by restricting the VLM to subtask-level prompts.
- Corrective steering: A pointing command can fail on the wrong banana, after which the VLM switches to atomic motion commands to guide the robot toward the correct object.This example illustrates in-context discovery of a more effective steering abstraction.
- Corrective steering: In-context learning lets VLMs revise ineffective commands by changing the steering abstraction level.The VLM uses task history to adjust its prompting strategy when commands cause errors or fail to make progress.
- Fine-grained correction: Steerable Policies let VLMs convert semantic and physical scene understanding into corrective commands such as moving toward the correct hammer handle.The VLM reasons about both object identity and end-effector clearance before issuing the correction.
- Abstraction selection: VLMs can sequence steering abstractions, using a pointing command for an ambiguous cluttered reach before switching to a higher-level grasp instruction.This demonstrates dynamic selection of command style based on the current state.
VII. DISCUSSION AND FUTURE WORK
The paper presents Steerable Policies as a broad command interface for transferring VLM capabilities into robotics. It argues that future applicability depends on behavioral diversity and better VLM understanding of steering affordances.
- Contributions: Steerable Policies support subtasks, motions, and grounded pixel coordinates, enabling high-level VLMs to transfer capabilities into robotic control.The paper demonstrates this through embodied-reasoning fine-tuning and off-the-shelf VLM in-context learning for multi-step problem-solving.
- Future work: The authors expect Steerable Policies to become more applicable as robot datasets increase behavioral diversity.They connect this expectation to the compositionality needed for increasingly complex and open-world tasks.
- Future work: Future VLMs may learn steering affordances through reinforcement learning and cross-task in-context learning rather than relying only on zero-shot prompting.Rollouts could help models identify when each steering style is effective.
APPENDIX A SYNTHETIC GENERATIONS IN BRIDGE
The appendix describes how Bridge trajectories are transformed into synthetic steering data and how the in-context VLM evaluation is configured. The pipeline combines subtask decomposition, object and gripper localization, segmentation, and prompt-based command selection.
- Synthetic generation: Each Bridge episode is decomposed into subtasks with corresponding steering commands and post-hoc rationales.Gemini generates subtask decompositions and explains why each subtask fits its starting observation.
- Coordinate extraction: Object coordinates are identified with Molmo models, propagated through trajectories with SAM 2, and converted into PaliGemma-compatible location and segmentation tokens.The pipeline first detects relevant objects, predicts initial coordinates, and then maintains temporal localization.
- Coordinate extraction: Gripper positions are labeled by a DETR model trained on 100 manually annotated Bridge images.The trained detector supplies gripper centroids across the dataset.
- VLM evaluation: The in-context VLM receives task instructions, command-style explanations, examples, and interaction history before producing a command executed for N = 20 steps.The VLM is then queried again, and grounded descriptions can be converted into coordinates through another Gemini call.
- Baselines: The SayCan-like baseline preserves the prompt structure and reasoning budget but restricts examples to subtask-level commands.The full method and non-reasoning baseline differ in access to command abstractions or explicit reasoning instructions, respectively.
- Evaluation design: Evaluation covers in-distribution, motion, spatial, and semantic generalization, with more than 650 robot rollout episodes across the evaluation suites.The in-context experiments replace in-distribution tasks with long-horizon tasks and use rubric-based task progress.
A. Human Oracle Experiments
The experiments evaluate human-oracle steering and compare hierarchical methods across generalization and long-horizon task suites. Trials use rubric-based progress, while the paper also provides examples of synthetic steering commands and reasoning traces.
- Experimental scope: Human-oracle experiments evaluate steering across in-distribution, motion, spatial, and semantic generalization settings.The human-oracle tasks differ from the embodied-reasoner tasks to broaden the tested settings.
- Long-horizon evaluation: The in-context VLM suite replaces in-distribution tasks with two long-horizon tasks that compose many in-distribution behaviors.Its motion-generalization tasks also involve a novel behavior requiring challenging multi-step motions.
- Evaluation protocol: Each trial scores rubric items as success or failure, reports average successes as task progress, and penalizes later actions that undo completed progress.Policies run for 20 high-level steps, or 25 for Long Horizon 2, unless all rubric items are completed.
- Reasoning traces: The appendix includes full reasoning traces for the in-context method and points readers to corresponding SayCan-like baseline examples.These examples expand the abbreviated reasoning illustrations shown earlier.
- Synthetic commands: Figure 10 visualizes subtasks alongside diverse steering commands extracted from training data, although labels and points are not present in robot-training images.More than five commands may correspond to a subtask, but only a subset is shown for readability.
APPENDIX E FURTHER DISCUSSIONS
The paper frames steerability as a training-time property shaped by behavioral diversity, while showing that command abstraction affects both intelligent action selection and failure modes. It also identifies limits from ambiguous motions, high-level affordance misunderstandings, and narrow datasets.
- Training versus inference-time steering: Inference-time steering methods optimize generated samples without modifying pretrained policy weights, whereas this work improves steerability through training.The related approaches optimize criteria such as classifier scores, likelihoods, reference similarity, or Q-functions.
- The manifold of reasonable actions: Atomic motion commands can produce image-conditioned actions aligned with reasonable tasks rather than blind movement in the commanded direction.The policy attends to the scene and samples actions that may accomplish tasks compatible with the requested motion.
- The manifold of reasonable actions: Ambiguous motions can select the wrong object when multiple objects satisfy the same directional command, motivating multiple steering styles.Other command abstractions can supplement motion commands when their underspecification creates confusion.
- Failure modes: Subtask-only interfaces may leave a low-level VLA unable to execute corrective behavior even when the high-level VLM diagnoses the error.The paper contrasts repeating a subtask with issuing a more specific motion such as moving up and left.
- Failure modes: Failure modes often arise because the high-level VLM does not understand which command abstraction the low-level policy can follow at the current state.The VLM may issue the correct next step while the VLA repeats or undoes prior progress, even though a motion command could avoid the failure.
- Dataset scope: The approach depends on behavioral diversity in training data and is difficult to apply to narrow datasets such as LIBERO.The paper attributes this boundary to unimodal trajectories, minimally randomized starts, and degraded performance under small perturbations.
A. Steerable Policy Training
The Steerable Policy is trained by adapting OpenVLA and randomly substituting task, subtask, and generated steering commands for each trajectory frame. A second policy based on π0.5 follows the same command-training strategy with its respective recipe.
- OpenVLA-based policy: The OpenVLA-based Steerable Policy trains for 80k steps with batch size 256 across 8 H100 GPUs, using a DINOv2-SigLIP encoder and Llama2 7b VLM.These settings adapt OpenVLA’s Bridge training configuration.
- Command sampling: For each frame, training uniformly samples among Bridge task language, subtask language, and corresponding generated steering commands.The sampled command replaces the standard task-level label while retaining OpenVLA’s next-token prediction loss.
- Action representation: The action representation discretizes each of seven action dimensions into 256 bins corresponding to selected Llama vocabulary tokens.This is the standard OpenVLA action-tokenization scheme used by the policy.
- π0.5-based policy: The π0.5-based Steerable Policy uses the same training procedure with OpenPi’s DROID fine-tuning defaults, running 30k gradient steps on 8 H200 GPUs.Training takes less than a day, and the policy omits proprioception while not using the Knowledge Insulation trick.
B. High-level Embodied Reasoner Training
The high-level embodied reasoner is trained to map observations and task instructions to steering commands, using rationales and synthetic command annotations derived from robot trajectories. The annotation prompts cover subtasks, motions, grounded positions, and combinations of these modalities.
- Reasoner objective: The high-level reasoner is trained with next-token prediction to generate a rationale followed by an appropriate steering command.It uses altered OpenVLA training and predicts the reasoning-command sequence from the task and current observation.
- Reasoner objective: A non-reasoning ablation removes rationale supervision while retaining the same training procedure for direct observation-to-command prediction.This isolates the contribution of intermediate reasoning generations.
- Rationale supervision: The pipeline also supplies task and subtask context for rationale generation, including prompts that identify the current subtask and prior subtasks.Separate examples show rationale supervision for tasks such as placing broccoli on a towel and related object-placement tasks.
- Subtask supervision: Subtask annotations are generated by decomposing demonstrations into object-specific steps such as reaching, grasping, lifting, moving, and releasing.The prompts ask for subtask lists and timestamp groupings based on motion sequences.
- Grounded supervision: The annotation prompts use frame-level gripper motion, gripper coordinates, and object positions to generate possible instructions for each subtask.Generated answers are requested as mappings from subtasks to lists of concrete instructions with numerical positions.
- Steering command modalities: Steering annotations include high-level task instructions, low-level motions, object positions, gripper traces, and combinations of these inputs.Examples range from moving down and closing the gripper to grasping at an object coordinate or following several gripper positions.