Source-linked AI summary
Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, Chelsea Finn
TL;DR
Generalist robots need to reason over complex instructions and feedback in physical contexts, beyond executing atomic commands. Hi Robot uses hierarchical VLMs to translate open-ended prompts and interactions into executable low-level actions, and its evaluation across diverse robots and tasks reports stronger alignment with human intent and task success than API-based VLMs and flat VLA policies.
Problem
Robotic instruction-following methods largely focus on atomic commands, while open-world tasks require situated reasoning over complex prompts, multi-stage goals, and user feedback.
Method
Hi Robot hierarchically uses a high-level VLM to interpret observations, prompts, and feedback and a low-level VLA to execute generated atomic commands.
Results
Hi Robot surpasses API-based VLMs and flat VLA policies in alignment with human intent and task success across diverse robots, tasks, and user constraints.
Takeaways & Limitations
The system supports open-ended instruction following, situated feedback adaptation, and new skill combinations across single-arm, dual-arm, and mobile robots.
Takeaways & Limitations
High-level training relies partly on prompt engineering for synthetic examples, and the high- and low-level models are decoupled from each other’s capabilities.
Abstract
from arXiv · showhide
Generalist robots that can perform a range of different tasks in open-world settings must be able to not only reason about the steps needed to accomplish their goals, but also process complex instructions, prompts, and even feedback during task execution. Intricate instructions (e.g., "Could you make me a vegetarian sandwich?" or "I don't like that one") require not just the ability to physically perform the individual steps, but the ability to situate complex commands and feedback in the physical world. In this work, we describe a system that uses vision-language models in a hierarchical structure, first reasoning over complex prompts and user feedback to deduce the most appropriate next step to fulfill the task, and then performing that step with low-level actions. In contrast to direct instruction following methods that can fulfill simple commands ("pick up the cup"), our system can reason through complex prompts and incorporate situated feedback during task execution ("that's not trash"). We evaluate our system across three robotic platforms, including single-arm, dual-arm, and dual-arm mobile robots, demonstrating its ability to handle tasks such as cleaning messy tables, making sandwiches, and grocery shopping. Videos are available at https://www.pi.website/research/hirobot
1. Introduction
Hi Robot targets the gap between simple atomic instruction following and open-ended robotic tasks requiring situated reasoning, multi-stage planning, and adaptation to feedback. It introduces a hierarchical VLM-based system and evaluates it across diverse robots and tasks, reporting improved alignment with human intent and task success.
- Motivation: Open-world robots must interpret diverse commands, corrections, and constraints while adapting their behavior to unfamiliar situations.The motivating example is table tidying with conditional object removal, user corrections, and delicate objects requiring special handling.
- Problem: Complex prompts require contextual language understanding and composition of existing skills, unlike atomic commands such as “pick up the coke can.”The paper uses vegetarian and separate meat sandwiches as an example of context-dependent, multi-stage instruction following.
- Problem: Prior robotic instruction-following work largely addressed System 1-level behaviors based on atomic instructions, leaving higher-level reasoning over long-horizon tasks and feedback less addressed.System 2 is characterized as parsing complex tasks, interpreting feedback, and selecting an appropriate course of action.
- Approach: Hi Robot combines a high-level VLM for complex prompts and feedback with a low-level VLA for atomic robot actions.The high-level policy generates verbal responses and commands that the low-level policy executes.
- Evaluation: Hi Robot is evaluated on single-arm, dual-arm, and mobile robots across table cleaning, sandwich making, and grocery shopping.The evaluation includes new combinations of skills and scenarios involving diverse objects and user constraints.
- Results: Hi Robot surpasses API-based VLM and flat VLA approaches in alignment with human intent and task success.The reported capabilities include handling complex prompts and incorporating feedback during execution.
2. Related Work
Related work includes directly trained VLM controllers, language-model planners over predefined skills, and systems for language interaction and feedback. These approaches generally address simpler commands, have limited visual-context reasoning, or rely on specialized recovery mechanisms, whereas Hi Robot targets open-ended prompts and corrections.
- VLMs for Robotic Control: Directly trained VLM controllers have demonstrated generalization and instruction following but typically use relatively simple commands.Examples include commands such as placing a cup on a plate.
- VLMs with Predefined Skills: Language-model planners decompose multistage tasks into simpler steps, but earlier systems had limited ability to incorporate complex image observations into reasoning.These systems combined language models with learned or hand-designed skills.
- Language Interaction: Robot language-interaction methods include symbolic scene grounding and newer hierarchical learning systems that process feedback directly.The cited work spans model-based and learning-based approaches.
- Positioning: Hi Robot uses VLMs and a new data-generation scheme to support diverse prompts and open-ended corrections from real robot demonstrations.Unlike RACER, the approach does not use a physics simulator to construct recovery behaviors.
3. Preliminaries and Problem Statement
The paper formulates robot control as predicting action chunks from visual observations, robot state, and language. It motivates VLA models for grounding complex language in images while noting that standard VLAs generally remain limited to atomic commands.
- Policy Formulation: A learned policy maps observation inputs to an action chunk containing the next H actions to execute.The action chunk is denoted At = [at, at+1, ..., at+H−1].
- Policy Formulation: Robot observations combine multiple camera images, robot configuration, and a language prompt to define the policy input.The robot configuration includes joint and gripper positions.
- VLMs and VLAs: Vision-language models represent distributions over language conditioned on image and language inputs, while VLAs adapt this framework to predict robot actions.Standard VLAs represent actions as tokens in the language suffix, and π0 additionally supports continuous action chunks through flow matching.
- Problem Setting: Complex multi-stage tasks require policies to interpret intricate language and ground it through observations of the environment.This requirement motivates using VLA models initialized with vision-language pretraining.
- Hierarchical Formulation: The hierarchical VLA overview separates a high-level policy that generates language commands from a low-level policy that converts those commands, images, and robot states into actions.The low-level policy can also produce verbal responses.
- Problem Setting: Standard VLAs can follow varied language prompts but are typically limited to simple, atomic commands and do not handle the complex prompts and feedback studied here.The distinction motivates adding a higher-level reasoning process.
4. Hi Robot
Hi Robot decomposes control into slower high-level reasoning over open-ended instructions and interactions and faster low-level action generation. It trains the high-level policy with human demonstrations augmented by synthetic prompts and responses, then uses a VLA for execution.
- Hierarchical Policy: The low-level VLA produces action chunks from simpler language commands, while the high-level VLM translates the overall prompt and context into those commands.The two processes run at different rates, with low-level inference operating more frequently.
- Hierarchical Policy: The high-level policy uses images and the open-ended prompt as input and produces an intermediate language command for the low-level policy.The intermediate command is the high-level VLM’s output suffix.
- Interaction Timing: High-level inference is recomputed after one second or immediately when a new user interaction occurs.This simple schedule provides reactive behavior to feedback while maintaining implementation simplicity.
- User Interaction: Users can intervene with text or speech, and the high-level policy recomputes commands while using current images to ground contextual feedback such as “that’s not trash.”The policy may also include verbal confirmations or clarifications.
- Data Generation: Training starts from teleoperated demonstrations segmented into short skills, then uses a VLM to synthesize plausible user prompts, interjections, and robot utterances.The synthetic examples pair visual context and skill labels with imagined interactions.
- Training: The high-level policy trains on synthetic and labeled image-language data, whereas the low-level policy trains on labeled demonstrations with a flow-matching objective.The resulting framework uses distinct learning objectives for reasoning and action generation.
- Implementation: Both policies start from PaliGemma-3B, with π0 supplying the low-level continuous-action VLA and a fine-tuned VLM predicting high-level commands.The framework remains modular and can integrate alternative language-conditioned policies.
5. Experiments
The experiments evaluate Hi Robot on complex instructions, live feedback, and novel task variations across table bussing, sandwich making, and grocery shopping. Comparisons and ablations examine performance against prior methods and the roles of synthetic data and hierarchical reasoning.
- Evaluation goals: The evaluation combines multi-stage instructions, live mid-task feedback, and novel task variations across three complex problem domains.The domains include table bussing, sandwich making, and grocery shopping.
- Task domains: Table bussing requires sorting dishes and trash while handling difficult grasps, object singulation, and contextual corrections such as “that’s not trash.”The robot must sometimes alter default cleanup behavior and leave specified objects untouched.
- Comparisons: Hi Robot outperforms GPT-4o and flat VLA across Table Bussing, Sandwich Making, and Grocery Shopping, averaging over 40% higher instruction accuracy than GPT-4o.The comparison also reports that Hi Robot approaches expert human guidance.
- Task domains: Sandwich making tests complex ingredient instructions and mid-task updates while requiring careful manipulation of deformable, delicate ingredients.Examples include adding lettuce and then stopping when the user says, “this is all I want.”
- Core results: Across tasks and robot platforms, Hi Robot achieves higher Instruction Accuracy and Task Progress while respecting object-specific constraints and changing user requests.The system adapts to feedback, whereas GPT-4o and the flat baseline often lose context or revert to default behaviors.
- Ablations: Synthetic interactions improve language flexibility, and hierarchical reasoning outperforms a flat policy trained on the same data for clarifications, partial instructions, and mid-task updates.Without synthetic data, the ablated model ignores clarifications or includes forbidden items; the flat model often reverts to clearing all items.
6. Discussion and Future Work
Hi Robot separates deliberative reasoning over prompts and feedback from reactive execution of atomic actions using two VLM-based layers. The authors identify tighter coupling and more adaptive cross-level processing as future directions.
- Discussion: Hi Robot uses a high-level VLM to interpret complex prompts and feedback, then a low-level VLM to execute atomic robot actions.The architecture instantiates deliberative System 2 and reactive System 1 roles.
- Discussion: The high-level policy uses semantic and visual knowledge from web-scale pre-training to reason about user interactions and select the next step.
- Future Work: The current system runs high-level inference at a fixed lower frequency; adaptive asynchronous processing could support more flexible multi-level reasoning.
- Future Work: The training process relies partly on prompt engineering for synthetic examples and decouples the high- and low-level models during training.The layers are not aware of one another’s capabilities except through training examples.
- Future Work: Future work could couple the layers more directly by making the high-level policy aware of how successfully the low-level policy completes each command.
Impact Statement
The paper concerns advancing machine learning and robotics and acknowledges potential societal consequences without identifying any that require specific emphasis.
- The authors state that the work aims to advance machine learning and robotics.
- The impact statement acknowledges potential societal consequences but does not highlight any specific consequence.
A. Synthetic Data Generation
The synthetic-data pipeline generates diverse, context-aware user interactions for training high-level policies, while the reported setup uses separate data and policies per task. The system also integrates speech interaction and real-time control infrastructure.
- Scenario Classification: Synthetic interactions are categorized into negative tasks, situated corrections, and specific constraints to improve data quality and diversity.
- Prompt Design: Prompt generation conditions responses on task descriptions, visual observations, and prior context, enabling interactions that reflect world knowledge and constraints.The paper gives dietary constraints in sandwich-making as an example.
- Episode Consistency: Conditioning generation on prior skill labels produces coherent commands that account for earlier actions in a multi-step episode.
- Task Scope: The authors generate separate synthetic datasets and high-level policies for each task, while noting that the architecture could support unified multi-task training.They suggest this could facilitate knowledge transfer between task domains.
- System Integration: The system supports speech-based interaction through local Whisper large-v2 transcription and Cartetia text-to-speech synthesis.
- System Integration: Real-time inference uses one to two consumer-grade NVIDIA GeForce RTX 4090 GPUs.
High-Level Policy (Single Decoding Step)
The system is evaluated with three robot configurations spanning single-arm, bimanual, and mobile manipulation capabilities. Latency measurements indicate real-time feasibility, with action chunking supporting a higher control rate.
- Latency: 47 ms prefill plus 13.2 ms decode on RTX 4090, versus 17.3 ms plus 5.7 ms on H100, are the reported latency measurements.
- Latency: The measurements support approximately 10 Hz control, while action chunking enables control at 50 Hz.
- Robot Configurations: The UR5e uses a 6-DoF arm, parallel jaw gripper, wrist and over-the-shoulder cameras, and a 7-dimensional configuration and action space.
- Robot Configurations: Bimanual ARX combines two 6-DoF arms, three cameras, and a 14-dimensional configuration and action space for bimanual manipulation.
- Robot Configurations: Mobile ARX adds a mobile base to two 6-DoF arms, yielding a 14-dimensional configuration space and 16-dimensional action space.Its wrist and base cameras provide visual feedback for navigation and manipulation.
C. Model and Experiment Details
The method can use different VLM backbones, with PaliGemma selected in practice, and is trained with specific optimization settings.
- PaliGemma, an open-source 3-billion-parameter VLM, is used as the base model for computational efficiency and performance balance.
- The full model is unfrozen during fine-tuning, allowing all PaliGemma parameters to be updated.
- Training uses AdamW with β1 = 0.9, β2 = 0.95, no weight decay, gradient clipping at 1, and EMA decay of 0.999.
- The learning rate warms up for 1,000 steps before remaining constant at 1 × 10^-5, with batch size 512.
C.3. Training Duration and Resources
The high-level policy trains efficiently, while low-level training time depends on dataset size and task complexity.
- Approximately 2 hours on 8×H100 GPUs are required to train the high-level policy.
- Low-level policy training follows a similar pipeline, but its duration varies with dataset size and the complexity of target action-prediction tasks.
C.4. Failure Cases
The reported failure cases include long-context reasoning difficulties and low-level instruction-following and recovery errors; proposed mitigations target modeling and data diversity.
- Failure Cases: The high-level policy struggles with instructions requiring long-context reasoning because the current system lacks memory.
- Failure Cases: The low-level policy may temporarily ignore user instructions, such as grabbing cheese despite a stated lactose intolerance constraint.
- Failure Cases: Low-level errors can accumulate and leave the robot needing out-of-distribution recovery, including after dropped objects.
- Mitigations: Suggested mitigations include stronger or long-context instruction-following models, adversarial edge-case data, and diverse recovery-focused data collection.
- GPT-4o Baseline Prompt: The GPT-4o baseline uses a task- and robot-specific system prompt with a provided instruction list, camera descriptions, and strict output formatting.