PaperlayerView exact revision on arXiv

Source-linked AI summary

HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation

Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, Ankit Goyal

arXiv:2502.05485v4cs.ROcs.AIcs.CV

TL;DR

Robot manipulation lacks sufficiently broad and affordable data, limiting open-world generalization. HAMSTER hierarchically fine-tunes a VLM to predict coarse 2D paths that guide a 3D-aware low-level policy, and reports improved generalization over OpenVLA. The approach uses off-domain data while retaining precise low-level control.

  • Problem

    Robot datasets are expensive to collect on hardware and remain limited in size, quality, and diversity, hindering broad manipulation generalization.

  • Method

    HAMSTER connects a fine-tuned VLM predicting coarse 2D end-effector paths with a low-level policy that produces actions conditioned on those paths.

  • Results

    20% average success-rate improvement over OpenVLA across seven generalization axes represents a 50% relative gain; HAMSTER also outperforms monolithic VLAs by over 2x on average.

  • Takeaways & Limitations

    Hierarchical decomposition enables off-domain data to support real-world control while specialist low-level policies execute precise, spatially aware actions.

  • Takeaways & Limitations

    The 2D-path interface provides no native 3D prediction and cannot communicate nuances such as force or rotation.

Abstract

from arXiv · show

Large foundation models have shown strong open-world generalization to complex problems in vision and language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is the lack of robotic data, which are typically obtained through expensive on-robot operation. A promising remedy is to leverage cheaper, off-domain data such as action-free videos, hand-drawn sketches or simulation data. In this work, we posit that hierarchical vision-language-action (VLA) models can be more effective in utilizing off-domain data than standard monolithic VLA models that directly finetune vision-language models (VLMs) to predict actions. In particular, we study a class of hierarchical VLA models, where the high-level VLM is finetuned to produce a coarse 2D path indicating the desired robot end-effector trajectory given an RGB image and a task description. The intermediate 2D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Doing so alleviates the high-level VLM from fine-grained action prediction, while reducing the low-level policy's burden on complex task-level reasoning. We show that, with the hierarchical design, the high-level VLM can transfer across significant domain gaps between the off-domain finetuning data and real-robot testing scenarios, including differences on embodiments, dynamics, visual appearances and task semantics, etc. In the real-robot experiments, we observe an average of 20% improvement in success rate across seven different axes of generalization over OpenVLA, representing a 50% relative gain. Visual results, code, and dataset are provided at: https://hamster-robot.github.io/

1 INTRODUCTION

HAMSTER addresses limited and costly robot data by hierarchically combining large VLMs with dexterous low-level policies through 2D path guidance. This design uses inexpensive off-domain data while improving generalization across substantial domain gaps.

  • Motivation: Robot datasets remain limited in size, quality, and diversity because on-robot observation-action pairs are expensive to collect.Monolithic VLAs depend on such datasets to predict robot actions directly.
  • Motivation: Small policy models provide dexterous, precise control but are often brittle to major environmental or task-semantic changes.They can also struggle with simulation-to-real transfer because of visual and dynamics gaps.
  • Approach: HAMSTER connects a fine-tuned VLM to a low-level policy through embodiment-agnostic 2D path representations.The paths encode coarse end-effector trajectories and gripper-state changes.
  • Approach: The hierarchy lets the high-level VLM use cheap videos or simulations while the low-level policy exploits rich 3D and proprioceptive inputs for precise actions.This separates semantic trajectory prediction from fine-grained control.
  • Results: 20% average success-rate improvement over OpenVLA across seven generalization axes corresponds to a 50% relative gain.HAMSTER is presented as a family of hierarchical VLA models that can use abundant off-domain data for real-world control.

2 RELATED WORK

Related work spans monolithic VLAs, intermediate visual representations, trajectory-conditioned policies, and simulation-based robot learning. HAMSTER builds on these directions while emphasizing hierarchical VLA training with off-domain data.

  • Monolithic VLAs: Monolithic VLA models directly produce robot actions from task descriptions and image observations, typically using large on-robot datasets.These models are often constructed from pretrained VLMs.
  • Intermediate representations: Prior VLM-based methods predict intermediate representations such as keypoint affordances instead of directly predicting actions.These representations can be obtained with detectors, prompting, or semantic fine-tuning.
  • Trajectory-based predictions: RT-Trajectory uses trajectory-based task specifications to condition low-level policies and discusses trajectory sketches generated by pretrained VLMs.HAMSTER focuses instead on hierarchical VLA design for leveraging off-domain data.
  • Trajectory-based predictions: Track-any-point models have enabled policies conditioned on object trajectories or image-grid points, while HAMSTER currently predicts end-effector trajectories.The framework can naturally extend to object trajectories or other motion cues.
  • Simulation data: Simulation is used in reinforcement and imitation learning to reduce reliance on expensive real-robot teleoperation data.HAMSTER targets off-domain-data use within a hierarchical VLA architecture.

3 BACKGROUND

The background describes supervised imitation learning and VLMs as foundations for studying improved generalization in robot policies. It frames HAMSTER as using VLMs to enhance imitation learning under novel visual and semantic variations.

  • Imitation Learning via Supervised Learning: Supervised imitation learning trains a policy from expert demonstrations containing proprioceptive inputs, perceptual observations, task instructions, and actions.The policy is optimized by maximum likelihood over the demonstration dataset.
  • Imitation Learning via Supervised Learning: Despite advances such as 3D policy representations, generalization to novel semantic or visual variations remains challenging.The paper explores VLMs as a way to enhance imitation-learning models for better generalization.
  • Vision-Language Models: VLMs accept vision and text tokens, are pretrained on extensive multimodal datasets, and are later fine-tuned on task-specific data.They autoregressively generate text responses from shared multimodal representations.

4 HAMSTER: HIERARCHICAL ACTION MODELS FOR ROBOTIC LEARNING

HAMSTER is a hierarchical VLA framework that uses off-domain data to train a high-level VLM for 2D path guidance and a low-level policy for conditioned action generation. The framework combines semantic reasoning with 3D-aware, locally precise control while supporting cross-domain manipulation.

  • Architecture: HAMSTER connects a high-level VLM trained on off-domain data to a low-level policy that produces actions conditioned on intermediate 2D paths.The two models divide task-level path prediction and low-level action generation.
  • 2D Path Representation: The VLM predicts a coarse end-effector path from a monocular RGB image and language instruction, including normalized image-plane positions and gripper-state changes.The path can represent either a robot end-effector or a human hand in videos.
  • Off-Domain Data: Doff combines pixel-point prediction, simulated robotics, and real-robot trajectory data, enabling VLM training without deployment-environment data.The datasets cover object locations, simulated manipulation paths, and trajectories from robot embodiments distinct from the test robot.
  • Off-Domain Data: 770k pixel-point tasks, approximately 320k RLBench tuples, and 110k real-robot paths comprise the three main off-domain data categories.The real-robot paths come from Bridge and DROID trajectories, while the simulated paths come from RLBench.
  • Path Processing: The framework simplifies very long paths with the Ramer-Douglas-Peucker algorithm so high-level reasoning remains at a coarser scale than low-level control.The authors note that extracted paths can exceed one hundred steps.
  • Low-Level Control: The low-level policy receives 2D paths alongside proprioceptive and perceptual inputs, allowing it to focus on local geometric action prediction rather than long-horizon semantic reasoning.HAMSTER supports 3D perceptual inputs such as depth images and point clouds.

5 EXPERIMENTAL EVALUATION

HAMSTER is evaluated in simulation and real-world settings across visual, semantic, spatial, task, and demonstration-efficiency challenges. Across these evaluations, its hierarchical path-guided design improves generalization, robustness, and data efficiency relative to monolithic VLAs and unguided 3D policies.

  • Evaluation Setup: HAMSTER is evaluated across unseen scenarios, cross-domain generalization, non-prehensile and long-horizon tasks, demonstration efficiency, and visual-semantic reasoning.The evaluation combines real-world and Colosseum simulation experiments with comparisons against OpenVLA, RVT-2, and 3D-DA.
  • Real-World Results: HAMSTER demonstrates long-horizon manipulation on tasks absent from the in-domain policy training set, including towel unfolding, drawer operation, wiping, and table cleaning.These qualitative tasks involve changing lighting, clutter, and semantic understanding.
  • Real-World Results: Over 2x and 3x average performance gains are reported over monolithic VLA models and non-VLM 3D policies, respectively, under substantial visual and semantic changes.HAMSTER outperforms OpenVLA across pick-and-place, press-button, and knock-down tasks.
  • Simulation Evaluation: 31% average improvement over vanilla 3D-DA is observed across Colosseum visual variations, attributed to path-drawn training that focuses policies on paths rather than extraneous visual features.The controlled simulation uses five seeds and evaluates pick-place and non-prehensile tasks.
  • Simulation Evaluation: With 50% of the demonstrations, HAMSTER+3D-DA achieves 2x the success rate of standard 3D-DA on five Colosseum tasks.This result supports demonstration efficiency for downstream imitation learning.
  • VLM Generalization Studies: HAMSTER remains robust to novel camera angles, and concatenating a dedicated path channel gives the best performance while being less compatible with some policy architectures.The dedicated channel preserves path guidance more effectively, but 3D-DA expects a pretrained three-channel image encoder.

6 CONCLUSION AND LIMITATIONS

HAMSTER combines a finetuned VLM that predicts 2D paths with a low-level policy that uses them to generate actions, supporting generalization across domain shifts. The approach remains limited by its 2D-only, bandwidth-limited interface.

  • Conclusion: HAMSTER combines a finetuned VLM that predicts 2D paths with a low-level policy that learns to generate actions from them.The architecture supports semantic reasoning and visual generalization while allowing specialist policies conditioned on 3D inputs to execute low-level actions.
  • Conclusion: The hierarchical design enables visual generalization and semantic reasoning across considerable domain shifts while retaining specialist low-level control.
  • Limitations: The method generates points in 2D space rather than native 3D predictions, preventing the VLM from having true spatial 3D understanding.
  • Limitations: The 2D-path interface is bandwidth limited and cannot communicate nuances such as force or rotation.The paper identifies learnable intermediate interfaces and direct training from large-scale human video datasets as future directions.

A VLM FINETUNING DATASET DETAILS

The VLM finetuning data combine point and VQA supervision with trajectories from simulation and real-robot datasets. Real-robot processing filters or corrects data using camera calibration and depth–RGB alignment checks.

  • Pixel Point Prediction Data: The point-prediction dataset contains 770k samples with unordered points or bounding-box labels, while the VQA dataset contributes 660k natural-language samples.The VQA data are retained to support semantic reasoning and visual generalization, and HAMSTER’s VLM is finetuned on the entire Robopoint dataset.
  • Simulation Data: The simulation dataset contains around 320k trajectories from 81 selected RLBench tasks.Tasks with poor visibility from the front camera were removed from the original 103 tasks.
  • Real Robot Data: Bridge trajectories are calibrated by estimating episode-specific extrinsics, projecting gripper poses into images, and filtering episodes with significant misalignments.The process uses manually labeled gripper centers and a Perspective-n-Point projection approach.
  • Real Robot Data: DROID trajectories with poor camera extrinsics are filtered using projected-depth and RGB-image alignment, producing approximately 45k trajectories.These correspond to approximately 22k unique trajectories because each trajectory has two camera viewpoints.

B IMPLEMENTATION AND ARCHITECTURE DETAILS

The implementation trains a VLM to output 2D paths and gripper-state tokens, then conditions low-level policies on those paths alongside their task-specific inputs. Training includes path simplification and noise to accommodate long trajectories and prediction errors.

  • VLM Prompt: HAMSTER’s VLM receives an image and prompt, and outputs gripper changes as separate Open Gripper or Close Gripper language tokens.The same prompt is used for simulation and real-robot finetuning and evaluation, except for Pixel Point Prediction data, which retain their dataset prompts.
  • VLM Trajectory Processing: Long path labels are simplified with the Ramer-Douglas-Peucker algorithm before training.The algorithm reduces curves to similar curves composed of fewer points and is applied to simulation and real-robot paths.
  • VLM Training Details: The VLM is VILA1.5-13B, trained for about 30 hours on eight NVIDIA A100 GPUs with effective batch size 256 and learning rate 1 × 10^-5.The entire model, including the vision encoder, is updated during finetuning.
  • Low-Level Policies: RVT2 and 3D-DA serve as the lower-level policies, with their inputs modified to incorporate the 2D path.RVT2 removes the language instruction when conditioned on HAMSTER paths, whereas 3D-DA retains language in simulation because its visual attention benefits from detailed instructions.
  • Low-Level Policy Training: Low-level policies train on projected ground-truth paths plus N(0, 0.01) noise on image-plane coordinates to improve robustness to VLM path errors.No noise is added to gripper opening or closing indicators.
  • Real-World Architecture: Real-world low-level training simplifies language, reduces transformer embedding dimension from 120 to 60, removes past-timestep proprioception, and reduces heads from 12 to 6.These changes are made to encourage trajectory following and prevent overfitting.
  • Training Tasks: The real-robot training tasks are pick and place with 220 episodes, knock down objects with 50 episodes, and press button with 50 episodes.The tasks use toy objects, varied objects, and four colored buttons respectively.
  • Robot Constraints: The robot arm input is a 2D pose with gripper status represented as (x, y, gripper open == 1).

C.2 BASELINE TRAINING DETAILS

The baseline comparisons include parameter-efficient OpenVLA finetuning and two RT-Trajectory variants using GPT-4o. Real-world evaluation covers multiple task and generalization variations, with success scored by task sub-actions and VLM trajectories judged by human experts.

  • OpenVLA: OpenVLA is trained with LoRA parameter-efficient finetuning at rank r=32 and 360 x 360 resolution.The training follows the recommended practice of continuing until token accuracy surpasses 95%.
  • RT-Trajectory: The first RT-Trajectory baseline uses GPT-4o directly to generate 2D paths from a prompt similar to HAMSTER’s.
  • RT-Trajectory: The second RT-Trajectory baseline combines Code-as-Policies with GPT-4o and uses OWLv2 open-vocabulary object detection for scene descriptions.The scene description lists detected objects and their bounding boxes before prompting RT-Trajectory.
  • Evaluation: The evaluation includes HAMSTER without RLBench finetuning, HAMSTER finetuned on all Section 4.1 data, and two GPT-4o-powered RT-Trajectory variants.
  • Evaluation: Evaluation covers pick and place, knock down object, and press button tasks across varied generalization challenges.Success is scored by sub-actions, while human experts assess VLM trajectory correctness.

D.1 IMPACT OF DESIGN DECISIONS ON VLM PERFORMANCE

The study evaluates how training-data choices affect VLM-generated 2D paths, using human rankings across real-world image-instruction pairs. It also documents failure modes spanning trajectory prediction, adherence, and execution.

  • Human evaluation: Cross-domain finetuning outperforms zero-shot path generation, including GPT-4o assisted by Code-as-Policies.The comparison tests transfer from RLBench to real-world images.
  • Training-data choices: Including low-fidelity simulation trajectories during finetuning improves the VLM’s real-world path-generation performance.The result supports transfer across substantial perceptual differences between training and testing domains.
  • Evaluation protocol: Human rankings are used instead of MSE because multiple trajectories can accomplish the same task.Evaluators rank trajectories by their perceived likelihood of task success and may assign ties.
  • Human evaluation: HAMSTER evaluates four path-generation methods on 48 image-question pairs using rankings from five robot-learning researchers.Evaluated methods include HAMSTER without RLBench, HAMSTER, Code-as-Policy, and GPT4o.
  • Failure modes: The reported failure analysis separates errors into trajectory prediction, trajectory adherence, and action execution.Examples include language-goal misunderstanding, environmental changes, 2D ambiguity, trajectory deviation, and incorrect grasp angles.

E.2 FAILURE ANALYSIS

Failure attribution differs substantially between RVT and 3DDA: RVT failures are dominated by trajectory adherence, whereas 3DDA failures are dominated by execution.

  • Failure attribution: 72% of RVT failures stem from low-level trajectory non-adherence, compared with 28% from execution failures.The authors hypothesize that RVT’s re-projection step complicates trajectory adherence.
  • Failure attribution: 90% of 3DDA failures are execution failures, while 10% are related to trajectory adherence.The authors hypothesize that 3DDA’s original-image vision tower simplifies trajectory interpretation.

F SIMULATION EXPERIMENT DETAILS

The simulation study uses Colosseum, an RLBench-based benchmark with visual and task variations, to evaluate manipulation-policy generalization under a front-camera setup.

  • Benchmark: Colosseum is an RLBench-based simulator containing visual and task variations for testing robot-manipulation generalization.The study retains 14 of 20 tasks after removing tasks without a clear front-camera view.
  • Evaluation setup: Each Colosseum task has 100 training episodes without visual variations and 25 evaluation episodes for each variation.The reported results remove variations without visual changes, such as object friction.

G DIFFERENT WAYS OF REPRESENTING 2D PATHS

The study compares simplified RDP paths with uniformly sampled 20-point paths across two VILA backbones on the close-jar task. RDP is especially beneficial for the smaller model.

  • Path representations: RDP paths simplify short-horizon trajectories to 3–5 points, while 20p paths use 20 equally spaced samples.Both representations retain points where the gripper opens or closes.
  • Backbone comparison: With VILA1.5-13B, both RDP and 20-point path representations work perfectly on the evaluated close-jar task.The experiment trains on 80 RLBench tasks and evaluates on 25 episodes of close jar.
  • Backbone comparison: VILA1.5-3B performs significantly better with RDP paths than with fixed 20-point paths.The authors attribute this to fewer points allowing more attention to gripper open and close positions.
Loading 2502.05485v4…