PaperlayerView exact revision on arXiv

Source-linked AI summary

3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

Dongyoon Hwang, Byungkun Lee, Dongjin Kim, Hyojin Jang, Hoiyeong Jin, Jueun Mun, Minho Park, Hojoon Lee, Hyunseung Kim, Jaegul Choo

arXiv:2606.31329v2cs.ROcs.AI

TL;DR

Hierarchical VLAs often use 2D trajectories to guide 3D point-cloud policies, but missing depth can geometrically distort the intended path. 3D HAMSTER predicts metrically reliable 3D trajectories with a depth-augmented VLM and outperforms proprietary VLMs and 2D-guided baselines across trajectory prediction, simulation, and real-world manipulation.

  • Problem

    2D trajectory guidance lacks depth for 3D point-cloud policies, so lifting waypoints from scene surfaces can distort the intended manipulation path.

  • Method

    3D HAMSTER augments a VLM with a depth encoder and dense depth reconstruction to predict 3D end-effector trajectories directly consumable by a point-cloud policy.

  • Results

    Across trajectory prediction, simulation, and real-world manipulation, 3D HAMSTER consistently outperforms proprietary VLMs and 2D-guided baselines, especially under appearance and geometry shifts.

  • Takeaways & Limitations

    Aligning planning and control in metric 3D space supports robust manipulation across distribution shifts and transfers to real-world tasks.

Abstract

from arXiv · show

Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation. Recent work in this paradigm uses 2D end-effector trajectories predicted by a Vision-Language Model (VLM) as explicit guidance for a downstream policy. However, state-of-the-art low-level policies operate in 3D metric space on point clouds, and feeding them 2D guidance that lacks depth forces each waypoint to be assigned the depth of whatever scene surface lies beneath it, producing geometrically distorted trajectories. We propose 3D HAMSTER, a hierarchical framework that closes this gap by having the planner directly output metrically reliable 3D trajectories. We augment a VLM with a dedicated depth encoder and a dense depth reconstruction objective to predict 3D waypoint sequences, which are directly integrated into a pointcloudbased low-level policy. Across 3D trajectory prediction, simulation, and real-world manipulation, 3D HAMSTER consistently outperforms proprietary VLMs and 2D-guided baselines, with the largest gains under appearance-altering shifts and unseen language, spatial, and visual conditions. The project page is available at https://davian-robotics.github.io/3D_HAMSTER/.

I. INTRODUCTION

Hierarchical VLAs separate semantic planning from motor control, but 2D waypoint guidance is misaligned with 3D point-cloud policies and can distort trajectories. 3D HAMSTER addresses this gap by predicting metrically reliable 3D end-effector trajectories for direct low-level execution.

  • Motivation: Hierarchical VLAs use a VLM to produce 2D image keypoints while a low-level controller converts that guidance into motor commands.This decoupling leverages VLM semantic generalization while separating planning from control.
  • Motivation: 3D-native policies operate on point clouds, creating a representational misalignment when planners reason in 2D pixel coordinates.The cited passage states that 3D architectures outperform 2D alternatives in spatial precision and viewpoint robustness.
  • Motivation: Depth-lifting 2D waypoints samples whatever scene surface lies beneath each pixel, producing trajectories that cling to surfaces rather than freely traversing 3D space.This surface-following behavior is described as a graffiti effect that makes control difficult.
  • 3D HAMSTER: 3D HAMSTER trains a VLM backbone to predict metrically reliable 3D end-effector trajectories in (u, v, d), directly aligning planner guidance with 3D policy operation.Here, (u, v) denote image-plane coordinates and d denotes metric depth.
  • 3D HAMSTER: The framework combines a depth encoder, dense reconstruction loss, and curated data mixture to enable a pretrained VLM to generate metric 3D trajectories.The paper evaluates this approach across three settings and reports robust manipulation under distribution shifts where 2D guidance fails.

II. RELATED WORK · A. Hierarchical Vision-Language-Action (VLA) Models

Hierarchical VLA frameworks separate high-level planning from low-level control using intermediate representations such as language, 2D keypoints, and 2D end-effector trajectories. Because manipulation operates in 3D, these 2D representations create ambiguity under occlusion and viewpoint changes, motivating 3D-aware planning.

  • A. Hierarchical Vision-Language-Action (VLA) Models: Hierarchical VLA frameworks decouple high-level planning from low-level control through language, 2D keypoints, and 2D end-effector trajectories.These intermediate representations are designed to be compatible with vision-language models.
  • A. Hierarchical Vision-Language-Action (VLA) Models: 2D intermediate representations introduce ambiguity for fine-grained manipulation under 3D-sensitive shifts such as occlusion or viewpoint changes, motivating 3D-aware planning.The limitation arises because manipulation fundamentally operates in 3D space.

B. Vision-Language Models (VLMs) for Spatial Reasoning … A. 3D HAMSTER

The paper situates 3D HAMSTER within the shift from point-based to trajectory-level VLM spatial reasoning and from 2D to 3D-native manipulation policies. Its method combines a metrically reliable 3D trajectory planner with a trajectory-conditioned point-cloud policy for closed-loop control.

  • B. Vision-Language Models (VLMs) for Spatial Reasoning: VLM spatial prediction has progressed from point estimates to trajectory-level guidance, alongside emerging rough 3D capabilities from geometry-rich pretraining.Examples include 3D bounding box prediction in Qwen3-VL and joint 3D scene reconstruction with spatial question answering in G2VLM.
  • C. 3D Low-Level Manipulation Policies: 2D-image manipulation policies struggle with depth ambiguities and perspective distortions, motivating 3D-native architectures operating on metric representations such as point clouds.DP3 uses point-cloud state representations, while Act3D and RVT-2 use 3D feature fields and multi-view rendering.
  • A. 3D HAMSTER: 3D HAMSTER’s planner fuses RGB, depth, and language to autoregressively generate metrically reliable end-effector trajectories, regularized by dense depth reconstruction.The dense reconstruction objective preserves metrically accurate scene geometry while aligning depth features with the VLM space.
  • C. 3D Low-Level Manipulation Policies: 3D Diffuser Actor and 3D FlowMatch Actor extend diffusion-style control with 3D scene representations for precise or efficient end-effector trajectory prediction.3D FlowMatch Actor replaces diffusion with rectified flow matching for faster and more efficient prediction.
  • A. 3D HAMSTER: Given calibrated RGB-D observations and language, the problem is decomposed into a high-level planner predicting end-effector waypoints and a low-level policy producing motor actions.The trajectory contains pixel coordinates and metric depth, while the low-level policy conditions action chunks on the trajectory.
  • A. 3D HAMSTER: The low-level policy executes closed-loop control by generating action chunks from the current point-cloud observation while conditioning on the planned trajectory.Re-planning occurs at each closed-loop control step.
  • A. 3D HAMSTER: The low-level policy unprojects planned waypoints into world coordinates, appends them to the RGB-D-derived scene point cloud, and predicts actions through rectified flow matching.Known camera intrinsics and extrinsics are used for unprojection, creating a unified 3D representation for action prediction.

B. 3D Trajectory Planner

The 3D trajectory planner adapts Qwen3-VL to generate coherent metric 3D waypoints by combining RGB, depth, language, and dense geometric supervision. Its training uses mixed capability-preservation data and a two-stage procedure to retain pretrained vision-language abilities.

  • Motivation: Qwen3-VL’s pretrained 3D knowledge does not directly transfer to coherent end-effector waypoint generation, yielding near-zero accuracy without targeted adaptation.The model’s prior 3D knowledge came from object-level localization tasks, including single-image 3D bounding-box prediction.
  • Training Data: The planner trains on eight data sources combining RGB-depth trajectories and spatial reasoning with RGB-only preservation tasks for referring, pointing, detection, and VQA.3D capability data channels metric trajectory prediction, while preservation data prevents original vision-language capabilities from being overwritten.
  • Depth Encoder: A dedicated depth pathway fuses RGB visual tokens, depth tokens, and language context before autoregressively generating τ = {(u_t, v_t, d_t)}_t^T.The depth encoder addresses the inability of RGB features alone to reliably recover metric depth from a single view.
  • Depth Reconstruction Loss: Dense depth reconstruction supplements sparse trajectory language modeling, preserving metrically faithful geometric features and providing a scene-level geometric prior.The combined loss uses λ = 0.1 for the reconstruction component.
  • Two-stage Training: Two-stage training first aligns depth representations by updating only the depth projector and decoder, then fine-tunes the task while freezing both visual encoders.Stage 1 freezes the RGB encoder, depth encoder, and LLM backbone; stage 2 retains the encoders’ learned representations.

C. Trajectory-conditioned 3D Low-level Policy

The low-level controller uses 3DFA to condition point-cloud action prediction on planner-generated world-frame trajectories. By fusing trajectory guidance with scene geometry in a shared metric representation, it executes plans directly without implicit 2D-to-3D lifting.

  • Action Prediction: The 3DFA backbone predicts actions with rectified flow matching by denoising Gaussian noise into clean action chunks while supervising both motion and gripper state.The policy learns a velocity field and includes a separate gripper open/close supervision term.
  • Trajectory-Scene Fusion: The planner’s trajectory is unprojected into world coordinates using camera intrinsics and extrinsic parameters before conditioning the point-cloud policy.The unprojection follows pcam = d · K−1[u, v, 1]⊤ and pworld = R pcam + tcam.
  • Trajectory-Scene Fusion: The world-frame trajectory is appended to the scene point cloud, with temporal color coding and learned modality embeddings distinguishing trajectory points from scene observations.This creates a unified 3D representation for trajectory following and scene understanding.
  • Trajectory-Scene Fusion: All trajectory points are preserved during point-cloud subsampling, ensuring the guidance signal remains available regardless of scene complexity.Dropped trajectory points replace the lowest-priority scene points.
  • Action Prediction: Because trajectory guidance and scene observations share a metric coordinate frame, the controller executes plans directly without implicit 2D-to-3D lifting.This avoids the geometric conversion required by 2D guidance approaches.

IV. EXPERIMENTS

The experiments evaluate the importance of depth for metric 3D trajectory prediction and test whether 3D trajectory guidance improves robustness in simulation and transfers to real-world manipulation.

  • RQ1: The experiments ask how depth affects 3D end-effector trajectory prediction and what adaptations enable a VLM to generate metric waypoints.This is evaluated through RQ1.
  • RQ2: The experiments test whether 3D trajectory guidance improves manipulation robustness under controlled visual and physical perturbations in RLBench.This is evaluated through RQ2.
  • RQ3: The experiments assess whether the benefits of 3D guidance transfer to real-world manipulation across diverse generalization axes.This is evaluated through RQ3.

A. Experimental Setup

The evaluation spans 3D trajectory prediction, simulated manipulation, and real-world manipulation, using benchmarks and hardware designed to test accuracy, robustness, and generalization. Comparisons isolate the effect of trajectory guidance by holding the low-level policy and training data fixed across guided and unguided variants.

  • Evaluation: Experiments cover 3D trajectory prediction, simulation, and real-world manipulation.The three settings correspond to RQ1, RQ2, and RQ3.
  • Trajectory prediction: DroidSpatial-Bench contains 148 held-out DROID pick-and-place episodes and evaluates grasp, placement, and combined trajectory accuracy at δ ∈{5, 10} cm.Models receive an RGB-D image and language instruction while predicting a 3D trajectory.
  • Simulated manipulation: Colosseum evaluates 11 front-camera-visible tasks across 14 perturbation axes, reporting success over 25 episodes per perturbation.The perturbations span object appearance, scene context, camera pose, and their aggregate.
  • Real-world manipulation: Real-world tests use a Franka Panda and external RGB-D camera across button pressing, pouring, and pick-and-place under language, spatial, visual, and multiple generalization axes.The task families include 3 buttons, 2 cups and 3 bowls, and 10 objects with 3 bowls.
  • Baselines: End-to-end comparisons hold the 3DFA low-level policy and demonstration data constant while testing no guidance, HAMSTER 2D trajectories, and 3D HAMSTER 3D trajectories.Real-world experiments additionally compare against π0.5 fine-tuned on the same demonstration dataset.

B. RQ1: 3D Trajectory Prediction Accuracy

3D HAMSTER improves 3D trajectory prediction by combining targeted trajectory training, depth encoding, and dense reconstruction. It outperforms proprietary and open-source baselines and remains metrically consistent across viewpoints, including unseen real-world scenes.

  • Baseline comparison: 3D HAMSTER surpasses RoboBrain-2.5-8B across all reported DroidSpatial-Bench metrics, while Gemini-3.0-Pro reaches only 16.2% on strict 5 cm Both.RoboBrain-2.5-8B reaches 39.2% on 5 cm Both and 60.1% on 10 cm Both using RGB input alone.
  • Component analysis: 5 cm Both rises from 0.7% to 27.7% when targeted 3D trajectory-dataset supervision is added to Qwen3-VL-8B.The result confirms that targeted training is required to channel the base VLM’s general 3D knowledge into effective trajectory prediction.
  • Component analysis: Dense reconstruction loss anchors depth representations to a scene-level geometric prior, counteracting depth drift over the full trajectory sequence.The passage attributes this limitation to sparse trajectory supervision alone being insufficient to maintain depth fidelity.
  • Qualitative results: Baseline trajectories that appear plausible from one viewpoint reveal substantial depth errors from another, whereas 3D HAMSTER remains metrically consistent across both.The qualitative comparison includes DroidSpatial-Bench and zeroshot real-world scenes unseen during training, where the gap widens.

C. RQ2: Robustness Under Systematic Distribution Shifts

3D HAMSTER improves robustness under distribution shifts by replacing geometrically ambiguous 2D guidance with metrically reliable 3D trajectories. It achieves the highest average success rate and improves both unperturbed performance and appearance-shift robustness.

  • Performance under distribution shifts: 44.8% average success rate is highest for 3D HAMSTER, exceeding 2D guidance (HAMSTER) by 6.0% and the unguided baseline by 8.2%.Table III reports 44.8% for 3D HAMSTER, compared with 2D guidance and unguided performance.
  • Performance under distribution shifts: 2D guidance lowers unperturbed performance to 49.5% versus 53.8% without guidance, while 3D guidance raises it to 62.9%.The degradation is attributed to projecting plans into 2D pixel coordinates, which introduces geometric ambiguity for the 3D low-level controller.
  • Real-world execution: 3D HAMSTER’s metrically reliable trajectories avoid the scene-surface adhesion of HAMSTER’s 2D trajectories, enabling successful real-world execution.The real-world rollouts illustrate 2D trajectories clinging to scene surfaces and 3D trajectories remaining metrically reliable.
  • Appearance invariance and geometric robustness: +15.6% on lighting, +11.3% on manipulated object (MO), and +8.7% on background texture (BG Tex.) are the largest gains of 3D over 2D guidance.These perturbations corrupt color and texture features used by 2D planners for implicit depth inference, whereas 3D trajectories specify metric-coordinate paths.

D. RQ3: Real-World Transfer Across Generalization Axes

3D HAMSTER achieves the strongest real-world transfer across button pressing, pouring, and pick-and-place, with particular advantages under out-of-distribution and compounding shifts. Its metrically grounded 3D waypoints preserve the planner’s intended free-space trajectory, whereas 2D guidance can pin waypoints to scene surfaces and distort execution.

  • Real-world success across task families: 80%, 68%, and 62% are 3D HAMSTER’s success rates for button pressing, pouring, and pick-and-place, exceeding HAMSTER’s 60%, 45%, and 46%.It also exceeds monolithic π0.5 at 74%, 41%, and 40% across the same task families.
  • Generalization under distribution shifts: 68% versus 45% in pouring highlights 3D HAMSTER’s advantage when tilting the cup requires precise height control encoded by 3D waypoints.Under compounding shifts, 3D guidance degrades gracefully and consistently outperforms or matches all baselines.
  • Trajectory execution fidelity: 2D guidance assigns each waypoint the depth of the underlying scene surface, pinning trajectories to the point cloud instead of tracing free paths through 3D space.The resulting execution can look plausible from one view while revealing geometric distortion from another.

V. CONCLUSION

3D HAMSTER aligns hierarchical planning and low-level control in metric 3D space by generating metrically reliable trajectories that directly guide a pointcloud-based policy.

  • 3D HAMSTER is a hierarchical framework aligning high-level planning with low-level control in metric 3D space.
  • A pretrained VLM with a depth encoder and dense depth reconstruction loss generates metrically reliable (u, v, d) trajectories.
  • The generated trajectories are directly integrated into a pointcloud-based low-level policy.
  • Experiments show that each depth-augmented planner component provides complementary gains in 3D trajectory prediction accuracy.
Loading 2606.31329v2…