PaperlayerView exact revision on arXiv

Source-linked AI summary

HiWE: Hierarchical World Knowledge Model with Visual Keypoint Enhancement for Zero-Shot 3D Path Planning

Guoqing Ma, Mingqi Yuan, Chen Gao, Jiayu Chen, Shan Yu

arXiv:2609.39323v1cs.ROcs.AI

TL;DR

Robot demonstrations require locating interactions, relating them to scene geometry, and executing suitable motions, while deployment-specific demonstrations remain costly. HiWE links visual grounding to planning and grasp execution through a point-based interface. It has the highest reported mean on 10 tested tasks, though its advantage is not uniform.

Abstract

from arXiv · show

Robot demonstration generation requires a system to identify where an interaction should occur, plan a feasible motion, and execute the required contact. HiWE connects these decisions through a point-based interface between visual grounding and language-based planning. PointVLM is instruction-tuned to associate task-relevant objects with image coordinates using a mixture of point annotations, segmentation-derived samples, robot observations, and visual question answering data. Depth measurements lift these predictions into a semantic 3D representation. A language planner, 3DLLM, uses this representation to specify end-effector waypoints and gripper commands, while a hybrid grasping module resolves local grasp poses. The evaluation covers 14 simulated manipulation tasks and four physical-robot tasks, together with ablations of the visual training data, spatial inputs, and grasp selection. Here, zero-shot execution refers to deployment without task-specific demonstration training; the visual model uses existing robot data during fine-tuning. This paper describes the original point-based formulation of the framework; its relationship to the subsequent GeneralVLA extension is detailed in the introduction.

Introduction

Robot demonstrations require locating interaction regions, relating them to scene geometry, and executing suitable motions, while deployment-specific demonstrations remain costly. HiWE connects visual grounding to planning through a point-based interface, and the paper evaluates the original system separately from later GeneralVLA extensions.

Introduction

  • HiWE addresses the need to locate interaction regions, relate them to scene geometry, and execute motions that respect objects and surroundings.Errors at any of these stages can invalidate a demonstration.Source: Introduction · p. 1
  • HiWE is the original formulation; GeneralVLA is a subsequent extension that adds segmentation refinement and persistent experience retrieval while retaining shared components.The paper’s experiments characterize the original configuration and do not isolate the incremental benefit of those extensions.Source: Introduction · pp. 1–2
  • The paper’s “zero-shot” setting means deployment without task-specific demonstration training, not an entirely untrained system or one without robot data.PointVLM is fine-tuned using existing datasets, including robot observations from outside the deployment environment.Source: Introduction · pp. 1–2
  • The paper presents a point-based interface connecting an instruction-tuned visual model to a language planner and grasp execution module, with visual supervision separated from deployment-specific action demonstrations.Its implementation combines PointVLM training, depth-based grounding, 3DLLM waypoint generation, and HGM grasp selection.Source: Introduction · pp. 1–2
  • The evaluation covers simulated and physical manipulation tasks and ablates visual training data, planner inputs, and grasp selection.The paper also studies policy training from generated demonstrations.Source: Introduction · p. 2

Related Work

Related work spans language-based robot planning, action-predicting vision-language models, and intermediate point or trajectory representations. HiWE uses explicit spatial waypoints and gripper commands, while GeneralVLA is the subsequent architectural extension that retains the planning and execution backbone.

Language models as robot planners

  • Language-based planners use different interfaces: Code-as-Policies represents plans as executable code, VoxPoser uses 3D value maps, and Scaling-up uses language-guided interaction to obtain policy-learning data.HiWE instead uses ordered spatial waypoints and gripper commands and compares complete systems under the experiment-specific inputs.Source: Related Work · p. 2

Learning actions from vision and language

  • RT-1, OpenVLA, and LLARVA learn action predictions or trajectories from vision and language, whereas HiWE exposes an intermediate spatial description to separate planning and execution modules.This changes the interfaces and supervision used; it does not guarantee better reasoning, visual accuracy, or execution speed in every setting.Source: Related Work · p. 2

Points and trajectories as intermediate representations

  • Point-based supervision and trajectory specifications provide intermediate representations, while GeneralVLA is the closest architectural comparison and a subsequent extension of HiWE.GeneralVLA retains the planning and execution backbone while developing perception and experience-reuse mechanisms.Source: Related Work · pp. 2–3, Figure 2 · p. 3

HiWE: Point-Based Grounding and Planning System interfaces

HiWE passes explicit spatial information from visual grounding to planning and execution. PointVLM predicts object-associated image points, which depth lifts into 3D geometric anchors for the planner.

HiWE: Point-Based Grounding and Planning System interfaces

  • PointVLM receives an image and task instruction and returns object-associated image points; depth converts them to 3D coordinates for a semantic scene description to 3DLLM.3DLLM returns end-effector waypoints and gripper commands; HGM resolves local grasp poses before motion-planner execution.Source: System interfaces · p. 3

PointVLM: predicting interaction locations

  • PointVLM represents each relevant object by multiple image coordinates, giving the planner information about its spatial extent after depth projection.Its output is a coordinate sequence; HiWE does not use GeneralVLA’s ASM segmentation decoder or segmentation-feedback loop.Source: PointVLM: predicting interaction… · p. 3
  • PointVLM’s training mixture includes point-prediction examples, segmentation-mask samples paired with object semantics, robot observations, and visual question-answering conversations.The sources include 347k RoboPoint point-prediction examples and 667K visual question-answering conversations.Source: PointVLM: predicting interaction… · p. 3

3DLLM: planning from a spatial description

3DLLM plans from a semantic 3D scene description, while HGM resolves local grasp geometry and selects a candidate pose for execution.

3DLLM: planning from a spatial description

  • 3DLLM receives an instruction and compact semantic scene description, then specifies ordered spatial waypoints interleaved with gripper commands.Multiple object points describe orientation and extent, while obstacle points provide route context; these inputs do not guarantee collision-free execution, and planning uses the current scene rather than persistent experience retrieval.Source: 3DLLM: planning from a spatial… · p. 3

HGM: resolving local grasp geometry

  • HGM uses task-relevant 3D points to restrict an RGB-D region, generates candidate poses in an object-centered point cloud, rejects collisions, and selects the remaining pose nearest the object center.This is a selection heuristic, not a guarantee of a globally optimal grasp; the chosen pose joins the planned waypoints for motion-planner execution.Source: HGM: resolving local grasp geometry · pp. 3–4

Experiments

HiWE is evaluated using its original PointVLM and current-scene 3DLLM configuration in simulation and on a physical robot. It reports broad but nonuniform advantages over listed baselines, with comparison limits tied to the tested tasks and baseline implementations.

Experiments

Zero-shot performance in simulation

  • HiWE has a nonzero success rate on all 14 simulated tasks and the highest reported mean on 10, but its advantage is not uniform.Non-prehensile interactions and fine manipulation expose the need for more accurate pose estimates or execution adjustments.Source: Zero-shot performance in simulation · p. 4

Real-world experiments

  • Across four hardware tasks, HiWE succeeds in some episodes of each and reports higher success rates than both listed baselines.RoboPoint lacks trajectory planning in the evaluated configuration, while CAP lacks a supplied drawer-opening primitive, constraining the comparison.Source: Real-world experiments · p. 4

Ablation Study

The ablations examine PointVLM localization and training data, 3DLLM spatial inputs, HGM grasp selection, and policy training with generated demonstrations. Results favor combined supervision and the full tested planner and grasp configurations, while input benefits vary by task.

Point location accuracy of PointVLM

Information required by 3DLLM

  • The full 3DLLM representation performs best or ties for best on both reported tasks, while obstacle information helps Take umbrella but not Put block.The ablations vary coordinate dimensionality, points per object, and obstacle context; the reported benefits are task-dependent rather than universal.Source: Information required by 3DLLM · pp. 4–5, Table 6 · p. 5

Ablation study of the HGM module

  • The full HGM configuration has the highest reported mean on Play jenga and Take umbrella; removing target-point guidance leaves no successful trials.Other tested ablations reduce success without eliminating it, and these findings are limited to the evaluated tasks.Source: Ablation study of the HGM module · p. 5, Table 7 · p. 5

Data scaling

  • The reported linear-fit slopes are 0.543 for HiWE-generated demonstrations and 0.156 for RLBench-generated demonstrations, summarizing only the evaluated range.The coefficients are not presented as a general scaling law.Source: Data scaling · p. 5

Conclusion

HiWE connects visual grounding, spatial language planning, and grasp execution through a point-based pipeline; successful executions also provide demonstrations for policy training. The evaluated configuration retains point-localization, geometric-description, and grasp-execution limitations, and its zero-shot scope excludes task-specific demonstration training but not existing robot observations.

Conclusion

  • Successful executions in HiWE’s point-based pipeline provide demonstrations for policy training.The original system uses fine-tuned PointVLM for geometric anchors, 3DLLM for current-scene planning, and HGM for grasp poses.Source: Conclusion · p. 6
  • The evaluated HiWE configuration examines manipulation success, visual supervision, and planner spatial information; GeneralVLA later adds affordance segmentation and persistent experience retrieval.Those GeneralVLA mechanisms are outside the HiWE configuration evaluated here.Source: Conclusion · p. 6
  • HiWE remains limited by point-localization errors, incomplete geometric descriptions, and grasp failures; zero-shot deployment still uses existing robot observations for visual training.Zero-shot refers specifically to deployment without task-specific demonstration training.Source: Conclusion · p. 6
Loading 2609.39323v1…