PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing What Matters: Visual Cue Guided Video Planning for Generalizable Robot Navigation

Hojin Lee, Sizhe Lester Li, Maximilian Hilger, Susie Lu, Achim J. Lilienthal, Vincent Sitzmann, Daniel A. Duecker

arXiv:2609.16737v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Existing video navigation methods often use short-horizon directional guidance, omit robot physical characteristics, and translate plans through geometric waypoints rather than directly grounded continuous control. CueNav conditions a generative video planner on BEV task context and body-visible embodiment cues, then uses an embodiment-specific flow-based IDM to convert predicted flow fields into robot actions. CueNav retains 55% success in larger mazes and completes 70% of trials in the 1.0 m narrow passage with body-aware observations and the IDM.

  • Problem

    Existing video navigation methods often use short-horizon directional guidance, omit robot physical characteristics, and translate plans through geometric waypoints rather than directly grounded continuous control.

  • Method

    CueNav conditions a generative video planner on BEV task context and body-visible embodiment cues, then uses an embodiment-specific flow-based IDM to convert predicted flow fields into robot actions.

  • Results

    CueNav retains 55% success in larger mazes and completes 70% of trials in the 1.0 m narrow passage with body-aware observations and the IDM.

  • Takeaways & Limitations

    Visual cue-guided planning with embodiment-specific action grounding supports longer-horizon, precise, and cross-platform robot navigation.

Abstract

from arXiv · show

Generative video models can serve as a promising backbone for robot navigation by predicting future observations as video plans. Recent approaches often condition video planning on short-horizon guidance and recover geometric waypoints through scene reconstruction, leaving longer-horizon planning and precise video-to-action translation less explored. We present CueNav, a video model-based navigation framework combining visual cue guided video planning with an embodiment-specific Inverse-Dynamics Model (IDM). As visual cues, we use a Bird's-Eye View (BEV) map to convey global task context and retain part of the robot body in the egocentric observation to expose embodiment context. These cues guide the video planner, while the IDM translates dense flow fields extracted from the video plan into robot actions. With the visual cue encoding global task context, CueNav achieves nearly 2x higher success in maze navigation than planning without the cue. The body-aware view with the IDM enables precise navigation with 70% success in a narrow passage where comparison methods largely fail to complete the task. We further demonstrate zero-shot semantic-conditioned navigation and deployment of the same video planner across different robot platforms. Our results show that visual cue-guided video planning with embodiment-specific action grounding paves the way toward a generalizable navigation framework for longer-horizon planning and embodiment-aware control. Additional results and code are available on our project website: https://cuenav.github.io.

I. INTRODUCTION

CueNav addresses limitations in video-based robot navigation by adding task and embodiment visual cues and grounding predicted visual motion in continuous robot control.

  • I. INTRODUCTION: Existing video planners rely on granular directional instructions, limiting their use for navigation beyond local guidance.The introduction identifies left- or right-turn instructions as an example of this constraint.
  • I. INTRODUCTION: Existing planners also overlook robot physical characteristics, although size and kinematics matter for precise navigation through constrained spaces.CueNav exposes embodiment information visually to address this limitation.
  • I. INTRODUCTION: CueNav combines visual task and embodiment cues with an embodiment-specific IDM to translate predicted visual motion into continuous robot actions.The framework uses visual cues for planning and flow-based action grounding across simulation and real-world experiments.
  • I. INTRODUCTION: Waypoint-based action translation requires a downstream controller and does not directly ground generated visual motion in robot kinematics and dynamics.CueNav instead uses an embodiment-specific inverse-dynamics model for action grounding.
  • I. INTRODUCTION: BEV maps provide global task context for high-level navigation, while retaining part of the robot body exposes geometry and obstacle relationships for embodiment-aware planning.These cues extend beyond local egocentric guidance and avoid requiring a separate embodiment representation.

II. RELATED WORK

Related navigation foundation models provide semantic reasoning but require large robot-specific datasets, motivating video models that separate visual planning from robot control.

  • II. RELATED WORK: Large pretrained VLAs support unified navigation tasks and strong semantic reasoning, but learning robot-specific control still requires extensive paired observation-action data.This data requirement motivates alternatives that do not directly predict robot actions.
  • II. RELATED WORK: Navigation foundation models use millions of samples, often from simulation or reconstructed environments that can introduce sim-to-real gaps and couple policies to particular action spaces and embodiments.Reported training scales range from 8M navigation samples to 30M supervised samples.
  • II. RELATED WORK: Video models learn semantic and motion priors from internet-scale videos without robot action labels, planning in visual space while robot-specific control is learned separately.This separation avoids committing the visual planner directly to one robot action space.

B. Video Models for Robot Navigation

Video navigation methods predict future visual behavior but commonly depend on local directional guidance and geometric waypoint recovery; CueNav instead combines global task context with direct flow-based action grounding.

  • B. Video Models for Robot Navigation: Existing video planners largely rely on local motion guidance and recover camera motion or geometric waypoints for downstream control.These approaches leave task-level navigation without explicit directional cues less explored and add geometric estimation or tracking stages.
  • B. Video Models for Robot Navigation: CueNav uses global task context to plan from high-level objectives without explicit local directional guidance.This design targets navigation decisions that extend beyond short-horizon directional commands.
  • B. Video Models for Robot Navigation: CueNav’s embodiment-specific flow-based IDM grounds generated visual motion directly into continuous robot commands without explicit geometric reconstruction or joint video-action training.The planner predicts visual futures, while the IDM converts their flow fields into executable control.
  • B. Video Models for Robot Navigation: The video-to-navigation formulation must solve both visual-plan generation and translation of those plans into executable continuous control commands.This separates the planning question from the action-grounding question.

B. Visual Cue Guided Planning with a Video Model

CueNav exposes task and embodiment context directly in the video model’s visual input, then uses action-free video post-training and closed-loop IDM-based execution.

  • B. Visual Cue Guided Planning with a Video Model: The planner predicts short-horizon visual futures from recent observations and a navigation prompt while directly exposing task and embodiment context through visual cues.This visual-input design is intended to support generalizable navigation.
  • B. Visual Cue Guided Planning with a Video Model: A BEV map embeds robot and goal locations while leaving the connecting path unspecified, providing global situational awareness beyond the egocentric view.For embodiment-aware navigation, the camera retains part of the robot body in the observation.
  • B. Visual Cue Guided Planning with a Video Model: Retaining part of the robot body exposes its geometry and spatial relationship to nearby obstacles and free space without requiring a separate embodiment representation.The body-visible observation is the embodiment cue used by CueNav.
  • B. Visual Cue Guided Planning with a Video Model: CueNav adapts an open-weight video diffusion transformer for video-to-video prediction using navigation videos without robot action labels.The training objective is applied only to future frames under a diffusion-forcing formulation.
  • B. Visual Cue Guided Planning with a Video Model: At deployment, predicted frames pass to a flow-based IDM for continuous commands, and newly observed frames update the history for closed-loop receding-horizon replanning.This execution loop connects visual prediction to repeated action updates.

C. Flow-based Inverse-Dynamics Model

CueNav translates predicted visual motion into robot actions using dense flow fields and an embodiment-specific inverse-dynamics model. The closed-loop procedure repeatedly updates plans from observations and executes robot-specific control inputs.

  • CueNav’s IDM converts dense flow fields from predicted frames into sequences of robot actions.The flow fields are computed from consecutive frames in the combined observed-and-predicted sequence.
  • Temporal padding supplies additional flow context so the IDM can account for dynamics such as delayed control responses.The IDM uses q + 2p flow fields, subject to p + q ≤ M and p ≤ N − 1.
  • A convolutional encoder and spatiotemporal transformer jointly process flow-field tokens before predicting q actions.The resulting features are mapped to actions by a feed-forward network trained with mean-squared error.
  • Separate IDMs support different robot embodiments and action spaces, including planar wheeled and lateral-motion-capable legged controls.Wheeled robots use (v_x, ω_z), whereas legged robots use (v_x, v_y, ω_z).
  • CueNav executes predicted actions in a receding-horizon loop, then acquires new observations and replans.The procedure computes flow fields, infers actions, executes them, appends observations, and advances by q steps.

IV. EXPERIMENTS

The experiments test CueNav’s semantic generalization, embodiment-aware precision, and ability to plan beyond local directional guidance. Together, these questions target the framework’s principal claims about generalizable navigation.

  • Semantic-conditioned closed-loop navigation: Semantic-conditioned closed-loop navigation tests whether CueNav can use video-model generalization to navigate toward described objects in real environments.
  • Precise embodiment-aware navigation: Precise embodiment-aware navigation tests whether retaining embodiment cues improves performance in geometrically constrained spaces.
  • Planning beyond local directional guidance: Planning beyond local directional guidance tests whether global task context enables navigation beyond short-range directional instructions.

A. Experimental Setup

CueNav is evaluated on two robot platforms using RGB navigation data, pretrained video-model adaptation, and comparisons with recent vision-language navigation baselines. The setup includes semantic goal navigation with standard success, efficiency, error, and path-length metrics.

  • CueNav is evaluated on Clearpath Husky A300 and Unitree Go2 robots using RGB observations paired with executed actions.Each platform contributes approximately two hours of indoor and outdoor navigation data, while the planner is post-trained using RGB observations from both platforms.
  • The planner uses a pretrained Wan2.2-5B video model at 192 × 128 resolution and 10 Hz, with LoRA applied to its diffusion transformer.
  • CueNav conditions on 21 context frames and generates 16 future frames; onboard planning and IDM inference take approximately 4 s per step.
  • The system executes 15 predicted actions after each planning step with one temporal padding flow field.
  • Semantic navigation uses three unseen targets, 10 runs per target, and reports SR, SPL, final NE, and TL over 30 trials per method.Success requires termination within 1.5 m of the target center; NE and TL are defined as final target distance and total traveled distance.
  • CueNav achieves the highest SR and SPL, while CueNav without the body view attains the lowest final navigation error.Removing the IDM and using geometry-based camera-pose reconstruction substantially reduces SR and SPL.

C. Precise Embodiment-aware Navigation

CueNav uses embodiment-visible observations and an inverse-dynamics model to improve precise navigation in zero-shot narrow passages. It maintains strong progress as the passage narrows and aligns realized motion more closely with generated plans.

  • 70% of trials complete the 1.0 m narrow track, while CueNav completes all trials in the 1.5 m setting.The Husky is 0.7 m wide, and comparison methods frequently terminate before reaching the end.
  • The narrow-track task tests zero-shot traversal under collision constraints, with trials ending at completion or immobilization.The tracks are excluded from post-training, making the evaluation a precise-navigation test in unseen environments.
  • CueNav maintains close alignment between generated and realized observations, unlike CueNav w/o IDM.The comparison indicates more accurate translation of the visual plan into robot motion.
  • The evaluation measures maximum normalized progress over 10 runs in 1.5 m and 1.0 m tracks.Progress ranges from 0 at the start to 1 at successful completion.

D. Planning beyond Local Guidance with Global Task Context

CueNav adds a BEV map showing the agent and goal to the egocentric observation, enabling global task context during maze navigation. This improves zero-shot generalization to larger mazes compared with egocentric input alone.

  • The BEV map indicates agent and goal locations while leaving the solution path unspecified, complementing the egocentric view.The combined observation is tiled into a single 832 × 480 pixel input.
  • Training uses successful trajectories from randomly generated 3 × 3 mazes, followed by zero-shot evaluation on maze sizes up to 6 × 6.The maze task uses continuous forward velocity and yaw-rate commands at 10 Hz, without turn-by-turn directional commands.
  • The ablation compares CueNav with a map-free variant that receives only the egocentric view.This isolates the contribution of the visual cue providing global task context.
  • 55% success is retained in unseen 6 × 6 mazes with global context, compared with 30% without the map.CueNav with global task context consistently outperforms CueNav (w/o map), with the gap increasing in larger mazes.
  • Global task context produces more goal-directed trajectories, whereas the no-map variant shows more local wandering and less consistent progress.The trajectories support planning beyond local directional guidance in larger unseen environments.

E. Case

CueNav reuses the same video planner on the Unitree Go2 while using a platform-specific IDM for its different morphology and action space. In an unseen environment, the Go2 traverses randomly placed obstacles without collision.

  • The shared video planner is deployed zero-shot on the Unitree Go2 with a separately trained, embodiment-specific IDM.The Go2 differs from the Husky A300 in morphology and action space.
  • The Go2 traverses randomly placed obstacles without collision in an environment unseen during post-training.The embodiment-visible observation supplies information about the executing robot and its relation to nearby obstacles.

V. CONCLUSION AND FUTURE WORK

CueNav combines task- and embodiment-relevant visual cues with a flow-based IDM to translate predicted visual motion into continuous robot actions. The paper reports longer-horizon, precise, and cross-platform navigation, while identifying computational, temporal, and camera-configuration limits.

  • CueNav conditions video planning on task and embodiment cues and grounds predicted visual motion into continuous actions with a flow-based IDM.
  • The framework supports navigation beyond local directional guidance, precise embodiment-aware control, and reuse across robot platforms.
  • Computationally expensive inference limits deployment replanning rates, while short-horizon predictions and finite observation windows limit longer-horizon reasoning and memory.The paper suggests few-step or latent-space distillation and longer-term visual context or hierarchical memory as possible remedies.
  • The IDM is tied to a fixed camera configuration, so its flow-to-action mapping may not transfer directly across camera poses.Camera-conditioned or adaptive IDMs are proposed to improve sensor-configuration generalization.
Loading 2609.16737v1…