PaperlayerView exact revision on arXiv

Source-linked AI summary

ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics

Jie Chen, Yuxin Cai, Yizhuo Wang, Ruofei Bai, Yuhong Cao, Jun Li, Wei-Yun Yau, Guillaume Sartoretti

arXiv:2603.13833v2cs.RO

TL;DR

Vision-language navigation still faces limits from simulation diversity and the cost of robot demonstrations. ImagiNav plans in visual space by generating instruction-conditioned future video and grounding it into trajectories, and the authors report zero-shot transfer from human video data to robots.

  • Problem

    Vision-language navigation has relied on expensive, embodiment-specific robot data, while simulation’s limited scene diversity and visual fidelity constrain generalization to open-world environments.

    Source: §I · p. 1

  • Method

    ImagiNav generates language-conditioned future egocentric video as a plan, uses inverse dynamics to extract trajectories, and trains from in-the-wild human videos with geometry-first annotation.

    Source: §I · pp. 1–2

  • Results

    The authors report zero-shot transfer to robot navigation from in-the-wild human videos and better out-of-distribution video-generation generalization for human data than simulation data.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that diverse, kinematically natural human video can support robust zero-shot generalization to physical robot navigation.

    Source: §VI · p. 7

    Video-generation latency restricts the system to low-frequency control, limiting reactivity to sudden dynamic hazards.

    Source: §VI · p. 7

Abstract

from arXiv · show

Enabling robots to navigate open-world environments via natural language is critical for general-purpose autonomy. Yet, Vision-Language Navigation has relied on end-to-end policies trained on expensive, embodiment-specific robot data. While recent foundation models trained on vast simulation data show promise, the challenge of scaling and generalizing persists due to the limited scene diversity and visual fidelity in simulation. To address this gap, we propose ImagiNav, a novel hierarchical paradigm that formulates navigation in visual space. Instead of predicting waypoints, ImagiNav synthesizes a future egocentric video conditioned on language instructions, serving as a high-level plan, interpreted by an inverse dynamics model to extract metric trajectories for execution. By decoupling planning from robot actuation, the paradigm enables direct utilization of diverse in-the-wild navigation videos. To support this, we develop an auto-labeling data pipeline that enhances motion annotation accuracy. ImagiNav demonstrates strong zero-shot transfer to robot navigation without requiring robot demonstrations, paving the way for generalist robots that learn navigation directly from unlabeled, open-world data. The project page is available at: https://j1dan.github.io/ImagiNav

I. INTRODUCTION

ImagiNav addresses the limits of simulation-trained navigation and costly robot demonstrations by planning in visual space. It uses generated future observations as plans, grounds them into trajectories, and trains from in-the-wild egocentric video.

Motivation

  • Simulation’s limited scene diversity and visual fidelity, alongside the high cost of real-world robot demonstrations, constrain scalable open-world navigation.In-the-wild egocentric video offers diverse environments and navigation intent, but lacks action labels for conventional policy training.Source: §I · p. 1

ImagiNav

  • ImagiNav generates future egocentric observations as a plan, then uses inverse dynamics to extract pose trajectories for a low-level controller.The visual-to-geometric interface separates planning from robot actuation and is described as embodiment-agnostic.Source: §I · pp. 1–2

Scalable data collection

  • A geometry-first pipeline extracts motion primitives before semantic annotation, enabling training from human egocentric videos without precise state estimation.The pipeline is presented as a way to use in-the-wild data without robot teleoperation, precise localization, or metric calibration.Source: §I · p. 2

Results

  • The authors report zero-shot transfer of navigation priors from in-the-wild human videos to robots.They also report that video-generation evaluations show better out-of-distribution generalization for human data than simulation data.Source: §I · p. 2

II. RELATED WORK

Prior VLN work has relied on reconstructed, synthetic, or simulated scenes and often retains robot-data or action-supervision requirements. ImagiNav instead uses video generation as a language-conditioned navigation policy and inverse dynamics as a geometric grounding layer.

A. Vision-Language Navigation

  • VLN approaches span reconstructed and synthetic environments, while recent methods use pretrained language models but remain reliant on robot data, often from simulation.Web-video approaches still typically require precise state estimation for action labels or retain action-space supervision.Source: A. Vision-Language Navigation · p. 2

B. Generative Models

  • Unlike video models used for simulation or trajectory scoring, ImagiNav directly uses video generation as a language-conditioned policy that imagines a future trajectory.The related work also situates generative modeling in image and video creation, robotic action distributions, and video-model planning benchmarks.Source: B. Generative Models · pp. 2–3

C. Inverse Dynamics from Visual Observations

  • ImagiNav treats inverse dynamics as geometric grounding that translates visual trajectories into executable motion without explicit action supervision.The passage distinguishes this role from using visual odometry or structure-from-motion only for localization.Source: C. Inverse Dynamics from Visual… · p. 3

III. IMAGINAV

ImagiNav is a hierarchical navigation architecture that converts semantic intent into imagined visual trajectories and then interfaces those plans with downstream control.

III. IMAGINAV

  • The framework comprises semantic reasoning, visual imagination, and geometry inverse dynamics as three interconnected modules.The pretrained vision-language model conditions its natural-language prompt on the current egocentric frame and the global instruction.Source: §III · p. 3, A. Semantic Reasoning · p. 3

B. Visual Imagination

  • A generative video model predicts future egocentric frames from the current observation and a textual instruction, encoding the plan as expected visual progression rather than control signals or waypoints.The imagined trajectory spans a horizon of H steps.Source: B. Visual Imagination · p. 3
  • The model is based on pretrained LTX-2B and is finetuned on human-collected video; two experts handle forward motion and specialize in left and right turns.At inference, the semantic reasoner routes generation to the expert matching the intended subgoal; the synthesized video can be decoded by an IDM for downstream control.Source: B. Visual Imagination · p. 3

C. Geometry Inverse Dynamics

ImagiNav grounds imagined video in relative ego-motion and uses geometry-first annotation to make in-the-wild human walking videos usable for training without robotic hardware or precise state estimation.

C. Geometry Inverse Dynamics

  • An inverse dynamics model decodes the synthesized video into relative waypoints, which a low-level controller converts into robot commands using scale calibration and kinematic constraints.The trajectory is an embodiment-agnostic navigation interface.Source: C. Geometry Inverse Dynamics · p. 3

IV. SCALABLE DATA COLLECTION PIPELINE

  • Human walking videos captured with commodity cameras provide ego-motion and diverse layouts without requiring robotic hardware, localization systems, or camera calibration.Visual-space supervision shifts data collection away from embodiment-specific action labels or pose annotations and toward curation of egocentric video.Source: §IV · p. 3

A. Auto-Labeling via IDM-Guided Annotation

  • Because raw human videos lack instructions and VLM annotation can hallucinate directions, the pipeline extracts geometric motion primitives before semantic labeling.The ordering is intended to keep textual descriptions physically grounded.Source: A. Auto-Labeling via IDM-Guided… · pp. 3–4

1) Geometric Motion Primitive Extraction:

The pipeline extracts motion primitives from image-space geometry before using those primitives to guide semantic captioning. The experiments then evaluate zero-shot transfer in a simulated embodied benchmark and compare finetuning data sources.

1) Geometric Motion Primitive Extraction:

  • A heuristic classifier labels IDM-recovered relative ego-motion as forward, rotational, or compound primitives without camera calibration or robot-specific kinematics.Raw videos are first segmented into fixed-length clips; motion labels are derived directly from image-space geometry.Source: 1) Geometric Motion Primitive Extraction · p. 4

2) Motion-Primitive–Guided Captioning:

  • Geometric motion priors constrain LLM-generated scene descriptions, including by mapping navigation verbs to camera-motion terms such as “dolly” and “pan.”This produces clips paired with descriptions intended to keep visual trajectories physically consistent with text.Source: 2) Motion-Primitive–Guided Captioning · p. 4

V. EXPERIMENTS

  • The evaluation tests zero-shot transfer from human videos to unseen simulated environments and compares visual quality and kinematic realism across finetuning data sources.Source: §V · p. 4

A. Simulation Experimental Setup

  • The benchmark uses 100 InteriorNav episodes in Isaac Sim with a simulated Unitree H1 and an RL-trained locomotion policy.Reported metrics are standard navigation and stability measures.Source: A. Simulation Experimental Setup · p. 4

B. Implementation Details

Implementation uses a pilot dataset of handheld videos and foundation models for reasoning, video generation, and inverse dynamics. Baselines include modular and domain-trained systems, plus a simulation-data variant of ImagiNav.

B. Implementation Details

  • Geometric mirroring augments approximately 33 minutes of iPhone video into 2,558 clips of 121 frames, balancing motion-primitive coverage.The dataset statistics figure shows raw-video environments and the augmented dataset’s motion-primitive distribution.Source: B. Implementation Details · p. 4, Figure 2 · p. 5
  • The system uses Gemini 3.0 Flash for task decomposition, LoRA-finetuned LTX-Video-2B for video generation, and VGGT to extract camera poses.The controller maps predicted displacements to velocity commands using a proportional controller.Source: B. Implementation Details · p. 4

C. Baselines

  • Baselines include NavDP and InternVLA-N1, while ImagiNav-Sim is finetuned on size- and motion-matched simulation data for comparison with human-video training.Zero-shot evaluation uses open-loop local execution and replans after each action buffer is consumed.Source: C. Baselines · pp. 4–5

D. Simulation Results

In zero-shot VLN-PE evaluation, ImagiNav-Real slightly exceeds NavDP, while InternVLA-N1 sets the performance upper bound and ImagiNav-Sim narrowly exceeds ImagiNav-Real. The reported explanations distinguish semantic-to-coordinate difficulty from differences in simulation and human-video motion dynamics.

D. Simulation Results

  • Success Rate measures stopping within a goal threshold, while SPL weights success by path length relative to the optimal path.The evaluation also reports Oracle Success Rate, Navigation Error, and Trajectory Length.Source: D. Simulation Results · p. 5
  • ImagiNav-Real slightly outperforms NavDP in zero-shot VLN, while InternVLA-N1 sets the upper bound and ImagiNav-Sim narrowly exceeds ImagiNav-Real.Source: D. Simulation Results · pp. 5–6, Table I · p. 6
  • The authors attribute NavDP’s slight underperformance to VLM difficulty grounding semantic subgoals in precise metric coordinates, which ImagiNav addresses through visual imagination.The generated future video provides a dense visual trajectory rather than a coordinate abstraction.Source: D. Simulation Results · p. 5
  • Despite negligible exposure to residential indoor scenes, ImagiNav-Real transfers to VLN-PE, and imagined frames correspond closely to actual observations and feasible executed plans.Source: D. Simulation Results · p. 5, Figure 3 · p. 5
  • The authors attribute ImagiNav-Sim’s narrow advantage to in-domain layout priors, while human videos provide more natural continuous momentum than deterministic graph-planner trajectories.Source: D. Simulation Results · p. 6

E. Evaluation of Video Generation

The video-generation model is evaluated on a real-world test set distinct from training, with diverse scenes and lighting. The test uses 710 clips sampled from about 12.3 minutes of raw video.

E. Evaluation of Video Generation

1) Quantitative Results:

Real-world finetuning leads the reported model comparisons, while the evaluation distinguishes visual quality from kinematic and behavioral fidelity. The section defines trajectory-based measures to assess whether generated videos represent intended motion, not just plausible frames.

Quantitative Results

  • FVD measures spatiotemporal distribution divergence, LPIPS perceptual consistency, and PSNR and SSIM low-level reconstruction fidelity.The evaluation reports these video-generation metrics to assess visual quality.Source: 1) Quantitative Results · p. 6
  • RPE measures geometric trajectory deviations, while Motion Fidelity compares sequences of Forward, Turn Left, Turn Right, or Static primitives using normalized Levenshtein distance.RPE is sensitive to domain-specific kinematic shifts; Motion Fidelity evaluates behavioral alignment.Source: 1) Quantitative Results · p. 6
  • 72.72 vs. 65.39 FVD and 0.67 vs. 0.72 Motion Fidelity: Sim-Finetuning trails Real-Finetuned, which dominates all reported categories.Table II evaluates the LTX-2B video model with simulation and real-world data.Source: 1) Quantitative Results · pp. 6–7, Table II · p. 7
  • AC-MoE outperforms a single-model baseline in FVD and Motion Fidelity, while Base LTX-2B performs poorly across metrics.Sim-Finetuning improves PSNR, SSIM, and LPIPS but lags in spatiotemporal consistency and kinematic realism.Source: 1) Quantitative Results · p. 6

2) Qualitative Results:

The qualitative results contrast real- and simulation-trained predictions in static and dynamic scenes, and show that language-conditioned generations adapt their paths to scene geometry.

Qualitative Results

  • ImagiNav-Real follows the walkway, while ImagiNav-Sim generates a trajectory that collides with the guardrail in Scene A.The comparison identifies the walkway as a navigable affordance.Source: 2) Qualitative Results · p. 7
  • In Scene B, ImagiNav-Real anticipates pedestrian motion, whereas ImagiNav-Sim treats pedestrians as static objects.The passage attributes the simulation model's behavior to a training distribution lacking dynamic actors.Source: 2) Qualitative Results · p. 7
  • For a left-turn instruction, the model first steers right around a stone bench, grounding the requested turn in scene geometry for collision avoidance.The model generates distinct futures for forward, left-turn, and right-turn instructions from the same start frame.Source: 2) Qualitative Results · p. 7

VI. CONCLUSION

ImagiNav treats video generation as an embodiment-agnostic planner and uses in-the-wild human data to support zero-shot generalization to physical robot navigation. Its pipeline remains constrained by inference latency, RGB-space geometric hallucinations, and the overhead of its preliminary ambiguity-mitigation strategy.

VI. CONCLUSION

  • In-the-wild human data’s environmental diversity and kinematic naturalness support robust zero-shot generalization to physical robot navigation.The framework treats video generation as an embodiment-agnostic planner and aims to let robots learn from human visual experience.Source: §VI · p. 7
  • Video-generation cost restricts ImagiNav to low-frequency control, limiting reactivity to sudden dynamic hazards.The conclusion identifies inference latency as the pipeline’s primary limitation.Source: §VI · p. 7
  • RGB-only planning can produce visually plausible but physically unsafe trajectories through geometric hallucinations.The authors suggest a visual plan scorer or traditional safety fallbacks as possible mitigations.Source: §VI · p. 8
  • AC-MoE mitigates spatial ambiguity but remains preliminary and adds expert-management overhead; future work targets real-time distillation, explicit depth, and unified directional experts.The proposed directions address control frequency, geometric consistency and safety, and directional accuracy.Source: §VI · p. 8
Loading 2603.13833v2…