PaperlayerView exact revision on arXiv

Source-linked AI summary

An Efficient and Multi-Modal Navigation System with One-Step World Model

Wangtian Shen, Ziyang Meng, Jinming Ma, Mingliang Zhou, Diyun Xiang

arXiv:2601.12277v1cs.RO

TL;DR

Navigation policies often lack 3D spatial reasoning, while existing world models are too slow for real-time planning. The paper introduces a one-step 3D U-Net world model with efficient spatial-temporal attention and anchor-based optimization planning, achieving superior efficiency and robustness in multi-modal closed-loop navigation. Its remaining challenges are action-space exploration and the trade-off between generation quality and inference speed.

  • Problem

    Existing end-to-end policies struggle with 3D spatial reasoning, while diffusion and autoregressive world models impose prohibitive latency for real-time navigation.

    Source: §1 · p. 1

  • Method

    The paper combines one-step future-observation generation, a 3D U-Net with spatial-temporal attention, and optimization-based planning with anchor-based initialization.

    Source: §1 · pp. 1–2

  • Results

    The proposed system achieves superior efficiency and robust multi-modal goal navigation across simulation and real-world closed-loop experiments.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    The system provides a unified world-model-based framework for image-goal, language-goal, and point-goal navigation.

    Source: §1 · pp. 1–2

    CEM-based planning still requires exploration of the full action space, and iterative optimization trades off generation quality against inference speed.

    Source: §7 · p. 7

Abstract

from arXiv · show

Navigation is a fundamental capability for mobile robots. While the current trend is to use learning-based approaches to replace traditional geometry-based methods, existing end-to-end learning-based policies often struggle with 3D spatial reasoning and lack a comprehensive understanding of physical world dynamics. Integrating world models-which predict future observations conditioned on given actions-with iterative optimization planning offers a promising solution due to their capacity for imagination and flexibility. However, current navigation world models, typically built on pure transformer architectures, often rely on multi-step diffusion processes and autoregressive frame-by-frame generation. These mechanisms result in prohibitive computational latency, rendering real-time deployment impossible. To address this bottleneck, we propose a lightweight navigation world model that adopts a one-step generation paradigm and a 3D U-Net backbone equipped with efficient spatial-temporal attention. This design drastically reduces inference latency, enabling high-frequency control while achieving superior predictive performance. We also integrate this model into an optimization-based planning framework utilizing anchor-based initialization to handle multi-modal goal navigation tasks. Extensive closed-loop experiments in both simulation and real-world environments demonstrate our system's superior efficiency and robustness compared to state-of-the-art baselines.

1 Introduction

The paper addresses limited 3D spatial reasoning and high latency in navigation policies by combining a one-step world model with optimization-based planning for unified multi-modal, closed-loop navigation.

  • Existing end-to-end policies often lack explicit 3D spatial reasoning, while diffusion and autoregressive world models incur latency and accumulated errors that hinder real-time deployment.These limitations become especially important when planning evaluates multiple candidate actions.Source: §1 · p. 1
  • One-step generation, spatial-temporal attention, and nonautoregressive prediction reduce inference latency while improving predicted-sequence quality for high-frequency control.The model uses a 3D U-Net backbone and predicts future frames simultaneously rather than through iterative diffusion or frame-by-frame autoregression.Source: §1 · pp. 1–2
  • The proposed system combines task-specific planning losses with anchor-based initialization to refine action samples for multi-modal goals.The contribution is presented as a unified navigation system grounded in world-model prediction and planning.Source: §1 · pp. 1–2
  • The system integrates a lightweight one-step navigation world model with planning to support image-goal, language-goal, and point-goal navigation in one framework.The framework is evaluated through closed-loop experiments in simulation and the real world.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Closed-loop experiments in simulation and the real world demonstrate the system’s effectiveness and efficiency.Source: §1 · p. 2

2 RELATED WORK

Prior navigation research advances learning-based generalization and world-model planning, but remains limited by spatial reasoning, real-time computational cost, and unified support for different goal modalities.

2.1 Learning Based Visual Navigation

  • Learning-based visual navigation methods improve generalization and support image-goal, language-conditioned, or omni-modal tasks, but limited spatial reasoning still constrains performance.Examples include GNM, ViNT, NoMaD, EffoNAV, LeLaN, and OmniVLA.Source: §2.1 · p. 2

2.2 World Model for Embodied Intelligence

  • World models generate future observations for interaction and planning, extending from reinforcement-learning methods to visual generation and navigation systems.Representative approaches include Pathdreamer, UniPi, Genie, NWM, and 3D-memory-based methods.Source: §2.2 · p. 2
  • Generative world-model planning remains difficult to deploy in real time because iterative generation is computationally expensive, while latent-space approaches may lack explicit 3D prediction and unified goal integration.Source: §2.2 · p. 2

3 METHODS

The method uses a one-step world model with efficient 3D spatiotemporal modeling, then plans actions by scoring candidate trajectories and initializing them with anchors.

3.1 One-Step World Model for Navigation

  • The world model predicts a future observation from the current observation and action, using shortcut training to replace iterative sampling with one-step generation.For larger step sizes, bootstrap consistency distills two smaller substeps into one step.Source: §3.1 · p. 3

3.2 Architecture of Navigation World Model

  • A 3D U-Net predicts 11 future frames simultaneously, combining CNNs and transformers with latent-space generation through a pretrained VAE.This nonautoregressive design avoids frame-by-frame prediction and its associated long-horizon error accumulation.Source: §3.2 · p. 3, Figure 2 · p. 3
  • Spatial and temporal dependencies are decoupled: frame-wise modules model intra-frame structure, while window-based temporal attention propagates information locally across frames.Restricting temporal attention to spatial windows reduces context length while maintaining spatiotemporal consistency.Source: §3.2 · pp. 3–4

3.3 Training Details

  • Training uses random trajectory sampling from 500 indoor trajectories, and pretraining on a large public navigation dataset improves performance after scene-specific fine-tuning.Source: §3.3 · p. 4

3.4 Navigation with World Model

  • The planner uses the world model to score candidate action samples with goal-specific losses and optimizes them with the Cross-Entropy Method.For point-goal navigation, the loss combines terminal-point distance with a depth-based obstacle-proximity penalty.Source: §3.4 · p. 4
  • Anchor-based initialization uses fixed linear and angular velocity sets because random initialization with 32 samples may undersample the action space and degrade navigation performance.Source: §3.4 · p. 4, Figure 3 · p. 4

4 Generation Experiments

The generation experiments evaluate a lightweight navigation world model against NWM and ablations, finding higher-quality predictions with substantially lower inference time.

4.1 Experimental Setup

  • The evaluation uses MP3D trajectories from Habitat, training independently on indoor scenes and testing on unseen trajectories.Source: §4.1 · p. 4

4.2 Generation Performance

  • 45% lower inference time for 11 future frames (0.076s vs. 0.138s) than NWM, with a similar parameter scale.Source: §4.2 · pp. 4–5
  • The proposed model surpasses NWM in per-frame quality and spatiotemporal consistency across the evaluated generation metrics.Source: §4.2 · p. 4
  • Qualitative results show generated frames more closely match ground truth than NWM, whose autoregressive predictions exhibit severe position drift.Source: Figure 4 · p. 5, §4.2 · p. 5
  • Random trajectory sampling and public-dataset pretraining improve generalization and the visual quality of generated images.Source: §4.2 · p. 5

5 Navigation Experiments

Navigation experiments combine the world model with task-specific optimization for image-, language-, and point-goal tasks, with ablations examining initialization and planning losses.

5.1 Experimental Setup

  • Evaluation covers 150 episodes across five scenes, using SR, SPL, and inference time to compare methods.Source: §5.1 · p. 5
  • The framework uses LPIPS for image-goal scoring, SigLIP for language-goal scoring, and depth-based obstacle-aware loss for point-goal planning.Source: §5.1 · pp. 5–6

5.2 Navigation Performance Resluts

  • Image-goal navigation reaches 72.67% SR, exceeding NWM by 29.3% and OmniVLA by 36.0%; language-goal navigation reaches 69.33% SR.Source: §5.2 · p. 6, Table 2 · p. 6
  • The method significantly outperforms NWM, NoMaD, and OmniVLA on evaluated multi-modal navigation tasks, while matching NWM on point-goal performance and outperforming OmniVLA.Source: §5.2 · p. 6

5.3 Ablations

  • Anchor-based initialization produces smoother, more concentrated trajectory samples and improves planning compared with random initialization.Source: §5.3 · p. 7, Figure 6 · p. 7
  • Language-goal navigation is highly sensitive to the planning loss, with SigLIP outperforming BLIP and CLIP.Source: Table 3 · p. 6, §5.3 · p. 7

6 Real-world Experiments

Real-world closed-loop experiments evaluate the system on a mobile robot using networked inference and report consistent advantages over baselines.

  • The evaluation uses 25 trials per method on each task with a mobile robot equipped with an Intel RealSense camera.Source: §6 · p. 7
  • A client-server system transmits observations to a cloud server, which returns linear and angular velocity commands after inference.Source: §6 · p. 7
  • The proposed approach consistently outperforms baselines in physical environments, demonstrating robustness and generalization.Source: §6 · p. 7, Table 4 · p. 7

7 Conclusion

The proposed system combines a low-latency navigation world model with anchor-based planning to achieve robust multi-modal navigation in simulation and real-world closed-loop experiments. However, complex trajectory search and the quality–speed trade-off remain limitations.

7 Conclusion

  • The system combines one-step generation, a 3D U-Net with spatial-temporal attention, and anchor-based planning for multi-modal goal navigation.It is evaluated in simulation and real-world closed-loop experiments.Source: §7 · p. 7
  • Its planning integration achieves superior results across multi-modal goal navigation in simulation and real-world closed-loop experiments.Source: §7 · p. 7
  • CEM-based planning must explore the whole action space, making complex trajectory search difficult and forcing a trade-off between generation quality and inference speed.The authors propose latent action spaces and a fast-slow dual system as future directions.Source: §7 · p. 7
Loading 2601.12277v1…