Source-linked AI summary
NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance
Wenzhe Cai, Jiaqi Peng, Yuqiang Yang, Yujian Zhang, Meng Wei, Hanqing Wang, Yilun Chen, Tai Wang, Jiangmiao Pang
TL;DR
Robots need scalable navigation that generalizes across dynamic open-world scenes and embodiments despite limitations of modular systems and scarce real-world demonstrations. NavDP trains an end-to-end transformer diffusion policy entirely in simulation, using privileged-information-guided trajectory generation and contrastive critic evaluation from local RGB-D observations. It achieves zero-shot sim-to-real and cross-embodiment generalization, outperforming prior methods in simulation and real-world experiments.
Problem
Dynamic open-world navigation requires generalization across diverse scenes and robot embodiments, but modular systems and limited real-world demonstrations constrain existing approaches.
Method
NavDP jointly learns RGB-D-conditioned trajectory generation and critic evaluation using diffusion modeling, privileged simulation guidance, and contrastive positive and negative trajectories.
Results
NavDP outperforms the previous state-of-the-art learning-based method by 6.3% in simulation Success Rate and improves cross-embodiment real-world performance by 23.0% on average.
Takeaways & Limitations
NavDP demonstrates strong zero-shot sim-to-real and cross-embodiment generalization with real-time path planning and collision avoidance in static and dynamic scenarios.
Abstract
from arXiv · showhide
Learning to navigate in dynamic and complex open-world environments is a critical yet challenging capability for autonomous robots. Existing approaches often rely on cascaded modular frameworks, which require extensive hyperparameter tuning or learning from limited real-world demonstration data. In this paper, we propose Navigation Diffusion Policy (NavDP), an end-to-end network trained solely in simulation that enables zero-shot sim-to-real transfer across diverse environments and robot embodiments. The core of NavDP is a unified transformer-based architecture that jointly learns trajectory generation and trajectory evaluation, both conditioned solely on local RGB-D observation. By learning to predict critic values for contrastive trajectory samples, our proposed approach effectively leverages supervision from privileged information available in simulation, thereby fostering accurate spatial understanding and enabling the distinction between safe and dangerous behaviors. To support this, we develop an efficient data generation pipeline in simulation and construct a large-scale dataset encompassing over one million meters of navigation experience across 3,000 scenes. Empirical experiments in both simulated and real-world environments demonstrate that NavDP significantly outperforms prior state-of-the-art methods. Furthermore, we identify key factors influencing the generalization performance of NavDP. The dataset and code are publicly available at https://wzcai99.github.io/navigation-diffusion-policy.github.io.
I. INTRODUCTION
Dynamic open-world navigation requires zero-shot generalization across embodiments and unstructured scenes, but modular systems and real-world data collection limit scalability. NavDP addresses this with an end-to-end simulation-trained framework that combines diffusion-based trajectory generation with critic guidance from privileged simulation information.
- Dynamic open-world navigation demands zero-shot generalization across different robot embodiments and unstructured scenes.
- Modular navigation systems suffer system latency and compounding errors, while learning-based methods face scarce high-quality training data.
- Simulation provides diverse, scalable data and can generate customized navigation trajectories across scenes, observations, and goals.
- NavDP combines end-to-end transformer modeling, diffusion-based multimodal trajectory generation, and critic prediction for positive and negative trajectories.
- Privileged simulation information guides trajectory generation with a global-optimal planner and critic learning with contrastive samples and a global ESDF.
II. RELATED WORKS
Prior work spans diffusion policies, end-to-end visual navigation, and planning-based approaches, but scalable simulation training and safety-aware policy outputs remain important challenges. NavDP uses simulation data and a critic function to improve generalization and distinguish safer actions.
- Robot Diffusion Policy: Diffusion policies model multimodal robot behaviors but commonly depend on offline imitation-learning datasets that are labor-intensive to collect and scale.
- NavDP is presented as a simulation-trained framework for zero-shot sim-to-real policy transfer and cross-embodiment generalization.
- Robot Diffusion Policy: NavDP trains robot policies entirely from scalable simulation datasets rather than relying on difficult-to-scale real-world demonstration datasets.
- Robot Diffusion Policy: The critic function estimates policy-output safety and uses prioritized simulation data to represent action consequences during diffusion-policy learning.
- End-to-End Visual Navigation Models: End-to-end visual navigation models demonstrate cross-embodiment adaptation and multitask generalization, while vision-language-action models add language-based task specification.
III. DATAENGINE
NavDP’s data engine generates collision-free trajectories and photorealistic RGB-D observations across diverse 3D scenes. It combines robot and camera randomization with ESDF-based planning, waypoint refinement, and trajectory smoothing to support cross-embodiment training.
- Robot Model: The simulated robot uses a cylindrical body with two-wheel differential drive, while randomized height and camera pitch imitate observation variation across embodiments.The robot safe radius is set to rb = 0.25m and height is randomized from 0.25m to 1.25m.
- Trajectory Generation: Scene meshes are voxelized to estimate an ESDF, with height thresholds defining navigable and obstacle areas and truncation preventing collisions.The voxel size is 0.05m, and the navigable ESDF is downsampled to 0.2m for efficient A* planning.
- Trajectory Generation: A* plans paths between random navigable start and target points, after which local ESDF search moves waypoints away from obstacles and cubic splines smooths the trajectory.
- Scene Assets and Render Engine: The pipeline renders photorealistic RGB and depth images along navigation trajectories across more than 3,000 scenes from six 3D scene datasets.
IV. NAVIGATION DIFFUSION POLICY
NavDP uses a unified transformer architecture to generate and evaluate navigation trajectories from RGB-D observations. Its jointly trained actor and critic support diffusion-based generation and safety-oriented selection.
- Architecture: NavDP combines a multi-modal RGB-D encoder with a unified transformer for trajectory generation and critic value prediction.The generation head plans dense waypoints, while the critic predicts trajectory safety scores.
- Inputs: The encoder processes historical RGB frames, a depth frame, and a point-goal embedding, while no-goal tasks use a zero goal embedding.RGB and depth tokens are compressed before fusion with the navigation goal representation.
- Training: During training, the actor predicts DDPM-injected trajectory noise while the critic scores augmented collision-free and collision trajectories contrastively.The two objectives are jointly optimized in one training stage.
- Inference: At inference, NavDP generates candidate trajectories and selects the best one using the evaluation head.This combines diffusion-based trajectory generation with trajectory evaluation for safer execution.
- Training: The actor objective uses weighted MSE noise-prediction losses for point-goal and no-goal tasks.The loss conditions on denoising steps, depth, RGB history, navigation goals, expert trajectories, and the dataset.
- Training: The critic label incorporates absolute ESDF values and ESDF changes along augmented trajectory waypoints, with dsafe defining the collision-radius threshold.The critic loss trains a score-prediction network against these labels.
A. Evaluation and Metrics
NavDP is evaluated on point-goal and no-goal navigation in simulation and real-world settings across multiple robot embodiments. The benchmarks measure task success, path efficiency, collision-free exploration time, and explored area.
- Evaluation Settings: The evaluation covers point-goal navigation and no-goal exploration in simulation and real-world environments across four robot embodiments.Simulation uses IsaacSim and a ClearPath Dingo, while real-world evaluation includes Unitree Go2, Turtlebot4, and Unitree G1.
- Results: 6.3% success rate and 4.0% SPL are the reported simulation improvements over the previous state-of-the-art method for point-goal navigation.The reported real-world point-goal improvement is 23.4% success rate.
- Results: Nearly 2.9x exploration time and 3.1x exploration area improvements are reported in simulation, with 3.8x exploration-time improvement in real-world evaluation.These comparisons are reported against the previous method for no-goal navigation.
- Metrics: Point-goal navigation is measured by Success Rate and Success Weighted by Path Length, which capture completion and path efficiency.The real-world point-goal benchmark uses three indoor scenarios and ten episodes per scene with one embodiment.
- Metrics: No-goal navigation is measured by exploration Time before collision and explored Area.Cross-embodiment evaluation uses corridor, hall, and meeting-room scenarios.
B. Experiment Analysis
Experiments evaluate NavDP across robot platforms, environments, baselines, and ablations. Results highlight strong cross-embodiment transfer and identify RGB-D fusion, temporal context, critic-based selection, trajectory augmentation, and domain-randomized data as important factors.
- Baseline comparison: NavDP outperforms the previous state-of-the-art learning-based method by 6.3% in simulation SR and improves cross-embodiment real-world SR by 23.0%.The comparison covers PointGoal navigation and multiple baseline approaches.
- Baseline comparison: NavDP achieves 2.9x longer exploration time and 3.1x larger exploration area than NoMaD in simulation, plus 3.8x exploration time in the real world.These results come from no-goal navigation experiments.
- Ablation analysis: RGB-D fusion is essential: removing depth drops success rate by 10.3%, removing RGB drops it by 5.1%, and removing multi-frame RGB reduces it by 2.8%.The ablations compare RGB-only, depth-only, and single-frame variants with the original model.
- Ablation analysis: Critic-based trajectory selection decreases simulation success rate by 7.8% when removed, while removing critic-training trajectory augmentation decreases home-scene success by 3.0%.The results support the critic function and contrastive trajectory training as safety-related components.
- Ablation analysis: No-goal auxiliary objectives increase point-goal success rate by 2.1% and SPL by 1.8%.The auxiliary task is jointly trained with point-goal navigation.
- Cross-embodiment generalization: Without cross-embodiment data, Galaxea R1 success rate falls from 90% to 20%, while Unitree-Go2 performance remains maintained.The result indicates that domain-randomized cross-embodiment data supports robot-specific detour behavior.
VI. CONCLUSION & FUTURE WORKS
NavDP combines scalable simulation data with RGB-D fusion and contrastive critic training to support zero-shot sim-to-real and cross-embodiment navigation. Its generalization depends strongly on cross-embodiment data, while future work targets broader goals, post-training, and long-term exploration.
- NavDP demonstrates zero-shot sim-to-real and cross-embodiment generalization with real-time path planning and collision avoidance in static and dynamic scenarios.
- 70% success-rate drop on Galaxea R1 occurs when cross-embodiment data is removed, highlighting its importance for cross-embodiment performance.
- Its two key ingredients are an automatic simulation data-generation pipeline with a large-scale navigation dataset, plus efficient RGB-D fusion and contrastive critic training.
- Future work includes efficient post-training, natural-language navigation goals, and global memory for long-term exploration and holistic navigation.