Source-linked AI summary
NavGen: Visual Generative Models as a Scalable Data Engine for Embodied 3D Navigation
Xijie Huang, Yongyang Wan, Chengbin Dong, Zimo Ding, Mo Zhu, Yijin Wang, Zhiyang Liu, Fei Gao, Yuze Wu, Xin Zhou
TL;DR
Embodied 3D navigation data are difficult to scale while remaining realistic and diverse. NavGen composes visual generative models to create 400K navigation episodes, including diversified long-tail tasks. Models trained on this data outperform prior-dataset baselines on reported comparisons, and the deployed model achieves a 75% zero-shot success rate in unseen environments.
Problem
Navigation generalization depends on training environments and behaviors, yet existing sources struggle to provide data that are simultaneously realistic, diverse, and scalable.
Source: §I · p. 1Method
NavGen composes visual generative models to produce 300K common-task episodes and expands 10K long-tail samples to 100K through style diversification.
Source: §I · p. 1Results
17% higher navigation progress score than the best baseline, alongside 2.8× higher diversity and the smallest sim-to-real gap among non-real datasets; performance improves with dataset scale.
Source: §I · pp. 1–2Takeaways & Limitations
NavGen’s generated data support improved benchmark performance and real-world transfer, with 75% zero-shot success in unseen environments.
Source: §I · pp. 1–2
Abstract
from arXiv · showhide
General-purpose robot models increasingly rely on large and diverse datasets. For embodied 3D navigation, however, existing data sources face a fundamental trade-off: simulated data can be generated at scale but often suffer from the visual sim-to-real gap, whereas real-world flight data provide realistic observations but are costly to collect. This paper studies another direction: the use of high-fidelity visual generative models as scalable data engines for embodied 3D navigation. We introduce NavGen, a text-to-video data generation pipeline that produces diverse vision-language navigation (VLN) episodes across indoor and outdoor scenes. We also propose a style-diversification method that scales up long-tail data that are difficult and costly to collect. The resulting dataset contains approximately 400K navigation episodes. We evaluate our dataset against existing UAV navigation datasets across multiple metrics, and find that the model trained on our data generally improves with scale, outperforming those trained on existing datasets. To validate real-world transferability, we deploy the trained model in world-action-model paradigm to real-world flying experiments. The final model achieves a 75\% success rate across different navigation tasks and environments.
I. INTRODUCTION
NavGen uses visual generative models to build navigation data at scale, addressing the difficulty of making it simultaneously realistic, behaviorally rich, and diverse. Its 400K-episode dataset outperforms prior datasets on reported benchmark comparisons and supports real-world UAV navigation.
I. INTRODUCTION
- Navigation generalization depends on the environments and behaviors represented in training data, which must also be visually realistic and behaviorally rich.These data requirements make scaling navigation datasets difficult.Source: §I · p. 1
- Existing sources struggle to provide navigation data that are simultaneously realistic, diverse, and scalable.Simulation offers scale but has visual and dynamic gaps; real-world collection is costly, and larger homogeneous trajectory sets may add limited generalization supervision.Source: §I · p. 1
- NavGen combines 300K common-task episodes with 10K generated long-tail samples expanded to 100K, yielding a 400K-episode dataset.The long-tail samples are scaled through style diversification.Source: §I · p. 1, Figure 1 · p. 2
- 75% zero-shot success rate in unseen environments validates the trained model in real-world flying experiments.The reported deployment uses a physical UAV.Source: §I · p. 2
- 17% higher navigation progress score than the best baseline, with 2.8× higher diversity and the smallest sim-to-real gap among non-real datasets.The paper also reports that navigation performance consistently improves as the generated dataset grows.Source: §I · pp. 1–2
II. RELATED WORK
Prior navigation datasets rely on simulation or manual piloting, while generative-model research has explored visual augmentation and synthesized trajectories. The related work identifies limited realism, diversity, scalability, and navigation-specific generation as open challenges.
A. Embodied 3D Navigation Datasets
- Simulation- or manually piloted datasets, including TravelUAV and IndoorUAV, are limited in realism, scalability, and diversity.OpenFly reduces human operation cost but still depends on simulator appearance and manually designed task rules; UAV-Flow uses real trajectories from limited campus scenes, while VLA-AN and FlyMirage use 3DGS environments.Source: A. Embodied 3D Navigation Datasets · p. 2
B. Visual Generative Models for Embodied AI
- Visual augmentation methods improve appearance diversity but remain tied to the initial demonstration distribution.The cited examples include ROSIE [16], RoboTransfer, and RoboCurate [18].Source: B. Visual Generative Models for… · p. 2
- Trajectory-generation work demonstrates generative models’ potential, but diverse, high-quality dataset generation and its transfer to embodied navigation remain underexplored.DreamGen [19] requires manually provided high-quality, diverse initial frames; X-Humanoid [20] and Lucid-XR [21] target humanoid or manipulation policies.Source: B. Visual Generative Models for… · pp. 2–3
III. METHOD
NavGen builds a balanced data-generation pipeline for navigation videos, using diverse descriptions, motion rebalancing, and scalable video generation. Its methods address prompt homogeneity and motion-distribution bias while covering common and underrepresented motions.
III. METHOD
- The pipeline combines large-scale diverse descriptions, motion rebalancing, long-tail style diversification, video re-annotation, and a low-latency inference system.The section overview assigns these components to the following method stages.Source: §III · p. 3
A. Diverse Embodied 3D Navigation Descriptions for Text-to-Video Generation
- To reduce homogeneous prompts at hundreds-of-thousands scale, NavGen filters a 1M-object dataset to 300K targets and uses an embodied-navigation expert to generate detailed events around them.The descriptions include environment style, background objects, spatial layout, and target-related context; videos are generated with Wan2.2 14B T2V.Source: A. Diverse Embodied 3D Navigation… · p. 3
1) Motion Rebalancing:
- Motion rebalancing counters prompt-following errors that skew generated motion distributions, using horizontal mirroring for left-right imbalance and temporal reversal for forward-backward imbalance.Pi3 decodes each video's overall motion direction; the strategy improves balance without additional video generation.Source: 1) Motion Rebalancing · p. 3
2) Style Diversification:
NavGen expands difficult long-tail tasks through style diversification and applies filtering and motion-guided reannotation to improve data quality. Its training system uses supervised fine-tuning followed by inference acceleration for deployment.
2) Style Diversification:
- Style diversification expands 10K long-tail samples, including narrow-space and orbiting maneuvers, to 100K while preserving the main object shape and varying scene appearance.The process edits first frames and generates reference-conditioned videos using depth guidance from the original videos.Source: 2) Style Diversification · p. 3
C. Video Filtering and Description
- The pipeline filters videos below a VBench threshold or navigation grade L2, regenerates them, and reannotates videos using 11 uniformly downsampled frames plus Pi3-decoded motion direction.The motion cue improves the quality and consistency of the generated descriptions.Source: C. Video Filtering and Description · p. 3
D. Training System Design
- For real-world deployment, NavGen trains a lightweight Wan2.2 5B model through bidirectional long-video generation and supervised fine-tuning, then accelerates inference with distillation and a lightweight VAE.The evaluation adopts NavDreamer’s video-prediction and action-decoding pipeline.Source: D. Training System Design · p. 4
1) Bidirectional Long-Video Generation via Clean-Prefix History Conditioning:
The paper uses bidirectional generation with clean-prefix history conditioning to extend navigation videos across clips, then trains the student on sparse, reduced-resolution video data. Randomized prefix lengths unify image-to-video and video-to-video training, while supervised fine-tuning initializes the student before distillation.
1) Bidirectional Long-Video Generation via Clean-Prefix History Conditioning:
- Randomized clean-prefix lengths let one bidirectional model handle image-to-video and video-to-video continuation across clips.The method addresses lower success rates from autoregressive generation and inconsistent motion when each clip is conditioned only on its anchor frame.Source: 1) Bidirectional Long-Video Generation… · p. 4, Figure 2 · p. 4
2) Supervised Fine-Tuning:
- Training at 640 × 384 with 33 frames preserves task completion while reducing spatial and temporal redundancy.Videos are resized and downsampled from 81-frame outputs; this design reduces generation time from 171.57s to 12.25s.Source: 2) Supervised Fine-Tuning · p. 4
- Supervised fine-tuning adapts the student to generated navigation data and provides its initialization before step distillation.The flow-matching objective uses the clean video latent, text condition, diffusion timestep, and Gaussian noise.Source: 2) Supervised Fine-Tuning · p. 4
3) Step Distillation:
The pipeline combines consistency and distribution-matching distillation, plus an adversarial loss, to accelerate video generation. Sparse low-resolution generation, step reduction, and a lightweight VAE bring inference to 0.7 seconds per clip, or 47 FPS.
3) Step Distillation:
- A 2.5K-step sCM warmup stabilizes the student before stronger distribution-level distillation.The warmup uses a continuous-time consistency objective under the TrigFlow parameterization.Source: 3) Step Distillation · p. 4
- DMD2-style distillation matches the student’s generated video-latent distribution to the teacher through velocity differences.The student and fake models start from the sCM-warmed checkpoint, while the teacher uses the SFT checkpoint.Source: 3) Step Distillation · p. 4
- An adversarial loss on diffused samples is added to the student objective to improve visual quality.The final student objective combines DMD and GAN losses.Source: 3) Step Distillation · p. 4
- 0.7 seconds per clip, or 47 FPS, is achieved by combining sparse low-resolution generation, step distillation, and a lightweight VAE.The optimizations include a 14× speedup from sparse 640 × 384, 33-frame processing and a 17× reduction in DiT computation from 30 denoising steps to 4 with CFG removed.Source: 3) Step Distillation · pp. 4–5
IV. EXPERIMENTS
The dataset analysis finds continuous egocentric trajectories with diverse rotations and vertical motion, while metric-scale correction supports quantitative analysis of generated motion.
A. Dataset Analysis
- Pi3 trajectory scale ambiguity is corrected by aligning MoGe-2 metric depth with rendered point-cloud depth, then applying the resulting scale factor to the trajectory.This produces physically accurate motion sequences for quantitative analysis.Source: A. Dataset Analysis · p. 5
- Generated trajectories are mainly short-range but include diverse yaw rotations and vertical movements; motion rebalancing corrects biases toward particular directions.Video generation can deviate from specified motion trends, creating distributional imbalance.Source: A. Dataset Analysis · p. 5
- Figure 4 summarizes motion distribution, text descriptions, and analyses of three specific dimensions.Source: Figure 4 · p. 5
B. Datasets Benchmark
NavGen scores highest on semantic diversity among the compared datasets and has the lowest CLIP-feature distance to real deployment images among non-real-world datasets.
1) Datasets Diversity and Sim-to-real Benchmark:
- Figure 5 presents Vendi Score, FD CLIP, and a CLIP-feature distribution visualization for comparing dataset diversity and real-world alignment.Source: Figure 5 · p. 6
- Our dataset surpasses the next-highest competitor, IndoorUAV (16.8), by a factor of 2.8×, indicating substantially richer coverage across scene types, lighting conditions, and viewpoints.Source: 1) Datasets Diversity and Sim-to-real… · p. 6
- 0.806 FDCLIP is the lowest among non-real-world datasets and approaches real-flight UAV-Flow's 0.691.The distance is computed over CLIP features against 1,000 held-out real-world deployment images.Source: 1) Datasets Diversity and Sim-to-real… · p. 6
2) Video Model Trained With Different Datasets Benchmark:
Across the dataset comparisons, NavGen training improves navigation progress under both training paradigms and reaches higher scores than the alternatives. The benchmark also evaluates generated-video quality and checks agreement between automatic and human navigation ratings.
2) Video Model Trained With Different Datasets Benchmark:
- The benchmark fine-tunes Wan2.2 5B separately on five datasets and evaluates visual quality on 1,000 real-world text-image pairs using VBench.Training uses autoregressive buckets for videos grouped by similar length.Source: 2) Video Model Trained With Different… · p. 6
- Navigation evaluation uses 1,000 authentic text-image pairs and a four-level rubric, with L3 requiring the prompted motion and target view to be reached.Source: 2) Video Model Trained With Different… · p. 6
- 56.9% of automatic scores exactly match the human expert, and 89.7% agree within one rubric level.The 89.7% includes an additional 32.8% differing by exactly one level.Source: 2) Video Model Trained With Different… · p. 6
- At one epoch, NavGen reaches around 63% progress score autoregressively and 79% bidirectionally, above all comparison datasets.UAV-Flow stays below 30%, TravelUAV reaches around 36%, IndoorUAV saturates around 45%, and OpenFly fluctuates between 33% and 46%.Source: 2) Video Model Trained With Different… · p. 6
1) Motion Rebalancing:
The experiments assess motion rebalancing, style diversification, and real-world deployment. Rebalancing and diversification improve their reported navigation measures, and the deployed model succeeds on 75% of real-world trials.
1) Motion Rebalancing:
- Motion rebalancing improves navigation success rate by approximately 15% versus training without rebalancing.The ablation compares separately trained models under identical conditions.Source: 1) Motion Rebalancing · p. 6
2) Style Diversification:
- 93.75% progress score at 100K diversified samples is a 15.6-point improvement over the original 10K-data setting.Without diversification, progress plateaus at 78.1% after 50K samples and gains nothing further.Source: Figure 7 · p. 7, 2) Style Diversification · p. 7
1) Real-World Deployment:
- Real-world deployment sends camera frames to server-side video generation, decodes generated video into waypoints, and uses Ego-Planner to optimize a collision-free trajectory.The setup uses a DJI Osmo Action 5 Pro for RGB observations and LiDAR for pose estimation.Source: 1) Real-World Deployment · p. 7
D. Real-World Experiment; 1) Real-World Deployment:
- 75% overall success is achieved across 20 real-world tasks with five trials each.Failures primarily involve targets that are too small or have little color contrast with the background.Source: Figure 8 · p. 7, 1) Real-World Deployment · p. 7
3) Long Horizon Experiment:
NavGen’s generated data supports stronger navigation benchmark performance and approximately 75% zero-shot task success in real-world deployment. Long-horizon generation maintains stable success when the target is clearly visible, but success drops when it is ambiguous.
3) Long Horizon Experiment:
- Long-horizon generation maintains stable success when the target is clearly visible in the first frame, but success drops to 50% when the target is ambiguous.Odometry retrieves previously observed RGB frames from the onboard buffer as context for resuming generation, helping new segments inherit the preceding state and maintain motion continuity.Source: 3) Long Horizon Experiment · p. 7, Figure 9 · p. 8
V. CONCLUSIONS
- NavGen combines 300K episodes from an open-source video model with long-tail samples from a closed-source model, expanded through style diversification to 400K episodes total.Motion rebalancing corrects distribution biases in the generated data.Source: §V · p. 7, 2) Short Horizon Experiment · p. 8
2) Short Horizon Experiment:
- Models trained on NavGen data consistently outperform models trained on prior datasets on video-quality metrics and navigation success, with approximately 75% task success in zero-shot deployment.The deployment spans diverse indoor and outdoor environments.Source: 2) Short Horizon Experiment · p. 8