NavDreamer: Video Models as Zero-Shot 3D Navigators

1 Introduction

• Robotic demonstrations are costly to collect, and existing datasets remain centered on constrained environments despite the need for environmental and object diversity.Simulation and domain randomization scale data generation, but the sim-to-real gap can degrade real-world deployment.Source: §1 · pp. 1–2
• Static image-text backbones lack the expressiveness to represent temporal dependencies and physical events such as collisions and falling.The paper describes this as a lossy representation that limits world reasoning.Source: §1 · p. 2
• Generative video offers internet-scale data, spatiotemporal representations, and physical-dynamics understanding as a basis for zero-shot generalization.The paper presents video as a shared pixel-space state-action interface across robotic platforms.Source: §1 · p. 2
• NavDreamer uses generative video as a high-level navigation planner, with VLM-based sampling selection and inverse dynamics decoding executable waypoints.Metric depth priors rectify absolute scale ambiguity before calibrated waypoints reach a reactive controller.Source: §1 · p. 2
• The evaluation suite covers object navigation, precise navigation, spatial grounding, scene reasoning, and language control.The authors also study design choices through ablations.Source: §1 · p. 3

2.1 Generative Video Models for Robotics

• Video-model robotics work spans trajectory synthesis for policy pretraining, direct-training VLA paradigms, and visual planning that decodes actions from generated future frames.Action extraction includes inverse dynamics models [29] and mappings into latent action spaces via VQ-VAEs [46] [47].Source: §2.1 · p. 3

2.2 3D Navigation and Autonomous Systems

• Aerial navigation systems achieve precision in specialized tasks, but open-world semantic reasoning and broad task generalization remain objectives.AerialVLN [52] and CityNav [53] provide outdoor benchmarks; other approaches rely on annotated post-training data, rule-based cues, or constrained image-goal settings [37].Source: §2.2 · p. 3

3 Methods

• Given an image and instruction, NavDreamer generates a predictive video, uses Qwen3-VL to select a trajectory, and decodes sampled frames into waypoints with π3.The selected waypoints guide low-level planning, with metric depth estimates used to address π3's outdoor scale ambiguity.Source: §3 · pp. 3–4

3.1 Optimization through Generative Sampling

• NavDreamer generates K video candidates, filters for task-consistent, collision-free execution, scores valid samples on safety, scene consistency, and task performance, then selects the highest-reward video.If no valid candidate exists, a human supervisor decides whether to resample or terminate.Source: §3.1 · p. 4

3.2 High-Level Action From Video

• π3 point maps and metric-depth references support a global scale estimate that transforms normalized waypoints into metric-space trajectories.A per-frame mask filters outliers and extreme distances, typically restricting observations to 0.5–30 meters; median consensus makes the scale estimate robust to depth noise and geometric misalignment.Source: §3.2 · pp. 4–5

3.3 Low-level Trajectory Generation

• Ego-Planner converts 3D waypoint goals into real-time collision-free trajectories, advancing to the next waypoint when the current target is within a distance threshold.Trajectory optimization uses maximum velocity and acceleration from the waypoint sequence as adaptive constraints, with yaw controlled independently.Source: §3.3 · p. 5

4.1 Evaluating Video Models

• Wan 2.6 leads nearly all metrics; HunyuanVideo-1.5 has target-centric strengths but weaker scene reasoning, Cosmos 2.5 falters on rare actions, and LVP remains still.HunyuanVideo-1.5 surpasses Wan 2.2 in object navigation and spatial grounding; LVP’s high consistency and feasibility coincide with a low success rate.Source: Table 1 · p. 6, §4.1 · p. 7

4 Experiments

• The evaluation spans 15 tasks across five dimensions and compares open- and closed-source models, alongside questions on action extraction, generalization, sampling, and prompt design.Source: §4 · pp. 5–6, §4.1 · p. 6

4.1 Evaluating Video Models

• Scores assess visual consistency, dynamic feasibility, and task completion; five independent samples are used per model and task, with experienced UAV pilots providing absolute ratings.The authors report that Qwen3-VL supports relative candidate assessment but lacks calibration for isolated absolute scores.Source: §4.1 · pp. 6–7

4.2 Real-World Results

• Real-world tests report zero-shot generalization across indoor and outdoor navigation scenes, with generated video decoded into waypoints for low-level trajectory planning.The platform uses a RealSense camera, LiDAR, and Fast-LIVO2 pose estimation; low-level planning and pose estimation run on a Jetson Orin NX.Source: §4.2 · p. 7
• Video models also zero-shot restore high-definition scenes from degraded inputs and expand the observation space through dreaming.Source: §4.2 · p. 7

4.3 Ablation Studies

• Outdoor scale correction reduces relative error to the 10% level after π3’s decoded physical scale is approximately 46% of ground truth.The authors attribute the initial discrepancy to difficulty finding implicit references in unseen outdoor environments.Source: §4.3 · p. 7, Figure 4 · p. 8
• Increasing the sampling budget significantly improves task success; evaluating multiple predicted futures with a VLM helps select high-value actions and avoid unsafe behavior.Source: Figure 5 · p. 8, §4.3 · p. 8
• Dynamic-aware prompts beat simple prompts by 6% in dynamic feasibility and 5% in task success; decomposed prompts improve success by 27%, while rewritten prompts trail them by 16%.The authors attribute the decomposed-prompt gain to detailed action decomposition and say automated rewriting can introduce semantic ambiguity when the initial prompt lacks detail.Source: Table 2 · p. 8, §4.3 · p. 8

5 Conclusion

• NavDreamer combines generative video models, VLM-based trajectory selection, and an inverse dynamics model for zero-shot 3D navigation.The framework also uses metric depth priors for scale-accurate navigation.Source: §5 · pp. 8–9
• The benchmark spans object navigation, precise navigation, spatial grounding, scene reasoning, and language control.Source: §5 · p. 9
• Experiments demonstrate robust generalization to unseen environments, bridging visual imagination and physical action.Source: §5 · p. 9

6 Limitation

• High-agility, high-precision tasks such as aerobatics and narrow-hoop traversal remain challenging, and 1–2-minute video synthesis latency constrains high-reactivity scenarios.The authors propose specialized aerial-robot fine-tuning and quantization or compression as future directions.Source: §6 · p. 10

A Overview

• The supplement covers four failure modes, benchmark specifications, robust-perception experiments, prompt-design ablations, structured VLM prompts, and video results.Source: A Overview · p. 14

B Experiment Details

• Figure 6 identifies four main failure modes in 3D navigation.Source: Figure 6 · p. 14

B.1 Failure Modes of Video Models in 3D navigation

• Pattern bias makes models follow common training-data patterns rather than current scene logic, such as animating an unpowered fan or stationary curtains.Source: §B.1 · p. 14
• Reward hacking can satisfy visual requirements without real movement, including hallucinating a nearby tree or zooming while the camera remains static.Source: §B.1 · pp. 14–15
• Mode collapse destabilizes rare actions and major scene transitions, fragmenting hoops or duplicating branches during orbital flight.These fluctuations indicate loss of spatial and temporal coherence under complex dynamics.Source: §B.1 · p. 15
• Instruction neglect can omit requested constraints, including vertical motion, and sampling-based optimization effectively mitigates this behavior.Source: §B.1 · p. 15

• Table 3 presents the task suite for 3D drone navigation.Source: Table 3 · p. 15

B.3 Emergent Perception in Dreaming

• Video models reconstruct high-definition scenes from blurred, low-resolution, or poorly lit inputs, enhancing drone-navigation robustness and reliability in complex environments.Source: §B.3 · p. 16
• During the downstairs task, the model tilts its synthesized view downward to imagine observations beyond the initial camera field of view.The authors describe this as enumerating future environment observations from a partial snapshot.Source: §B.3 · p. 16

B.4 Prompt Design Ablation Experiment

• The prompt-design ablation lists simple, refined, and detailed prompts for a 360-degree orbit around a green tree.The prompts progressively specify motion smoothness, camera height and orientation, speed, endpoint, and scene consistency.Source: §B.4 · p. 16, Table 4 · p. 17

Prompt for Video Quality Ranking and Task Verification

• Video ranking first checks task completion and failure conditions, then scores task performance, action safety, and scene consistency, weighting task performance most heavily.The total score is (TP * 1.4 + AS * 0.8 + SC * 0.8) / 3.Source: Prompt for Video Quality Ranking and… · p. 18

• Figure 11 presents visual samples of the 3D navigation benchmark tasks.Source: Figure 11 · p. 19
• One task instruction is to fly through the middle of the right hoop.Source: §B.6 · p. 20
• Figures 12–15 visualize generated videos and real-world performance from different perspectives.Source: Figure 12 · p. 20, Figure 13 · p. 21, Figure 14 · p. 22