Source-linked AI summary
Navigation World Models
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, Yann LeCun
TL;DR
Visual navigation models often struggle to incorporate new constraints and adapt computation during planning. The paper introduces NWM, a controllable video world model built with CDiT, and shows that it can plan or rank trajectories in familiar environments while improving prediction in unfamiliar ones using unlabeled video.
Problem
Existing supervised navigation policies are hard-coded after training, making new constraints difficult to introduce and limiting dynamic computation for hard problems.
Method
NWM predicts future visual states from past states and navigation actions, using a scalable CDiT trained on diverse human and robotic egocentric videos.
Results
NWM achieves state-of-the-art standalone visual-navigation performance, improves external-policy ranking, and produces more accurate predictions in unfamiliar environments with additional Ego4D video.
Takeaways & Limitations
World-model simulation supports navigation planning independently or alongside existing policies, while learned visual priors support trajectory imagination from single images in unfamiliar environments.
Takeaways & Limitations
Out-of-distribution generation can lose context and exhibit mode collapse, while temporal dynamics such as pedestrian motion remain difficult to simulate.
Abstract
from arXiv · showhide
Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional Diffusion Transformer (CDiT), trained on a diverse collection of egocentric videos of both human and robotic agents, and scaled up to 1 billion parameters. In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal. Unlike supervised navigation policies with fixed behavior, NWM can dynamically incorporate constraints during planning. Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy. Furthermore, NWM leverages its learned visual priors to imagine trajectories in unfamiliar environments from a single input image, making it a flexible and powerful tool for next-generation navigation systems.
1. Introduction
The paper introduces NWM to address navigation models’ limited flexibility under new constraints and limited compute allocation. NWM predicts future visual states with a scalable CDiT and supports trajectory planning, external-policy ranking, and prediction in unfamiliar environments.
- Motivation: Current supervised navigation policies cannot easily incorporate new constraints after training or dynamically allocate more computation to difficult problems.The paper motivates NWM with examples such as imposing “no left turns” during planning.
- Navigation World Model: NWM predicts future image representations from past image representations and navigation actions, using video footage and navigation actions from diverse robotic agents.Its world-model formulation supports controllable prediction of the agent’s next visual state.
- Navigation Planning: NWM plans trajectories by simulating candidate navigation plans and checking whether they reach a target, either independently or through ranking trajectories from an external policy.The ranking setup can select among pools of sampled trajectories, while the planning setup uses model predictive control.
- Conditional Diffusion Transformer: CDiT scales to 1B parameters across diverse environments and embodiments while using 4× fewer FLOPs than standard DiT and achieving better future prediction results.Its computational complexity is linear in the number of context frames.
- Unfamiliar Environments: Additional unlabeled, action- and reward-free Ego4D videos improve NWM’s video prediction and generation performance in unfamiliar environments.With this data, NWM produces more accurate predictions on the held-out Stanford Go dataset.
2. Related Work
Related work frames goal-conditioned navigation as a joint perception-and-planning problem and distinguishes world models from policies by their ability to simulate future states. Prior research spans analytical planning, reinforcement learning, shared or language-conditioned world models, diffusion-based game engines, and video-generation applications.
- Goal-Conditioned Navigation: Goal-conditioned visual navigation models generate paths toward specified goals or explore unfamiliar environments, requiring both perception and planning.Prior approaches include diffusion policies, neural SLAM with analytical planners, and reinforcement-learned policies.
- World Models: World models simulate an environment by predicting the next state and associated reward from the current state and action, unlike policies that directly learn behavior.Earlier work studied joint policy–world-model learning for improved sample efficiency across Atari, simulated robotics, and real-world robots.
- World-Model Extensions: Previous world-model research introduced shared models across tasks through action and task embeddings, language-described actions, or latent actions.Diffusion-based world models also learned game engines for environments such as Atari and Doom.
- Video Generation: Video-generation models have been applied to action-controlled synthesis, reinforcement-learning rewards and pretraining, manipulation planning, and indoor path generation.This work connects those video-generation applications with visual-navigation planning.
3. Navigation World Models
The Navigation World Model predicts future visual states from observations and navigation actions, then supports trajectory planning through autoregressive simulation. Its formulation adds temporal control and uses a Conditional Diffusion Transformer to model stochastic dynamics efficiently.
- 3.1. Formulation: NWM maps current visual state and navigation action to the next world state from the agent’s point of view.Actions specify translation and yaw rotation, while observations are represented as pretrained-VAE latents.
- 3.1. Formulation: The model supports a single world model across environments and embodiments and can extend to more complex action spaces.The formulation is designed to share dynamics without task or action embeddings.
- 3.1. Formulation: Temporal shifts let NWM learn environment dynamics while controlling how far forward or backward the model moves in time.Training permits time shifts of up to ±16 seconds, with navigation actions approximated across the shifted interval.
- 3.1. Formulation: Sampling multiple goals per state encourages natural counterfactuals and mitigates entanglement between actions and time.Without such variation, the model may rely solely on temporal cues or ignore subsequent actions.
- 3.2. Diffusion Transformer as World Model: CDiT models stochastic future states with temporally autoregressive attention and cross-attention from the target frame to past-frame tokens.The diffusion process denoises a noisy target state conditioned on past states, actions, and the diffusion timestep.
- 3.2. Diffusion Transformer as World Model: CDiT reduces attention complexity from O(m^2n^2d) for full DiT attention to O(mn^2d), linear in the number of context frames.The block avoids expensive self-attention over context tokens, enabling longer context sizes.
- 3.3. Navigation Planning with World Models: NWM plans by autoregressively rolling out candidate action sequences and minimizing an energy that combines goal similarity with action and state constraints.Goal similarity is measured after decoding predicted and target latents to pixels; invalid actions or unsafe states incur large indicator penalties.
4. Experiments and Results
Experiments evaluate NWM’s prediction, synthesis, ablation, and planning capabilities across known and unknown environments. Results show strong future prediction, efficient CDiT scaling, improved trajectory planning, constraint adherence, and benefits from unlabeled video data, with degradation during long imagined traversals.
- Experimental setting: The evaluation covers robotics datasets, unlabeled Ego4D videos, and the unknown GO Stanford environment, using trajectory, perceptual, pixel-level, and distributional metrics.Metrics include ATE, RPE, LPIPS, DreamSim, PSNR, FID, and FVD.
- Ablations: CDiT performs better than DiT at comparable model sizes while using less than 2× FLOPs, and equal-parameter CDiT-L is 4× faster than DiT-XL.The comparison measures LPIPS for four-second future prediction on RECON.
- Ablations: Using four goals and more conditioning frames improves prediction, while time-only conditioning performs poorly and omitting time causes a smaller performance drop.These ablations report four-second-ahead prediction on RECON.
- Video prediction: NWM predictions are significantly more accurate than DIAMOND over time, although accumulated errors and lost context eventually make 4 FPS preferable after eight seconds.The comparison evaluates predictions at 1 and 4 FPS through 16 seconds on RECON.
- Video synthesis: NWM generates higher-quality 16-second videos than DIAMOND when autoregressively predicting at 4 FPS under ground-truth actions.Video quality is evaluated with FVD.
- Planning: Standalone planning produces competitive goal-conditioned navigation and maintains effective performance under action constraints with only minor drops.Constraints are enforced by zeroing specific actions, and performance is measured by final position and yaw differences relative to unconstrained planning.
- Planning: NWM-based ranking improves navigation performance when selecting among NoMaD trajectories, with more sampled trajectories yielding better results.Trajectories are simulated and ranked using LPIPS similarity to the goal image, with ATE and RPE reported on in-domain datasets.
- Unknown environments: Training with unlabeled Ego4D video improves prediction and generation metrics on unknown environments, but imagined traversals break faster and hallucinate paths compared with in-domain results.Evaluation includes GO Stanford and random images.
5. Limitations
NWM has limitations on out-of-distribution prediction, temporal dynamics, and action dimensionality. In unfamiliar settings, it may lose context and resemble its training data.
- In out-of-distribution data, NWM can gradually lose context and produce states resembling its training data, a failure known as mode collapse.The paper illustrates this failure case in Figure 10.
- NWM struggles to simulate temporal dynamics such as pedestrian motion, although it succeeds in some cases.
- The current model uses 3 DoF navigation actions, while extending it to 6 DoF or robotic-arm joint control remains future work.
6. Discussion
NWM provides a scalable, data-driven approach to visual-navigation world modeling without explicitly using a structured environmental map. The authors suggest that egocentric next-frame prediction may support allocentric representations.
- NWM learns a scalable, data-driven world model for visual navigation without explicitly utilizing a structured map of the environment.
- The authors hypothesize that egocentric next-frame prediction can drive the emergence of allocentric representations.
- The approach connects learning from video, visual navigation, and model-based planning, potentially supporting self-supervised systems that perceive and plan.
7. Standalone Planning Optimization
Standalone planning uses the Cross-Entropy Method to optimize trajectory endpoints by repeatedly sampling, simulating, evaluating, and selecting candidate solutions under navigation constraints.
- Standalone goal-conditioned planning uses a pretrained NWM and the Cross-Entropy Method to optimize navigation trajectories.
- The unconstrained trajectory is modeled as a straight line, with optimization over one translation and one yaw-rotation endpoint variable.
- The optimizer maps the endpoint tuple into eight evenly spaced delta steps, applying the yaw rotation at the final step with k = 0.25 seconds between steps.
- Each iteration samples N = 120 candidate solutions, simulates them with NWM, and scores them using averaged LPIPS over repeated evaluations.
- Navigation constraints zero out parts of the trajectory; in the forward-first scenario, translation uses u = (∆x, 0) for five steps, then u = (0, ∆y) for three.
8. Experiments and Results
Experiments evaluate trajectory accuracy, perceptual and pixel-level prediction quality, datasets, planning, runtime, and adaptation. NWM improves navigation results, benefits from added unlabeled data in unknown environments, and can reach real-time inference with efficiency techniques.
- Evaluation Metrics: Trajectory evaluation uses ATE for overall Euclidean trajectory accuracy and RPE for consistency of consecutive relative poses.
- Evaluation Metrics: Image and video quality are assessed with LPIPS, DreamSim, PSNR, FID, and FVD, with lower FID and FVD indicating higher visual quality.
- Datasets: The robotics data combines SCAND, TartanDrive, RECON, and HuRoN, while Ego4D contributes unlabeled videos using time shift as the action.
- Evaluation Sets: Evaluation sets select 100 trajectories least predictable by forward-only motion, while time-prediction tests randomly select 500 examples per dataset.
- Additional Unlabeled Data: Training with additional unlabeled Ego4D data improves prediction on the unknown Stanford Go environment, while in-domain data generally performs better in known environments except SCAND.
- Planning: NWM improves all reported ATE and RPE metrics over NoMaD and GNM, and ranking 32 trajectories improves over ranking 16.
- Planning: Standalone planning outperforms other approaches, including previous hard-coded policies, while TartanDrive results reflect its strong forward-motion bias.
- Real-World Applicability: Existing efficiency methods enable real-time NWM applications at 2-10HZ, with time skipping preserving navigation performance and distillation causing minor visual-quality loss.