Pathdreamer: A World Model for Indoor Navigation

1. Introduction

• Pathdreamer targets visual world modeling for indoor navigation by generating plausible high-resolution observations at unvisited viewpoints in buildings unseen during training.The model supports RGB, depth, and semantic segmentation observations for future navigation trajectories.Source: §1 · pp. 1–2
• The task requires completing partially observed scenes, imagining entire unseen rooms around corners, and balancing stochastic novelty with consistency in revisited regions.These challenges include single-observation completion, the room reveal problem, and temporal consistency across predictions.Source: §1 · pp. 1–2
• Pathdreamer uses a hierarchical two-stage design: Structure Generator predicts depth and semantics with latent stochastic information, while Image Generator renders RGB using those predictions and 3D context.Back-projected point clouds are re-projected into image space to maintain long-term consistency.Source: §1 · p. 2
• The model generates plausible views under large viewpoint changes and can hypothesize the likely contents of an unseen room, such as a kitchen.Experiments evaluate the model on Matterport3D using 360° observations, baselines, and ablations.Source: §1 · p. 2
• Pathdreamer predictions retain around half the benefit of looking ahead at actual observations for Vision-and-Language Navigation planning.This downstream result motivates visual world models for embodied tasks including object navigation, instruction-guided navigation, and dialog-guided navigation.Source: §1 · p. 2

Video Prediction

• Pathdreamer adapts hierarchical video prediction and 3D point-cloud representations to action-conditional future-view generation along specified trajectories.Unlike typical video generation, it receives future viewpoints rather than relying on constant frame-rate camera motion.Source: §2 · p. 2

Action-Conditional Video Prediction

• Prior action-conditional video prediction largely addressed simple games, tabletop robotics, and low-resolution imagery, whereas Pathdreamer targets building-scale environments with 1024×512 images.The paper positions this as the first investigation of action-conditional video prediction in such settings.Source: §2 · p. 2

World Models and Navigation Priors

• Other world models predict compact latent states, task-specific measurements, rewards, or navigation priors, while Pathdreamer explicitly predicts generic, human-interpretable visual observations.The pixel-level output is intended to support varied downstream tasks and applications.Source: §2 · pp. 2–3

Embodied Navigation Agents

• Embodied navigation research has mainly used model-free approaches, motivating visual world models that encode environmental regularities for tasks such as ObjectNav and VLN.Ground-truth look-ahead with beam search improves R2R VLN success rates by 17-19%, underscoring the potential value of predictive visual planning.Source: §2 · p. 3

Novel View Synthesis

• Novel view synthesis methods include point clouds, layered depth images, meshes, and NeRFs, but NeRFs typically require many views and generalize poorly to unseen environments.Pathdreamer instead aims to generalize across unseen buildings from as little as one previous observation.Source: §2 · p. 3

3. Pathdreamer

• Pathdreamer generates realistic RGB, semantic, and depth observations for future viewpoints, including unseen regions and complete room reveals.Inputs include previous multimodal observations, camera poses, and a future trajectory; outputs can be generated iteratively or from a trajectory specified in advance.Source: §3 · p. 3

3.1. Structure Generator: Segmentation & Depth

• The Structure Generator uses reprojected 3D semantic and depth context plus latent noise to produce diverse plausible segmentation and depth layouts.The latent distribution is trained against ground-truth-conditioned posteriors and sampled from a learned prior during inference.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2. Image Generator: RGB

• The Image Generator converts predicted semantic and depth images into realistic RGB views with an image-to-image translation GAN.SPADE-based blocks condition generation on predicted structure and sparse RGB guidance reprojected from the accumulated point cloud.Source: §3.2 · p. 4
• The model can sample multiple plausible scene alternatives, and its RGB outputs closely reflect the corresponding generated semantic layouts.This stochasticity supports alternative predictions for the same navigation trajectory, especially around unseen corners.Source: §3.1 · p. 4, Figure 3 · p. 5

3.3. Training and Inference

• Training uses Matterport3D RGB panoramas and 400k trajectories sampled using R2R navigation graphs, with 5–8 panoramas per trajectory and viewpoints averaging 2m apart; stages are trained separately.The Image Generator uses 1024×512 RGB panoramas, while the Structure Generator uses 512×256 panoramas; predictions degrade after 8.6m in an example sequence.Source: §3.3 · p. 5, Figure 4 · p. 6

4.1. Pathdreamer Results

• Pathdreamer performs far better than nearest-neighbor interpolation across rollout lengths and context sizes, with higher performance in seen than unseen environments.Source: §4.1 · p. 6, Table 1 · p. 7
• Recurrent training improves longer-rollout results in Val-Seen but not Val-Unseen, suggesting its learned error compensation does not generalize easily.Source: §4.1 · p. 6
• Pathdreamer generates diverse semantic scenes through noise interpolation, with RGB outputs reflecting those alternatives for the same trajectory.Source: §4.1 · p. 6
• RGB context improves Val-Unseen generation, while removing semantic inputs substantially degrades longer-horizon image quality; ground-truth semantics further improve FID.Source: §4.1 · p. 7, Table 2 · p. 7

4.2. VLN Results

• The VLN system ranks beam-search trajectories with an instruction-trajectory compatibility model and evaluates ground-truth, Pathdreamer, repeated-pano, and blank-pano future observations over one to three steps.Source: §4.2 · pp. 7–8
• Ground-truth look-ahead raises VLN success from 44.6% with one planning step to 59.3% with three, while Pathdreamer reaches 50.4% versus 40.6% for the repeated-pano baseline at three steps.Source: §4.2 · p. 8, Table 3 · p. 8

• Pathdreamer synthesizes realistic and diverse 360° panoramic images for unseen trajectories in real buildings.Source: §5 · p. 8
• In VLN, Pathdreamer captures around half the benefit of looking ahead at actual observations from the environment.Source: §5 · p. 8
• Its VLN efficacy may stem from modeling real-world constraints that reduce the need to learn buildings’ geometric, visual, and semantic structure.Source: §5 · p. 8

• Val-Unseen tests Pathdreamer in environments not seen during training, directly probing generalization.Source: §A.1 · p. 12
• Figure 7 presents cherry-picked examples of Pathdreamer-generated sequences on Val-Unseen.Source: §A.1 · p. 12
• Figure 8 presents randomly chosen Pathdreamer-generated sequences on Val-Unseen.Source: §A.1 · p. 12

• The Structure Generator and Image Generator perform significantly better on Val-Seen because they can memorize training environments with less generalization required.Source: §A.2 · p. 12
• Val-Seen contains novel navigation trajectories through environments seen in the training split.Source: §A.2 · p. 12
• Generation results maintain high fidelity even at large distances from the input observation.Source: §A.2 · p. 12

• Interpolating different noise vectors produces diverse and plausible outcomes in environment areas not seen from the previous view.Source: §A.3 · p. 12
• The Structure Generator generates alternative room reveals for a given scene.Source: §A.3 · p. 12
• Figures 11 and 12 show cherry-picked and randomly selected sequences conditioned on different or random noise vectors.Source: §A.3 · p. 12

• Pathdreamer generates continuous video sequences by sequencing generated images with small viewpoint changes.Videos show generated results for several unseen environments.Source: B. Generated Videos · p. 12
• The paper compares multi-observation Pathdreamer renderings with Habitat’s mesh-based rendering and reports a favorable quality comparison.Source: B. Generated Videos · p. 12
• The generated videos extend Pathdreamer’s image-generation capability to unseen indoor environments.Source: B. Generated Videos · p. 12

Training configuration

• The model uses separate training configurations for the Structure Generator and Image Generator, including different batch sizes, epoch counts, optimizers, and loss settings.The Structure Generator uses batch size 64 over 50 epochs, while the Image Generator uses batch size 128 over 500 epochs.Source: C. Implementation Details · p. 12

Training procedure

• Pathdreamer is trained first with teacher forcing and later recurrently, using previous predictions as inputs during rollout.Teacher forcing lasts 30 epochs before switching to the recurrent setting.Source: C. Implementation Details · p. 12

Evaluation details

• FID evaluation uses 10,000 random samples per prediction-sequence step, with augmentation for the Val-Unseen and Val-Seen validation sets.Source: C. Implementation Details · p. 12
• Checkpoint selection evaluates every 2,000 training steps and separately reports the best Val-Seen and Val-Unseen checkpoints.Val-Seen can overfit while Val-Unseen generalization deteriorates.Source: C. Implementation Details · p. 12

Qualitative limitations

• Some sequences preserve segmentation features up to 4.7m before diverging after the model misses a stairwell.Source: C. Implementation Details · p. 14
• Other examples nearly recreate navigation sequences, but miss details such as sliding glass doors, likely because glass produces poor depth returns in training data.Source: C. Implementation Details · p. 15

Qualitative results

• Qualitative examples show realistic view synthesis across potentially unbounded viewpoint changes, including kitchen-counter views at 5.1m and 7.9m.Source: C. Implementation Details · p. 16
• Pathdreamer can generate diverse plausible layouts for unseen regions, including a complete bed, cushion, and chair configuration.Different noise vectors produce alternative scene layouts.Source: C. Implementation Details · p. 17, Figure 11 · p. 17, Figure 12 · p. 18