PaperlayerView exact revision on arXiv

Source-linked AI summary

RAE-NWM: Navigation World Model in Dense Visual Representation Space

Mingkun Zhang, Wangtian Shen, Fan Zhang, Haijian Qin, Zihao Pei, Ziyang Meng

arXiv:2603.09241v2cs.CVcs.RO

TL;DR

Visual navigation world models need reliable long-horizon action-conditioned predictions, but VAE compression can lose fine-grained structure. RAE-NWM instead learns continuous transitions in dense DINOv2 space with CDiT-DH and time-driven gating, improving rollout stability and downstream navigation performance.

  • Problem

    VAE-based navigation world models can lose geometric information through spatial compression, undermining long-horizon structural consistency and action controllability.

    Source: §1 · p. 2

  • Method

    RAE-NWM models action-conditioned transitions in dense DINOv2 representations using a CDiT-DH backbone and time-driven gating for kinematic conditioning.

    Source: §7 · p. 15

  • Results

    RAE-NWM improves long-horizon rollout stability and provides more reliable predictions for downstream planning and navigation.

    Source: §1 · p. 3, §7 · p. 15

  • Takeaways & Limitations

    Dense representation-space world modeling supports structurally stable sequential rollouts for visual navigation.

    Source: §1 · p. 3, §7 · p. 15

    DINOv2 representations can overlook high-frequency stochastic textures, and the experiments do not fully isolate representation, architecture, and model-capacity effects through scale-matched training.

    Source: §6 · p. 15

Abstract

from arXiv · show

Visual navigation requires agents to reach goals in complex environments through perception and planning. World models address this task by simulating action-conditioned state transitions to predict future observations. Current navigation world models typically learn state evolution under actions within the compressed latent space of a Variational Autoencoder, where spatial compression often discards fine-grained structural information and hinders precise control. To better understand the propagation characteristics of different representations, we conduct a linear dynamics probe and observe that dense DINOv2 features exhibit stronger linear predictability for action-conditioned transitions. Motivated by this observation, we propose the Representation Autoencoder-based Navigation World Model (RAE-NWM), which models navigation dynamics in a dense visual representation space. We employ a Conditional Diffusion Transformer with Decoupled Diffusion Transformer head (CDiT-DH) to model continuous transitions, and introduce a separate time-driven gating module for dynamics conditioning to regulate action injection strength during generation. Extensive evaluations show that modeling sequential rollouts in this space improves structural stability and action accuracy, benefiting downstream planning and navigation.

1 Introduction

Visual navigation world models must preserve geometric stability and action controllability over long horizons, but VAE-based latent rollouts can structurally degrade. RAE-NWM addresses this by modeling action-conditioned transitions in dense DINOv2 features with a gated CDiT-DH generator.

Long-horizon generation

  • At k = 12s and 16s, the VAE-based baseline shows severe structural degradation, whereas RAE-NWM maintains strong structural integrity.This qualitative comparison is shown in Figure 1.Source: Figure 1 · p. 2

Motivation

  • VAE-based rollouts can suffer structural collapse and kinematic deviation over long horizons, reducing reliability for downstream path planning.The problem arises because spatial compression can discard critical geometric information.Source: §1 · p. 2

Representation motivation

  • DINOv2 features show strong linear predictability for action-conditioned transitions, motivating generative modeling directly in dense visual representation space.This representation-space choice is the central basis for RAE-NWM.Source: §1 · p. 3

Method

  • RAE-NWM uses frozen DINOv2 and RAE components, while training a CDiT-DH transition model with time-driven gating for adaptive kinematic conditioning.The gate balances visual generation fidelity and action controllability during the probability flow.Source: §1 · p. 3

Contributions

  • RAE-NWM improves long-horizon rollout stability and performance in open-loop trajectory evaluation and downstream navigation planning tasks.The contribution statement reports these benefits across the evaluated tasks.Source: §1 · p. 3

2 Related Work

Prior work uses world models to simulate action-conditioned dynamics, but many methods rely on compressed or deterministic representations. RAE-NWM extends representation-level modeling toward generative first-person navigation rollouts in dense visual features.

Visual Representation Space

  • Representation Autoencoders reconstruct images from uncompressed visual foundation-model features, offering high-fidelity spatial reconstruction without compression.RAE combines uncompressed representations with a vision-transformer decoder.Source: §2 · p. 3

World Models for Embodied Intelligence

  • World models explicitly simulate environmental dynamics for decision-making, while many navigation methods condition future predictions on physical actions.This distinguishes world-model planning from end-to-end policies.Source: §2 · p. 4
  • VAE-based latent spaces can degrade spatial structure, while projection-based, hybrid, and representation-level alternatives address this limitation with different computational or predictive designs.The related methods include future-view projection, autoregressive-diffusion hybrids, and self-supervised feature-space models.Source: §2 · p. 4
  • Existing feature-space models mostly use deterministic prediction or non-navigation settings, whereas RAE-NWM models generative first-person navigation rollouts with continuous-time dynamics.The paper positions this as modeling smooth action-induced geometric changes for stable long-horizon planning.Source: §2 · p. 4

3 Task Definition and Representation Analysis

The task is to predict future visual states from context, action, and horizon within a chosen representation space. Representation analysis finds that dense DINOv2 tokens support action-conditioned dynamics better than compressed or spatially disrupted alternatives, motivating the RAE-NWM architecture.

3.1 Task Definition

  • A navigation world model predicts the future representation from historical context, planar motion, and a relative prediction horizon.The motion condition contains planar translation and yaw rotation.Source: §3.1 · p. 4

3.2 Representation Analysis

  • The representation space determines whether the predictor can extract action-conditioned dynamics directly or must model unrelated visual variation.The linear dynamics probe isolates this property by freezing the encoder and training only a linear predictor.Source: §3.2 · pp. 4–5
  • The probe predicts future tokens from current tokens and intermediate motion using learnable linear transformations, testing whether the feature space natively supports the dynamics.The probe is intentionally restricted to a simple linear formulation.Source: §3.2 · p. 5
  • DINOv2’s uncompressed token space maintains consistently high global R2 predictability across the horizon, while VAE, MAE, SigLIP, and ResNet50 representations perform substantially worse.Spatially shuffling DINOv2 tokens substantially reduces performance, linking the advantage to spatial structure rather than static global semantics alone.Source: §3.2 · p. 5
  • RAE-NWM encodes context frames as frozen-DINOv2 tokens, injects motion and horizon through time-driven gating, and uses CDiT-DH to predict flow velocity.The architecture therefore combines dense spatial tokens with adaptive dynamics conditioning.Source: Figure 3 · p. 6, §3.2 · p. 6

4 RAE-NWM

RAE-NWM models action-conditioned navigation dynamics in dense DINOv2 representation space using a CDiT-DH transition model and time-driven conditioning. It trains with flow matching and performs sequential ODE rollouts directly in representation space, decoding only for visualization and pixel metrics.

4.1 State Representation

  • RAE-NWM extracts 256 uncompressed DINOv2 patch tokens of dimension 768 and uses temporal context as conditioning for generation.The frozen encoder discards the [CLS] token and retains spatial patch tokens.Source: §4.1 · pp. 6–7

4.2 Generative Backbone: CDiT-DH

  • CDiT-DH predicts the velocity field of a noisy future representation while conditioning on context tokens and a unified action- and horizon-control vector.Its transformer backbone uses spatial positional embeddings, self-attention, cross-attention, and AdaLN-based global conditioning.Source: §4.2 · p. 7
  • The DDT head is shallow and wide, enabling velocity prediction for high-dimensional token representations without significantly increasing computational cost.It uses spatial AdaLN modulation under deep backbone features.Source: §4.2 · p. 7

4.3 Dynamics Conditioning Module

  • Time-driven gating combines action and horizon dynamics with flow time so early noisy stages receive stronger kinematic control while later stages refine visual details.The gated condition is injected through AdaLN, with the gate constrained to (0, 1).Source: §4.3 · pp. 7–8

4.4 Training Objective and Sequential Rollout

  • Flow matching trains the velocity field along a noise-to-representation probability path, while an ODE solver sequentially generates future states in a closed representation-space rollout loop.Generated states enter the sliding context for subsequent predictions; decoding is reserved for visualization and pixel-level metrics, while planning operates directly in representation space.Source: §4.4 · pp. 8–9, Figure 4 · p. 8

5 Experiments

Across real-world and simulated navigation evaluations, RAE-NWM is tested for long-horizon generation, trajectory planning, closed-loop control, and component-level effects. The results show improved rollout stability, planning accuracy, and success rate, with representation, transition objective, gating, and DDT-head choices materially affecting performance.

5.1 Experimental Setup

  • The evaluation combines three real-world datasets, held-out trajectories, and Matterport3D Habitat experiments, using perceptual, DINO-feature, trajectory, planning, and navigation metrics.SACSoN, RECON, and SCAND cover dynamic interactions, unstructured terrain, and socially normative demonstrations.Source: §5.1 · pp. 9–10

5.2 Open-Loop Generation

  • RAE-NWM maintains temporal stability and structural consistency over 16-second sequential rollouts across SACSoN, RECON, and SCAND.LPIPS rises gradually from 16s to 32s—0.387→0.438, 0.458→0.517, and 0.523→0.614—rather than collapsing abruptly.Source: §5.2 · pp. 10–11, Figure 6 · p. 12

5.3 Trajectory and Planning Accuracy

  • ATE 2.91 and RPE 0.70: RAE-NWM improves on NWM on SACSoN, while achieving ATE 1.36 and RPE 0.37 on RECON and outperforming GNM and NoMaD.On RECON, NWM has lower short-horizon errors, likely because VAE latents preserve more texture-sensitive cues than DINOv2 features.Source: §5.3 · pp. 12–13, Table 2 · p. 12

5.4 Simulation in Habitat

  • 78.95% Success Rate: RAE-NWM outperforms existing visual navigation methods in closed-loop Habitat image-goal navigation.Its SPL is slightly below One-Step WM, while its Success Rate is higher under iterative dense-space rollouts.Source: §5.4 · p. 13, Table 3 · p. 13

5.5 Ablation Study

  • Flow-matching transitions, DINOv2 representations, and the DDT head each support performance: DINO-Reg is worse on most errors, SD-VAE degrades more over time, and removing DDT worsens long-horizon LPIPS.DINO-Reg is marginally better on RECON RPE, 0.36 versus 0.37, while deterministic regression also produces blurry ghosting artifacts.Source: §5.5 · pp. 13–14, Figure 9 · p. 15
  • The learned dynamics gate achieves the lowest SACSoN planning errors, with ATE 2.91 and RPE 0.70, while maintaining more stable long-horizon LPIPS and FID than alternative injection strategies.The comparison includes addition, MLP fusion, and scheduled gating.Source: Table 4 · p. 13, §5.5 · p. 14

6 Discussion and Limitations

RAE-NWM trades some texture fidelity for more stable long-horizon spatial structure and uses fewer resources than NWM, though scale-matched training remains unresolved.

Discussion and Limitations

  • RAE-NWM ends sequential rollouts at LPIPS 0.472 versus NWM’s 0.554, despite starting from a higher LPIPS of 0.230.DINOv2-based semantics improve long-horizon structural stability but overlook high-frequency stochastic textures; static FID is 5.47 for DINOv2+RAE versus 4.14 for SD-VAE.Source: §6 · p. 15
  • RAE-NWM uses 350M parameters versus NWM’s 1B and requires 8.97 versus 36.10 TFLOPs, but scale-matched training is not evaluated.Profiling also reports 19.4 versus 28.8 ms and 0.73 versus 1.93 GB on one A800.Source: §6 · p. 15

7 Conclusion

RAE-NWM models action-conditioned navigation transitions in dense DINOv2 features with frozen encoder and decoder components, CDiT-DH, and time-driven gating. The resulting design improves long-horizon stability, preserves spatial structure, and supports downstream planning.

Conclusion

  • RAE-NWM learns action-conditioned transitions in dense DINOv2 space using a frozen encoder, frozen RAE decoder, CDiT-DH, and time-driven gating.The gate adaptively modulates kinematic conditioning along the probability flow.Source: §7 · p. 15
  • Experiments on real-world navigation datasets and Habitat Image-Goal navigation show improved long-horizon rollout stability and more reliable predictions for planning.Future work targets texture fidelity and faster iterative rollouts.Source: §7 · p. 15

A Implementation Details

The appendix specifies the CDiT-DH architecture, optimization and inference settings, cross-dataset action normalization, DINO-distance computation, and linear-probe implementation.

A.1 Network Architecture

  • CDiT-DH uses 12 DiT-B Transformer blocks, while its DDT head uses two shallow, wide blocks; inputs include four context frames.The backbone has hidden size 768 with 12 heads, and the DDT head has hidden size 2048 with 16 heads.Source: §A.1 · p. 21

A.2 Training and Optimization

  • Training runs for 50 epochs with AdamW, batch size 96, and a learning rate decaying linearly from 2 × 10^-4 to 2 × 10^-6.Weight decay is set to 0.Source: §A.2 · p. 21

A.3 Inference and Training Resources

  • Inference uses a 50-step Euler ODE solver, while joint real-world training spans SACSoN, RECON, and SCAND on two A800 GPUs for approximately two days.Source: §A.3 · p. 21

A.4 Action Normalization

  • Actions are normalized by each dataset’s average training-split inter-frame step size so joint training uses comparable physical motion scales.Source: §A.4 · p. 21

A.5 DINO Representation Distance

  • DINO distance extracts patch tokens with a frozen encoder, computes per-token cosine distance after ℓ2 normalization, and averages across tokens.The token sequence has L patches of feature dimension d.Source: §A.5 · pp. 21–22

B.1 Implementation of Linear Transformations

  • The linear dynamics probe uses two bias-free linear operators: a 1 × 1 convolution for spatial features and an action projection broadcast over the patch grid.The action vector contains planar translation and yaw rotation (u_x, u_y, ω).Source: §B.1 · p. 22

B.2 Training Configuration and Dataset Specifics

The appendix details probe evaluation, deterministic DINO-token regression, dynamics-conditioning analyses, and extended sequential-rollout measurements across navigation datasets.

B.2 Training Configuration and Dataset Specifics

  • Probe training separates trajectories from evaluation and freezes all visual encoders; it uses Huber loss with non-trivial physical displacement steps.Average metric displacement per frame is approximately 0.25 m for RECON, 0.255 m for SACSoN, and 0.36 m for SCAND.Source: §B.2 · p. 22

B.3 Calculation of the Global R2 Score

  • The global R2 score flattens predicted and ground-truth latent states across batch, spatial, and channel dimensions before computing 1 - SSE/SST.Source: §B.3 · p. 23

C Details of Deterministic DINO-Token Regression

  • Deterministic DINO-token regression keeps the representation, context, actions, decoder, rollout protocol, and evaluation setting fixed while replacing flow matching with direct future-token regression.It improves over several navigation baselines, but flow matching achieves stronger overall performance and more stable rollouts.Source: C Details of Deterministic DINO-Token… · p. 23

D Extended Details on the Dynamics Conditioning Module

  • The time-driven gate weights kinematic conditioning more strongly at high-noise flow times and reduces it near the clean data manifold, with stable allocation across samples.This provides structural guidance for global topology before allowing visual-detail refinement.Source: D Extended Details on the Dynamics… · p. 24

E Extended Evaluation on Sequential Rollouts

E.1 Quantitative Performance

  • On RECON, RAE-NWM reduces rapid structural degradation versus NWM, while on SCAND it shows better temporal stability and lower error accumulation across perceptual and structural metrics.SCAND DINO Distance is computed by re-encoding decoded observations with the frozen DINOv2 encoder.Source: §E.1 · p. 25, Figure 13 · p. 25

E.2 Failure Case Analysis

RAE-NWM exhibits a representative SCAND failure after collision-induced context ambiguity, while qualitative rollouts are assessed across four navigation datasets with dataset-specific fidelity challenges.

E.2 Failure Case Analysis

  • Collision-induced context ambiguity can lead RAE-NWM to hallucinate a plausible but domain-inconsistent indoor layout on SCAND.The case indicates that dynamic multi-agent scenes and abrupt contact events remain challenging for long-horizon representation-space rollouts.Source: §E.2 · p. 25

E.3 Qualitative Results of Rollout

  • Qualitative rollout visualizations cover SACSoN, SCAND, RECON, and Matterport3D, with fine-grained vegetation less faithfully preserved in RECON.Source: §E.3 · p. 26, Figure 16 · p. 27, Figure 17 · p. 27
  • In SCAND, visual information from the initial context can gradually fade during dynamic multi-person scenes and with limited data scale.Source: §E.3 · p. 26

Related papers

Loading 2603.09241v2…