PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniNWM: Omniscient Driving Navigation World Models

Bohan Li, Zhuang Ma, Dalong Du, Baorui Peng, Zhujin Liang, Zhenqiang Liu, Xianda Guo, Zheng Zhu, Chao Ma, Yueming Jin, Xin Jin, Hao Zhao, Wenjun Zeng

arXiv:2510.18313v6cs.CV

TL;DR

Existing autonomous-driving world models struggle to unify multimodal state prediction, precise action control, long-horizon consistency, and intrinsic policy evaluation. OmniNWM addresses these gaps with a joint state-action-reward framework, and experiments report SOTA generation quality and control precision with robust zero-shot generalization across datasets, camera rigs, and trajectories.

  • Problem

    Existing methods face fragmented multimodal modeling, long-horizon exposure-bias drift, imprecise geometry-dependent control, and limited intrinsic mechanisms for policy evaluation.

  • Method

    OmniNWM jointly models panoramic RGB, semantics, depth, and occupancy, using structured panoramic forcing, normalized Plücker ray-maps, and occupancy-grounded rewards in a closed-loop simulator.

  • Results

    OmniNWM achieves SOTA generation quality and control precision while generalizing zero-shot across unseen datasets, camera rigs, novel trajectories, and horizons beyond ground truth.

  • Takeaways & Limitations

    The unified framework provides a grounded basis for long-horizon generation, cross-rig trajectory control, and closed-loop planning-agent evaluation.

Abstract

from arXiv · show

Autonomous driving world models are expected to work effectively across three core dimensions: state, action, and reward. However, existing methods are typically restricted to fragmented modality modeling, short-horizon drift, and imprecise action control, while lacking intrinsic mechanisms for policy evaluation. In this paper, we introduce OmniNWM, an Omniscient panoramic Navigation World Model that addresses all three dimensions within a consistent probabilistic framework. For State, OmniNWM generates panoramic videos of RGB, semantics, metric depth, and 3D occupancy, ensuring pixel-level alignment across modalities with joint distribution modeling. To mitigate autoregressive exposure bias, we propose a structured panoramic forcing strategy to stabilize long-horizon generation via stochastic manifold thickening. For Action, we introduce canonical geometric action encoding with normalized panoramic Plücker ray-maps. This representation decouples motion dynamics from sensor intrinsics, enabling precise, zero-shot trajectory control across heterogeneous datasets and camera configurations. For Reward, we derive intrinsic occupancy-grounded dense rewards directly from generated 3D volumes, establishing a reliable closed-loop simulation cycle for evaluating diverse planning agents. Extensive experiments demonstrate that OmniNWM achieves SOTA performance in generation fidelity and control precision, with remarkable zero-shot robustness to novel scenes on NuPlan and in-house datasets with distinct camera rigs. Project page is available at https://arlo0o.github.io/OmniNWM/.

1 Introduction

OmniNWM unifies state, action, and reward modeling for autonomous driving in a joint probabilistic framework, addressing multimodal inconsistency, long-horizon drift, and geometric control limitations. It combines panoramic multimodal generation, canonical action encoding, structured forcing, and occupancy-grounded rewards, with experiments reporting SOTA quality and robust zero-shot generalization.

  • OmniNWM unifies the state-action-reward triad within a single probabilistic world-model framework for autonomous driving.
  • It generates pixel-aligned panoramic RGB, semantics, metric depth, and 3D occupancy while using structured panoramic forcing to stabilize long-horizon forecasting.
  • Normalized panoramic Plücker ray-maps decouple motion dynamics from sensor geometry, enabling precise control across novel trajectories and camera rigs.
  • Occupancy-grounded dense rewards form a closed-loop cycle in which planning trajectories condition generation and generated 3D occupancy evaluates policies.
  • Experiments report SOTA generation quality and control precision, with zero-shot robustness across nuPlan, in-house datasets, varied camera rigs, novel trajectories, and horizons beyond ground truth.

2 Related Work

Prior autonomous-driving world models have advanced visual realism and controllability through multi-stage generation, multimodal modeling, cross-view attention, and latent replacement strategies. The cited work situates OmniNWM within this progression of driving-world-model research.

  • Driving world models have progressed toward visual realism and controllability through multi-stage diffusion, multimodal appearance-depth generation, cross-view attention, and latent replacement.

3 Methodology

OmniNWM unifies multimodal state generation, canonical geometric action control, intrinsic occupancy-grounded rewards, and closed-loop simulation in one probabilistic framework. Its components jointly support consistent 3D scene modeling, trajectory conditioning, policy evaluation, and interactive generation.

  • Comprehensive Generation within OmniNWM: OmniNWM jointly forecasts panoramic RGB, semantic, and metric-depth states, then lifts them into visually grounded 3D occupancy for consistent simulation.A shared latent representation promotes pixel-level alignment across modalities, while geometric lifting produces occupancy volumes from RGB, depth, and semantic features.
  • Comprehensive Generation within OmniNWM: Its occupancy-grounded reward combines collision, drivable-area, and velocity penalties to evaluate planning trajectories against physical scene constraints.The resulting dense utility supports closed-loop evaluation of planning agents, including obstacle and road-boundary feasibility.
  • Canonical Encoding via Normalized Panoramic Ray-maps: OmniNWM closes the simulation loop by feeding generated states to planning agents and encoding their trajectories as canonical normalized panoramic Plücker ray-maps for next-state generation.The integrated simulator maps agent actions into a canonical geometric space before conditioning the subsequent state transition.
  • Canonical Encoding via Normalized Panoramic Ray-maps: Pose- and scale-normalized panoramic ray-maps decouple motion dynamics from sensor geometry, unify trajectories in an invariant 3D Plücker space, and enrich trajectory diversity.The representation uses camera-rig geometry and a canonical reference frame to improve robustness across configurations.
  • Canonical Encoding via Normalized Panoramic Ray-maps: Restricting explicit control to the ego vehicle lets surrounding agents reactively emerge from the learned equilibrium distribution, such as yielding during an ego cut-in.This design avoids heuristic scripting of surrounding-agent behavior while preserving physically consistent interactive responses.

4 Experiments

OmniNWM achieves strong generation, control, occupancy, planning, and zero-shot generalization performance, while ablations identify the contributions of canonical normalization, occupancy inputs, and structured panoramic forcing.

  • 5.45 FID and 23.63 FVD establish state-of-the-art video generation quality using normalized panoramic ray-maps without heavy volumetric inputs.
  • 1.18 m TransErr versus 7.56 m for UniScene demonstrates substantially lower camera-control drift with normalized Plücker ray-maps.
  • 87.3% SPR surpasses Impromptu-VLA and Qwen-2.5-VL in closed-loop trajectory planning across 150 nuScenes validation scenes.
  • 90.6% and 87.9% reductions in rotation and translation errors follow canonical panoramic normalization, which also supports stable 200-frame zero-shot generation.
  • Generated-occupancy rewards correlate with GT occupancy and LiDAR+Box rewards at rGT = 0.96 and rLiDAR = 0.94, supporting occupancy-grounded evaluation.
  • Structured panoramic forcing maintains 25.22 FVD over 201 frames, whereas the comparison reaches 178.65 FVD at 201 frames.

5 Conclusion

OmniNWM unifies state, action, and reward modeling through cross-modal consistency and canonical geometric action encoding. The framework supports zero-shot generalization across diverse datasets and camera configurations.

  • OmniNWM approximates the joint multimodal posterior of the state-action-reward triad to address fragmentation in autonomous driving world models.
  • Normalized panoramic ray-maps decouple motion dynamics from sensor intrinsics, enabling zero-shot generalization across diverse datasets and camera rigs.

A More Related Works and Discussions

Related work spans reasoning-oriented and end-to-end vision-language-action agents, semantic occupancy prediction, and driving world models. OmniNWM differs from direct occupancy regression by using a generative-first strategy grounded in jointly generated visual modalities.

  • Reasoning-oriented VLA methods provide semantic rationales but often lack the geometric precision required for low-level control.
  • Semantic occupancy prediction combines geometric reconstruction with semantic labeling, but camera-based discriminative methods can suffer geometric collapse in textureless or distant regions.
  • OmniNWM adopts a generative-first strategy rather than directly regressing 3D occupancy from RGB.
  • Table 9 benchmarks OmniNWM against LiDAR-based, vision-centric discriminative, and generative occupancy methods, with OmniNWM reported as state-of-the-art in vision-based occupancy prediction.

B More Details of Occupancy Evaluations

The occupancy evaluation compares OmniNWM with multiple baseline categories and examines fine-grained structural robustness. Results support strong performance on both small dynamic objects and large-scale scene structure.

  • Table 9 compares OmniNWM with LiDAR-based, vision-centric discriminative, and generative occupancy baselines on NuScenes-Occupancy.
  • 12.9 IoU on bicycles versus 9.1 for OccGen shows stronger performance on small dynamic objects.
  • 53.1 IoU on drivable surfaces versus 34.8 for OccGen demonstrates stronger large-scale structural fidelity.

C More Implementation Details on OmniNWM-VLA

OmniNWM-VLA maps high-dimensional multimodal observations to precise trajectory actions through efficient context fusion and canonical geometric control.

  • OmniNWM-VLA Architecture: The planner combines a Tri-Modal Mamba Interpreter with a Canonical Action Head for multimodal fusion and geometric trajectory prediction.The interpreter is built on Qwen-2.5-VL, while the action head interfaces with the Normalized Plücker Ray-map control representation.
  • Tri-Modal Feature Fusion: RGB, metric depth, and semantic inputs are aligned into a shared feature space before Mamba-based selective state-space fusion.Modality-specific encoders and MLP projections preserve modality-specific priors while producing tokens for fusion.
  • Tri-Modal Feature Fusion: The Mamba fusion block compresses token sequences into a recurrent state that allows RGB context to gate geometric and semantic features.This design avoids treating high-resolution multimodal tokens independently through quadratic self-attention.
  • Tokenized Rationale Injection: Tokenized Rationale injection aggregates fused context through learnable query tokens and projects it into the VLM embedding space as semantic anchors.The enriched tokens bridge dense Mamba features and the language model’s discrete token space.
  • Canonical Action Prediction: The prediction head regresses dense trajectory tuples including heading angle and lifts waypoints into the Normalized Plücker Ray-map representation.Full-pose prediction keeps planner decisions compatible with OmniNWM’s canonical geometric manifold.

D More Ablations

Ablations show that joint RGB, semantic, and depth supervision improves generation quality, while reward weights require balancing safety and traffic efficiency for strong planning performance.

  • Joint Multi-modal Supervision: Joint RGB, semantic, and depth supervision achieves the best generation performance, improving FID and FVD over RGB-only supervision.Semantic and depth maps are training supervision rather than external inference conditions, and all modalities are jointly generated from noisy latents.
  • Reward Hyperparameters: Planning performance is highly sensitive to the distribution of collision, drivable-area, and traffic-flow reward weights.The study evaluates weight combinations using Scenario Pass Rate and Collision Rate across 150 NuScenes validation scenes.
  • Reward Hyperparameters: The configuration αcol = 0.5, αbd = 0.3, αvel = 0.2 achieves the highest Scenario Pass Rate of 87.3%.The selected weights provide the reported optimal balance between safety boundaries and navigation momentum.

E High-Fidelity Data Curation

The implementation combines multimodal generative, planning, and occupancy objectives, with data curation and joint losses designed to preserve semantic and geometric consistency.

  • Semantic Data Curation: Semantic segmentation is trained across Cityscapes, Mapillary Vistas, Waymo Open, Woodscape, and BDD100k, with translated low-light samples for nighttime domain coverage.The data-centric pipeline targets robust semantic supervision for NuScenes across diverse structural environments.
  • Generative Backbone: Rectified Flow Matching trains the generative backbone to predict the velocity field transporting a Gaussian prior toward the data distribution.The objective uses interpolated latents and canonical ray-map and reference-frame conditions, supporting deterministic stable sampling for long-horizon generation.
  • Planning Policy: The planning policy is trained with causal language modeling to maximize next-token likelihood for waypoint coordinates and heading angle.Including heading angle makes predicted actions compatible with the Normalized Plücker Ray-map representation.
  • Occupancy Training Objective: The occupancy generator uses a compound objective combining depth, semantic, geometric, and class-balanced cross-entropy losses.This objective enforces photometric consistency, volumetric accuracy, structural completeness, and robustness to sparse small objects.
  • Occupancy Training Objective: The multi-task occupancy objective keeps generated 3D volumes aligned with the visual features produced by the PDiT.Depth and semantic losses jointly constrain the geometric and semantic structure of the lifted scene representation.

G.1 Derivation of the Canonical Projection Divergence Bound

The canonical Plücker projection aligns trajectory distributions across camera rigs, while structured panoramic forcing is empirically associated with contractive long-horizon dynamics.

  • Canonical Projection Divergence Bound: The normalized Plücker ray-map acts as a projection operator that makes projected variables conditionally independent of raw camera extrinsics.This establishes the geometric basis for transferring trajectory control across distinct camera rigs.
  • Canonical Projection Divergence Bound: The projected target and source marginal distributions approximately align on the canonical manifold.The alignment replaces disjoint raw geometric supports with a shared representation for motion dynamics.
  • Canonical Projection Divergence Bound: Canonical projection reduces target-source trajectory divergence to the irreducible uncertainty bound ε by aligning camera-conditioned motion distributions.The normalized Plücker ray-map removes dependence on raw extrinsics and maps valid camera configurations to a shared manifold.
  • Contractive Property: Structured panoramic forcing changes the empirical local Lipschitz constant from approximately 1.08 to 0.94 over 150 validation sequences.The unregularized model exhibits compounding errors and a 200-frame collapse, whereas structured noise produces a restorative mapping.
  • Training Strategy: The model is trained progressively from single-view control to six-view multimodal generation and then variable-length, variable-resolution sequences.The three stages use 10K, 3K, and 3K iterations respectively to improve panoramic and scenario adaptability.
  • Training Strategy: A decoupled-then-coupled training strategy first converges the generative backbone and geometric lifter before fine-tuning the planner against frozen world-model representations.The procedure is intended to reduce distribution shift between perception and control.

I More Visualization Results

Additional visualizations show that OmniNWM maintains panoramic fidelity, cross-view structure, and temporal coherence during long-horizon generation while supporting precise control over varied trajectories.

  • OmniNWM better preserves panoramic fidelity, cross-view structural consistency, and long-horizon temporal coherence than UniScene, especially in later autoregressive frames.The comparison highlights degradation differences that become more visible as rollouts extend.
  • Additional visualizations show that normalized Plücker ray-maps maintain geometric consistency for complex non-monotonic and reversing trajectories.The examples include reversing maneuvers and further panoramic camera-control cases.
  • 321-frame navigation sequences remain temporally coherent and structurally intact beyond the training sequence length under structured panoramic forcing.The result supports extended closed-loop evaluation beyond ground-truth horizons.
  • Normalized Plücker ray-maps enable consistent multi-view videos from different input trajectories given the same conditional frame.The representation converts trajectory differences into pixel-level control signals.
Loading 2510.18313v6…