PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation

Xingtai Gui, Meijie Zhang, Tianyi Yan, Wencheng Han, Jiahao Gong, Feiyang Tan, Cheng-zhong Xu, Jianbing Shen

arXiv:2603.14948v1cs.CV

TL;DR

Existing driving world models primarily optimize visual scene representation, leaving motion representations insufficiently shared with precise planning. WorldDrive unifies vision and motion through trajectory-conditioned scene generation, inherited encoders, multimodal planning, and future-latent trajectory re-scoring, and reports leading vision-only planning performance while retaining high-fidelity action-controlled video generation.

  • Problem

    Existing driving world models focus primarily on visual scene representation, while motion representation is not explicitly designed to be shared with and inherited by planners.

  • Method

    WorldDrive trains a trajectory-aware driving world model, transfers its vision and motion encoders to a multimodal planner, and uses a future-aware rewarder to re-score trajectories from distilled future latents.

  • Results

    WorldDrive achieves strong planning performance on NAVSIM, NAVSIM-v2, and nuScenes while supporting high-fidelity, motion-consistent action-controlled scene synthesis.

  • Takeaways & Limitations

    The framework demonstrates a unified representation for connecting future scene generation with multimodal, real-time planning.

Abstract

from arXiv · show

End-to-end autonomous driving aims to generate safe and plausible planning policies from raw sensor input. Driving world models have shown great potential in learning rich representations by predicting the future evolution of a driving scene. However, existing driving world models primarily focus on visual scene representation, and motion representation is not explicitly designed to be planner-shared and inheritable, leaving a schism between the optimization of visual scene generation and the requirements of precise motion planning. We present WorldDrive, a holistic framework that couples scene generation and real-time planning via unifying vision and motion representation. We first introduce a Trajectory-aware Driving World Model, which conditions on a trajectory vocabulary to enforce consistency between visual dynamics and motion intentions, enabling the generation of diverse and plausible future scenes conditioned on a specific trajectory. We transfer the vision and motion encoders to a downstream Multi-modal Planner, ensuring the driving policy operates on mature representations pre-optimized by scene generation. A simple interaction between motion representation, visual representation, and ego status can generate high-quality, multi-modal trajectories. Furthermore, to exploit the world model's foresight, we propose a Future-aware Rewarder, which distills future latent representation from the frozen world model to evaluate and select optimal trajectories in real-time. Extensive experiments on the NAVSIM, NAVSIM-v2, and nuScenes benchmarks demonstrate that WorldDrive achieves leading planning performance among vision-only methods while maintaining high-fidelity action-controlled video generation capabilities, providing strong evidence for the effectiveness of unifying vision and motion representation for robust autonomous driving.

1. Introduction

WorldDrive addresses the disconnect between visually optimized scene generation and motion planning by unifying vision and motion representations. It couples trajectory-conditioned future scene synthesis with inherited representations, multimodal planning, and future-aware trajectory selection.

  • Research Gap: Existing driving world models emphasize visual representation, while planners separately encode motion for action regression, creating representation misalignment between scene generation and planning.This disconnect limits planners’ use of learned world-model dynamics and motion priors.
  • WorldDrive: WorldDrive introduces a Trajectory-aware Driving World Model that couples visual dynamics with motion intentions through predefined trajectory anchors.The structured latent space supports action-controllable future scene generation with physically consistent trajectories.
  • WorldDrive: The downstream planner inherits the world model’s learned vision and motion encoders, so planning operates in a latent space pre-aligned by future scene generation.A lightweight multimodal planner fuses historical visual context with structured motion priors to generate diverse trajectories.
  • WorldDrive: The Future-aware Rewarder distills future latents from the frozen world model to evaluate trajectory candidates without generating explicit videos for every candidate.This design uses the world model’s predictive information for real-time trajectory re-scoring.
  • Reported Capabilities: WorldDrive is presented as a unified framework supporting motion-consistent future scene generation and multimodal, real-time planning.The stated contributions include evidence on NAVSIM, NAVSIM-v2, and nuScenes, alongside conditional scene-generation evaluation.

2. Related Works

Related work develops driving world models for future-scene prediction and increasingly connects them to planning. WorldDrive is positioned within this progression as a holistic architecture that combines visual simulation, inherited representations, and trajectory optimization.

  • End-to-End Driving: End-to-end driving systems commonly combine perception, prediction, and planning using cascaded or parallel architectures with structured BEV features.Sparse representations have also been proposed to reduce reliance on dense BEV features.
  • Driving World Models: Driving world models predict scene evolution and support long-horizon, high-fidelity, and corner-case scenario generation under motion conditions.Prior methods use motion losses, trajectory prompts, motion priors, simulator data, or discrete actions to control generated scenarios.
  • WorldDrive: WorldDrive’s architecture uses a visual-simulation phase to train TA-DWM and a trajectory-optimization phase that inherits frozen visual and trajectory encoders.The planning phase adds a learnable multimodal planner after representation transfer.
  • World Models for Planning: World-model-based planning methods use predicted scenes, visual reasoning, or autoregressive representations to optimize planning policies.Drive-WM introduced driving world models into end-to-end planning, while later methods explored visual chain-of-thought and unified generation-planning models.
  • World Models for Planning: Latent world-model methods reduce the latency of explicit scene generation while learning scene representations through self-supervision in latent space.Examples include LAW, World4Drive, and WorldRFT.

3. Method

WorldDrive couples future-scene simulation and trajectory planning through shared vision and motion representations. Its TA-DWM, inherited planner, and Future-aware Rewarder support trajectory-conditioned generation and real-time candidate selection.

  • Trajectory-aware Driving World Model: TA-DWM adapts a video diffusion model with a trajectory vocabulary and multimodal trajectory encoder to condition future-scene generation on motion intentions.The visual encoder produces historical spatio-temporal latents, while anchor and offset encoders produce trajectory embeddings for diffusion conditioning.
  • Multi-modal Trajectory Planner: The planner inherits frozen visual and trajectory encoders from TA-DWM, then combines visual context, ego status, and trajectory-anchor queries through attention.Ego status includes velocity, acceleration, and driving command; the decoder outputs enriched trajectory features.
  • Future-aware Rewarder: FAR trains with latent feature alignment and preference ranking, then maps each candidate’s future feature to a scalar reward for real-time trajectory selection.The alignment objective uses predicted and target features, while ranking contrasts positive and suboptimal candidates; the highest-reward trajectory is selected.
  • Multi-modal Trajectory Planner: The planner estimates imitation, simulation, and offset-refinement outputs for trajectory anchors, selects the top-K combined scores, and refines them into diverse candidates.These candidate trajectories are passed to the Future-aware Rewarder for subsequent scoring.
  • Future-aware Rewarder: FAR distills planning-relevant future latents from the frozen TA-DWM during training, avoiding full diffusion sampling for every candidate at inference.Future Scene Queries and a lightweight decoder align predicted features with world-model future latents before a trajectory-conditioned decoder queries the distilled features.

4. Experiments

Experiments evaluate WorldDrive across planning, representation, rewarder, latency, motion sensitivity, and scene-generation settings. Results show strong planning performance, real-time future-aware scoring, motion-controllable generation, and diverse feasible trajectories.

  • Planning performance: WorldDrive achieves the best PDMS among vision-only methods on NAVSIM, reaching 88.1 with single-view input and 89.0 when trained on the full navtrain split.It also surpasses prior single-view and leading multi-view approaches, while remaining competitive with multi-modal methods.
  • Planning performance: 93.6 PDMS is achieved in the best-of-6 oracle setting, probing the upper bound and verifying that WorldDrive generates a multi-modal candidate set.The oracle scorer selects the best trajectory from six candidates and is not used during inference.
  • Planning performance: WorldDrive achieves the best EPDMS on the NAVSIM-v2 navhard split using a single-view camera, with gains across most compliance and safety-related sub-metrics.Comfort-related metrics remain challenging, indicating room to improve long-horizon efficiency while preserving safety.
  • Representation and rewarder analysis: Inheriting TA-DWM vision and motion representations improves planner performance by 0.9 and 1.1 PDMS beyond the 3D VAE initialization.The ablation attributes these step-wise gains to better alignment between vision-motion features and downstream planning requirements.
  • Representation and rewarder analysis: FAR substantially improves overall performance while increasing both safety and efficiency, and distilled future latents avoid explicit future-scene generation during inference.WorldDrive therefore maintains strong planning performance with low latency and real-time inference.
  • Generation and motion sensitivity: Latent similarity decreases as trajectory geometric distance increases, while longer training amplifies sensitivity and Top-5 conditioning improves discrimination, especially for small deviations.Qualitative results further show that FAR selects safer candidates and that TA-DWM generates plausible futures closely following expert and counterfactual trajectories.

5. Conclusion

The conclusion presents WorldDrive as a unified framework that couples visual dynamics with motion intentions for planning and scene generation. It combines representation inheritance and future-aware reward scoring to support strong planning and action-controllable synthesis with real-time inference.

  • Unified representation: WorldDrive bridges the representational gap between scene generation and planning by coupling visual dynamics with motion intentions in TA-DWM.Its unified feature space is intended to support both future-scene generation and downstream decision-making.
  • Unified representation: Representation inheritance initializes a lightweight planner with mature features learned from scene generation.The planner reuses the vision and motion representations established by TA-DWM.
  • Future-aware planning: The Future-aware Rewarder leverages driving-world-model foresight while maintaining real-time inference.It uses future information without requiring explicit future-scene generation at inference.
  • Overall outcome: Extensive experiments show strong planning performance together with action-controllable scene synthesis.The conclusion positions generative world models as a foundation for safe and interpretable autonomous driving research.

6. Further Implementation Details

WorldDrive combines frozen visual encoding, trajectory-aware motion supervision, and Future-aware Rewarder training to support efficient planning and future-scene modeling. Its implementation uses staged world-model pretraining and preference-based trajectory selection.

  • Trajectory-aware Driving World Model: The trajectory-aware world model encodes historical and future frames with a frozen 3D Causal VAE, while a Visual Adapter refines historical latent features.The diffusion process uses these processed representations to model future scene evolution.
  • Multi-modal Trajectory Planner: Trajectory anchors receive imitation supervision from expert-distance rewards and simulation supervision based on safety and efficiency metrics.The simulation metrics include collisions, drivable-area compliance, time-to-collision, comfort, and ego progress.
  • Multi-modal Trajectory Planner: The final trajectory reward combines the supervision signals through a weighted log-sum, with hyperparameters ω1 = 0.1, ω2 = ω3 = 0.5, and ω4 = 1.The weights balance the reward components used for trajectory evaluation.
  • Multi-modal Trajectory Planner: The planner trains imitation scores with cross-entropy, simulation rewards with binary cross-entropy, and positive-anchor offsets with L1 regression.The positive anchor is the trajectory closest to the expert trajectory.
  • Future-aware Rewarder: Future-aware Rewarder preference pairs use the planner’s top candidate, three lowest-simulation-score hard negatives, and three randomly sampled trajectories.These pairs are optimized with Bradley–Terry loss, and the highest-reward candidate is selected at inference.
  • Training Setup: The world model is pretrained for 200,000 iterations on nuPlan and fine-tuned for 100,000 iterations on nuScenes at 256 × 512 resolution and 10Hz.The first stage predicts 17 future frames from 8 historical frames.

7. Further Ablation Study

Ablations show that Future-aware Rewarder performance depends on the candidate-set size, while the full WorldDrive pipeline maintains low-latency inference. The default K=5 setting provides the reported balance between trajectory diversity and precision.

  • Future-aware Rewarder Candidates: Increasing FAR’s candidate count from K=1 to K=5 consistently improves PDMS, whereas K=10 causes a performance drop.The authors attribute the drop to additional low-quality trajectories distracting the rewarder and compromising safety metrics.
  • Future-aware Rewarder Candidates: WorldDrive sets K=5 because it offers the best trade-off between trajectory diversity and precision.The candidate set is drawn from the trajectories with the highest predicted planner scores.
  • Inference Efficiency: WorldDrive meets real-time requirements in planning inference mode using lightweight transformer decoders and MLP layers in the planner and FAR.Latency is measured for the full forward pipeline on a single NVIDIA A800 GPU.

8. Further Qualitative Comparison

Qualitative evaluations show WorldDrive jointly producing expert-aligned planning candidates and action-controlled future scenes. FAR improves behavior selection in complex cases, while generated scenes remain geometrically consistent with commanded motion.

  • Planning and Scene Generation: On NAVSIM, the planner proposes trajectory candidates that closely align with expert decisions, while FAR selects favorable behavior when complex scenarios cause deviations.The qualitative results connect candidate generation with reward-based corrective selection.
  • Planning and Scene Generation: Generated sequences exhibit geometric consistency with the input motion commands on the NAVSIM qualitative evaluation.This supports motion-consistent scene dynamics rather than mere texture memorization.
  • Action-Controlled Generation: WorldDrive’s action-controllable future-scene generation is qualitatively evaluated on the nuScenes validation set.The evaluation focuses on generated future scenes conditioned by driving actions.
Loading 2603.14948v1…