PaperlayerView exact revision on arXiv

Source-linked AI summary

UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving

Zhexiao Xiong, Xin Ye, Burhan Yaman, Sheng Cheng, Yiren Lu, Jingru Luo, Nathan Jacobs, Liu Ren

arXiv:2601.04453v4cs.CV

TL;DR

Existing VLM-based driving systems often separate perception, prediction, and planning, while text-only intermediates can lose visual–geometric information. UniDrive-WM unifies scene understanding, trajectory planning, and trajectory-conditioned future image generation, achieving high-fidelity visual prediction and improved planning accuracy with collision reduction.

  • Problem

    Many VLM-based driving approaches separate perception, prediction, and planning, with text-only intermediates creating an information bottleneck for visual–geometric cues.

  • Method

    UniDrive-WM jointly performs scene understanding, trajectory planning, and future image generation within a VLM-centric architecture using planning-conditioned visual predictions.

  • Results

    UniDrive-WM achieves high-fidelity planning-conditioned image generation and improves planning accuracy and collision reduction on driving benchmarks.

  • Takeaways & Limitations

    The unified world-model pipeline supports direct visual reasoning and uses future visual predictions to enhance trajectory planning.

  • Takeaways & Limitations

    Deterministic reconstruction during inference reduces diversity, although it provides stable and accurate future-frame prediction.

Abstract

from arXiv · show

World models have become central to autonomous driving, where accurate scene understanding and future prediction are crucial for safe control. Recent work has explored using vision-language models (VLMs) for planning, yet existing approaches typically treat perception, prediction, and planning as separate modules. We propose UniDrive-WM, a unified VLM-based world model that jointly performs driving-scene understanding, trajectory planning, and trajectory-conditioned future image generation within a single architecture. UniDrive-WM's trajectory planner predicts a future trajectory, which conditions a VLM-based image generator to produce plausible future frames. These predictions provide additional supervisory signals that enhance scene understanding and iteratively refine trajectory generation. We further compare discrete and continuous output representations for future image prediction, analyzing their influence on downstream driving performance. Experiments on the challenging Bench2Drive benchmark show that UniDrive-WM produces high-fidelity future images and improves planning performance by 7.3% in L2 trajectory error and 10.4% in collision rate over the previous best method. These results demonstrate the advantages of tightly integrating VLM-driven reasoning, planning, and generative world modeling for autonomous driving. The project page is available at https://unidrive-wm.github.io/UniDrive-WM.

1 Introduction

UniDrive-WM addresses the information bottleneck of staged autonomous-driving pipelines by unifying scene understanding, trajectory planning, and future image generation in a VLM-centric world model. It also compares discrete autoregressive and continuous AR+diffusion pathways for trajectory-conditioned visual prediction.

  • Motivation: Existing VLM-based driving approaches often use text-only intermediates between future-scene reasoning and trajectory or image decoding, losing visual–geometric information across stages.Generative models can produce plausible frames but typically lack explicit state estimation and reasoning.
  • UniDrive-WM: UniDrive-WM jointly performs scene understanding, trajectory planning, and future image generation within a single VLM-centric architecture.The framework is designed to connect perception, reasoning, planning, and generation in one world model.
  • Architecture: A trajectory planner produces a differentiable latent distribution over waypoints, bridging language–vision reasoning and numeric action spaces.The model encodes multi-view observations, temporal history, and perception cues before planning.
  • Visual prediction: Trajectory-conditioned future image prediction uses both a discrete autoregressive pathway and a continuous AR+diffusion pathway.The discrete pathway expands a visual codebook and detokenizes with MoVQGAN, while the continuous pathway predicts latent features.
  • Reported outcomes: The unified framework achieves high-fidelity, planning-conditioned visual generation and improves planning accuracy and perception performance on standard driving benchmarks.The introduction presents these improvements as evidence for the framework’s effectiveness.

2 Related Work

Prior autonomous-driving world models infer current and future states to support planning, while unified image models combine visual understanding and generation. UniDrive-WM extends these directions by coupling trajectory planning with predicted visual futures under a shared VLM backbone.

  • World Models for Autonomous Driving: Autonomous-driving world models infer ego status and dynamic environments from past observations to support future planning and reduce human control.The related work frames world modeling as understanding the current state and predicting future states.
  • World Models for Autonomous Driving: UniDrive-WM jointly performs trajectory planning and future image generation within a unified world-model framework.Its shared VLM backbone couples planned motion with predicted visual futures.
  • Unified Image Understanding and Generation: Unified image-understanding and generation studies commonly use discrete visual representations with autoregressive token prediction.These methods quantize images into discrete tokens that LLMs can interpret and generate.

3 Method

UniDrive-WM unifies scene understanding, trajectory planning, and future image generation in a VLM-centered world model. It predicts future states and trajectories jointly while comparing discrete autoregressive and continuous AR+Diffusion image-generation designs.

  • Framework overview: The framework combines a QT-Former encoder, an LLM reasoning core, and an output layer for planning trajectories and future images.The output layer bridges reasoning, planning, and image spaces.
  • Problem formulation: The task jointly predicts future scene states and planning trajectories from multimodal state information and high-level language or instruction conditions.The state includes multiview images, historical context, and perception features; future front-view images form part of the predicted state.
  • Vision-language model: The vision encoder extracts current and historical information through perception and history queries, then converts them into reasoning-space tokens for the LLM.The QT-Former uses cross-attention with image features and stores updated history queries in a memory bank.
  • Trajectory planner: The trajectory planner learns a multimodal waypoint distribution conditioned on VLM reasoning embeddings, creating a differentiable bridge to continuous action prediction.A Gaussian latent variable captures future-motion stochasticity, while omitting explicit KL regularization improves training stability in the stated setup.
  • Future image generation: Future image prediction conditions on multimodal state and VLM reasoning embeddings, with discrete AR and continuous AR+Diffusion architectures representing the same conditional distribution.The discrete pathway expands a joint language–vision codebook and autoregressively predicts visual tokens conditioned on planning tokens.
  • Future image generation: The AR+Diffusion pathway uses deterministic reconstruction and CLIP supervision to produce stable, semantically aligned future frames in a continuous latent space.Deterministic reconstruction reduces diversity while supporting stable and accurate future-frame prediction.

4 Experiments

Experiments evaluate UniDrive-WM on Bench2Drive and nuScenes across closed-loop driving, open-loop planning and perception, future-frame generation, and VQA. The unified model outperforms prior methods across planning and detection, generates trajectory-consistent frames, and exposes speed–fidelity trade-offs between AR and AR+Diffusion.

  • Experimental Setup: UniDrive-WM is trained and evaluated on Bench2Drive and nuScenes, using open-loop and closed-loop protocols.Bench2Drive uses 950 training scenes, 50 open-loop validation scenes, and 220 closed-loop routes across 44 interactive scenarios.
  • Planning and Perception: UniDrive-WM outperforms prior end-to-end and VLM-guided methods in closed-loop Bench2Drive evaluation.The reported gains cover methods conditioned on target points and navigation commands.
  • Planning and Perception: UniDrive-WM achieves better planning and detection performance on both Bench2Drive and nuScenes open-loop evaluations.The consistent gains across synthetic Bench2Drive and real-world nuScenes are reported as evidence of strong generalization.
  • Future Image Generation: UniDrive-WM generates coherent future frames that match scene evolution and remain consistent with predicted trajectories across AR and AR+Diffusion settings.These properties correspond to competitive or superior image-generation quality measured with FID.
  • AR and AR+Diffusion: The AR branch runs at 2 fps, whereas AR+Diffusion runs at 0.4 fps on an A100, making AR faster and AR+Diffusion more suitable for high-fidelity generation.The paper identifies accelerating AR+Diffusion inference as future work and describes the branches as a deployment split.
  • VQA: Adding image generation consistently improves all reported VQA metrics compared with the model variant without the image-generation module.The authors attribute this improvement to additional structural cues from future-frame prediction.

5 Conclusion and Future Work

UniDrive-WM integrates scene understanding, trajectory planning, and visual generation in one VLM-based world-model pipeline. Its planning-conditioned visual forecasts improve planning accuracy and reduce collisions, while future work targets more interactive and long-horizon driving.

  • Conclusion: UniDrive-WM unifies scene understanding, trajectory planning, and visual generation within one world-model-driven VLM pipeline.The model uses current and historical multi-view observations together with planning tokens to predict future frames.
  • Conclusion: Planning-conditioned visual predictions improve planning accuracy and reduce collisions in the reported experiments.The framework connects reasoning, action, and visual imagination through rich multi-view inputs, temporal history, and perception features.
  • Future Work: Future work will extend UniDrive-WM to more interactive and long-horizon driving scenarios.

Appendix

The supplied appendix passage identifies Table 9 as reporting multi-ability results on the Bench2Drive base set.

  • Appendix: Table 9 reports multi-ability results on the Bench2Drive base set.The table includes notation for expert feature distillation, camera/LiDAR inputs, navigation commands, and target points.

A Evaluation Metrics

The evaluation uses task-specific metrics for closed-loop driving, open-loop planning, 3D detection, future-frame prediction, and VQA. These metrics cover route success and quality, trajectory and collision performance, detection quality, visual fidelity, and language-based understanding.

  • Closed-Loop Driving: Closed-loop driving is evaluated with Driving Score, Success Rate, Efficiency, and Comfortness.Success Rate measures completed routes within the time limit, while Driving Score combines route completion with violation penalties.
  • Open-Loop Planning and Detection: Open-loop planning uses L2 trajectory error and collision rate, while 3D detection uses mAP, mATE, mASE, mAOE, mAVE, and NDS.The detection metrics measure accuracy, translation, scale, orientation, velocity, and overall detection quality.
  • Future Frame Prediction: Future-frame prediction is evaluated with Fréchet Inception Distance (FID) to measure similarity between generated and ground-truth frame distributions.
  • VQA: VQA evaluation uses BLEU, ROUGE-L, CIDEr, ChatGPT score, and multiple-choice accuracy.The first three metrics assess language generation, while ChatGPT score covers open-ended answering.

B.1 Additional Closed-Loop Evaluation Results

UniDrive-WM achieves stronger closed-loop performance than baseline methods across several Bench2Drive driving abilities, reflected in a higher mean score.

  • UniDrive-WM performs better in Merging, Overtaking, Emergency Brake, and Traffic Sign tasks than baseline methods.The reported gains are summarized by a higher mean score in the closed-loop multi-ability evaluation.

B.2 Further Visualization

Additional evaluations show that UniDrive-WM supports scene understanding, future-state reasoning, and plausible future-frame generation, while its perception backbone supplies structured features for downstream tasks.

  • Further Visualization: Open-loop VQA examples show scene understanding and reasoning about future states in complex driving scenarios and challenging weather.The responses are described as decision-oriented and focused on the ego vehicle’s movement and driving behavior.
  • Further Visualization: Both AR and AR+Diffusion architectures generate plausible future frames across diverse scenes and weather conditions.Qualitative results are provided on nuScenes for the AR architecture and the AR+Diffusion architecture.
  • QT-Former Backbone: QT-Former compresses multi-view observations and temporal history into structured visual representations for reasoning and task-specific perception heads.Scene, perception, and history queries interact with image features and a long-term memory bank before projection into the LLM reasoning space.
  • QT-Former Backbone: Auxiliary perception heads estimate 3D objects, lanes, traffic state, and dynamic-agent motion.These heads complement the detection head within the perception branch.
  • QT-Former Backbone: Detection supervision is reported through 3D detection metrics, while other heads provide auxiliary supervision for shared perception features and traffic-aware scene structure.The remaining heads are not used to report the main-paper 3D detection metrics.
  • QT-Former Backbone: During joint planning and image-generation training, the pretrained QT-Former is initialized and the detection head is frozen.Joint optimization therefore focuses on downstream planning and image-generation objectives.

D Additional Details on Trajectory Planner

The trajectory planner maps VLM reasoning and multimodal state representations into continuous future waypoints, while trajectory-conditioned image generation feeds supervisory information back into planning.

  • Trajectory Planner: The planner models a conditional distribution over future waypoints from the current multimodal state and VLM reasoning embedding.This establishes the planner as a differentiable bridge between semantic reasoning and continuous action spaces.
  • Trajectory Planner: The planner projects VLM reasoning into a Gaussian latent action space using lightweight MLP layers.The latent action variable is sampled from a distribution whose mean and variance are predicted from planning-related hidden representations.
  • Trajectory Planner: Reparameterization enables backpropagation through stochastic latent-action sampling.The sampled variable is expressed using the predicted mean, standard deviation, and Gaussian noise.
  • Trajectory Planner: A recurrent waypoint decoder regresses m continuous 2D BEV waypoints representing the future ego trajectory.The latent-variable design provides a smoother interface between VLM semantic intent and numerically precise continuous trajectories.
  • Trajectory Planner: UniDrive-WM omits explicit KL regularization because enforcing a standard-normal latent space was found to destabilize large-scale multimodal training.The planning head is instead optimized with the stated planning losses.
  • Trajectory Planner: The joint planning objective combines collision, boundary, and MSE losses to supervise safe, road-constrained trajectory prediction.Collision and boundary losses use perception outputs and road topology, while MSE supplies the primary expert-trajectory signal.
  • Trajectory Planner: Trajectory-conditioned future image generation provides additional supervisory signals that improve planning quality and visual quality in ablations.This links planned action to future observation within the unified world-model pipeline.
Loading 2601.04453v4…