Source-linked AI summary
World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model
Yupeng Zheng, Pengxuan Yang, Zebin Xing, Qichao Zhang, Yuhang Zheng, Yinfeng Gao, Pengfei Li, Teng Zhang, Zhongpu Xia, Peng Jia, Dongbin Zhao
TL;DR
End-to-end autonomous driving needs informative world models that capture complex scenes and planning intentions without costly perception annotations. World4Drive uses vision foundation model priors to build an intention-aware latent world model that generates and evaluates multi-modal trajectories. It achieves state-of-the-art annotation-free planning on nuScenes and NavSim, with lower displacement error and collision rate than a prior unsupervised method.
Problem
End-to-end autonomous driving requires modeling spatial, semantic, and temporal scene information under uncertain planning intentions, while existing approaches often depend on costly perception annotations.
Method
World4Drive uses physical and intention encoders with vision foundation model priors to construct latent world representations and evaluate multi-modal trajectories under different intentions.
Results
18.2% lower average planning displacement error and 46.7% lower average collision rate are reported than LAW, with state-of-the-art annotation-free planning on nuScenes and NavSim.
Takeaways & Limitations
World4Drive demonstrates that tightly coupling driving intentions with a spatial-semantic physical latent world model supports annotation-free end-to-end planning across open-loop and closed-loop benchmarks.
Takeaways & Limitations
Single-modal latent features from images struggle to capture spatial-semantic scene information and multi-modal driving intentions.
Abstract
from arXiv · showhide
End-to-end autonomous driving directly generates planning trajectories from raw sensor data, yet it typically relies on costly perception supervision to extract scene information. A critical research challenge arises: constructing an informative driving world model to enable perception annotation-free, end-to-end planning via self-supervised learning. In this paper, we present World4Drive, an end-to-end autonomous driving framework that employs vision foundation models to build latent world models for generating and evaluating multi-modal planning trajectories. Specifically, World4Drive first extracts scene features, including driving intention and world latent representations enriched with spatial-semantic priors provided by vision foundation models. It then generates multi-modal planning trajectories based on current scene features and driving intentions and predicts multiple intention-driven future states within the latent space. Finally, it introduces a world model selector module to evaluate and select the best trajectory. We achieve perception annotation-free, end-to-end planning through self-supervised alignment between actual future observations and predicted observations reconstructed from the latent space. World4Drive achieves state-of-the-art performance without manual perception annotations on both the open-loop nuScenes and closed-loop NavSim benchmarks, demonstrating an 18.1\% relative reduction in L2 error, 46.7% lower collision rate, and 3.75 faster training convergence. Codes will be accessed at https://github.com/ucaszyp/World4Drive.
1. Introduction
World4Drive addresses the challenge of modeling spatial, semantic, temporal, and multi-modal driving information without perception annotations by combining driving intentions with a latent world model. It achieves annotation-free planning with strong benchmark performance and faster convergence.
- End-to-end autonomous driving must model spatial, semantic, and temporal scene information while handling uncertainty in planning intentions.
- Existing approaches commonly require perception annotations such as 3D bounding boxes and HD maps, limiting scalability.
- World4Drive integrates multi-modal driving intentions with a latent world model to support rational planning and simulate future physical-world evolution.
- World4Drive combines a physical latent encoder and intention encoder to represent spatial-semantic-temporal scene context and multi-modal driving intentions.
- 18.2% lower average planning displacement error and 46.7% lower average collision rate are achieved than LAW, without perception annotations on nuScenes and NavSim.
- World4Drive achieves state-of-the-art planning without perception annotations and significantly accelerates convergence by using vision foundation model priors.
2. Related Works
Related work spans diverse end-to-end scene representations and world models for autonomous driving. Annotation-free latent-world approaches avoid perception labels but may struggle to represent spatial-semantic information and multi-modal driving uncertainty.
- End-to-end driving methods use BEV-centric, vector-based, sparse-centric, generative, and parallelized scene or trajectory representations.
- World-model research includes image-based video generation, 3D representations such as point clouds and occupancy grids, and latent feature-based future-world generation.
World Model Selector
World4Drive’s intention-aware latent world model predicts future latent states under multiple driving intentions and evaluates trajectories for selection.
- The model generates, evaluates, and ranks multi-modal trajectories under multi-modal driving intentions.
- Unlike single-modal latent prediction, the approach models future scene representations across multi-modal driving intentions for trajectory evaluation.
3.1. Overview
World4Drive has two tightly coupled modules: Driving World Encoding extracts intentions and physical world latents, while the Intention-aware World Model predicts future latents and scores planning trajectories.
- Driving World Encoding extracts driving intentions and physical world latent representations from RGB images and a trajectory vocabulary.
- The Intention-aware World Model predicts future world latents under multi-modal intentions and scores candidate trajectories with a world model selector.
3.2. Driving World Encoding
Driving World Encoding extracts intentions from trajectory vocabulary and builds physical world latents enriched with spatial, semantic, and temporal context from multi-view images.
- Intention Encoder: The intention encoder derives driving intentions from a trajectory vocabulary containing clustered endpoint alternatives for left, right, and straight commands.The vocabulary contains N trajectories with S waypoints, while K intentions are retained per command type.
- Physical World Latent Encoding: The physical latent encoder combines a context encoder with temporal aggregation to represent the dynamic driving environment.The context encoder incorporates spatial and semantic priors, while temporal aggregation enhances temporal context.
- Context Encoder: The context encoder uses a 3D spatial encoding module and a semantic understanding module to achieve holistic physical-world understanding.It uses metric depth and vision-language priors to enrich visual features with spatial-semantic information.
- Temporal Aggregation: Temporal cross-attention aggregates the prior timestep’s visual features into current features, enriching world latents with temporal information.The resulting representation supports imagining future states of the dynamic driving environment.
3.3. Planning with Intention-aware World Model
The intention-aware world model generates trajectories under multiple driving intentions, predicts their future latent states, and selects a trajectory using latent similarity and learned scores.
- Intention-aware World Model: The model generates multimodal planning trajectories from intention-aware planning queries and encodes each intention as an action token.A cross-attention layer aggregates scene context before an MLP produces trajectories and action tokens.
- Intention-aware World Model: Future latent prediction models how the world evolves under each intention by combining action tokens with the current world latent.A learnable future query and multilayer cross-attention predict future world latents at a specified timestamp interval.
- World Model Selector: The World Model Selector compares predicted and actual future latent distances and selects the modality with the minimum distance.Its selected latent and corresponding trajectory provide the final modality during training-time selection.
- World Model Selector: During inference, the selector chooses the trajectory associated with the highest predicted score.The scoring network is trained with focal loss against the selected modality index, while latent distance uses MSE.
3.4. Training Loss
World4Drive is trained end to end with trajectory supervision and a composite loss whose default weighting emphasizes the trajectory objective.
- Training Loss: The selected planning trajectory is guided toward the expert trajectory using an L1 trajectory loss.This objective is part of World4Drive’s end-to-end trainable formulation.
- Training Loss: The final end-to-end training objective combines the trajectory loss with additional loss terms using fixed default weights.The defaults are α = 0.2, β = 0.2, γ = 0.5, and η = 1.0.
4. Experiments
World4Drive is evaluated on open-loop nuScenes and closed-loop NavSim, including annotation-free comparisons, component ablations, diverse conditions, maneuvers, and scalability settings. It achieves strong planning and safety performance while showing benefits from intention modeling, vision foundation-model priors, and increased model capacity.
- Benchmarks and setup: nuScenes uses L2 displacement error and collision rate for trajectories sampled at 2Hz over a 3-second horizon, while NavSim evaluates closed-loop PDMS over five factors.NavSim trajectories are sampled at 2Hz over 4 seconds and interpolated with an LQR controller.
- Main results: 18.0% lower L2 error and 46.7% lower collision rate are reported against strong baselines, with the lowest collision rate among all methods.World4Drive achieves SOTA among perception annotation-free approaches and requires no manual perception annotations for training or inference.
- Main results: World4Drive achieves competitive closed-loop PDMS, improves TTC and DAC over the baseline, and surpasses annotation-dependent methods except DiffusionDrive.TTC and DAC evaluate time-to-collision and drivable-area compliance, respectively.
- Ablation studies: Adding vehicle intention or vision foundation-model and vision-language priors improves planning over the single-modal LAW baseline.Spatial priors improve positional awareness and trajectory fitting, while semantic priors reduce collision rates by improving obstacle understanding.
- Ablation studies: Combining intentions with world modeling improves planning quality, whereas intentions alone without world modeling degrade performance.The world model evaluates and ranks multi-modal intentions to select safer trajectories.
- Robustness and scalability: Collision rates decrease by 63.7% at night and 68.8% in rain versus LAW, while performance remains safer across diverse driving maneuvers.The method also shows comparable improvements when scaling either the image backbone or hidden dimension.
4.5. Qualitative Results
Qualitative visualizations compare World4Drive with LAW during turning and show how its world model selector chooses among multi-modal planning intentions. The visualization renders ground-truth annotations as perception results because the method does not predict explicit perception outputs.
- Qualitative Results: World4Drive produces safer planning than LAW during turning maneuvers in the qualitative visualization.
- Qualitative Results: Ground-truth annotations are rendered as perception results because World4Drive does not predict explicit perception outputs.
- Qualitative Results: The world model selector selects the most reasonable trajectory from multi-modal planning intentions across diverse scenarios.
5. Conclusion
World4Drive combines driving intentions with a physical latent world model and uses vision foundation-model priors plus temporal aggregation. Experiments on nuScenes and NavSim support its planning effectiveness and physical-world modeling.
- 5. Conclusion: World4Drive combines driving intentions with a latent world model to generate, evaluate, and select multi-modal trajectories.
- 5. Conclusion: Its physical world latent encoding module incorporates spatial and semantic vision foundation-model priors and aggregates temporal information.
- 5. Conclusion: Experiments on nuScenes and NavSim demonstrate the framework’s planning effectiveness and comprehensive physical-world understanding.