Source-linked AI summary
Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, Zhaoxiang Zhang
TL;DR
Out-of-distribution cases challenge end-to-end autonomous-driving planners, motivating a world model that can forecast controllable multiview futures. Drive-WM combines joint spatial-temporal modeling with view factorization and image-based rewards for planning, generating high-quality, consistent, and controllable videos while improving planning soundness and robustness.
Problem
End-to-end planners may struggle with out-of-distribution driving cases, while driving world models must support high-resolution pixel-space modeling and consistent multiview prediction.
Method
Drive-WM jointly models multiview video spatially and temporally, factorizes view prediction for consistency, supports heterogeneous conditions, and evaluates maneuver futures with image-based rewards.
Results
Drive-WM generates high-quality, controllable, and consistent multiview driving videos and improves planning soundness and robustness in out-of-distribution situations.
Takeaways & Limitations
The results support using Drive-WM for multiview driving-scene simulation and safer end-to-end planning.
Abstract
from arXiv · showhide
In autonomous driving, predicting future events in advance and evaluating the foreseeable risks empowers autonomous vehicles to better plan their actions, enhancing safety and efficiency on the road. To this end, we propose Drive-WM, the first driving world model compatible with existing end-to-end planning models. Through a joint spatial-temporal modeling facilitated by view factorization, our model generates high-fidelity multiview videos in driving scenes. Building on its powerful generation ability, we showcase the potential of applying the world model for safe driving planning for the first time. Particularly, our Drive-WM enables driving into multiple futures based on distinct driving maneuvers, and determines the optimal trajectory according to the image-based rewards. Evaluation on real-world driving datasets verifies that our method could generate high-quality, consistent, and controllable multiview videos, opening up possibilities for real-world simulations and safe planning.
1. Introduction
End-to-end planners can struggle with out-of-distribution driving situations, motivating Drive-WM, a controllable multiview world model for forecasting and safer planning.
- End-to-end autonomous driving models jointly optimize sensor processing and planning but may generalize poorly beyond expert-trajectory distributions.The paper highlights difficulty handling out-of-distribution cases when the ego vehicle deviates laterally from the lane center.
- Real-world driving world models must represent high-resolution pixel-space events and maintain consistency across multiple views.The paper identifies pixel-space fidelity and multiview consistency as central challenges for driving-scene generation.
- Drive-WM jointly models multiple views and frames, factorizing view prediction through adjacent views to improve multiview consistency.Its unified condition interface supports images, text, 3D layouts, and actions for flexible conditional generation.
- Drive-WM generates high-quality, controllable, and consistent multiview driving videos, with experiments on nuScenes evaluating quality, controllability, and consistency.The paper also reports improved planning soundness and robustness in out-of-distribution situations.
2. Related Works
Prior work spans several video-generation paradigms, video prediction, and world models, while autonomous-driving methods have largely remained monocular.
- Video-generation research includes VAE-, GAN-, flow-, autoregressive-, and diffusion-based methods.Diffusion models have recently improved realism and controllability in image and video generation.
- Driving video-prediction methods use observations or vehicle actions to generate future frames, but prior systems are limited to monocular videos.The paper positions multiview generation as a response to their inability to represent overall 3D surroundings.
- World models learn representations of environments and predict future states resulting from action sequences, with prior successes in games and laboratory robotics.Dreamer-family systems perform visual control by modeling dynamics and values in latent spaces.
3. Multi-view Video Generation
Drive-WM jointly models multiview temporal data, improves consistency through factorized generation, and integrates heterogeneous conditions through a unified interface. Its pipeline combines temporal and cross-view modeling with controllable video generation.
- 3.1. Joint Modeling of Multiview Video: Drive-WM adapts image diffusion to multiview-temporal data by adding temporal and multiview layers.The denoising model operates on encoded, diffused video latents with spatial, temporal, and multiview parameters.
- 3.1. Joint Modeling of Multiview Video: The model jointly generates multiple frames and views, using temporal encoding for frame dynamics and multiview attention for cross-view information exchange.Temporal layers operate across frames, while multiview attention operates across views.
- 3.2. Factorization of Joint Multiview Modeling: Factorized generation partitions views into reference and stitched groups, generating stitched views conditionally from reference views to improve consistency in overlapping regions.In nuScenes, reference views include {F, BL, BR}, while stitched views include {FL, B, FR}.
- 3.3. Unified Conditional Generation: The framework overview combines a three-view output with factorized generation to produce additional views and enhance multiview consistency.Figure 3 also depicts the training and inference pipeline and the unified controls for multiview video generation.
- 3.2. Factorization of Joint Multiview Modeling: Previous frames are added as context, so reference-view generation and conditional stitched-view generation preserve temporal coherence.The context can include the last two frames from previously generated clips.
- 3.3. Unified Conditional Generation: A unified condition interface maps images, layouts, text, and actions into a shared feature space before cross-attention with the latent video representation.The supported conditions include context frames, text descriptions, ego actions, 3D boxes, BEV maps, and reference views.
4. World Model for End-to-End Planning
Drive-WM evaluates candidate future scenarios before selecting trajectories, using image-based rewards to support safer end-to-end planning. Its pixel-space modeling also supports rewards and counterfactual cases that are difficult to represent with vectorized perception.
- Motivation: World-model planning is motivated by the danger of choosing actions without anticipating consequences and is presented as an under-explored application in autonomous driving.The section evaluates future assessment and out-of-domain planning improvement.
- Planning with world models: Drive-WM generates predicted future scenarios for planner-sampled trajectory candidates, evaluates them with image-based rewards, and selects the optimal trajectory for tree-based rollout.The process repeats across timestamps to extend the planning tree.
- Tree-based rollout with actions: The planning tree represents predicted ego trajectories evolving over time, with actions defined as successive changes in ego x and y locations.This action representation is designed to remain compatible with mainstream planners.
- Image-based reward function: The total reward multiplies map and object rewards, encouraging drivable-area and lane consistency while avoiding collisions with other road users.The highest-reward ego prediction is selected before planning advances to the next timestamp.
- Image-based reward function: Pixel-space rewards can cover non-vectorized events such as sprinkler water and damaged road surfaces, which supervised perception models struggle to represent.The paper connects this capability to world-model training on massive unlabeled data and possible use of foundational models.
5. Experiments
Experiments on nuScenes evaluate generation quality, controllability, consistency, ablations, and planning. Drive-WM produces strong multiview video results, improves consistency through factorization, and supports tree-based planning and recovery from out-of-domain ego deviations.
- Main results of multiview video generation: 15.8 FID and 122.7 FVD are achieved for video generation, alongside 12.99 FID for single-view image generation.The paper reports significant quality improvements over prior single-view video-generation methods and consistent multiview video generation.
- Main results of multiview video generation: Drive-WM surpasses other methods in nuScenes multiview video quality and controllability evaluation.The evaluation uses models trained on the nuScenes training set for each task.
- Ablation study for multiview video generation: Layout conditioning improves generated-video quality and consistency, while temporal embedding further enhances video quality.These findings come from layout-based video-generation ablations on the nuScenes validation set.
- Ablation study for multiview video generation: Factorized generation increases multiview consistency from 45.8% to 94.4% compared with joint modeling while preserving image and video quality.Qualitative comparisons are provided in Figure 6.
- Exploring planning with world model: The tree-based planner outperforms random driving commands and approaches performance using the ground-truth command.Combined map and object rewards outperform either sub-reward, especially for object collision.
- Exploring planning with world model: Generated-video supervision improves planning performance toward normal levels after a 0.5-meter lateral ego-location deviation causes VAD performance to decrease.Drive-WM also generates counterfactual turning-around and non-drivable-area events absent from the training data.
6. Conclusion
Drive-WM is presented as the first multiview world model for autonomous driving, generating controllable, high-quality, and consistent videos. Experiments on nuScenes indicate improved planning soundness and robustness in out-of-distribution situations.
- The model generates high-quality and consistent multiview videos under diverse conditions using textual descriptors, layouts, or ego actions.
- Factorized generation significantly enhances spatial consistency across various views.
- Experiments on nuScenes show improved planning soundness and robustness in out-of-distribution situations.
Supplementary Material
Supplementary examples show Drive-WM forecasting multiple maneuver-dependent futures and generating diverse multiview scenes. The model also produces rare nighttime and rainy scenarios despite limited training exposure.
- Drive-WM forecasts multiple plausible futures consistent with initial observations and planner maneuvers.Examples include left lane changes, keeping the lane, roadside driving, straight driving, and intersection turns.
- Drive-WM can generate diverse multiview video forecasts from temporal layouts and novel combinations of objects, motions, and scenes.The examples use the nuScenes validation set although training used its training set.
- The model generates diverse multiview forecasts from layout conditions in normal scenes.
- Drive-WM produces high-quality videos for rare nighttime and rainy driving conditions despite limited exposure during training.The passage describes this as generalization beyond daytime scenarios dominant in the training distribution.
A.3. Visual Element Control
Drive-WM supports interactive visual control through text, ego actions, and 3D layouts. Qualitative results show preserved scene structure under weather changes and maneuver-, vehicle-, foreground-, and pedestrian-level control.
- Drive-WM supports conditional generation through text prompts, ego-vehicle actions, and 3D boxes.These controls modify weather and lighting, driving maneuvers, and foreground layouts, respectively.
- Weather and lighting can change while the road structure and foreground objects remain the same.The passage identifies diverse weather and lighting generation as having potential for data augmentation.
- Given steering signals, the model generates turning-left and turning-right videos from the same initial frame.It also predicts surrounding-vehicle positions consistent with accelerating and decelerating signals.
- Drive-WM provides fine-grained foreground control through modified lateral and longitudinal conditions.The resulting images correspond to the specified layout changes.
- Drive-WM generates pedestrians in scenes involving crossing and bus-stop interactions.These results indicate potential for detailed multi-agent interaction generation.
A.4. End-to-end Planning Results for Out-of-domain Scenarios
Drive-WM is used to explore out-of-distribution planning cases, evaluate maneuver-dependent futures, and generate forecasts across datasets. These experiments expose planner recovery failures and support image-based safety assessment.
- Existing end-to-end planners trained on lane-centered expert trajectories struggle with off-center deviations, known as the lack-of-exploration problem.The planner from [30] cannot recover when evaluated on generated out-of-distribution cases.
- Drive-WM simulation generates more planned trajectories deviating from the lane center, supporting exploration of corner cases and robustness improvement.
- Figure 8 organizes plausible futures by maneuver, including lane changes, roadside or straight driving, and intersection turns.
- GPT-4V evaluates safety across Drive-WM futures with varying road conditions and agent behaviors using holistic image-based rewards.The passage contrasts this with vectorized reward functions and describes broader scene understanding of hazards.
- GPT-4V critique of Drive-WM forecasts provides more robust feedback for improving autonomous-driving safety under diverse real-world conditions.
- On the Waymo Open Dataset, Drive-WM generates realistic and diverse 768×512 driving forecasts using the same hyper-parameters as nuScenes.The examples are presented as evidence of adaptation across datasets and resolutions.
B. Implementation Details
The implementation jointly models multiview frames and temporal sequences, using factorization and heterogeneous conditions to generate consistent videos. Inference uses DDIM-based sampling and autoregressive frame conditioning.
- Model implementation: The unified condition interface supports images, text, 3D layouts, and actions for flexible conditional generation.The implementation conditions image diffusion on HD maps, BEV segmentation, 3D bounding boxes, and text descriptions.
- Inference: During inference, the model uses 50 sampling steps, stochasticity η=1.0, and CFG=5.0, generating later video frames from preceding generated frames.Long-video generation repeatedly uses the generated frame as the next condition.
- Conditional generation: The implementation also demonstrates conditional changes in weather, lighting, foreground layouts, and turning behavior across generated views.Captions describe sunny-to-rainy, daytime-to-nighttime, box-layout, and steering-controlled generation examples.
- Model implementation: Drive-WM combines joint multiview-temporal modeling with a factorization model that generates stitched views from adjacent reference views and previously generated frames.The factorized model generates a single stitched view per iteration during training and conditions on paired reference clips during inference.
- Factorized inference: For nuScenes factorization, front, back-left, and back-right views are selected as reference views before stitched views are generated.The stitched views are conditioned on paired reference videos and previously generated views.
C.1. Data Preparation
Data preparation uses nuScenes multiview scenes and Waymo front-camera data, while action resampling addresses severe imbalance in steering-angle and speed combinations.
- Datasets: nuScenes provides 360-degree camera coverage, with 700 scenes used for training and 150 for validation.The preparation processes 3D boxes, HD maps, BEV segmentation, and text conditions.
- Datasets: Waymo Open Dataset training uses only the front camera at 768 × 512 resolution, with map processing following OpenLane.This provides an additional driving dataset and front-view video setting.
- Action distribution: nuScenes action data is heavily imbalanced toward steering angles below 30 degrees and speeds of 10-20 m/s, weakening generalization to rare action combinations.The imbalance motivates balancing the training distribution by resampling rare ego actions.
- Action distribution: The curation procedure splits trajectories into behavior-specific clips and clusters them by average steering angle and speed.The procedure produces 1048 unique clips before grid-based resampling.
- Action distribution: Sampling N = 36 clips from each bin of the 2D 32 × 11 grid yields 7272 clips in the balanced training dataset.Bins with too many clips are randomly subsampled, while undersized bins are looped until they reach N samples.
D. Metric Evaluation Details
Evaluation measures video quality, multiview consistency, controllability, and planning behavior using generated examples and dedicated metrics. The evaluation includes both standard video metrics and keypoint-based consistency analysis.
- Standard metrics: FID and FVD are computed on 150 nuScenes validation clips decomposed into six views, producing 900 video segments of 40 frames.The evaluation follows the VideoLDM calculation process and uses the official UCF FVD code.
- Multiview consistency: The KPM score evaluates multiview consistency through keypoint matching in overlapping image regions between adjacent views.LoFTR matches keypoints between the current view and its two neighboring views.
- Planning evaluation: GPT-4V is used as an image-based reward function that provides more reasonable rewards for situations such as a puddle ahead.This supports reward-based evaluation of imagined driving futures.
- Planning evaluation: VAD produces reasonable trajectories under normal data but irrational trajectories when the ego location is shifted 0.5m to create an out-of-domain case.The comparison contrasts top-row normal-data predictions with bottom-row out-of-distribution behavior.