PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving

Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, Liping Jing, Yiming Nie, Bin Dai

arXiv:2405.04390v1cs.CV

TL;DR

Vision-centric autonomous-driving pre-training largely uses 2D or 3D tasks, leaving temporal 4D scene understanding under-addressed. DriveWorld uses a world model with separate temporal and spatial latent modeling plus task prompts to learn spatio-temporal representations from multi-camera videos. It improves performance across detection, mapping, tracking, forecasting, occupancy prediction, and planning, including a 7.5% mAP gain for 3D object detection and a 0.34m reduction in average L2 planning error.

  • Problem

    Existing vision-centric autonomous-driving pre-training predominantly uses 2D or 3D pre-text tasks and overlooks 4D spatio-temporal representation learning.

    Source: §1 · pp. 1–2

  • Method

    DriveWorld uses a world model with a Memory State-Space Model for temporal dynamics and spatial statics, together with Task Prompts for task-aware representations.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    DriveWorld improves vision-centric perception, prediction, and planning tasks; it achieves a 7.5% mAP increase for 3D object detection and reduces average planning L2 error by 0.34m.

    Source: §1 · p. 2, §4.3 · pp. 7–8

  • Takeaways & Limitations

    World-model-based 4D pre-training provides a unified spatio-temporal representation that supports strong performance across autonomous-driving tasks.

    Source: §1 · p. 2, §5 · p. 8

    DriveWorld uses LiDAR-based annotations and has only been validated with a lightweight ResNet101 backbone.

    Source: §5 · p. 8

Abstract

from arXiv · show

Vision-centric autonomous driving has recently raised wide attention due to its lower cost. Pre-training is essential for extracting a universal representation. However, current vision-centric pre-training typically relies on either 2D or 3D pre-text tasks, overlooking the temporal characteristics of autonomous driving as a 4D scene understanding task. In this paper, we address this challenge by introducing a world model-based autonomous driving 4D representation learning framework, dubbed \emph{DriveWorld}, which is capable of pre-training from multi-camera driving videos in a spatio-temporal fashion. Specifically, we propose a Memory State-Space Model for spatio-temporal modelling, which consists of a Dynamic Memory Bank module for learning temporal-aware latent dynamics to predict future changes and a Static Scene Propagation module for learning spatial-aware latent statics to offer comprehensive scene contexts. We additionally introduce a Task Prompt to decouple task-aware features for various downstream tasks. The experiments demonstrate that DriveWorld delivers promising results on various autonomous driving tasks. When pre-trained with the OpenScene dataset, DriveWorld achieves a 7.5% increase in mAP for 3D object detection, a 3.0% increase in IoU for online mapping, a 5.0% increase in AMOTA for multi-object tracking, a 0.1m decrease in minADE for motion forecasting, a 3.0% increase in IoU for occupancy prediction, and a 0.34m reduction in average L2 error for planning.

1. Introduction

Vision-centric autonomous driving needs spatio-temporal representations for perception, prediction, and planning, but existing pre-training mainly uses 2D or 3D tasks rather than 4D scene understanding. DriveWorld addresses this gap with world-model-based 4D pre-training and reports improvements across autonomous-driving tasks.

1. Introduction

  • 4D scene understanding requires robust spatio-temporal representations for perception, prediction, and planning, yet spatio-temporal pre-training for autonomous driving remains limited.Source: §1 · p. 1
  • Existing vision-centric pre-training predominantly relies on 2D tasks such as classification and depth estimation or 3D tasks such as scene reconstruction, overlooking 4D representation.Source: Figure 1 · p. 1, §1 · p. 2
  • The framework targets driving uncertainty by learning temporal-aware dynamics for future-state prediction, spatial-aware statics for scene context, and task-specific features through prompts.Source: §1 · p. 2, Figure 2 · p. 3
  • DriveWorld introduces world-model-based 4D pre-training that learns compact spatio-temporal representations from multi-camera driving videos.Source: §1 · p. 2

2.1. Pre-training for Autonomous Driving

  • DriveWorld is presented as the first world-model-based 4D pre-training method for real-world vision-centric autonomous driving, with experiments reporting state-of-the-art results across perception, prediction, and planning.Source: §1 · p. 2

2.3. World Models

  • World-model research spans reinforcement learning, in-the-wild video pre-training, and generative driving models, while DriveWorld focuses on 4D representation learning for autonomous-driving pre-training.Source: §1 · p. 2, §2.3 · p. 3

3. DriveWorld

DriveWorld learns a compact spatio-temporal BEV representation by predicting current and future 3D occupancy and actions from multi-camera observations. Its Memory State-Space Model separates temporal dynamics from spatial statics, while task prompts adapt features for downstream objectives.

3. DriveWorld

  • DriveWorld transforms multi-camera observations into a world-model representation that predicts current and future 3D occupancy and actions.Source: §3 · p. 4

3.1. Memory State-Space Model

  • The Memory State-Space Model separates temporal-aware dynamics from spatial-aware statics to model moving objects and persistent scene context.Source: §3.1 · p. 4, Figure 3 · p. 4
  • The probabilistic model matches posterior states inferred from observed images and actions with prior states predicted from historical states and actions, enabling future prediction without images.Source: §3.1 · p. 4
  • The Dynamic Memory Bank uses motion-aware layer normalization and memory interaction to encode object motion and propagate temporal information.Source: §3.1 · pp. 4–5
  • The Static Scene Propagation module preserves spatial information in BEV features, combines static and dynamic representations without warping, and passes the result to the decoder.Source: §3.1 · p. 5

3.2. 3D Occupancy Prediction

  • 3D occupancy provides a full geometric scene structure, retaining height and object extent beyond depth surfaces, 2D reconstruction, or BEV segmentation targets.Source: §3.2 · p. 5

3.3. Task Prompt

  • Task Prompts use semantic cues from a pre-trained text encoder to guide task-aware feature extraction because downstream tasks prioritize different temporal and spatial information.Source: §3.3 · p. 5

3.4. Pre-training Objective

  • DriveWorld pre-training combines KL divergence for posterior-prior state matching with cross-entropy and L1 losses for occupancy and action prediction, adding L2 occupancy-flow loss on OpenScene.Source: §3.4 · p. 5

4. Experiments

DriveWorld is evaluated through controlled ablations, multi-task benchmarks, and qualitative comparisons against established pre-training methods. Across tasks, 4D world-model pre-training improves perception, temporal prediction, occupancy, and planning, while task prompts and larger pre-training data further benefit performance.

4.1. Experimental Setup

  • The evaluation pre-trains on nuScenes and OpenScene, retains the pre-trained encoder for downstream fine-tuning, and uses BEVFormer or UniAD task frameworks.The default pre-training uses four observed and four future steps, with ResNet101-DCN as the backbone.Source: §4.1 · p. 6

4.2. Ablation Studies

  • Ablations show that SSP improves performance by approximately 1%, DMB trades detection and mapping performance for tracking gains, and MLN improves all perception tasks.Task prompts provide further perception gains by decoupling information for different downstream tasks.Source: §4.2 · p. 6, Table 1 · p. 6
  • Increasing pre-training data improves downstream performance, while using 75% of fine-tuning data remains comparable and reduces annotation requirements by 25%.The study frames this as a practical and economic advantage of 4D pre-training.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Main Results

  • DriveWorld improves 3D detection by 7.5% mAP, online mapping by 3% IoU, tracking by 5.3% AMOTA, and motion forecasting by 0.1m minADE after OpenScene pre-training.The comparisons use the same decoder-head setup as UniAD and contrast 4D pre-training with 2D, 3D occupancy, and distillation baselines.Source: §4.3 · pp. 7–8

4.4. Qualitative Results

  • Qualitative comparisons show fewer distant-object false positives and more accurate turning-vehicle trajectory predictions after DriveWorld pre-training.These examples connect the quantitative gains to improved detection and future-change prediction.Source: Figure 4 · p. 8, §4.4 · p. 8

4.3. Main Results

  • OpenScene pre-training raises occupancy performance by 2.8% IoU-near, 5% IoU-far, 3.4% VPQ-near, and 3.4% VPQ-far.The result is presented as evidence of more comprehensive 4D scene reconstruction.Source: Table 7 · p. 7, §4.3 · p. 8
  • A 0.34m reduction in average L2 error and an average Collision rate of 0.12 establish new state-of-the-art planning results over UniAD.The paper attributes the gain to combining improved current-scene reconstruction with future-scenario prediction.Source: Table 8 · p. 7, §4.3 · p. 8

5. Conclusion

DriveWorld is a world model-based 4D pre-training method that learns compact spatio-temporal BEV representations and task-specific features for vision-centric autonomous driving. Its validation shows improved performance across autonomous driving tasks, while current limitations include LiDAR-based annotation and evaluation only with ResNet101.

5. Conclusion

  • DriveWorld predicts 3D occupancy from past multi-camera images and actions while learning temporal and spatial representations through a Memory State-Space Model.The model includes Dynamic Memory Bank and Static Scene Propagation modules, plus a Task Prompt for adaptive task-specific representations.Source: §5 · p. 8
  • DriveWorld significantly enhances performance across various autonomous driving tasks and represents 4D world knowledge for vision-centric driving.Source: §5 · p. 8
  • DriveWorld currently relies on LiDAR point-cloud annotations and has been validated only with the lightweight ResNet101 backbone.The authors identify self-supervised learning, larger datasets, and larger backbones as future directions.Source: §5 · p. 8

6. Pre-training Objective

The pre-training objective models DriveWorld as a probabilistic world model with deterministic and stochastic latent states. It jointly predicts past and future occupancy and actions while matching inferred stochastic states to their priors.

6. Pre-training Objective

  • The objective predicts past and future 3D occupancy and actions, combining reconstruction terms with posterior-prior KL matching for stochastic states.The resulting losses include occupancy, action, future occupancy, future action, and KL terms.Source: §6 · p. 13
  • DriveWorld parameterizes BEV, stochastic-state, transition, static-propagation, action-decoding, and 3D-occupancy components within a unified generative model.Source: §6 · p. 13
  • The model uses deterministic hidden states and inferred stochastic states to represent spatio-temporal driving dynamics from observations and actions.Because h_t is deterministic, variational inference is used to infer the latent stochastic variables s_t.Source: §6 · p. 13
  • Gaussian stochastic states, Laplace actions, and categorical occupancy labels yield closed-form KL, L1, and cross-entropy losses, respectively.Source: §6 · p. 13

7. Lower Bound Derivation

The lower-bound derivation converts the world-model likelihood into an evidence lower bound over latent states. The bound separates predictive log-likelihood terms from a sum of KL divergences matching inferred stochastic states to their transition priors.

7. Lower Bound Derivation

  • The variational lower bound consists of expected log-likelihood for observed and future occupancy and actions minus a KL divergence between variational and prior latent-state distributions.Source: §7 · p. 13
  • The expected log-likelihood decomposes into per-step past predictions and future predictions conditioned on the terminal latent state.Past terms use h_t and s_t, whereas future terms use h_T and s_T.Source: §7 · p. 14
  • Because the deterministic-state transition distributions match, the KL term reduces to a sum matching each inferred stochastic state posterior with its prior conditioned on previous latent states.Source: §7 · p. 14
  • Recursive decomposition expresses the latent-state divergence as expected per-time-step KL divergences between q(s_t|o_≤t,a_<t) and p(s_t|h_t−1,s_t−1).Source: §7 · p. 14
  • The final bound combines past occupancy and action likelihoods, future occupancy and action likelihoods, and the accumulated posterior-prior KL penalty.Source: §7 · pp. 13–15

8. Dataset

DriveWorld uses nuScenes and OpenScene to support autonomous-driving pre-training and 4D supervision. OpenScene supplies semantic occupancy and occupancy-flow labels, while dense occupancy ground truth is derived from fused LiDAR frames.

8. Dataset

  • nuScenes contains 700 training, 150 validation, and 150 testing sequences recorded across Boston and Singapore under diverse conditions.Source: §8 · p. 15
  • OpenScene provides over 120 hours of occupancy labels across Boston, Pittsburgh, Las Vegas, and Singapore, including semantic occupancy and occupancy-flow information.DriveWorld uses both supervision types for 4D pre-training.Source: §8 · p. 15
  • Dense 3D occupancy ground truth is obtained by fusing multiple LiDAR point-cloud frames, covering occluded regions more comprehensively than single-frame point clouds.Source: §8 · p. 15
  • The Memory State-Space Model represents deterministic states with squares, stochastic states with circles, observed states in grey, and generative versus variational paths with solid versus dotted lines.Source: Figure 6 · p. 15

9. Task Prompt

Task prompts adapt shared BEV features to the semantic needs of each downstream task, reducing redundant information and emphasizing task-relevant spatial or temporal cues.

9. Task Prompt

  • Task prompts are added to BEV maps before each downstream decoder to decouple task-aware features across tasks.The prompt encoder transfers to downstream tasks, while fine-tuning includes task-specific prompts.Source: §9 · p. 15
  • The task prompts include explicit descriptions for 3D object detection and planning with current and future scenes.The current design is relatively simple, and autonomous-driving tasks remain limited in number.Source: §9 · p. 15
  • Online mapping prompts emphasize current spatio-aware information, whereas motion forecasting prompts preserve both spatial and temporal information.These guided features respectively improve target localization precision and support more robust prediction.Source: §9 · p. 15
  • Figure 7 qualitatively shows 3D occupancy predictions extending 2 seconds into the future.Source: Figure 7 · p. 16

10. Differences between RSSM and MSSM

MSSM is designed to address limitations of RSSM by preserving long-term information, retaining spatial context, and separating dynamic from static scene information.

10. Differences between RSSM and MSSM

  • MSSM models and preserves long-term information with a Dynamics Memory Bank, unlike RSSM’s RNN-based latent modeling.RSSM may encounter challenges retaining long-term information.Source: §10 · p. 16
  • MSSM uses context BEV features to reconstruct 3D scenes, whereas RSSM compresses features into a 1D tensor.Source: §10 · p. 16
  • MSSM separates dynamic and static information so they can be addressed independently.Source: §10 · p. 16

11. Graphical Model

The Memory State-Space Model propagates temporal-aware features through historical memory while retaining spatial-aware information in BEV features.

11. Graphical Model

  • The deterministic state h_t is updated from historical states in the Dynamics Memory Bank to transmit temporal-aware features.Source: §11 · p. 16
  • Spatial-aware features are preserved by retaining the BEV feature ˆb.Source: §11 · p. 16

12. Qualitative Results

DriveWorld reconstructs current and future 3D scenes, qualitatively illustrating its ability to predict future changes after 4D pre-training.

12. Qualitative Results

  • Figure 7 reconstructs both current and future 3D scenes, demonstrating DriveWorld’s qualitative prediction of future changes.The authors connect this capability with improved downstream task performance after 4D pre-training.Source: §12 · p. 16

Related papers

Loading 2405.04390v1…