Source-linked AI summary
OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, Jiwen Lu
TL;DR
Existing autonomous-driving methods often require difficult stage-wise annotations and focus on object-box motion, leaving finer 3D scene evolution insufficiently modeled. OccWorld learns a 3D occupancy world model with discrete scene tokens and spatial-temporal autoregressive generation to forecast scenes and ego trajectories. On nuScenes, it reports effective occupancy forecasting and competitive planning without instance and map supervision, while longer-horizon planning degrades.
Problem
Existing methods require difficult stage-wise labels and usually predict object-box motion, which fails to capture fine-grained 3D scene evolution.
Method
OccWorld tokenizes 3D occupancy with a reconstruction-based VQVAE and uses a GPT-like spatial-temporal transformer to generate future scene and ego tokens.
Results
26.63 average IoU, 17.13 mIoU, and 1.16 L2 error are reported for 3s forecasting from 2s history without instance or map annotations.
Takeaways & Limitations
OccWorld models joint ego and surrounding-scene evolution in comprehensive 3D occupancy and shows potential for interpretable end-to-end autonomous driving without additional supervision signals.
Takeaways & Limitations
Planning is strong at 1s but worsens at longer horizons, reaching 1.99 L2 error at 3s versus 1.65 for UniAD; collision rate also slightly trails because freespace or box guidance is absent.
Abstract
from arXiv · showhide
Understanding how the 3D scene evolves is vital for making decisions in autonomous driving. Most existing methods achieve this by predicting the movements of object boxes, which cannot capture more fine-grained scene information. In this paper, we explore a new framework of learning a world model, OccWorld, in the 3D Occupancy space to simultaneously predict the movement of the ego car and the evolution of the surrounding scenes. We propose to learn a world model based on 3D occupancy rather than 3D bounding boxes and segmentation maps for three reasons: 1) expressiveness. 3D occupancy can describe the more fine-grained 3D structure of the scene; 2) efficiency. 3D occupancy is more economical to obtain (e.g., from sparse LiDAR points). 3) versatility. 3D occupancy can adapt to both vision and LiDAR. To facilitate the modeling of the world evolution, we learn a reconstruction-based scene tokenizer on the 3D occupancy to obtain discrete scene tokens to describe the surrounding scenes. We then adopt a GPT-like spatial-temporal generative transformer to generate subsequent scene and ego tokens to decode the future occupancy and ego trajectory. Extensive experiments on the widely used nuScenes benchmark demonstrate the ability of OccWorld to effectively model the evolution of the driving scenes. OccWorld also produces competitive planning results without using instance and map supervision. Code: https://github.com/wzzheng/OccWorld.
1. Introduction
Existing autonomous-driving pipelines require difficult stage-wise annotations and mainly model object-box motion, limiting their representation of fine-grained scene evolution. OccWorld instead models 3D occupancy to jointly forecast surrounding scenes and ego trajectories, achieving occupancy-forecasting and planning results without instance or map annotations.
- Conventional perception-prediction-planning pipelines require ground-truth labels at each stage, while instance boxes and high-definition maps are difficult to annotate.
- Object-box motion prediction misses fine-grained information about the evolving 3D scene.
- OccWorld uses 3D semantic occupancy to jointly model surrounding-scene evolution and the self-driving vehicle’s future trajectory.Occupancy describes finer 3D structure and can be learned from sparse LiDAR points.
- OccWorld tokenizes occupancy with a self-supervised VQVAE and uses a spatial-temporal generative transformer to predict future scene and ego tokens.
- 26.63 average IoU, 17.13 mIoU, and 1.16 L2 error are reported for 3s forecasting from 2s history without instance or map annotations.
2. Related Work
Related work covers 3D occupancy prediction, world models, and end-to-end autonomous driving, while OccWorld instantiates these ideas with a GPT-like occupancy forecasting and planning framework. Its framework combines discrete scene tokens with spatial-temporal autoregressive generation.
- 3D Occupancy Prediction: 3D occupancy prediction assigns occupancy and semantic labels to voxels, describing both dynamic and static elements of surrounding scenes.Recent work explores both vision-based and LiDAR-based occupancy prediction.
- World Models for Autonomous Driving: World models generate subsequent scene observations from actions and past observations, with recent driving methods using deep generative models.
- End-to-End Autonomous Driving: End-to-end autonomous-driving methods aim to produce ego-car planning outputs from sensor inputs, often through perception, prediction, and planning stages.
- OccWorld adapts a GPT-like architecture to predict the next occupancy scene autoregressively from previous scenes.
- Its key designs are a 3D occupancy scene tokenizer for discrete high-level representations and spatial mixing around spatial-wise temporal causal attention.
3. Proposed Approach
OccWorld replaces the conventional staged pipeline with a 3D-occupancy world model that jointly predicts scene evolution and ego movement. It tokenizes occupancy into discrete representations and uses a spatial-temporal transformer to forecast future scenes and trajectories.
- The conventional perception-prediction-planning pipeline requires stage-wise ground-truth labels and models only object-level movement, missing finer scene evolution.
- OccWorld uses 3D occupancy as its scene representation because it captures structural and semantic information while supporting efficient learning and both vision and LiDAR inputs.
- A self-supervised VQ-VAE converts low-level 3D occupancy into compact discrete scene tokens that encode higher-level concepts for modeling scene evolution.
- OccWorld jointly models surrounding-scene and ego evolution, and can operate with oracle, image-based, or LiDAR-based occupancy representations.
- The world model combines past scene tokens with an ego token representing vehicle position and autoregressively predicts subsequent scene and ego states.
- The spatial-temporal generative transformer models token dependencies within each scene and across time using spatial mixing, temporal causal attention, and multi-scale aggregation.
4. Experiments
OccWorld is evaluated on 4D occupancy forecasting and motion planning across multiple input-supervision settings. Results show strong occupancy forecasting and competitive planning without instance or map supervision, with performance depending on forecast horizon and tokenizer or transformer design.
- OccWorld evaluates 4D occupancy forecasting on Occ3D and motion planning on nuScenes, using IoU, mIoU, L2 error, and collision rate.
- OccWorld generates non-trivial future occupancy in end-to-end settings, including the self-supervised OccWorld-S model trained without 3D occupancy information.
- OccWorld demonstrates competitive planning using 3D occupancy, which can be efficiently obtained by accumulating LiDAR scans, while UniAD achieves the best overall performance with auxiliary supervision.
- Motion planning: OccWorld-O outperforms OccNet by a large margin without map or bounding-box supervision, and its collision rate is comparable to OccNet despite slightly worse collision performance overall.
- Longer-horizon planning degrades: OccWorld-O reaches 1.99 L2 error at 3 seconds versus 1.65 for UniAD, despite achieving the best 1-second L2 error.
- Scene tokenizer: A codebook larger than 512 overfits, while larger spatial resolution improves reconstruction but harms forecasting and planning because tokens become difficult to forecast.
- Spatial-temporal generative transformer: Spatial aggregation and temporal attention are vital for occupancy forecasting and planning, while jointly predicting ego motion improves performance over occupancy forecasting alone.
5. Conclusion
The paper presents OccWorld, a 3D occupancy world model that jointly models ego movements and surrounding-scene evolution. A scene tokenizer and spatial-temporal generative transformer enable autoregressive future prediction in comprehensive 3D semantic occupancy space.
- OccWorld jointly models ego movements and surrounding-scene evolution with a 3D occupancy world model.
- A 3D occupancy scene tokenizer extracts high-level concepts, while a spatial-temporal generative transformer predicts future tokens autoregressively.
- Quantitative and visualization results show effective prediction of future scene evolution in comprehensive 3D semantic occupancy space.
- The paper positions OccWorld as a path toward interpretable end-to-end autonomous driving without additional supervision signals.