Source-linked AI summary
BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving
Yunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, Jiwen Lu
TL;DR
BEVerse addresses the lack of a multitask paradigm for vision-centric autonomous driving, where prior systems commonly separate perception, prediction, and planning. It builds shared spatiotemporal BEV representations from multi-camera videos, aligns past features, and jointly decodes detection, semantic maps, and motion, using grid samplers and iterative flow. On nuScenes, the multitask framework achieves state-of-the-art results across the three tasks and is more efficient than the sequential paradigm.
Problem
Vision-centric autonomous-driving research had not previously unified perception and prediction, while sequential pipelines incur error propagation and repeated feature extraction.
Method
BEVerse extracts and lifts multi-timestamp, multi-view images into aligned spatiotemporal BEV features, then uses task-specific grid samplers and iterative flow for joint decoding.
Results
BEVerse achieves state-of-the-art performance on 3D object detection, semantic map construction, and motion prediction on nuScenes, while being more efficient than the sequential paradigm.
Takeaways & Limitations
Temporal information improves 3D object detection and semantic map construction, while joint perception-task learning can implicitly improve motion prediction.
Abstract
from arXiv · showhide
In this paper, we present BEVerse, a unified framework for 3D perception and prediction based on multi-camera systems. Unlike existing studies focusing on the improvement of single-task approaches, BEVerse features in producing spatio-temporal Birds-Eye-View (BEV) representations from multi-camera videos and jointly reasoning about multiple tasks for vision-centric autonomous driving. Specifically, BEVerse first performs shared feature extraction and lifting to generate 4D BEV representations from multi-timestamp and multi-view images. After the ego-motion alignment, the spatio-temporal encoder is utilized for further feature extraction in BEV. Finally, multiple task decoders are attached for joint reasoning and prediction. Within the decoders, we propose the grid sampler to generate BEV features with different ranges and granularities for different tasks. Also, we design the method of iterative flow for memory-efficient future prediction. We show that the temporal information improves 3D object detection and semantic map construction, while the multi-task learning can implicitly benefit motion prediction. With extensive experiments on the nuScenes dataset, we show that the multi-task BEVerse outperforms existing single-task methods on 3D object detection, semantic map construction, and motion prediction. Compared with the sequential paradigm, BEVerse also favors in significantly improved efficiency. The code and trained models will be released at https://github.com/zhangyp15/BEVerse.
1. Introduction
Autonomous-driving systems traditionally process perception, prediction, and planning sequentially, but this causes error propagation and repeated computation. BEVerse instead jointly performs vision-centric perception and prediction in a shared BEV framework.
- Motivation: Self-driving systems divide the problem into perception, prediction, and planning&control, with perception covering dynamic objects and static streets.Prediction estimates future obstacle movements, while planning and control determine and execute driving behavior.
- Limitations of Sequential Systems: The sequential paradigm feeds each subtask's output into the next, enabling task-specific research but exposing downstream tasks to propagated errors.Its sequential structure also introduces repeated feature extraction and feature propagation.
- Research Gap: Vision-centric autonomous driving uses multiple surrounding cameras as a cost-effective alternative to LiDAR, yet its multitask paradigm had not previously been discussed.Prior joint perception-and-prediction studies focused on LiDAR-centric systems, while camera methods were mainly studied for separate perception or prediction tasks.
- BEVerse: BEVerse jointly reasons about 3D object detection, semantic map construction, and motion prediction from consecutive multi-camera frames in a 4D BEV representation.It extracts multi-frame, multi-view features, transforms them into BEV, aligns past features using ego-motion, and applies a spatiotemporal BEV encoder before task decoding.
- Technical Contributions: The framework introduces grid samplers for task-specific BEV ranges and granularities and iterative flow for efficient future prediction and multitask learning.Its shared feature extraction and parallel multitask inference target a better performance–efficiency trade-off than sequential processing.
2. Related Work
Related work spans camera-based 3D detection, semantic map construction, BEV motion prediction, and multitask learning. BEVerse builds on these directions by unifying multi-camera BEV perception and prediction while addressing future-prediction memory consumption.
- 3D Object Detection: Monocular 3D detection methods such as FCOS3D and PGD process views separately and merge outputs with heuristic post-processing.These methods improve 3D attributes and depth estimation but do not fully exploit joint multi-view BEV representations.
- Semantic Map Construction: HD maps provide fine-grained road information, but manual annotation and LiDAR-based repeated scanning require costly data collection and long-term iteration.Online semantic maps from onboard sensors are presented as an affordable alternative.
- Unified BEV Framework: BEVerse processes consecutive surrounding-camera frames through view transformation, ego-motion alignment, temporal modeling, and multiple task decoders.This framework provides the shared spatiotemporal BEV representation used for joint perception and prediction.
- Motion Prediction: Camera-based motion-prediction methods include future visual prediction, joint perspective segmentation and optical flow, and BEV prediction from surrounding-camera videos.Some frameworks depend on detection results or HD maps, whereas FIERY predicts motion directly in BEV from camera videos.
- Multitask Learning: Multitask learning jointly solves related tasks through shared network structures and mutual promotion, with prior autonomous-driving work concentrating on LiDAR-centric systems.BEVerse extends this multitask direction to vision-centric multi-camera driving.
3. Approach
BEVerse processes multi-camera observations across timestamps into aligned spatio-temporal BEV features, then applies parallel task decoders for perception and future prediction. Its design includes task-specific BEV sampling and an iterative-flow predictor alongside ablations of temporal information and prediction approaches.
- BEVerse takes surrounding camera images from multiple timestamps, ego-motions, and camera parameters to produce present-frame perception outputs and future motion outputs.
- The architecture sequentially applies an image-view encoder, view transformer, spatio-temporal BEV encoder, and multi-task decoders.
- Image-view Encoder: The shared image-view encoder extracts multi-scale features across cameras and timestamps before view transformation.
- View Transformer: The view transformer predicts categorical depth distributions, lifts image features into 3D points, and uses pillar pooling to construct BEV features.
- Spatio-temporal BEV Encoder: Aligned features from past timestamps are processed by a spatio-temporal BEV encoder to extract spatial and temporal information for the task decoders.
- Task Decoders: Independent parallel task decoders use grid sampling and task-specific heads for detection, semantic maps, and motion prediction, including an iterative-flow future predictor.The iterative flow predicts a latent map for object-separated uncertainty and generates each next state by warping the current state with predicted flows.
4. Experiments
Experiments evaluate BEVerse on nuScenes across perception, prediction, ablations, efficiency, and qualitative outputs. The results show strong performance across tasks, with temporal information and joint reasoning affecting task performance differently.
- Ablation Studies: Past frames improve 3D detection mAP by 2.5 points, possibly because temporal clues reveal occluded objects.
- Ablation Studies: Joint reasoning improves motion prediction, while shared capacity reduces performance for 3D detection and semantic map construction relative to single-task models.
- Main Results: BEVerse-Small achieves 51.7 mIoU for semantic map construction, 7.1 points above the previous best method.
- Main Results: BEVerse-Tiny achieves 38.7 IoU and 33.3 VPQ for long-range motion prediction, outperforming FIERY.
- Qualitative Results: BEVerse-Small qualitatively produces accurate boxes, semantic maps, and future trajectories, but depth errors remain for faraway objects and traffic lines.
5. Conclusion
BEVerse unifies multi-camera 4D BEV representation learning with joint 3D detection, semantic map construction, and motion prediction. On nuScenes, it achieves state-of-the-art performance across the three tasks, improves efficiency over sequential processing, and shows that joint perception learning can benefit motion prediction.
- BEVerse generates 4D BEV representations from multi-camera videos and jointly reasons about 3D detection, semantic maps, and motion prediction.
- The multitask BEVerse achieves state-of-the-art performance on 3D object detection, semantic map construction, and motion prediction.
- BEVerse is more efficient than the sequential paradigm, while joint perception learning can implicitly improve motion prediction.