Source-linked AI summary
M$^2$BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation
Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, Jose M. Alvarez
TL;DR
Camera-based autonomous-vehicle perception must jointly understand 3D objects and the environment despite limited depth information. M²BEV uses a shared BEV representation for detection and segmentation, achieving state-of-the-art results on nuScenes while improving efficiency.
Problem
Camera-based AV perception must support both 3D object detection and BEV segmentation, but these tasks are commonly handled separately and depth estimation is ill-posed.
Method
M²BEV projects multi-view image features into a unified BEV representation and applies shared processing with efficient encoding, dynamic assignment, centerness re-weighting, and 2D auxiliary supervision.
Results
M²BEV achieves state-of-the-art performance on both 3D object detection and BEV segmentation on nuScenes.
Takeaways & Limitations
The unified BEV representation supports multi-task inference with little extra computational cost and 2D data can improve label efficiency for 3D tasks.
Takeaways & Limitations
Complicated road conditions produce failures, performance remains below LiDAR-based methods, and severe camera extrinsic noise degrades prediction quality.
Abstract
from arXiv · showhide
In this paper, we propose M$^2$BEV, a unified framework that jointly performs 3D object detection and map segmentation in the Birds Eye View~(BEV) space with multi-camera image inputs. Unlike the majority of previous works which separately process detection and segmentation, M$^2$BEV infers both tasks with a unified model and improves efficiency. M$^2$BEV efficiently transforms multi-view 2D image features into the 3D BEV feature in ego-car coordinates. Such BEV representation is important as it enables different tasks to share a single encoder. Our framework further contains four important designs that benefit both accuracy and efficiency: (1) An efficient BEV encoder design that reduces the spatial dimension of a voxel feature map. (2) A dynamic box assignment strategy that uses learning-to-match to assign ground-truth 3D boxes with anchors. (3) A BEV centerness re-weighting that reinforces with larger weights for more distant predictions, and (4) Large-scale 2D detection pre-training and auxiliary supervision. We show that these designs significantly benefit the ill-posed camera-based 3D perception tasks where depth information is missing. M$^2$BEV is memory efficient, allowing significantly higher resolution images as input, with faster inference speed. Experiments on nuScenes show that M$^2$BEV achieves state-of-the-art results in both 3D object detection and BEV segmentation, with the best single model achieving 42.5 mAP and 57.0 mIoU in these two tasks, respectively.
1 Introduction
M²BEV addresses multi-camera autonomous-vehicle perception by unifying 3D detection and BEV segmentation through a shared BEV representation. Its designs target accuracy and GPU memory efficiency for camera-based perception.
- 3D object detection predicts BEV-coordinate boxes from monocular cameras, while BEV segmentation predicts environmental semantics such as drivable areas and lane boundaries.
- Existing camera-based approaches process views separately, reconstruct voxels through predicted depth, or use 3D queries, creating fusion, supervision, or dense-BEV limitations.
- M²BEV transforms multi-view image features into a unified BEV representation so detection and segmentation can share one network with minimal additional computation.
- On nuScenes, M²BEV achieves state-of-the-art performance on both 3D object detection and BEV segmentation.
- The framework introduces an efficient BEV encoder, dynamic box assignment, BEV centerness re-weighting, and 2D detection pre-training with auxiliary supervision.
2 Related Work
Related work develops monocular and multi-view 3D detection alongside BEV segmentation, but existing methods use separate-view fusion, depth estimation, or architectures without dense BEV features.
- Monocular 3D detectors estimate 3D information from 2D images using box regression, cuboid scoring, or predicted dense depth.
- Multi-view detectors extend monocular methods by independently processing camera views and fusing predictions during post-processing.
- Post-processing introduces burdensome hyperparameters and is typically nondifferentiable, limiting end-to-end training with downstream planning tasks.
- BEV segmentation methods transform perspective-view features into BEV representations using MLPs, transformers, or implicit depth distributions.
3 Method
M2BEV transforms multi-view image features into a unified BEV representation for joint 3D detection and map segmentation, with designs targeting efficiency and camera-based perception accuracy.
- M2BEV Pipeline: M2BEV projects multi-view 2D features into a unified 3D voxel representation in ego-car coordinates, then reduces its height dimension to form BEV features.The BEV features support both 3D detection and map segmentation heads.
- Efficient 2D→3D Projection: The projection assumes a uniform depth distribution along each camera ray, copying a pixel's features to all voxels on that ray without learned depth estimation.Compared with LSS, this avoids a categorical depth volume and enables larger backbones and higher-resolution inputs with lower memory use.
- Efficient BEV Encoder: The S2C BEV encoder reshapes a 4D voxel tensor into a 3D tensor and applies 2D convolutions instead of memory-expensive 3D convolutions.This design is intended to improve speed and memory efficiency while reducing the voxel height dimension.
- Improvement Designs: Dynamic box assignment uses learning-to-match to select positive anchors from classification and localization quality rather than relying only on a fixed IoU threshold.The approach addresses less-accurate geometric information in camera-derived BEV features.
- Improvement Designs: BEV centerness assigns larger loss weights to predictions farther from the ego vehicle because distant BEV regions correspond to fewer image pixels.The centerness range is 1 to 2, and the reported IoU improvement increases with distance.
4 Experiments
Experiments on nuScenes evaluate M2BEV’s accuracy, efficiency, component ablations, and limitations. The unified framework achieves competitive results across 3D detection and BEV segmentation while reducing memory and inference costs.
- Implementation Details: M2BEV is evaluated on nuScenes using mAP and NDS for 3D detection, and IoU for BEV segmentation.The dataset provides six camera views per sample; the final comparisons use ResNeXt-101, while ablations use ResNet-50.
- Comparison with state of the art: 4.8% mAP and 4.2% NDS improve over PGD on the nuScenes validation set.On the test set, M2BEV also exceeds camera-only baselines by more than 1% mAP over DD3D and DETR3D.
- Comparison with state of the art: 3.0% and 18.1% IoU gains over LSS are reported for drivable area and lane boundary segmentation, respectively.The comparison indicates that depth estimation is not necessary for the reported BEV segmentation task.
- Ablation Studies: 7.8% mAP and 4.8% NDS improve when fixed IoU matching is replaced by dynamic anchor matching in the 3D detection ablation.NuImage pre-training adds 5.8% mAP and 6.0% NDS, S2C improves detection by more than 1%, and auxiliary supervision slightly improves both metrics without inference cost.
- Ablation Studies: 50% of the nuScenes data with nuImage pre-training achieves similar mAP and NDS to 100% with ImageNet pre-training.The experiment reports faster convergence and suggests that cheaper 2D labels can reduce reliance on 3D annotation.
- Ablation Studies: 6.2% and 6.0% segmentation IoU gains follow nuImage pre-training, while S2C adds 10.6% and 12.2% for drivable area and lanes.BEV centerness further improves results by increasing performance on far-away objects, and S2C permits deeper 2D-convolution refinement than fewer 3D convolutions.
- Ablation Studies: Joint training slightly hurts both detection and segmentation, although shared multi-task inference retains nearly the same speed as single-task inference.The authors relate this to weak correlation between object and map-location distributions and leave the challenge for future work.
- Implementation Details: 4× faster inference is reported for M2BEV than the naively combined FCOS3D+LSS system.The tasks share most features and use lightweight heads, so single-task and multi-task inference speeds are nearly the same.
5 Conclusion
The framework performs 3D object detection and map segmentation in one network using a unified BEV representation. The authors identify temporal tasks as future extensions.
- M2BEV projects multi-view image features into BEV space, where detection and segmentation branches perform both tasks in one network.The unified representation supports both tasks with shared processing.
- The framework is presented as potentially extensible to 3D tracking, motion prediction, and trajectory forecasting.These future tasks involve temporal information.
A Additional Implementation Details
Implementation uses deformable convolutions and synchronized batch normalization in the backbone, warm-up and mixed precision during training, specified detection-head settings, and six-view image normalization without additional augmentations.
- The backbone adds deformable convolutions at specified stages, while synchronized batch normalization is used in the backbone and pyramid features.ResNet-50 uses DCN in stages 3–4; ResNeXt-101 uses it in stages 2–4.
- Training uses 1000 warm-up iterations from learning rate 1e-6 to 1e-3 and mixed-precision training to reduce GPU cost and accelerate training.Training runs on 3 DGX nodes with 24 GPUs.
- The detection head applies 3D rotated NMS with threshold 0.2, box score threshold 0.05, and a maximum of 500 objects per frame.Anchors use four sizes and two rotations at each feature-map point.
- Six-view images use fixed mean and standard-deviation normalization without color jitter or random rescaling.The normalization follows the stated common setting.
B More Visualizations
The visualizations show joint detection and map segmentation in night, cross-camera, and crowded scenes. They highlight detections of tiny or truncated objects and robustness under occlusion.
- The visualizations include six groups of ground-truth and predicted results across Figures 9–11.These figures provide qualitative examples for the joint tasks.
- In a night scene, M2BEV detects a tiny distant car that human annotators missed in the ground truth.The example is presented as evidence that the model can see in the dark.
- M2BEV correctly localizes buses appearing between two cameras, illustrating the benefit of unified BEV representation for cross-camera objects.Figure 10 focuses on objects that require handling across camera views.
- In a crowded scene, M2BEV detects most objects and segments maps despite heavy occlusion.Figure 11 presents the crowded-scene example.