M$^2$BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation

• 3D object detection predicts BEV-coordinate boxes from monocular cameras, while BEV segmentation predicts environmental semantics such as drivable areas and lane boundaries.Source: §1 · p. 2
• Existing camera-based approaches process views separately, reconstruct voxels through predicted depth, or use 3D queries, creating fusion, supervision, or dense-BEV limitations.Source: §1 · pp. 2–3
• M²BEV transforms multi-view image features into a unified BEV representation so detection and segmentation can share one network with minimal additional computation.Source: §1 · p. 3
• On nuScenes, M²BEV achieves state-of-the-art performance on both 3D object detection and BEV segmentation.Source: §1 · p. 3
• The framework introduces an efficient BEV encoder, dynamic box assignment, BEV centerness re-weighting, and 2D detection pre-training with auxiliary supervision.Source: §1 · p. 3

• Monocular 3D detectors estimate 3D information from 2D images using box regression, cuboid scoring, or predicted dense depth.Source: §2 · p. 4
• Multi-view detectors extend monocular methods by independently processing camera views and fusing predictions during post-processing.Source: §2 · p. 4
• Post-processing introduces burdensome hyperparameters and is typically nondifferentiable, limiting end-to-end training with downstream planning tasks.Source: §2 · p. 4
• BEV segmentation methods transform perspective-view features into BEV representations using MLPs, transformers, or implicit depth distributions.Source: §2 · p. 4

M2BEV Pipeline

• M2BEV projects multi-view 2D features into a unified 3D voxel representation in ego-car coordinates, then reduces its height dimension to form BEV features.The BEV features support both 3D detection and map segmentation heads.Source: §3.1 · pp. 5–6

Efficient 2D→3D Projection

• The projection assumes a uniform depth distribution along each camera ray, copying a pixel's features to all voxels on that ray without learned depth estimation.Compared with LSS, this avoids a categorical depth volume and enables larger backbones and higher-resolution inputs with lower memory use.Source: §3.2 · p. 7

Efficient BEV Encoder

• The S2C BEV encoder reshapes a 4D voxel tensor into a 3D tensor and applies 2D convolutions instead of memory-expensive 3D convolutions.This design is intended to improve speed and memory efficiency while reducing the voxel height dimension.Source: §3.1 · p. 6, §3.3 · p. 7

Improvement Designs

• Dynamic box assignment uses learning-to-match to select positive anchors from classification and localization quality rather than relying only on a fixed IoU threshold.The approach addresses less-accurate geometric information in camera-derived BEV features.Source: §3.1 · p. 6, §3.3 · p. 7
• BEV centerness assigns larger loss weights to predictions farther from the ego vehicle because distant BEV regions correspond to fewer image pixels.The centerness range is 1 to 2, and the reported IoU improvement increases with distance.Source: §3.3 · p. 8

Implementation Details

• M2BEV is evaluated on nuScenes using mAP and NDS for 3D detection, and IoU for BEV segmentation.The dataset provides six camera views per sample; the final comparisons use ResNeXt-101, while ablations use ResNet-50.Source: §4.1 · pp. 9–10

Comparison with state of the art

• 4.8% mAP and 4.2% NDS improve over PGD on the nuScenes validation set.On the test set, M2BEV also exceeds camera-only baselines by more than 1% mAP over DD3D and DETR3D.Source: §4.2 · p. 11
• 3.0% and 18.1% IoU gains over LSS are reported for drivable area and lane boundary segmentation, respectively.The comparison indicates that depth estimation is not necessary for the reported BEV segmentation task.Source: §4.2 · p. 11

Ablation Studies

• 7.8% mAP and 4.8% NDS improve when fixed IoU matching is replaced by dynamic anchor matching in the 3D detection ablation.NuImage pre-training adds 5.8% mAP and 6.0% NDS, S2C improves detection by more than 1%, and auxiliary supervision slightly improves both metrics without inference cost.Source: §4.3 · p. 11
• 50% of the nuScenes data with nuImage pre-training achieves similar mAP and NDS to 100% with ImageNet pre-training.The experiment reports faster convergence and suggests that cheaper 2D labels can reduce reliance on 3D annotation.Source: Figure 6 · p. 11, §4.3 · p. 12
• 6.2% and 6.0% segmentation IoU gains follow nuImage pre-training, while S2C adds 10.6% and 12.2% for drivable area and lanes.BEV centerness further improves results by increasing performance on far-away objects, and S2C permits deeper 2D-convolution refinement than fewer 3D convolutions.Source: §4.3 · pp. 12–13
• Joint training slightly hurts both detection and segmentation, although shared multi-task inference retains nearly the same speed as single-task inference.The authors relate this to weak correlation between object and map-location distributions and leave the challenge for future work.Source: §4.3 · pp. 13–14

Implementation Details

• 4× faster inference is reported for M2BEV than the naively combined FCOS3D+LSS system.The tasks share most features and use lightweight heads, so single-task and multi-task inference speeds are nearly the same.Source: §4.3 · p. 14

• M2BEV projects multi-view image features into BEV space, where detection and segmentation branches perform both tasks in one network.The unified representation supports both tasks with shared processing.Source: §5 · p. 14
• The framework is presented as potentially extensible to 3D tracking, motion prediction, and trajectory forecasting.These future tasks involve temporal information.Source: §5 · p. 14

• The backbone adds deformable convolutions at specified stages, while synchronized batch normalization is used in the backbone and pyramid features.ResNet-50 uses DCN in stages 3–4; ResNeXt-101 uses it in stages 2–4.Source: A Additional Implementation Details · p. 15
• Training uses 1000 warm-up iterations from learning rate 1e-6 to 1e-3 and mixed-precision training to reduce GPU cost and accelerate training.Training runs on 3 DGX nodes with 24 GPUs.Source: A Additional Implementation Details · p. 15
• The detection head applies 3D rotated NMS with threshold 0.2, box score threshold 0.05, and a maximum of 500 objects per frame.Anchors use four sizes and two rotations at each feature-map point.Source: A Additional Implementation Details · p. 15
• Six-view images use fixed mean and standard-deviation normalization without color jitter or random rescaling.The normalization follows the stated common setting.Source: A Additional Implementation Details · p. 15

• The visualizations include six groups of ground-truth and predicted results across Figures 9–11.These figures provide qualitative examples for the joint tasks.Source: B More Visualizations · p. 15
• In a night scene, M2BEV detects a tiny distant car that human annotators missed in the ground truth.The example is presented as evidence that the model can see in the dark.Source: B More Visualizations · p. 15, Figure 9 · p. 16
• M2BEV correctly localizes buses appearing between two cameras, illustrating the benefit of unified BEV representation for cross-camera objects.Figure 10 focuses on objects that require handling across camera views.Source: B More Visualizations · p. 15, Figure 10 · p. 17
• In a crowded scene, M2BEV detects most objects and segments maps despite heavy occlusion.Figure 11 presents the crowded-scene example.Source: B More Visualizations · p. 15, Figure 11 · p. 18