Source-linked AI summary
BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection
Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, Zeming Li
TL;DR
Camera-based 3D detection depends on depth, yet existing Lift-splat depth estimates can be surprisingly inadequate. BEVDepth uses explicit point-cloud depth supervision, camera-aware prediction, and depth refinement, reaching 60.9% NDS on the nuScenes test set while maintaining high efficiency.
Problem
Existing Lift-splat depth estimates are surprisingly poor despite depth being essential for camera-based 3D detection.
Method
BEVDepth combines explicit supervision from point-cloud-derived depth with camera-aware depth prediction and a Depth Refinement Module.
Results
60.9% NDS was achieved on the nuScenes test set with customized Efficient Voxel Pooling and Multi-frame Fusion while maintaining high efficiency.
Takeaways & Limitations
BEVDepth generates robust, trustworthy depth predictions and establishes a new state of the art on the nuScenes camera 3D detection leaderboard.
Abstract
from arXiv · showhide
In this research, we propose a new 3D object detector with a trustworthy depth estimation, dubbed BEVDepth, for camera-based Bird's-Eye-View (BEV) 3D object detection. Our work is based on a key observation -- depth estimation in recent approaches is surprisingly inadequate given the fact that depth is essential to camera 3D detection. Our BEVDepth resolves this by leveraging explicit depth supervision. A camera-awareness depth estimation module is also introduced to facilitate the depth predicting capability. Besides, we design a novel Depth Refinement Module to counter the side effects carried by imprecise feature unprojection. Aided by customized Efficient Voxel Pooling and multi-frame mechanism, BEVDepth achieves the new state-of-the-art 60.9% NDS on the challenging nuScenes test set while maintaining high efficiency. For the first time, the NDS score of a camera model reaches 60%.
1 Introduction
Camera-based 3D detection offers lower-cost perception, but Lift-splat methods rely on depth estimates that are often inadequate. BEVDepth addresses this with explicit depth supervision and additional depth-aware modules, achieving strong nuScenes performance.
- Lower cost makes multi-view cameras attractive for autonomous-system 3D perception despite LiDAR’s demonstrated trustworthy detection.
- Lift-splat first lifts image features into 3D frustums using estimated depth, then splats them into a Bird’s-Eye-View representation for downstream tasks.
- Lift-splat depth is surprisingly poor: only a few regions predict reasonable depth, despite achieving 30 mAP on nuScenes.
- Indirect detection supervision causes inaccurate depth, depth-module over-fitting, and imprecise BEV semantics when features are unprojected into incorrect positions.
- Replacing learned depth with point-cloud-derived ground truth boosts mAP and NDS by nearly 20% and reduces mATE from 0.768 to 0.393.
- BEVDepth uses point-cloud depth supervision, camera-aware depth prediction, and Depth Refinement, reaching 60.9% NDS on nuScenes test with Efficient Voxel Pooling and Multi-frame Fusion.
2 Related Work
Vision-based 3D detection predicts 3D boxes from images, an ill-posed task because monocular depth is ambiguous and non-overlapping views remain challenging. Existing work spans image-domain prediction, 3D-space transformations, LiDAR-inspired representations, monocular depth, and cost-volume methods.
- Vision-based 3D object detection: Vision-based 3D detection aims to predict objects’ 3D bounding boxes from visual inputs.
- Vision-based 3D object detection: Monocular depth ambiguity makes vision-based 3D detection ill-posed, while areas without overlapping camera views remain challenging even with multiple cameras.
- Image-domain approaches predict 3D boxes or attributes directly from 2D features, using depth-aware convolutions, depth maps, geometric graphs, or depth pretraining.
- 3D-space approaches transform image features into pseudo-LiDAR, 3D voxels, orthographic maps, or other representations.
- LiDAR-based methods voxelize point clouds into dense representations and use structures such as sparse convolution for 3D detection.
- Depth-estimation research includes supervised regression, unsupervised disparity-and-reconstruction methods, pose-assisted prediction, and multi-view cost volumes.
3 Delving into Depth Prediction in Lift-splat
The Lift-splat baseline estimates depth indirectly for BEV construction, yet its depth can be poor, overfit, and produce imprecise BEV semantics. Experiments show that explicit depth supervision substantially improves depth quality and downstream classification, motivating BEVDepth.
- 3.1 Model Architecture for Base Detector: The baseline extracts multi-view image features, predicts depth distributions, unprojects features into 3D, pools them into BEV, and predicts 3D boxes.Its four components are an image encoder, DepthNet, view transformer, and 3D detection head.
- 3.2 Analysis of Lift-splat: Replacing predicted depth with a frozen random tensor reduces mAP only from 28.2% to 24.5%, exposing limited reliance on accurate depth.
- 3.3 Making Lift-splat work well is hard: 3.03 AbsRel for the Base Detector falls to 0.23 with the Enhanced Detector when evaluating all foreground regions on nuScenes val.The Enhanced Detector adds explicit depth supervision derived from point-cloud data.
- 3.3 Making Lift-splat work well is hard: Training depth indirectly through detection loss leaves most pixels poorly trained, raising concerns about generalization to image-size and camera-parameter changes.Figure 2 evaluates robustness to image sizes using mAP on nuScenes, with 256 × 704 used for training.
- 3.3 Making Lift-splat work well is hard: Poor depth causes incomplete feature unprojection, so voxel or pillar pooling aggregates only part of the available semantic information.Compared with the Base Detector, the Enhanced Detector retains more structure information during unprojection and provides more precise semantics.
- 3.3 Making Lift-splat work well is hard: The Enhanced Detector consistently outperforms the Base Detector across positive heatmap thresholds, supporting the benefit of better depth for classification.Classification is evaluated using CenterPoint heatmaps and TP / (TP + FN).
4 BEVDepth
BEVDepth improves camera-based 3D detection by supervising depth explicitly, modeling camera parameters in DepthNet, and refining unprojected frustum features. Its pipeline projects point-cloud data into image coordinates for depth targets before pooling refined features into BEV.
- Explicit Depth Supervision: BEVDepth uses explicit depth supervision to train intermediate depth predictions with ground-truth depth derived from point-cloud data.The base detector otherwise receives depth supervision only indirectly through the detection loss.
- BEVDepth Framework: The framework extracts multi-view image features, predicts context and depth, forms point features, and pools them into a unified BEV feature map.Voxel Pooling unifies point features in one coordinate system before BEV aggregation.
- Explicit Depth Supervision: Point clouds are projected into 2.5D image coordinates, points outside each camera view are discarded, and min pooling plus one-hot encoding align targets with predicted depth.The resulting ground-truth depth is used to compute a binary cross-entropy depth loss.
- Camera-aware Depth Prediction: Camera-aware DepthNet incorporates camera intrinsics and extrinsics so depth prediction accounts for differing camera properties and spatial locations.Intrinsics are expanded with an MLP and used to re-weight image features, while extrinsics are concatenated with intrinsics.
- Depth Refinement Module: The Depth Refinement Module reshapes unprojected frustum features, applies 3×3 convolutions along the depth plane, and feeds the refined features to voxel or pillar pooling.It is designed to aggregate features along the depth axis when depth confidence is low.
5 Experiment
Experiments on nuScenes evaluate BEVDepth’s components, efficiency techniques, and comparisons with leading camera-based 3D detectors. Explicit depth supervision, camera awareness, depth refinement, efficient voxel pooling, and multi-frame fusion collectively support strong validation and test-set results.
- Component Analysis: 4.0% mAP and 4.0% NDS improvements over baseline result from the combined proposed innovations.Depth Loss improves mAP by 2.2%, camera parameter modeling reduces mATE by 0.41, and Depth Refinement adds 0.8% mAP.
- Efficient Voxel Pooling and Multi-frame Fusion: 80× faster pooling and training reduced from 5 days to 1.5 days demonstrate the efficiency of customized Efficient Voxel Pooling.The method assigns each frustum feature a CUDA thread to add it to its corresponding BEV grid.
- nuScenes Validation Results: 2% mAP and 2% NDS gains over BEVDET4D-Base are reported despite its usually stronger backbone.
- nuScenes Test Results: 60.9% NDS is achieved on the nuScenes test set with ConvNeXT without extra data.The submitted single model reports 50.3% mAP and 60.0% NDS, while ConvNeXT raises NDS to 60.9%.
6 Conclusion
BEVDepth addresses unreliable depth in multi-view 3D detection with explicit supervision, camera-aware prediction, and depth refinement. Combined with multi-frame fusion and efficient voxel pooling, it achieves state-of-the-art nuScenes performance and is proposed as a strong baseline.
- BEVDepth introduces explicit depth supervision, camera-aware depth prediction, and depth refinement to generate robust depth for multi-view 3D detection.
- 60.9% NDS on the nuScenes leaderboard establishes BEVDepth as a new state-of-the-art camera-based detector.