Source-linked AI summary
DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
Yue Wang, Vitor Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, Justin Solomon
TL;DR
Camera-based 3D detection must recover 3D boxes from incomplete 2D image information. DETR3D uses sparse 3D queries and geometric back-projection to retrieve multi-view features, avoiding dense depth prediction and post-processing while remaining competitive on nuScenes.
Problem
Camera-based 3D detection is challenging because RGB images lack the 3D structure available from LiDAR.
Method
DETR3D uses sparse 3D object queries, projects their reference points into multiple cameras, retrieves 2D features, and predicts boxes with set-to-set loss.
Results
DETR3D is comparable with prior art on nuScenes without NMS and significantly outperforms other methods in camera-overlap regions.
Takeaways & Limitations
The approach eliminates intermediate depth maps or point clouds and uses information from multiple cameras through backward projection.
Takeaways & Limitations
The method still exhibits substantial translation error because depth estimation remains challenging, and single-point projection limits the receptive field.
Abstract
from arXiv · showhide
We introduce a framework for multi-camera 3D object detection. In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directly in 3D space. Our architecture extracts 2D features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices. Finally, our model makes a bounding box prediction per object query, using a set-to-set loss to measure the discrepancy between the ground-truth and the prediction. This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model. Moreover, our method does not require post-processing such as non-maximum suppression, dramatically improving inference speed. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.
1 Introduction
DETR3D addresses camera-based 3D detection by linking multi-view 2D features to 3D predictions through geometric back-projection and sparse object queries. It avoids dense depth prediction and post-processing while remaining competitive on nuScenes.
- Camera-based 3D detection is ill-posed because RGB images lack the explicit 3D structure available from LiDAR.
- Existing 2D-based pipelines predict 3D properties from images independently, then require cross-camera fusion and redundant-box removal.
- DETR3D links 2D feature extraction and 3D object prediction through geometric back-projection using camera transformation matrices.
- The model uses sparse object priors, projects decoded 3D reference points into each camera, retrieves image features, and predicts boxes with a set-to-set loss.
- DETR3D avoids explicit depth prediction and post-processing such as NMS, while achieving nuScenes performance comparable to prior methods that use NMS.
2 Related Work
Prior work spans proposal-based, set-based, and monocular 3D detection, while DETR3D uses sparse 3D queries to retrieve multi-view image features and make per-query predictions.
- 2D object detection: RCNN processes each pre-selected object proposal with a separate ConvNet pass, making it an order of magnitude slower than other approaches.
- Set-based object detection: DETR formulates detection as set-to-set prediction with a Transformer and avoids redundant-box filtering through ground-truth assignment.
- DETR3D overview: DETR3D encodes multi-view images with ResNet and FPN, decodes sparse object queries into 3D reference points, and projects them into image space to refine queries.
- DETR3D overview: The model produces one prediction per query and trains with a set-to-set loss.
- Monocular 3D object detection: Monocular 3D methods use 3D proposals, birds-eye-view representations, or 2D detections with projection-error minimization to estimate 3D boxes.
3 Multi-view 3D Object Detection
DETR3D alternates geometry-aware 2D feature sampling and 3D object-query refinement to predict a set of 3D boxes from calibrated multi-camera RGB images. It avoids dense scene-geometry estimation and post-processing while training with set-to-set supervision.
- Architecture: DETR3D takes calibrated multi-camera RGB images and outputs a set of 3D bounding box parameters.The inputs include camera projection matrices combining intrinsics and relative extrinsics.
- Architecture: The model incorporates 3D information during intermediate computations instead of relying on purely image-plane processing.It avoids estimating dense 3D scene geometry and associated reconstruction errors.
- Detection head: Projected features are bilinearly sampled across camera views and feature levels, while invalid projections are filtered before updating the next-layer object queries.Reference-point projections are normalized to [-1, 1], and binary visibility values suppress points outside image planes.
- Design rationale: Unlike bottom-up camera detectors, DETR3D avoids explicit depth prediction, point-cloud reconstruction, and NMS-based post-processing.On nuScenes, it is comparable to prior art with NMS without using NMS and significantly outperforms others in camera-overlap regions.
- Detection head: Each layer predicts reference points from object queries, projects them into camera feature maps, samples image features, and models object interactions with multi-head attention.The layers repeat feature sampling and query refinement before final box and label prediction.
- Training: Training uses a Hungarian-matched set-to-set loss combining focal classification loss with L1 bounding-box loss.Losses are computed from every layer, while inference uses only the final layer outputs.
4 Experiments
Experiments evaluate DETR3D on nuScenes against existing methods, overlap-region performance, pseudo-LiDAR, and design variations. DETR3D generally performs strongly without post-processing, especially where camera views overlap, while translation error remains substantial.
- Implementation Details: Experiments use nuScenes images from six cameras and report mAP, ATE, ASE, AOE, AVE, AAE, and NDS.The dataset provides camera intrinsics and extrinsics; evaluation follows the official nuScenes protocol.
- Implementation Details: The evaluated model uses a ResNet101-FPN backbone and a six-layer DETR3D detection head with feature refinement and multi-head attention.Training uses AdamW for 12 epochs on eight RTX 3090 GPUs, with no NMS during inference.
- Comparison to Existing Works: DETR3D outperforms CenterNet and FCOS3D despite using no post-processing, although FCOS3D performs better on mATE.The comparison attributes FCOS3D’s mATE advantage possibly to direct depth supervision and disentangled prediction heads.
- Comparison to Existing Works: On the test set, DETR3D outperforms all existing methods as of 10/13/2021 using the same backbone as DD3D for a fair comparison.This result is reported for the leaderboard comparison in Table 2.
- Comparison in Overlap Regions: In camera-overlap regions, DETR3D remarkably outperforms FCOS3D in NDS, supporting integrated multi-camera prediction.The overlap subset contains 18,147 boxes, or 9.7% of validation boxes, whose 3D centers are visible to multiple cameras.
- Comparison to pseudo-LiDAR Methods: Against a pseudo-LiDAR baseline using supervised PackNet depth and CenterPoint, DETR3D significantly outperforms the explicit-depth pipeline.The comparison is intended to test whether avoiding dense depth prediction improves multi-view 3D detection.
- Ablation & Analysis: Iterative object-query refinement significantly improves performance, while qualitative results show reasonable detections but substantial translation error.The translation error indicates that depth estimation remains a core challenge despite avoiding explicit depth prediction.
5 Conclusion
The paper frames DETR3D as a 3D-space approach to the ill-posed recovery of 3D information from images, using backward projection to retrieve multi-camera features without intermediate depth or point-cloud representations. Its visualizations show iterative refinement toward ground truth, while future work targets richer sampling, additional modalities, and broader application domains.
- Predicted bounding boxes move closer to ground truth across deeper DETR3D head layers, illustrating iterative object-query refinement.
- Backward projection retrieves image features from multiple cameras while eliminating intermediate depth maps or point clouds that can introduce compounding errors.
- Future work includes sampling multiple points per query, adding LiDAR or RADAR, and generalizing the pipeline beyond autonomous driving.
- The model can detect small objects and some unlabeled objects, but misses at least one far-ahead vehicle in the illustrated camera view.