Source-linked AI summary
MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection
Renrui Zhang, Han Qiu, Tai Wang, Ziyu Guo, Yiwen Tang, Xuanzhuo Xu, Ziteng Cui, Yu Qiao, Peng Gao, Hongsheng Li
TL;DR
Monocular 3D detection is limited by ill-posed depth estimation and center-guided methods’ reliance on local visual features. MonoDETR introduces a depth-guided DETR with foreground depth prediction, non-local depth encoding, and depth-guided object queries; it reports leading KITTI results and improves multi-view detectors on nuScenes without dense depth annotations.
Problem
Monocular images lack depth measurements and multi-view geometry, while center-guided methods rely on local features and miss long-range scene-level depth relations.
Method
MonoDETR uses parallel visual and depth encoders, a foreground depth map, and a depth-guided decoder to adaptively interact object queries with scene depth cues.
Results
+2.53%, +1.08%, and +0.85% over the second-best method across KITTI’s three difficulty levels, while its modules improve PETRv2 by +1.2% NDS and BEVFormer by +0.9% NDS on nuScenes.
Takeaways & Limitations
Depth guidance supports a DETR-based monocular detector and transfers as a plug-and-play module to multi-view 3D detection.
Takeaways & Limitations
The paper does not discuss incorporating multi-modal inputs such as LiDAR and RADAR into the transformer framework.
Abstract
from arXiv · showhide
Monocular 3D object detection has long been a challenging task in autonomous driving. Most existing methods follow conventional 2D detectors to first localize object centers, and then predict 3D attributes by neighboring features. However, only using local visual features is insufficient to understand the scene-level 3D spatial structures and ignores the long-range inter-object depth relations. In this paper, we introduce the first DETR framework for Monocular DEtection with a depth-guided TRansformer, named MonoDETR. We modify the vanilla transformer to be depth-aware and guide the whole detection process by contextual depth cues. Specifically, concurrent to the visual encoder that captures object appearances, we introduce to predict a foreground depth map, and specialize a depth encoder to extract non-local depth embeddings. Then, we formulate 3D object candidates as learnable queries and propose a depth-guided decoder to conduct object-scene depth interactions. In this way, each object query estimates its 3D attributes adaptively from the depth-guided regions on the image and is no longer constrained to local visual features. On KITTI benchmark with monocular images as input, MonoDETR achieves state-of-the-art performance and requires no extra dense depth annotations. Besides, our depth-guided modules can also be plug-and-play to enhance multi-view 3D object detectors on nuScenes dataset, demonstrating our superior generalization capacity. Code is available at https://github.com/ZrrSkywalker/MonoDETR.
1. Introduction
MonoDETR addresses monocular 3D detection’s difficulty with ill-posed depth and locally constrained center-guided methods by using a depth-guided DETR framework. It predicts foreground depth, learns non-local depth representations, and adaptively guides object queries, achieving strong KITTI and nuScenes results.
- Monocular 3D detection is difficult because single-view images lack depth measurements and multi-view geometry, making depth estimation ill-posed.
- Center-guided methods predict 3D properties from neighboring features around projected object centers, limiting long-range context and implicit geometric cues.
- MonoDETR introduces the first DETR framework for monocular 3D detection, using parallel visual and depth encoders plus a depth-guided decoder.
- A foreground depth map provides object-wise depth supervision from discrete labels, requiring no dense depth annotations during training.
- MonoDETR surpasses the second-best method by +2.53%, +1.08%, and +0.85% across KITTI’s three difficulty levels.
- On nuScenes, the depth-guided modules improve PETRv2 by +1.2% NDS and benefit BEVFormer by +0.9% NDS as plug-and-play components.
2. Related Work
Related work spans monocular and multi-view camera-based 3D detection, alongside efforts to transfer DETR architectures into 3D detection. MonoDETR is distinguished by combining depth guidance with depth-aware object queries.
- Monocular detectors use only front-view images, whereas multi-view detectors exploit surrounding views and cross-view dependence to understand 3D space.
- Most monocular detectors follow center-guided 2D-detection pipelines, with prior work adding discretized depth, depth-aware convolutions, efficient one-stage designs, or auxiliary data.
- DETR-based camera 3D detectors differ in their use of 2D, depth, 3D, and BEV feature or prediction spaces.
- MonoDETR is uniquely depth-guided among the compared DETR-based methods, using depth-aware object queries for camera-based 3D detection.
- Multi-view methods aggregate surrounding-view features through 3D queries, positional features, temporal information, or unified BEV representations.
3. Method
MonoDETR combines parallel visual and depth encoding with a depth-guided decoder to aggregate global scene geometry for monocular 3D detection. Its training uses object-wise depth supervision and geometry-aware bipartite matching, while the transformer can extend to multi-view detectors.
- Framework: MonoDETR extracts visual and depth features concurrently, then uses parallel encoders and a depth-guided decoder for global scene-level aggregation.The encoders produce non-local embeddings, while object queries adaptively capture scene information.
- Feature Extraction: A lightweight depth predictor generates depth features and a foreground depth map supervised only by discrete object-wise depth labels.Pixels inside object boxes share the corresponding object depth label; no dense depth annotations are required.
- Feature Extraction: Linear-increasing discretization divides foreground depth into k bins plus a background bin, using wider intervals for farther objects.The method limits foreground depths to [dmin, dmax] and assigns overlapping pixels the depth label of the nearest object.
- Depth-guided Transformer: Global self-attention encodes long-range depth dependencies, providing non-local geometric cues for scene-level 3D understanding.The depth encoder uses one transformer block, while the visual encoder uses three because visual appearances are richer than discrete foreground depth information.
- Depth-guided Transformer: Object queries first aggregate depth-guided features, interact through self-attention, and collect visual semantics before predicting 3D attributes.The decoder stacks three blocks and uses depth cross-attention to focus queries on informative image regions.
- Depth-guided Transformer: Learnable depth positional encodings replace sinusoidal functions and are interpolated from predicted pixel depths before depth cross-attention.Each depth-position embedding represents a meter between dmin and dmax.
- Detection Heads and Loss: Hungarian matching uses only the grouped 2D loss because unstable early 3D predictions could disturb query-label assignment.The 2D group contains category, 2D size, and projected 3D center; the 3D group contains depth, 3D size, and orientation.
- Multi-view Extension: The depth-guided transformer can be appended to multi-view detectors by sharing depth predictors and encoders across views and injecting depth cross-attention.This guides 3D or BEV object queries using surrounding-scene depth embeddings.
4. Experiments
Experiments evaluate MonoDETR on KITTI and nuScenes, compare its accuracy and efficiency with existing methods, and ablate its depth-guided components and design choices.
- Dataset: MonoDETR is evaluated on KITTI using AP3D and APBEV across easy, moderate, and hard difficulty levels.The benchmark contains 7,481 training and 7,518 test images, with 3,769 validation images split from training data.
- Comparison: MonoDETR achieves state-of-the-art KITTI performance, surpassing the second-best by +2.53%, +1.08%, and +0.85% in AP3D across the three difficulty levels.The corresponding APBEV gains are +2.94%, +1.73%, and +1.41%.
- Efficiency: MonoDETR achieves the best detection performance without consuming too much computational budget by processing feature maps at 1/16 and 1/32 downsample ratios.Other methods adopt 1/4 and 1/8 ratios, while Table 3 measures Runtime on one RTX 3090 GPU with batch size 1.
- Ablation Studies: Removing the depth-guided transformer greatly hurts performance, while removing either the transformer or depth guidance also degrades performance.These ablations support the significance of non-local geometric cues and depth-guided feature aggregation.
- Ablation Studies: A single-block Global SA depth encoder produces the best non-local depth embeddings for global geometry encoding.The study compares Global SA, deformable self-attention, convolutional layers, and directly feeding depth features into the decoder.
- Ablation Studies: The D → I → V decoder order performs best because queries first aggregate depth cues before the remaining operations in each decoder block.Here D denotes depth cross-attention, I inter-query self-attention, and V visual cross-attention.
- Ablation Studies: Foreground depth supervision better captures inter-object depth relations than dense supervision, while LID outperforms other discretization methods.The paper attributes LID’s advantage to linear-increasing intervals suppressing larger estimation errors for farther objects.
- Multi-view Generalization: Appending the depth-guided transformer to BEVFormer improves nuScenes validation performance by +0.9% NDS and +0.7% mAP.The result indicates benefits from auxiliary depth information for BEV-space feature encoding and demonstrates generalizability to multi-view detection.
5. Visualization
Attention visualizations show that target queries use depth-guided attention distributed across the image, including distant objects, rather than only neighboring visual features.
- Attention Maps: Figure 6 shows the input image followed by attention maps for target queries marked with white dots, with hotter colors indicating higher attention weights.The maps visualize attention AD in the depth cross-attention layer.
- Attention Maps: High-attention regions spread across the entire image and concentrate on other distant objects.This demonstrates that depth guidance lets object queries adaptively capture non-local depth cues.
6. Conclusion
MonoDETR is an end-to-end depth-guided DETR framework for monocular 3D detection that avoids additional inputs, anchors, and NMS. Its experiments support effectiveness on KITTI and nuScenes, while multimodal integration remains unaddressed.
- Conclusion: MonoDETR enables object queries to explore geometric cues adaptively from depth-guided regions and perform inter-object and object-scene depth interactions through attention.The framework is end-to-end and uses no additional input, anchors, or NMS.
- Conclusion: Experiments demonstrate MonoDETR’s effectiveness for both single-view KITTI and multi-view nuScenes input.The conclusion presents the method as a strong DETR baseline for future monocular 3D object detection research.
- Limitations: The paper does not discuss how to effectively incorporate multimodal input into the transformer framework.Future work is proposed around distilling geometric knowledge from LiDAR and RADAR modalities.