PaperlayerView exact revision on arXiv

Source-linked AI summary

MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer

Kuan-Chih Huang, Tsung-Han Wu, Hung-Ting Su, Winston H. Hsu

arXiv:2203.10981v2cs.CV

TL;DR

Monocular 3D detection needs depth cues, while existing depth-assisted methods can incur computational cost and rely on inaccurate depth priors. MonoDTR learns depth-aware features with auxiliary supervision and integrates them with context-aware features using a depth-aware transformer and DPE. On KITTI, it outperforms previous state-of-the-art monocular methods and achieves real-time detection.

  • Problem

    Monocular 3D detection remains limited without depth cues, while off-the-shelf depth-assisted methods suffer from inaccurate depth priors and additional computational cost.

  • Method

    MonoDTR uses auxiliary depth supervision for depth-aware feature learning, transformer-based context-depth integration, and depth positional encoding.

  • Results

    MonoDTR outperforms previous state-of-the-art monocular-based methods and achieves real-time detection on the KITTI dataset.

  • Takeaways & Limitations

    Its depth-aware modules can be plugged into existing image-only monocular 3D detectors to improve performance.

  • Takeaways & Limitations

    Depth-assisted approaches can risk inaccurate depth maps and added computational cost, while standard self-attention has O(N^2) time and memory complexity.

Abstract

from arXiv · show

Monocular 3D object detection is an important yet challenging task in autonomous driving. Some existing methods leverage depth information from an off-the-shelf depth estimator to assist 3D detection, but suffer from the additional computational burden and achieve limited performance caused by inaccurate depth priors. To alleviate this, we propose MonoDTR, a novel end-to-end depth-aware transformer network for monocular 3D object detection. It mainly consists of two components: (1) the Depth-Aware Feature Enhancement (DFE) module that implicitly learns depth-aware features with auxiliary supervision without requiring extra computation, and (2) the Depth-Aware Transformer (DTR) module that globally integrates context- and depth-aware features. Moreover, different from conventional pixel-wise positional encodings, we introduce a novel depth positional encoding (DPE) to inject depth positional hints into transformers. Our proposed depth-aware modules can be easily plugged into existing image-only monocular 3D object detectors to improve the performance. Extensive experiments on the KITTI dataset demonstrate that our approach outperforms previous state-of-the-art monocular-based methods and achieves real-time detection. Code is available at https://github.com/kuanchihhuang/MonoDTR

1. Introduction

Monocular 3D detection avoids costly depth sensors but remains limited by missing depth cues and inaccurate or expensive estimated-depth pipelines. MonoDTR addresses these issues with auxiliary depth-aware learning, transformer fusion, and depth positional encoding.

  • Image-only monocular 3D detection remains unsatisfactory without depth cues despite progress using geometric consistency.
  • Estimated-depth methods convert depth maps into pseudo-LiDAR or fuse depth and image features, but risk inaccurate priors and additional computation.
  • MonoDTR learns depth-aware features with auxiliary supervision in an end-to-end framework, avoiding off-the-shelf depth estimators.
  • A depth-aware transformer globally integrates context- and depth-aware features, while DPE injects depth positional hints into the transformer.
  • MonoDTR’s depth-aware modules are reported to improve existing image-only frameworks and support real-time detection on KITTI.

2. Related Work

Prior monocular detectors rely on geometric constraints or intermediate representations because images lack direct depth information. MonoDTR’s framework combines depth-aware enhancement and transformer-based integration to address these limitations.

  • Image-only monocular 3D object detection: Image-only methods use geometric consistency, voxel transforms, pairwise relations, feature alignment, or keypoints to predict 3D objects from a single image.
  • Image-only monocular 3D object detection: Purely monocular methods may localize objects inaccurately because images lack depth cues.
  • Depth-assisted monocular 3D object detection: Depth-assisted methods use pseudo-LiDAR or feature fusion, but estimated depth can be inaccurate and computationally costly.
  • Transformer-based integration: Transformer use in monocular 3D detection is motivated by globally modeling context- and depth-aware features under perspective-related object-size variation.
  • MonoDTR framework: MonoDTR’s framework uses DFE for auxiliary depth-aware learning, parallel context extraction, DTR integration, DPE, and a detection head.

3. Proposed Approach

MonoDTR combines auxiliary depth-aware feature learning with transformer-based global fusion for monocular 3D detection. Its DFE, DTR, and DPE modules integrate depth reasoning while retaining an image-only detection pipeline.

  • Framework: MonoDTR uses a backbone, DFE module, DTR module, DPE module, and 2D-3D detection head to process RGB images for monocular 3D detection.The backbone produces the feature map; DFE learns depth-aware features, DTR integrates context- and depth-aware features, and detection heads predict 2D and 3D outputs.
  • Depth-Aware Feature Enhancement: The DFE module predicts discretized depth-bin probabilities from backbone features, producing initial depth-aware features under auxiliary depth supervision.Linear-increasing discretization converts continuous depth into depth categories, while the intermediate feature map serves as the initial depth-aware representation.
  • Depth-Aware Feature Enhancement: DFE aggregates pixel features into depth prototypes, reconstructs prototype-enhanced features, and fuses them with the initial depth-aware features.The prototype representation captures global context for each depth category before reconstruction and 1 × 1 convolutional fusion.
  • Depth-Aware Transformer: The DTR module uses a transformer encoder-decoder to globally integrate context-aware and depth-aware features, using depth-aware features instead of conventional learnable object queries.The encoder processes flattened context-aware features with self-attention and a feed-forward network, while the decoder receives depth-aware features.
  • Depth Positional Encoding: DPE injects depth positional hints into the transformer by generating positional encodings from depth categories predicted by DFE.This design replaces purely pixel-wise positional information with depth-informed positional cues for transformer processing.
  • Computation Reduction: Linear attention replaces vanilla self-attention to reduce the stated computation from O(N^2) to O(N) for higher inference speed.The approximation combines transformed keys and values before multiplication, reducing the computational burden of attention.

4. Experiments

On KITTI, MonoDTR improves monocular 3D and BEV detection over image-only and depth-assisted baselines while retaining real-time speed. Ablations and qualitative comparisons attribute gains to its depth-aware modules and positional encoding.

  • Main Results: MonoDTR outperforms other monocular methods on the KITTI test set at the moderate difficulty level for both 3D and BEV detection.
  • Main Results: 2.59/1.76/2.38 AP3D improvements over DFRNet, 2.82/1.98/1.27 over CaDDN, and 2.28/2.61/2.93 over DDMP-3D are reported across the three difficulty settings.All comparisons use IoU threshold 0.7 for cars.
  • Main Results: 6.41/4.99/4.61 AP3D and 7.26/5.41/5.02 APBEV improvements over GUPNet are reported on the KITTI validation set at IoU 0.5.The values correspond to easy, moderate, and hard settings, respectively.
  • Running Time: MonoDTR achieves 27 FPS and runs 17× faster than CaDDN and 4.8× faster than DDMP-3D in the reported validation-set timing analysis.Timing uses batch size 1 on a single Nvidia Tesla V100 GPU; depth-estimator inference time is excluded from the comparison table.
  • Ablation Study: Replacing depth-aware features with pre-trained depth priors performs worse, while DTR, depth prototype enhancement, and DPE improve over simpler alternatives and the baseline.Figure 5 further reports that MonoDTR surpasses the baseline across different IoU thresholds and object depths.
  • Qualitative Results: Qualitative predictions from MonoDTR are closer to ground truth than predictions from the baseline without depth-aware modules, supporting more precise object localization.

5. Conclusion

MonoDTR is an end-to-end depth-aware transformer for monocular 3D object detection that learns depth-aware features without off-the-shelf depth estimation. On KITTI, it outperforms prior monocular methods and achieves real-time detection.

  • MonoDTR combines lightweight auxiliary-supervised depth-aware feature learning, transformer-based context-depth integration, and depth positional encoding.
  • The model avoids inaccurate depth priors and high computational cost from an off-the-shelf depth estimator while achieving real-time KITTI detection.
  • The proposed modules can be plugged into existing image-only monocular 3D detectors to improve performance.

A. Depth-Aware Transformer

The depth-aware transformer encodes context features and decodes fused context-depth features, using depth positional encoding to support 3D reasoning. Linear attention is adopted for real-time operation.

  • The encoder generates encoded context-aware features, while the decoder fuses context-aware and depth-aware features through multiple self-attention layers.
  • Depth positional encoding is added to both features before the transformer to provide depth hints for 3D reasoning.
  • Linear attention is almost 4× faster than vanilla self-attention with comparable performance, motivating its use for real-time transformers.

B. Auxiliary Depth Supervision

MonoDTR generates auxiliary depth supervision by projecting LiDAR into the image and discretizing continuous depth with linear-increasing discretization. The method uses 96 bins over depths from 1 to 80.

  • LiDAR signals are projected into the image plane to produce a sparse ground-truth depth map for auxiliary supervision.
  • Linear-increasing discretization converts continuous depth d into discretized depth bins using the bin index i.
  • The depth discretization uses D=96 bins over the range [1, 80], and out-of-range pixels are excluded from training optimization.
  • Linear-increasing discretization performs better than uniform and spacing-increasing discretization in the auxiliary-supervision comparison.

C. Results on nuScenes Dataset

On nuScenes validation, MonoDTR outperforms FCOS3D and PGD under matched backbone and training settings. Qualitative KITTI results further illustrate predictions across multiple object categories and comparisons with a baseline lacking depth-aware modules.

  • MonoDTR achieves better nuScenes validation performance than FCOS3D and PGD under the same backbone and training schedule.The models were trained for 12 epochs, with baseline results taken from MMDetection3D.
  • KITTI qualitative results cover car, pedestrian, and cyclist categories using orange, blue, and green annotations, respectively.
  • Compared with the baseline without depth-aware modules, MonoDTR generates higher-quality 3D bounding boxes with depth cues.
  • A representative failure case results from inaccurate object-depth prediction, producing a lower-quality 3D bounding box.The paper identifies inaccurate object depth as typical in monocular 3D object detection.

E. Broader Impacts

MonoDTR targets monocular 3D object detection for autonomous driving, but inaccurate depth predictions can produce incorrect downstream decisions and potential traffic accidents. The authors also identify further exploration of learned depth-aware features as an open direction.

  • The proposed model may generate inaccurate object-depth predictions that lead to incorrect downstream decision-making and potential traffic accidents.
  • The work offers a perspective on using learned depth-aware features to assist monocular 3D object detection.
  • Further research is warranted on learning depth-aware features that more effectively improve detection performance.This remains open despite progress from the lightweight depth-aware feature extraction module.
Loading 2203.10981v2…