PaperlayerView exact revision on arXiv

Source-linked AI summary

M3D-RPN: Monocular 3D Region Proposal Network for Object Detection

Garrick Brazil, Xiaoming Liu

arXiv:1907.06038v2cs.CV

TL;DR

Monocular 3D detection is limited by missing depth cues and has traditionally relied on expensive sensors or auxiliary networks. M3D-RPN reformulates it as a standalone end-to-end 3D region proposal network with shared 2D/3D structure and depth-aware convolutions. The method significantly improves monocular BEV and 3D detection on KITTI while using a shared multi-class model.

  • Problem

    Monocular image-only 3D detection lacks reliable depth cues, while prior methods often rely on external sub-networks, leaving a substantial gap relative to LiDAR-based detection.

  • Method

    M3D-RPN is a standalone end-to-end region proposal network using shared 2D and 3D anchors, depth-aware convolutions, and 3D→2D projection consistency for orientation optimization.

  • Results

    M3D-RPN significantly improves monocular Bird's Eye View and 3D object detection performance on KITTI across car, pedestrian, and cyclist classes.

  • Takeaways & Limitations

    A single multi-class network trained without external networks, data sources, or multiple stages can address monocular BEV and 3D detection on KITTI.

  • Takeaways & Limitations

    M3D-RPN relies on 3D box annotations and a known projection matrix for each sequence.

Abstract

from arXiv · show

Understanding the world in 3D is a critical component of urban autonomous driving. Generally, the combination of expensive LiDAR sensors and stereo RGB imaging has been paramount for successful 3D object detection algorithms, whereas monocular image-only methods experience drastically reduced performance. We propose to reduce the gap by reformulating the monocular 3D detection problem as a standalone 3D region proposal network. We leverage the geometric relationship of 2D and 3D perspectives, allowing 3D boxes to utilize well-known and powerful convolutional features generated in the image-space. To help address the strenuous 3D parameter estimations, we further design depth-aware convolutional layers which enable location specific feature development and in consequence improved 3D scene understanding. Compared to prior work in monocular 3D detection, our method consists of only the proposed 3D region proposal network rather than relying on external networks, data, or multiple stages. M3D-RPN is able to significantly improve the performance of both monocular 3D Object Detection and Bird's Eye View tasks within the KITTI urban autonomous driving dataset, while efficiently using a shared multi-class model.

1. Introduction

Monocular 3D detection remains substantially harder than LiDAR-based detection because images lack reliable depth cues. M3D-RPN addresses this gap with a standalone end-to-end network that shares 2D and 3D detection structure and adds depth-aware estimation designs.

  • Monocular image-only 3D detection lacks reliable depth cues, leaving a substantial performance gap versus LiDAR-based methods.
  • Prior monocular methods rely on external sub-networks for point clouds, segmentation, 2D detection, or depth estimation, increasing system complexity and component disconnection.
  • M3D-RPN reformulates monocular 3D detection as a single end-to-end region proposal network for multi-class 3D object detection.
  • Shared 2D and 3D anchors and classification targets allow image-space detection features to guide 3D localization in camera coordinates.
  • Depth-aware convolution learns spatially aware high-level features, while a 3D→2D projection consistency post-optimization improves orientation estimation.

2. Related Work

Prior monocular 3D systems commonly combine multiple stages and external networks, whereas M3D-RPN uses one network to generate 2D and 3D proposals simultaneously. Its design combines shared proposal generation with depth-aware feature extraction and direct 3D parameter prediction.

  • 2D region proposal networks use sliding-window detection over image locations and predefined anchor shapes to generate object proposals.
  • LiDAR-based 3D detection methods depend on depth information from sparse point clouds, making them unsuitable for camera-only applications.
  • Image-based 3D methods commonly use multiple sub-networks for depth, point-cloud, segmentation, or proposal generation because monocular images lack reliable depth.
  • Unlike prior approaches that estimate 3D parameters from cropped 2D proposals or auxiliary depth networks, M3D-RPN directly predicts 3D parameters and optimizes orientation post hoc.
  • The framework combines global spatial-invariant features with local depth-aware features to improve spatial awareness of high-level representations.
  • M3D-RPN is a single network trained only with 3D boxes that simultaneously generates 2D and 3D object proposals.

3. M3D-RPN

M3D-RPN is a standalone monocular 3D region proposal network that jointly operates in shared 2D and 3D detection spaces. It combines image-space 3D anchors, depth-aware features, and 3D-to-2D orientation optimization while requiring 3D annotations and a known projection matrix.

  • Framework: M3D-RPN adapts the region proposal network into a standalone monocular 3D detector with shared 2D and 3D proposal spaces.The network scans image locations using predefined anchor templates and regresses matched anchors into continuous object parameters.
  • Anchor Formulation: Each anchor jointly represents 2D dimensions, projected depth, 3D dimensions, orientation, and a shared image-space center.The projected 3D center is used because the network's convolutional features are defined in image space, then back-projected to camera coordinates during inference.
  • Anchor Formulation: Precomputed per-anchor means provide priors for projected depth, 3D dimensions, and orientation, giving discretized templates strong initial estimates under consistent scene geometry.The statistics are computed from matching ground truths whose 2D boxes have at least 0.5 IoU with the corresponding anchor.
  • 3D Detection: The detector predicts shared classification together with 2D box, projected-center, 3D-dimension, and orientation transformations for every anchor at every feature-map location.The outputs include c, [t_x, t_y, t_w, t_h]_2D, [t_x, t_y, t_z]_P, and [t_w, t_h, t_l, t_θ]_3D.
  • Depth-aware Convolution: Depth-aware convolution assigns distinct kernels to discretized row-wise depth bins, enabling location-specific high-level features while increasing layer memory by ×b.The theoretical convolution FLOPS remain consistent with shared kernels, although the implementation incurs a practical 10−20% reshaping overhead.
  • Scope: M3D-RPN relies on 3D box annotations and a known projection matrix for each sequence.Datasets lacking these inputs may require predicted camera intrinsics and weak supervision based on 3D-to-2D projection geometry.

4. Experiments

Experiments on KITTI evaluate M3D-RPN for BEV, 3D, multi-class, and 2D detection, alongside component ablations and runtime analysis. The method improves challenging 3D localization settings, while depth-aware convolution and post-optimization provide measurable gains with modest efficiency costs.

  • Bird’s Eye View: M3D-RPN improves BEV performance over image-only detectors across splits and protocols, reaching 21.18% on moderate and 17.90% on hard val1 at IoU ≥0.7.On the test split, it reaches 18.36% on moderate and 16.24% on hard under the same criterion.
  • 3D Object Detection: M3D-RPN improves 3D detection under strict IoU ≥0.7 settings, achieving 17.06% on moderate and 15.21% on hard val1.On test, the corresponding results are 15.70% and 13.32%, despite training as a shared multi-class model.
  • Multi-Class 3D Detection: The shared framework generalizes across cars, pedestrians, and cyclists, but is noticeably less stable for cyclists because of dataset bias toward cars and pedestrians.The multi-class evaluation uses a single shared model on KITTI test data under the moderate setting.
  • 2D Detection: M3D-RPN performs below other 3D detection systems on 2D detection, but its single-shot network is designed primarily to improve BEV and 3D localization.The method performs all tasks simultaneously without the external networks, data sources, and multiple stages used by prior work.
  • Ablations: Depth-aware convolution performs best with b = 32, improving APBEV by 3.71%, AP3D by 1.98%, and AP2D by 1.51% over the baseline without local features.The classification output assigns the highest bias toward local features, indicating reliance on depth position for semantic features.
  • Ablations: Post-optimization improves APBEV by 4.48% and AP3D by 4.09%, converging in approximately 8 iterations with 13 ms per-image overhead.It encourages consistency between 3D boxes projected into image space and predicted 2D boxes.

5. Conclusion

M3D-RPN reformulates monocular image-only 3D detection as a single-shot 3D region proposal network and improves both Bird's Eye View and 3D detection on KITTI across three object classes.

  • M3D-RPN reformulates monocular image-only 3D detection as a single-shot 3D RPN instead of using external networks, data sources, or multiple stages.
  • Shared 2D and 3D anchors leverage priors linked to the correlation between 2D scale and 3D depth.
  • Depth-aware convolution layers enable spatially-aware features for improved 3D parameter estimation.
  • Performance significantly improves on KITTI for both Bird's Eye View and 3D object detection across car, pedestrian, and cyclist classes.
Loading 1907.06038v2…