Source-linked AI summary
Multi-View 3D Object Detection Network for Autonomous Driving
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, Tian Xia
TL;DR
3D object detection for autonomous driving must combine LIDAR’s depth accuracy with cameras’ semantic detail, but existing multimodal models are not well designed for this challenging task. MV3D generates 3D proposals from bird’s-eye-view point clouds and deeply fuses region-based features across views, significantly outperforming existing methods on KITTI 3D localization and detection while remaining competitive for 2D detection.
Problem
Existing methods have complementary strengths across LIDAR and images, but 3D object detection requires a well-designed model to use both modalities effectively.
Method
MV3D generates 3D proposals from a bird’s-eye-view point-cloud representation and deeply fuses region-based features from multiple views for oriented 3D box regression.
Results
MV3D significantly outperforms existing LIDAR- and image-based methods on KITTI 3D localization and detection, with around 25% higher localization accuracy and 30% higher 3D AP, while achieving 10.3% higher 2D AP among LIDAR-based methods.
Takeaways & Limitations
The results support multi-view sensory fusion as an effective approach for high-accuracy 3D object detection in road scenes.
Abstract
from arXiv · showhide
This paper aims at high-accuracy 3D object detection in autonomous driving scenario. We propose Multi-View 3D networks (MV3D), a sensory-fusion framework that takes both LIDAR point cloud and RGB images as input and predicts oriented 3D bounding boxes. We encode the sparse 3D point cloud with a compact multi-view representation. The network is composed of two subnetworks: one for 3D object proposal generation and another for multi-view feature fusion. The proposal network generates 3D candidate boxes efficiently from the bird's eye view representation of 3D point cloud. We design a deep fusion scheme to combine region-wise features from multiple views and enable interactions between intermediate layers of different paths. Experiments on the challenging KITTI benchmark show that our approach outperforms the state-of-the-art by around 25% and 30% AP on the tasks of 3D localization and 3D detection. In addition, for 2D detection, our approach obtains 10.3% higher AP than the state-of-the-art on the hard data among the LIDAR-based methods.
1. Introduction
MV3D targets accurate 3D object detection by fusing LIDAR’s depth information with cameras’ semantic information. It uses compact multi-view point-cloud encoding, 3D proposal generation, and region-based feature fusion, achieving strong KITTI results across 3D and 2D detection tasks.
- Motivation: LIDAR provides accurate depth while cameras preserve detailed semantic information, motivating their fusion for autonomous-driving perception.The paper focuses on highly accurate 3D localization and object recognition in road scenes.
- Problem: 3D detection requires a model that exploits multiple modalities because LIDAR-based methods localize more accurately while image-based methods perform better on 2D box evaluation.Existing multimodal methods use early or late fusion for 2D detection, but 3D detection is more challenging.
- Method: MV3D predicts full 3D object extents from multimodal input using compact multi-view point-cloud encoding, a 3D Proposal Network, and region-based feature fusion.The network takes LIDAR bird’s-eye and front views plus an image, projects proposals into three views, and jointly predicts classes and oriented 3D boxes.
- Results: 99.1% and 91% 3D recall are achieved with only 300 proposals at IoU thresholds of 0.25 and 0.5, respectively.These results are reported on the KITTI object detection benchmark and exceed recent 3D proposal methods 3DOP and Mono3D.
- Results: 25% higher accuracy in 3D localization and 30% higher 3D AP in 3D object detection are achieved by the LIDAR-based variant over prior methods.MV3D also exceeds other LIDAR-based methods by 10.3% AP for 2D detection on KITTI’s hard test set, with further improvements from images.
2. Related Work
Related work spans point-cloud, image-based, multimodal, and 3D proposal methods for object detection. Existing approaches use voxel or projected representations, hand-crafted or learned proposal features, and limited multimodal fusion, motivating MV3D’s deep fusion design.
- 3D Object Detection in Point Cloud: Point-cloud methods primarily use voxel grids, while some project point clouds into front-view 2D maps.Sliding Shapes and Vote3D apply SVM classifiers to geometry features on 3D grids; newer methods use 3D convolutions but require expensive computation, while VeloFCN produces a 2D point map.
- 3D Object Detection in Images: Image-based methods estimate 3D structure or pose and use 3D candidate proposals for object recognition.3DVP uses 3D voxel patterns and ACF detectors, whereas 3DOP reconstructs stereo depth and Mono3D generates proposals from monocular images within R-CNN-style pipelines.
- Multimodal Fusion: Only a few autonomous-driving methods exploit multiple modalities, using mixture-of-experts or early RGB-depth fusion for 2D detection.The paper’s approach instead designs a deep fusion method inspired by FractalNet and Deeply-Fused Net.
- 3D Object Proposals: 3D proposal methods generate a small set of candidate boxes intended to cover most objects in 3D space.3DOP scores many candidates with stereo depth features, Mono3D uses ground-plane and segmentation features, and both rely on hand-crafted features.
3. MV3D Network
MV3D represents LIDAR point clouds from bird’s-eye and front views, generates 3D proposals from the bird’s-eye map, and fuses them with RGB features. A region-based fusion network then classifies proposals and regresses oriented 3D boxes.
- Multi-view representation: MV3D encodes LIDAR using bird’s-eye and front-view representations, then combines these with an image to support multi-view 3D detection.The bird’s-eye map uses height, intensity, and density, while the front-view map uses height, distance, and intensity.
- Oriented 3D box regression: The fused features jointly classify object proposals and regress oriented 3D boxes using normalized offsets for the boxes’ eight corners.The regression target is a 24-D vector of normalized corner offsets.
- 3D proposal network: The proposal network generates 3D candidate boxes from the bird’s-eye map using 3D prior boxes and removes empty anchors to reduce computation.Bird’s-eye projection preserves physical object sizes, and non-empty anchors are identified from the point-occupancy map.
- Region-based fusion: The region-based fusion network projects each 3D proposal into bird’s-eye, front-view, and RGB views, then applies ROI pooling to obtain fixed-length features.ROI pooling addresses differing feature-map resolutions across views and modalities.
- Deep fusion: Deep fusion hierarchically combines intermediate features from multiple views, enabling interactions beyond early input fusion or late prediction fusion.The deep-fusion join operation uses element-wise mean.
4. Experiments
Experiments on KITTI evaluate MV3D’s proposal recall, 3D localization, 3D detection, 2D detection, fusion strategies, and multi-view features. MV3D achieves strong recall and substantial gains over prior LIDAR- and image-based methods, while ablations support deep fusion and combining views.
- Experimental Setup: Experiments use KITTI car detection data, with roughly half of the 7,481 training images for training and half for validation, plus 7,518 test images.3D evaluations use the validation set, while 2D detection is evaluated on the KITTI test set.
- 3D Proposal Recall: 99.1% recall is achieved at IoU 0.25 and 91% at IoU 0.5 using only 300 proposals, versus 73.9% maximum recall for 3DOP at IoU 0.5.MV3D significantly outperforms 3DOP and Mono3D across IoU thresholds.
- 3D Localization: ∼25% higher APloc than VeloFCN is obtained by BV+FV at IoU 0.5, increasing to ∼45% at IoU 0.7 across easy, moderate, and hard regions.LIDAR-based approaches outperform the stereo-based 3DOP and monocular Mono3D methods.
- 3D Object Detection: ∼30% higher AP3D than VeloFCN is achieved by BV+FV at IoU 0.5, reaching 87.65% AP3D in the moderate setting.At IoU 0.7, the multimodal approach achieves 71.29% AP3D on easy data.
- Ablation Studies: Around 1% additional improvement is obtained by adding auxiliary loss to deep fusion, following its ∼0.5% gain over early and late fusion.Early and late fusion have very similar performance.
- 2D Detection and Feature Ablation: 10.31% AP2D improvement over 3D FCN is achieved by BV+FV in the hard setting among LIDAR-based methods, while combined views consistently outperform individual views.Bird’s eye view performs best and front view worst when used alone; combining BV, FV, and RGB is evaluated separately.
5. Conclusion
The paper proposes a multi-view sensory-fusion model for 3D object detection that combines LIDAR point clouds and images. It aligns modalities through 3D proposals and multi-view projection, then deeply fuses region-based features for oriented 3D box regression.
- Conclusion: The model combines LIDAR point clouds and images for 3D object detection in road scenes.This multi-view sensory-fusion design takes advantage of both sensing modalities.
- Conclusion: 3D proposals align modalities by being projected to multiple views for feature extraction.The proposal-based alignment connects the point-cloud and image representations.
- Conclusion: A region-based fusion network deeply combines multi-view information and performs oriented 3D box regression.The fusion network is the paper’s mechanism for integrating information across views.
- Conclusion: The approach significantly outperforms existing LIDAR-based and image-based methods on 3D localization and 3D detection tasks.The supplied conclusion states this comparative result without reporting the corresponding numerical gains.