Source-linked AI summary
A General Pipeline for 3D Detection of Vehicles
Xinxin Du, Marcelo H. Ang, Sertac Karaman, Daniela Rus
TL;DR
Autonomous driving needs accurate 3D vehicle perception, while existing 3D detection methods lag behind 2D detectors and do not readily reuse their advances. The paper fuses adaptable 2D detection networks with 3D point clouds through generalized car-model fitting and two-stage CNN refinement, achieving results close to MV3D on KITTI with both tested networks.
Problem
3D vehicle detection remains substantially weaker than 2D detection and existing 2D approaches do not readily provide the 3D information required for autonomous driving.
Method
The pipeline minimally adapts a 2D detector, fuses its boxes with 3D point clouds, fits boxes using three generalized car models and score maps, and refines them with a two-stage CNN.
Results
With PC-CNN and MS-CNN on KITTI, the pipeline significantly outperforms most compared methods and remains close to MV3D in bird’s-eye-view and 3D detection.
Takeaways & Limitations
Comparable results from two different 2D networks support the pipeline’s flexibility and adaptability for 3D vehicle detection.
Abstract
from arXiv · showhide
Autonomous driving requires 3D perception of vehicles and other objects in the in environment. Much of the current methods support 2D vehicle detection. This paper proposes a flexible pipeline to adopt any 2D detection network and fuse it with a 3D point cloud to generate 3D information with minimum changes of the 2D detection networks. To identify the 3D box, an effective model fitting algorithm is developed based on generalised car models and score maps. A two-stage convolutional neural network (CNN) is proposed to refine the detected 3D box. This pipeline is tested on the KITTI dataset using two different 2D detection networks. The 3D detection results based on these two networks are similar, demonstrating the flexibility of the proposed pipeline. The results rank second among the 3D detection algorithms, indicating its competencies in 3D detection.
I. INTRODUCTION
3D vehicle detection is essential for autonomous driving, but current 3D methods lag behind strong 2D detection and do not readily transfer 2D advances. The paper proposes a flexible fusion pipeline that adapts 2D detectors to 3D point clouds with limited changes.
- 3D vehicle detection provides dimensions, locations, and orientations needed for autonomous-vehicle planning and decision making, whereas state-of-the-art performance reaches only 62% AP compared with about 90% for 2D detection.
- Existing 3D vehicle detectors use mono images, stereo images, LiDAR, or camera–LiDAR fusion, each with documented depth, sparsity, projection, or information-loss challenges.
- Prior work has not effectively transferred advances from 2D detection to 3D detection or enabled existing 2D approaches to provide 3D information for autonomous vehicles.
- The proposed pipeline projects 2D detections into 3D, selects corresponding point-cloud subsets, fits 3D vehicle boxes, and refines them with a CNN.
- The method combines a general 2D-to-3D pipeline, three generalized car models with score maps, and a two-stage CNN refinement module.
- Using PC-CNN and MS-CNN on KITTI, the pipeline significantly leads most methods in bird’s-eye and 3D detection while achieving results comparable to MV3D.
II. RELATED WORKS
Related work spans model fitting, image-based, stereo, LiDAR, fusion, and 2D detection approaches. The proposed method distinguishes itself by using only three generalized vehicle models and by converting flexible 2D detections into 3D outputs.
- Mono Image Approaches: Mono-image methods estimate dimensions, orientations, probabilities, or keypoints from 2D boxes, but their accuracy depends on assumptions and parameter estimation.
- Stereo Image Approaches: Stereo and LiDAR methods derive depth or point-cloud representations for CNN processing, but stereo accuracy decreases with distance and direct 3D convolutions can be computationally expensive.
- Fusion Approaches: Earlier fusion methods transform point clouds into depth or HHA maps, while MV3D uses bird’s-eye-view input to generate 3D proposals directly without projective loss.
- 2D Detection: The proposed pipeline is flexible because existing 2D detectors need only a slight final-layer change, enabling leading networks to be incorporated for accuracy or speed.
- Model Fitting: Prior model-fitting methods use voxel, deformable-part, cuboid, or wireframe representations, often encoding visibility or occlusion explicitly.
- Model Fitting: Because vehicle types, sizes, and occlusions vary, prior approaches require many models, whereas this method uses three models and assigns occlusion patterns online.
III. TECHNICAL APPROACH
The technical pipeline starts from image-based vehicle proposals and uses camera–LiDAR geometry to extract 3D point subsets for subsequent box detection.
- 2D bounding boxes select projected point-cloud subsets, but perspective spread, non-vehicle points, and neighboring vehicles make 3D box detection challenging.
A. Car dimension estimation
The detector is extended with a vehicle-dimension regression branch that can be trained largely from pretrained weights while retaining the original detection objectives.
- An additional regression layer estimates vehicle dimensions at the end of the existing 2D detection network.
- The dimension-regression loss is weighted alongside the original classification and 2D box-regression losses, using car indicators and Smooth L1 regression.
- Pretrained weights can initialize the modified detector, with only selected post-ROI or detection-subnetwork layers retrained in MS-CNN and PC-CNN.
B. Vehicle model fitting
The pipeline generates 3D box proposals from point-cloud geometry, then scores them against generalized vehicle models using voxelized score maps and selects the best proposal.
- RANSAC-style sampling constructs vertical planes from randomly selected points and uses their intersections to generate candidate 3D boxes.Each iteration can generate up to 80 proposals after geometric filtering and box expansion.
- Three normalized models represent SUVs, sedans, and vans because within-category vehicle shapes and contours are similar after normalization.The models aggregate 58 SUV/hatchback, 65 sedan, and 10 van CAD models.
- Each generalized model is voxelized into an 8×18×10 grid whose shell receives positive scores while interior and exterior points receive increasingly negative scores.Bottom-layer points receive zero because they may correspond to either ground or tires.
- Self-occlusion is incorporated through view-dependent negative scores on occluded vertical facets during model fitting.The roof and bottom are excluded from self-occlusion analysis.
- The highest-scoring proposal among voxelized point-cloud boxes is selected after comparison with three generalized vehicle models.Proposals are rotated around the vertical axis by 180° to address orientation ambiguity before scoring.
C. Two-stage refinement CNN
A two-stage CNN refines model-fitted 3D boxes using normalized, voxelized point clouds and bounding-box context. The first stage regresses and classifies a box, while the second classifies the refined result.
- The two-stage CNN aligns detected 3D boxes to the point cloud while avoiding the slow, memory-intensive 3D convolutions used by common 3D CNNs.The paper reports that 2D CNNs are sufficient for this refinement task.
- The first CNN outputs a new 3D box from points inside the input box, and the second assigns a car probability to points inside the refined box.The stages are trained independently and do not share layers.
- The pipeline expands each box by 1.5, 1.5, and 1.6 along height, length, and width, then voxelizes the context into a 24×54×32 matrix.The resulting matrix is sparse, with approximately 0.6% occupied elements on average.
- The first stage has parallel regression and classification outputs, whereas the second stage has only a classification output.Both CNNs use SoftMax classification loss and SmoothL1 regression loss.
- Classification labels use bird’s-eye-view IoU thresholds of 0.5 for stage one and 0.7 for stage two.The second-stage threshold matches the KITTI benchmark criterion.
IV. EXPERIMENT RESULTS AND DISCUSSION
On KITTI, the pipeline achieves strong bird’s-eye-view and 3D detection performance, remains flexible across two different 2D networks, and generalizes qualitatively to Boston data. Ablations show that model fitting provides a sound baseline and the first refinement CNN supplies the main performance gains.
- Bird’s Eye View & 3D Box AP: At IoU = 0.7, both PC-CNN and MS-CNN are close to MV3D for bird’s-eye-view detection, except PC-CNN on the hard group, which is 7% worse.
- Bird’s Eye View & 3D Box AP: The pipeline significantly outperforms most compared methods and remains close to MV3D, with roughly 10% lower average 3D-box performance at both IoU thresholds.For moderate objects at IoU = 0.5 with MS-CNN, the gap is only 1.6%.
- Flexibility Analysis: Using either PC-CNN or MS-CNN yields comparable AP, and the refinement CNN transfers from PC-CNN to MS-CNN without further tuning.This supports the pipeline’s flexibility across different 2D detection network structures.
- Ablation Study: Model fitting alone performs better than all image-based algorithms and comparably to VeloFCN, supporting its effectiveness before CNN refinement.
- Ablation Study: The first refinement CNN improves bird’s-eye-view and 3D-box performance by approximately 10% and 30%, respectively, whereas the second CNN contributes insignificantly.The second CNN reshuffles first-stage box probabilities rather than regressing the 3D box.
- Qualitative Results: The pipeline works qualitatively on Boston data when trained on KITTI and applied without fine-tuning, while its KITTI results are illustrated with blue 3D boxes.
V. CONCLUSIONS
The paper proposes a flexible 3D vehicle-detection pipeline that adapts existing 2D networks with minimal changes and fuses them with point-cloud measurements.
- The pipeline adapts any 2D detection network to provide 3D vehicle information with minimal effort.Only one additional regression term is needed to estimate vehicle dimensions.
- It fits 3D bounding boxes from point clouds using generalized car models and score maps, then refines them with a two-stage CNN.
- Results from two different 2D networks indicate the pipeline's flexibility and capability for 3D vehicle detection.