Source-linked AI summary
3D Fully Convolutional Network for Vehicle Detection in Point Cloud
Bo Li
TL;DR
Point-cloud detection is important because it directly localizes objects in 3D, while image-plane evaluation does not fully reflect autonomous-driving needs. This paper extends fully convolutional networks to 3D point-cloud detection, using objectness and bounding-box predictions to detect vehicles as 3D boxes, and reports a significant improvement over previous point-cloud methods on KITTI.
Problem
Point-cloud detection is needed for robotic applications because it naturally provides 3D object coordinates, while 2D image-plane evaluation does not fully reflect the needs of subsequent autonomous-driving modules.
Method
The paper extends fully convolutional networks to 3D for end-to-end object detection and localization as 3D boxes, using objectness and bounding-box prediction maps whose positive regions are clustered into detections.
Results
The proposed method outperforms previous point-cloud detection approaches by a significant gap of > 20% on the KITTI evaluation.
Takeaways & Limitations
Embedding objects in 3D avoids perspective distortion and scale variation, enabling detection with a relatively simpler network structure and potential application to other point-cloud sources.
Abstract
from arXiv · showhide
2D fully convolutional network has been recently successfully applied to object detection from images. In this paper, we extend the fully convolutional network based detection techniques to 3D and apply it to point cloud data. The proposed approach is verified on the task of vehicle detection from lidar point cloud for autonomous driving. Experiments on the KITTI dataset shows a significant performance improvement over the previous point cloud based detection approaches.
I. INTRODUCTION
Point-cloud detection directly localizes objects in 3D, supporting downstream robotic tasks. The paper extends fully convolutional detection to 3D point clouds for vehicle detection.
- Point-cloud object detection naturally localizes 3D object coordinates for navigation and manipulation.
- The paper designs a 3D fully convolutional network to detect and localize objects as 3D boxes.
- The approach targets autonomous-driving vehicle detection using point clouds from a Velodyne 64E lidar.
- Earlier point-cloud detectors use proposal and classification stages based on segmentation, sliding windows, sampling, RPNs, shape, geometry, sparse coding, or deep learning.
- Projecting point clouds into depth maps or range scans benefits from 2D detectors but inevitably loses or distorts useful 3D spatial information.
B. Convolutional Neural Network and 3D Object Detection
The paper places 3D point-cloud detection within CNN-based approaches and transplants fully convolutional detection into 3D. Its design is inspired by DenseBox and aims at end-to-end 3D-box prediction.
- Prior CNN-based methods often embed 3D information in 2D projections, while relatively few works apply 3D CNNs directly.
- The paper transplants FCN to 3D for end-to-end detection and localization of objects as 3D boxes in point clouds.
- The proposed approach is inspired by DenseBox, an FCN variation for object detection.
A. FCN Based Detection Revisited
The FCN detection framework predicts objectness and bounding-box outputs over regions, combines their losses, and clusters positive-region predictions into detections. The 3D implementation uses an hourglass-style convolution/deconvolution network.
- FCN outputs: FCN detection uses an objectness map to identify object regions and a bounding-box map to predict object coordinates.
- Objectness prediction: The objectness output at each region can be encoded by softmax or hinge loss.
- Bounding-box prediction: The bounding-box output is encoded as coordinate offsets for the object bounding box.
- Objectness prediction: Each class uses one objectness label in this paper, whereas SSD and DenseBox can use multiple labels for scales or aspect ratios.
- Bounding-box prediction: The paper assumes one bounding-box map, while more sophisticated networks can predict multiple offsets for scales or aspect ratios.
- Training objective: The overall loss balances objectness and bounding-box losses with weight w.
- Deployment: At deployment, positive-objectness regions are selected, and their bounding-box predictions are collected and clustered as detections.
B. 3D FCN Detection Network for Point Cloud
The 3D FCN discretizes point clouds on square grids and predicts objectness and bounding-box information through a 3D hourglass-style network. At test time, positive regions generate candidate boxes that are scored, selected, and clustered into detections, although severe occlusion can distort unclustered boxes.
- 3D input representation: Point clouds are discretized on square grids into a 4D array of length, width, height, and channels.The simplest representation uses a binary channel indicating whether points occupy each grid element.
- Network structure: The 3D network downsamples and upsamples grid features, producing separate objectness and bounding-box output maps.The simplified hourglass structure applies ReLU after each layer and collects the two maps from deconvolution layers.
- Training targets: A 3D sphere centered on each object defines positive regions, while bounding-box predictions encode offsets to the eight box corners.The corner coordinates correspond to the object bounding box associated with region p.
- Inference: During testing, object-positive regions provide candidate boxes that are scored by neighboring candidates, with overlapping lower-scoring boxes suppressed.The remaining candidates are selected from the highest score and clustered as detection results.
- Failure case: Severely occluded vehicles can produce distorted, unclustered boxes because similar training examples are lacking.This failure mode is illustrated among the intermediate detection results.
C. Comparison with 2D CNN
Compared with 2D CNN processing, 3D CNNs require more memory and computation, but embedding objects naturally in 3D avoids perspective distortion and scale variation. This allows the approach to use a relatively simpler network structure.
- Computational trade-off: 3D CNNs consume more computational resources because 3D embedding grids increase memory costs and 3D kernels increase convolution costs.These costs are identified as the main computational difference from 2D CNNs.
- Representation advantage: Natural 3D embedding avoids perspective distortion and scale variation present in the 2D case.The comparison concerns the representation of objects before detection.
IV. EXPERIMENTS
The experiments evaluate 3D vehicle detection on KITTI using aligned images, lidar point clouds, and 2D and 3D object annotations. Training uses 6000 frames, while 1500 frames support offline validation in image and ground-plane spaces, alongside KITTI online evaluation.
- Dataset and task: The KITTI vehicle-detection task provides images aligned with point clouds and object information labeled by both 3D and 2D bounding boxes.The experiments primarily focus on the Car category.
- Labeling: Car-center regions inside a 3D center sphere are positive, while Van and Truck are ignored and other listed categories form negative background.The labeling scheme defines the objectness training regions for the experiment.
- Data split: 6000 of KITTI’s 7500+ frames are randomly selected for training, and the remaining 1500 frames are used for offline validation.Offline validation measures bounding-box overlap on both the image plane and ground plane; online evaluation uses image-space overlap.
- Difficulty levels: KITTI’s easy, moderate, and hard image-based difficulty levels are approximately reused for 3D detection and evaluation.The paper associates the 40-pixel easy threshold with objects within approximately 28 m and the 25-pixel moderate/hard threshold with objects within approximately 47 m.
A. Performance Analysis
The evaluation supplements KITTI’s image-plane criterion with ground-plane overlap to better reflect autonomous-driving needs in world space. It reports AP and AOS, and the proposed approach is described as more accurate than the compared method.
- Evaluation motivation: Image-plane evaluation is insufficient for planning and control because those modules operate in world space.The paper therefore validates detections in both image space and world space.
- Image-plane metric: Image-plane acceptance requires projected 3D boxes to achieve IoU greater than 0.7 with the ground-truth box.The projected box is represented by the minimum rectangle hull on the image plane.
- Ground-plane metric: Ground-plane acceptance also requires IoU greater than 0.7, emphasizing horizontal localization over vertical localization.The 3D box is projected orthogonally onto the 2D ground plane.
- Reported measures: Both Average Precision and Average Orientation Similarity are evaluated for the reported metrics.These metrics are used for the offline evaluation described in the section.
- Performance comparison: The proposed approach achieves much better detection accuracy than [16] while using fewer layers and connections.The paper attributes this mainly to reduced scale variation and occlusion in the 3D embedding.
B. KITTI Online Evaluation
On KITTI’s online evaluation, the proposed method is compared with earlier point-cloud detectors and achieves a substantial performance advantage, though it remains slightly behind image-based algorithms.
- KITTI’s image-plane bounding-box-overlap metric introduces misalignment when evaluating projected 3D bounding boxes.
- Image-based methods benefit from higher image resolution, especially for detecting far or occluded objects.
- The proposed method outperforms previous point-cloud detection methods by a significant gap of > 20%.
- Its performance is comparable to image-based algorithms, though not yet as good.
V. CONCLUSIONS
The paper presents a 3D FCN framework for end-to-end 3D object detection in point clouds. It reports significant improvement over previous point-cloud approaches and notes applicability beyond Velodyne-based autonomous-driving data.
- The paper proposes what it identifies as the first 3D FCN framework for end-to-end 3D object detection.
- The method achieves significant performance improvement compared with previous point-cloud-based detection approaches.
- Although experiments use Velodyne 64E point clouds for autonomous driving, the framework naturally applies to other sensors or reconstruction algorithms.