Source-linked AI summary
Vehicle Detection from 3D Lidar Using Fully Convolutional Network
Bo Li, Tianlei Zhang, Tian Xia
TL;DR
Vehicle detection from 3D lidar requires accurate 3D localization, while modern FCN detection techniques had not been fully transferred to range-scan perception. The paper projects Velodyne 64E scans into 2D point maps and uses an end-to-end FCN to predict objectness and full 3D boxes simultaneously. On KITTI, it reports strong detection performance, including higher easy-task AP than other compared methods and substantially better AOS than CSoR.
Problem
Accurate 3D localization is crucial for autonomous-vehicle planning and control, but a gap remained between vision-based FCN detection and robotic perception on range scans.
Method
The method projects Velodyne 64E range scans into 2D point maps and uses a unified FCN with objectness and 3D bounding-box prediction branches.
Results
The approach outperforms other compared methods in KITTI easy-task detection AP and significantly wins the AOS comparison against CSoR.
Takeaways & Limitations
The framework provides end-to-end vehicle detection from range scans and can be applied to similar lidar devices.
Abstract
from arXiv · showhide
Convolutional network techniques have recently achieved great success in vision based detection tasks. This paper introduces the recent development of our research on transplanting the fully convolutional network technique to the detection tasks on 3D range scan data. Specifically, the scenario is set as the vehicle detection task from the range data of Velodyne 64E lidar. We proposes to present the data in a 2D point map and use a single 2D end-to-end fully convolutional network to predict the objectness confidence and the bounding boxes simultaneously. By carefully design the bounding box encoding, it is able to predict full 3D bounding boxes even using a 2D convolutional network. Experiments on the KITTI dataset shows the state-of-the-art performance of the proposed method.
I. INTRODUCTION
3D lidar range scans provide spatial coordinates that support object localization, but applying modern vision detection methods to robotic perception remains limited. The paper addresses this gap by adapting an FCN for 3D vehicle detection from Velodyne 64E data.
- Background: Range-scan research has addressed localization, mapping, object detection, and scene parsing on 3D point clouds.
- Motivation: 3D range scans make detected-object pose and shape easier to obtain than camera images because they directly contain spatial coordinates.Accurate 3D localization is important for subsequent planning and control in autonomous vehicles.
- Contribution: The proposed work designs an FCN to detect and localize vehicles as 3D boxes from Velodyne 64E range scans, with possible application to similar lidar devices.
- Prior approaches: Traditional range-scan detectors often segment point clouds into candidate clusters before classifying them, making them vulnerable to incorrect segmentation.Segmentation may use rules, clustering, graphs, or hierarchical representations.
- Prior approaches: Earlier candidate classifiers relied on known shape models or hand-crafted geometric features such as spin images, shape factors, distributions, and FPFH.
B. Convolutional Neural Network on Object Detection
The approach represents lidar range scans as 2D maps and transplants unified FCN detection ideas from image and RGBD vision. It predicts objectness and 3D bounding boxes end to end from the resulting point map.
- Pipeline visualization: Figure 1 visualizes the pipeline from the input point map through confidence predictions, positive-point box candidates, and non-max-suppressed boxes.
- Detection framework: Unified FCN frameworks predict objectness confidence and bounding boxes simultaneously across the input, avoiding a separate candidate-verification stage.The paper adapts this design to range scans and predicts 3D object bounding boxes.
- Data representation: The method projects 3D range scans into 2D maps analogous to RGBD depthmaps for convolutional processing.The Velodyne 64E scan is discretized using point angles and map resolutions.
- Data representation: Each projected map element stores two channels, d and z, while empty positions are filled with (d, z) = (0, 0).The d channel couples x and y for rotation invariance around the z axis; nearer points are retained in projection collisions.
B. Network Architecture
The network uses a downsampling and upsampling convolutional trunk that splits into objectness and 3D box-regression branches. Point-wise outputs preserve the correspondence between map positions, vehicle confidence, and predicted boxes.
- Network architecture: The CNN trunk downsamples through convolutional layers, upsamples through deconvolutional layers, then splits into objectness classification and 3D box-regression branches.
- Point-wise prediction: The input, objectness, and bounding-box maps share width and height, enabling each map element to predict whether its corresponding point lies on a vehicle.For positive points, the corresponding box-map element predicts the 3D bounding box of that vehicle.
- Resolution design: The first layer uses horizontal and vertical downsampling factors of 4 and 2, matching the approximate Velodyne 64E resolution relation ∆φ = 2∆θ.Later convolutional and deconvolutional layers use equal horizontal and vertical resolutions with squared strides of (2, 2).
- Feature fusion: Feature maps from lower and higher layers are concatenated before subsequent layers to improve prediction of small objects and object edges.
C. Prediction Encoding
The FCN predicts foreground confidence and encodes each vehicle’s full 3D box through point-relative, rotation-aware corner coordinates. This design converts box prediction into a 24-dimensional representation at each point.
- Objectness encoding: The objectness output has foreground and background channels normalized by softmax to represent confidence that a point lies on a vehicle.
- Coordinate transform: A rotation matrix is formed from the point’s azimuth and elevation observation angles before transforming bounding-box corners.Its component axes align one direction with the observed point and another parallel to the horizontal plane.
- Bounding-box encoding: For a vehicle point, the box branch encodes the vehicle’s eight corners as a 24-dimensional vector.The corresponding deconvolutional output therefore contains 24 channels.
- Encoding rationale: The encoding uses point-relative corner offsets because they occupy a smaller range than absolute corner coordinates across the lidar perception area.The paper states that this smaller range is easier for the CNN to learn.
- Encoding rationale: Applying R⊤ makes corner-coordinate encoding rotation invariant when vehicles have the same observed range-scan appearance from different viewpoints.
D. Training Phase
The training data are augmented with random 3D transformations near identity before projection, preserving the lidar’s geometric properties.
- Random 3D transforms near identity augment range scans before point-map projection.Directly applying image-style zooming or rotation would create variable Δθ and Δφ and violate lidar geometry.
2) Multi-Task Training:
The network jointly predicts point-level vehicle objectness and corresponding 3D bounding boxes through separate classification and regression branches.
- The network has separate objectness-classification and bounding-box-regression branches.Their losses are accumulated during training.
- Each point’s objectness output labels it as background or vehicle point using a softmax loss.The ground-truth label is 0 for background and 1 for points on vehicles.
- The bounding-box branch uses an L2-norm loss for a 24-dimensional target vector.The bounding-box loss is computed only for points on vehicles.
3) Training strategies:
Training balances severe class and distance-related point imbalance by reweighting negative samples and vehicle losses according to object size.
- Negative background losses are reweighted because background points greatly outnumber vehicle points.With k = 4, the reweighted negative losses are equivalent on average to losses from k|V| negative samples while retaining all negatives.
- The method chooses k = 4 for balancing negative objectness losses.Unlike random negative-sample discarding, reweighting keeps more negative-sample information.
- Vehicle losses are reweighted by point count to reduce bias toward near vehicles and neglect of far or occluded vehicles.A vehicle’s point count also represents its area on the projected point map.
- The final training loss accumulates weighted objectness and bounding-box losses.The weight w_box balances the objectness loss against the bounding-box loss.
E. Testing Phase
During testing, positive point predictions generate 3D bounding-box candidates, which are transformed back to corners and consolidated by clustering with non-max suppression.
- The network produces an objectness map and bounding-box map from each test range scan.Only points predicted positive in the objectness map generate box candidates.
- Each positive point’s predicted box representation is converted into box corners by the inverse transform.The resulting 24-dimensional vector represents a bounding-box candidate.
- Bounding-box candidates are clustered and outliers are pruned using a non-max-suppression strategy.Candidates are ranked by neighboring-box counts; candidates scoring below 5 are discarded.
- Figure 4 illustrates detections in congested traffic and for far vehicles.
IV. EXPERIMENTS
The experiments evaluate vehicle detection on KITTI range scans using front-facing Velodyne data, with both image-space and world-space criteria.
- Evaluation Setup: The KITTI benchmark provides Velodyne 64E range scans with corresponding 3D vehicle boxes for training and testing.The dataset contains more than 7,500 frames, with 6,000 selected for training.
- Evaluation Setup: Experiments focus on the Car category, labeling points inside ground-truth car boxes as foreground and Truck or Van points as ignored.
- Evaluation Setup: Only the front part of each 360° range scan is used because KITTI annotations cover vehicles visible in the front camera.
- Evaluation Criteria: Table I reports average precision and average orientation similarity for the offline evaluation.
A. Performane Analysis on Offline Evaluation
Offline evaluation compares image-space and world-space criteria for 3D vehicle detections, showing stronger performance in world space but reduced performance for distant, sparsely scanned vehicles.
- Criterion Comparison: The offline evaluation derives image-space boxes by projecting detected and ground-truth 3D boxes and taking the minimum enclosing 2D rectangle.World-space overlap is computed after projecting the 3D boxes onto the ground plane, with at least 70% overlap required.
- Criterion Comparison: World-space evaluation produces slightly better performance than image-space evaluation because image annotations are often tighter than projected 3D boxes.The difference is especially pronounced for vehicles viewed diagonally, reducing image-space overlap.
- Difficulty Analysis: Performance drops from easy to moderate and hard evaluation because distant vehicles are represented by very few lidar points.Easy samples have a 40-pixel minimum height and approximately correspond to vehicles within 28 m; moderate and hard samples use 25 pixels and approximately 47 m.
- Detection Behavior: The method predicts complete boxes for partly visible vehicles and achieves maximum recall above 95% in the easy evaluation.False positives occur in some occluded scenes.
B. Related Work Comparison on the Online Evaluation
The online evaluation compares the proposed range-scan FCN with traditional-feature methods on KITTI. It performs best on easy detection and orientation similarity, while moderate and hard detection remain comparable to a prior method.
- Motivation for Comparison: Prior range-scan methods rank below state-of-the-art vision methods partly because image data have higher resolution for far and occluded objects.
- Online Results: Table II compares methods using average precision and average orientation similarity on the KITTI online evaluation.
- Online Results: The proposed approach outperforms the compared methods in easy-task detection average precision.The authors attribute this to CNN feature representation for nearby vehicles.
- Online Results: In moderate and hard detection tasks, the approach achieves similar average precision to Wang and Posner.The authors relate this to vehicles containing too few points for CNNs to embed complicated features.
- Online Results: For joint detection and orientation estimation, only the proposed approach and CSoR support orientation estimation, and the proposed approach significantly wins in average orientation similarity.
- Conclusion: The proposed method is presented as the first FCN detection framework for range-scan perception and is evaluated on Velodyne 64E data, with stated applicability to similar devices.