PointPillars: Fast Encoders for Object Detection from Point Clouds

Alex H. Lang; Sourabh Vora; Holger Caesar; Lubing Zhou; Jiong Yang; Oscar Beijbom

Source-linked AI summary

PointPillars: Fast Encoders for Object Detection from Point Clouds

Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, Oscar Beijbom

arXiv:1812.05784v2Dec 14, 2018cs.LGcs.CVstat.ML

TL;DR

Point-cloud detection requires an effective encoding because bird’s-eye-view representations are sparse and hand-crafted features may not generalize. PointPillars learns pillar-based features end to end with 2D convolutions, achieving higher KITTI BEV and 3D detection performance at faster speed, including 62 Hz inference.

Problem
Sparse bird’s-eye-view point clouds make convolutional processing inefficient, while hand-crafted encoders may not generalize without substantial engineering.
Method
PointPillars learns pillar-based point-cloud features end to end, converts them into a sparse pseudo-image, and uses a 2D convolutional detection network.
Results
62 Hz inference accompanies higher KITTI detection performance than existing methods on both bird’s-eye-view and 3D metrics.
Takeaways & Limitations
PointPillars offers a fast lidar-only architecture for 3D object detection with strong KITTI benchmark performance.
Takeaways & Limitations
KITTI timing uses only approximately 10% of the point cloud, while full-environment processing and embedded hardware may substantially increase runtime.

Abstract

from arXiv · show

Object detection in point clouds is an important aspect of many robotics applications such as autonomous driving. In this paper we consider the problem of encoding a point cloud into a format appropriate for a downstream detection pipeline. Recent literature suggests two types of encoders; fixed encoders tend to be fast but sacrifice accuracy, while encoders that are learned from data are more accurate, but slower. In this work we propose PointPillars, a novel encoder which utilizes PointNets to learn a representation of point clouds organized in vertical columns (pillars). While the encoded features can be used with any standard 2D convolutional detection architecture, we further propose a lean downstream network. Extensive experimentation shows that PointPillars outperforms previous encoders with respect to both speed and accuracy by a large margin. Despite only using lidar, our full detection pipeline significantly outperforms the state of the art, even among fusion methods, with respect to both the 3D and bird's eye view KITTI benchmarks. This detection performance is achieved while running at 62 Hz: a 2 - 4 fold runtime improvement. A faster version of our method matches the state of the art at 105 Hz. These benchmarks suggest that PointPillars is an appropriate encoding for object detection in point clouds.

1. Introduction

PointPillars addresses the sparsity and dimensionality challenges of lidar point-cloud detection with a learned pillar-based encoder and end-to-end 2D-convolutional processing. On KITTI, it achieves state-of-the-art BEV and 3D detection while enabling substantially faster inference.

Motivation: Lidar point clouds are sparse 3D representations, unlike dense 2D images, complicating direct application of conventional convolutional detection methods.Bird’s-eye-view projection reduces scale ambiguity and occlusion, but remains extremely sparse for direct convolutional processing.
Method: PointPillars learns features from vertical point-cloud columns, or pillars, to predict 3D oriented object boxes using only 2D convolutional layers.The learned encoder replaces hand-crafted feature extraction and enables end-to-end training.
Evaluation: A faster PointPillars setting matches state-of-the-art performance at over 100 Hz.The paper also reports released code for reproducing the results.
Efficiency: 62 Hz inference is achieved by posing all pillar computations as dense 2D convolutions, making the method 2-4 times faster than other methods.This design supports a speed-accuracy trade-off in the downstream detection network.
Evaluation: State-of-the-art results are demonstrated for cars, pedestrians, and cyclists on both BEV and 3D KITTI benchmarks using lidar point clouds alone.The network dominates current state-of-the-art methods, including methods that use lidar and images.

2. PointPillars Network

PointPillars encodes point clouds into a sparse pseudo-image, processes it with a 2D convolutional backbone, and uses an SSD-style detection head to estimate oriented 3D boxes. Its pillar encoder discretizes the x-y plane, applies PointNet-style feature aggregation, and scatters encoded features back to their original locations.

Network overview: PointPillars has three stages: feature encoding to a sparse pseudo-image, 2D convolutional processing, and 3D box detection and regression.It estimates oriented 3D boxes for cars, pedestrians, and cyclists.
Pillar encoding: The point cloud is discretized into B evenly spaced x-y pillars without a z-dimension binning hyperparameter.Each point is augmented with offsets describing distances to pillar and point means.
Pillar encoding: 6k-9k non-empty pillars arise at 0.162 m2 bins for an HDL-64E lidar, with ∼97% sparsity in the typical KITTI range.This sparsity motivates limiting non-empty pillars and points per pillar to form a dense tensor of size (D, P, N).
Pillar encoding: A simplified PointNet applies linear layers, BatchNorm, and ReLU per point, then max-pools over channels to produce a (C, P) tensor.The linear layer can be implemented as a 1x1 convolution for efficient computation.
2D backbone: Encoded pillar features are scattered to their original locations, producing a pseudo-image of size (C, H, W) for 2D convolution.The backbone combines increasingly low-resolution top-down features through transposed-convolution upsampling and concatenation.
Detection head: The SSD-style detection head matches prior boxes to ground truth using 2D IoU, while box height and elevation are regressed separately.Height and elevation are not used for matching.

3. Implementation Details

The implementation uses a shared three-block backbone design with task-specific initial strides, concatenated multiscale features, and SECOND-style losses. Training uses focal classification, discretized-direction supervision, Adam optimization, scheduled learning-rate decay, and fixed batch sizes.

Network architecture: The encoder outputs C = 64 features, and three blocks are upsampled and concatenated to produce 6C detection features.The blocks are Block1(S, 4, C), Block2(2S, 6, 2C), and Block3(4S, 6, 4C), with corresponding Up1, Up2, and Up3 steps.
Network architecture: The car and pedestrian/cyclist backbones differ only in the first-block stride: S = 2 for cars and S = 1 for pedestrians/cyclists.Both backbones use the same three-block and upsampling structure.
Loss function: The loss follows SECOND, combining localization regression, discretized-direction softmax classification, and focal object classification.The direction loss addresses flipped boxes that angle localization cannot distinguish.
Loss function: α = 0.25 and γ = 2 are used for focal loss, with total-loss weights βloc = 2, βcls = 1, and βdir = 0.2.The focal-loss probability is defined for each anchor, and Npos denotes the number of positive anchors.
Optimization: The model is trained with Adam at an initial learning rate of 2 ∗10−4, decayed by 0.8 every 15 epochs for 160 epochs.The batch size is 2 for validation and 4 for the test submission.

4. Experimental setup

Experiments use KITTI lidar point clouds with a defined train/validation split and fixed encoding, anchor, suppression, and class-specific spatial settings. Training augmentation combines object insertion, per-box perturbations, and global transformations.

Dataset: KITTI provides 7481 training and 7518 testing samples; experiments split training into 3712 training and 3769 validation samples, with lidar-only training.Comparisons include fusion methods using both lidar and images; a 784-sample minival set was created for test submission.
Experimental settings: 0.16 m xy resolution, 12000 maximum pillars, and 100 maximum points per pillar are used unless varied experimentally.These settings define the default point-cloud encoding configuration.
Experimental settings: Anchors use two orientations, 0 and 90 degrees, with 2D IoU matching; highest-overlap or above-threshold matches are positive, below-threshold matches negative, and others ignored.Each anchor specifies width, length, height, and z center, following the strategy of [31].
Experimental settings: 0.5 IoU is the axis-aligned NMS overlap threshold at inference, providing similar performance to rotational NMS while being much faster.The suppression choice is made for inference efficiency.
Data augmentation: 15, 0, 8 ground-truth samples for cars, pedestrians, and cyclists are randomly inserted per sample, respectively, using associated points from a 3D-box lookup table.The authors report these insertion settings outperform the settings proposed in.
Data augmentation: Per-box augmentation rotates boxes within [−π/20, π/20] and translates coordinates from N(0, 0.25), followed by global flipping, rotation, scaling, and translation from N(0, 0.2).Global transformations are jointly applied to the point cloud and all boxes; the final translation simulates localization noise.

5. Results

PointPillars outperforms published methods on KITTI detection benchmarks, achieving strong moderate-stratum BEV and 3D AP while producing accurate oriented 3D boxes. Its remaining errors mainly involve difficult, occluded, or distant objects and confusion with similar classes.

Quantitative Analysis: PointPillars outperforms all published methods in mAP, exceeding lidar-only methods across classes and difficulty strata except easy cars and surpassing fusion methods on cars and cyclists.The KITTI leaderboard ranks methods by moderate difficulty performance.
Quantitative Analysis: PointPillars significantly exceeds the only two oriented-box 3D detection methods on AOS across all difficulty strata.AOS evaluates orientation after projecting 3D boxes into the image and matching 2D detections.
Quantitative Analysis: 87.98, 63.55, and 69.71 are PointPillars’ moderate-stratum BEV AP values for cars, pedestrians, and cyclists, respectively.The corresponding moderate-stratum 3D AP values are 77.98, 57.86, and 66.02.
Qualitative Analysis: Qualitative results show tight oriented 3D bounding boxes, with particularly accurate car predictions.The visualizations present BEV and image perspectives despite training only on lidar point clouds.
Qualitative Analysis: Common qualitative failures are false negatives for partially occluded or faraway objects and false positives on similar classes.The reported predictions are visualized from both BEV and image perspectives.

6. Realtime Inference

PointPillars achieves substantial realtime-inference speed through a streamlined pipeline and design choices that reduce encoder and network runtime without affecting detection performance. TensorRT further accelerates the PyTorch pipeline, while a faster configuration reaches 105 Hz with limited accuracy loss.

Inference Steps: 2.9 ms is required to upload the PointPillar tensor to the GPU, compared with 2.7 ms for pillar organization and decoration.The pipeline also spends 1.4 ms loading and filtering the point cloud, 1.3 ms encoding it, and 0.1 ms scattering it to the pseudo-image.
Inference Steps: 48 ms is required by the compared encoder in its open-source implementation, suggesting PointPillars is substantially faster.The comparison is based on an architecture whose remaining components are similar, although its authors did not provide a runtime analysis.
Slimmer Design: 2.5 ms, 4.5 ms, and 3.9 ms were saved by using one PointNet, lowering the first-block dimensions, and halving upsampled feature-layer outputs to 128.These slimmer-design changes did not affect detection performance.
TensorRT: 45.5% speedup results from switching the 42.4 Hz PyTorch pipeline to NVIDIA TensorRT for the encoder, backbone, and detection-head GPU kernels.The final GPU kernels were built with TensorRT, an optimized GPU-inference library.
The Need for Speed: PointPillars reaches 105 Hz with limited loss of accuracy.The car network is the focus, and pedestrian and bicycle networks run at similar speeds; runtimes were measured using an Intel i7 CPU and 1080ti GPU.

7. Ablation Studies

The ablations show that PointPillars offers tunable speed–accuracy trade-offs and benefits from minimal augmentation, additional point decorations, and learned encoding. At comparable inference time, PointPillars provides a better operating point than slower learned alternatives.

Spatial binning: Larger pillars accelerate inference by reducing non-empty pillars and pseudo-image size, while smaller pillars improve localization through finer spatial resolution.The study varied spatial binning to quantify this speed–accuracy trade-off.
Spatial binning: 105 Hz was achieved at 0.282 m^2 with performance similar to previous methods, while performance declines mainly for pedestrians and cyclists and remains stable for cars.The grid-size sweep found that larger bin sizes produce faster networks.
Data augmentation: Minimal box augmentation outperformed extensive augmentation, with additional augmentation significantly degrading pedestrian detection.The authors hypothesize that ground-truth sampling reduces the need for extensive per-box augmentation.
Point decorations: Adding x_p and y_p offsets from each pillar’s x,y center to the VoxelNet decorations improved final detection performance by 0.5 mAP and made experiments more reproducible.These offsets were added during lidar point decoration.
Encoder comparison: Learned encoders strictly outperformed fixed encoders at every resolution, with the gap increasing for larger bin sizes.The comparison used the same network architecture and training procedure while changing only the encoder and xy resolution.
Encoder comparison: VoxelNet was marginally stronger than PointPillars among learned encoders, but PointPillars offered the better operating point at similar inference time because VoxelNet was orders of magnitude slower and larger.The comparison was performed in the official SECOND codebase with several encoders.

8. Conclusion

The paper introduces PointPillars, an end-to-end trainable deep network and encoder for lidar point clouds. On KITTI, it achieves higher BEV and 3D mAP at faster speed than existing methods, supporting its status as the best architecture so far for lidar-based 3D object detection.

Conclusion: PointPillars is a novel deep network and encoder trained end-to-end on lidar point clouds.The method combines the network and encoding process within an end-to-end trainable system.
Conclusion: On KITTI, PointPillars dominates existing methods with higher detection performance in BEV and 3D mAP at faster speed.The comparison covers both bird’s-eye-view and three-dimensional detection performance.
Conclusion: The results identify PointPillars as the best architecture so far for 3D object detection from lidar.This conclusion follows the reported KITTI performance and speed advantages over existing methods.