Source-linked AI summary
PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection
Guangsheng Shi, Ruifeng Li, Chao Ma
TL;DR
Top-performing LiDAR-based 3D detectors entail heavy computational and memory demands, motivating real-time onboard deployment. PillarNet addresses this with a 2D-convolutional encoder–neck detector and achieves state-of-the-art performance on nuScenes and Waymo while running in real time.
Problem
Top-performing LiDAR-based 3D detectors entail heavy computational loads and large memory requirements, motivating real-time onboard deployment.
Method
PillarNet uses hierarchical pillar feature encoding, multi-scale encoder–neck fusion, configurable pillar scales, and orientation-decoupled IoU-related modules.
Results
PillarNet achieves state-of-the-art performance on nuScenes and Waymo, with nuScenes variants running at 14, 13, and 12 FPS.
Takeaways & Limitations
PillarNet provides a scalable and flexible real-time one-stage detector using only 2D convolutions, supporting further pillar-based point cloud representation research.
Takeaways & Limitations
The IoU-related regression training can settle into a local optimum when box centers deviate far, and size variation affects orientation regression.
Abstract
from arXiv · showhide
Real-time and high-performance 3D object detection is of critical importance for autonomous driving. Recent top-performing 3D object detectors mainly rely on point-based or 3D voxel-based convolutions, which are both computationally inefficient for onboard deployment. In contrast, pillar-based methods use solely 2D convolutions, which consume less computation resources, but they lag far behind their voxel-based counterparts in detection accuracy. In this paper, by examining the primary performance gap between pillar- and voxel-based detectors, we develop a real-time and high-performance pillar-based detector, dubbed PillarNet.The proposed PillarNet consists of a powerful encoder network for effective pillar feature learning, a neck network for spatial-semantic feature fusion and the commonly used detect head. Using only 2D convolutions, PillarNet is flexible to an optional pillar size and compatible with classical 2D CNN backbones, such as VGGNet and ResNet. Additionally, PillarNet benefits from our designed orientation-decoupled IoU regression loss along with the IoU-aware prediction branch. Extensive experimental results on the large-scale nuScenes Dataset and Waymo Open Dataset demonstrate that the proposed PillarNet performs well over state-of-the-art 3D detectors in terms of effectiveness and efficiency. Code is available at \url{https://github.com/agent-sgs/PillarNet}.
1 Introduction
PillarNet targets real-time, high-performance 3D detection by strengthening pillar feature encoding while retaining an encoder–neck–head design based on 2D convolutions. Its staged architecture supports flexible pillar sizes and multiple backbone variants.
- Top-performing point-cloud detectors impose heavy computation and memory costs, motivating real-time onboard 3D detection.
- PillarNet attributes the main performance gap of prior pillar methods to insufficient pillar feature encoding.
- PillarNet combines hierarchical pillar feature extraction, multi-scale feature fusion, and a center-based detection head.
- PillarNet accommodates larger pillar sizes by detaching corresponding downsampled encoder stages, such as removing the 1x, 2x, and 4x stages for 8x pillars.
- PillarNet variants use only 2D convolutions, support VGG- and ResNet-based configurations, achieve state-of-the-art performance on two benchmarks, and run in real time.
2 Related Works
Point-cloud 3D detectors use point-based or grid-based representations, with grid-based methods commonly operating on 3D voxels or 2D pillars. Voxel methods dominate benchmark performance but incur resource costs, while pillar methods offer efficient 2D-convolution deployment.
- Point-cloud 3D object detectors are broadly categorized as point-based or grid-based methods.
- Point-based methods process irregular point clouds directly and predict 3D bounding boxes, with proposal generation and refinement used by representative systems.
- Grid-based methods discretize sparse point clouds into regular 3D voxels or 2D pillars before applying 2D or 3D convolutions.
- Voxel-based methods dominate most 3D detection benchmarks, but balancing fine voxel resolution against resource costs remains difficult.
- Pillar-based methods use ground-plane voxelization and per-pillar feature extraction to enable 2D-convolution deployment on resource-limited embedded systems.
- Multi-sensor approaches fuse camera and LiDAR information at stages including BEV representation, proposal generation, or feature projection.
3 PillarNet for 3D Object Detection
PillarNet addresses the accuracy gap of pillar-based detectors by strengthening sparse pillar encoding and spatial-semantic fusion while retaining 2D convolutions. It also introduces orientation-decoupled IoU regression to reduce coupling among box parameters.
- Architecture: PillarNet treats grid-based detectors as BEV-based systems and retains the encoder-neck-head architecture for pillar-based 3D detection.Its redesign focuses on separating the roles of encoder and neck networks.
- Architecture: Compared with SECOND, PillarNet uses sparse 2D rather than sparse 3D convolution in the encoder, while both architectures use standard 2D CNNs in the neck.On nuScenes, PillarNet uses a (0.075m, 0.075m) pillar size, whereas PointPillars uses (0.2m, 0.2m).
- Encoder design: The encoder uses hierarchical sparse 2D convolutions for deep pillar feature extraction, replacing the sparse 3D convolutions used by SECOND.The encoder progressively downsamples sparse pillar features and can incorporate VGGNet or ResNet-style backbones.
- Encoder design: Progressive downsampling decouples output feature-map size from the initial pillar projection scale, allowing PillarNet to operate with different pillar sizes.This addresses the limitation that previous pillar-based methods coupled pillar size to network design.
- Neck design: PillarNet fuses low-level spatial features from the sparse encoder with high-level semantic features from 16× downsampled dense maps in its neck.The neck uses convolution layers to exchange information between spatial and semantic features for robust feature extraction.
- Orientation-Decoupled IoU Regression Loss: The orientation-decoupled IoU-related regression loss separates orientation θ from the mutually coupled box parameters (x, y, z, w, l, h, θ).It extends IoU, GIoU, and DIoU regression losses to decouple orientation.
4 Experiments
PillarNet is evaluated on nuScenes and Waymo using real-time LiDAR-only detection settings, with experiments covering overall performance, ablations, pillar sizes, and runtime. The results show strong benchmark performance and gains from its encoder, neck, IoU-related modules, and scalable configurations.
- Overall Results: PillarNet variants surpass prior LiDAR-only non-ensemble methods on nuScenes while running at 12–14 FPS.PillarNet-18 exceeds AFDetV2 by +2.3% NDS or +2.6% mAP.
- Overall Results: PillarNet-34 outperforms previous single-frame and multi-frame LiDAR-only models for vehicle and pedestrian detection on Waymo at 19 FPS.PillarNet-vgg reaches comparable vehicle performance at 24 FPS.
- Overall Results: PillarNet-34 gains +1.24 mAPH for the vehicle category at LEVEL 2 difficulty on the Waymo validation set.PillarNet-18 achieves state-of-the-art vehicle performance, while two-frame inputs generally improve results except for cyclist detection.
- Ablation Studies: The encoder improves performance by about +7.61% mAP and +5.94% NDS over CenterPoint-PointPillars.Adding encoder stage 5 further boosts detection through enriched semantic features.
- Ablation Studies: The proposed neck improves substantially over a naive neck, while dense convolutions provide stronger semantic abstraction than sparse counterparts.The reported explanation attributes this to LiDAR points sparsely scattering on object surfaces.
- Ablation Studies: OD-DIoU provides the largest loss-related gain, improving either mAP or NDS by +0.24%.OD-IoU, OD-GIoU, and OD-DIoU all contribute to positioning accuracy.
- Ablation Studies: IoU-Aware rectification increases IoU-based mAP by +0.34% by alleviating misalignment between localization confidence and classification score.The rectification incorporates IoU information into confidence scores during post-processing.
- Ablation Studies: PillarNet scales with model capacity and pillar configuration, trading slightly more FLOPs and inference time for higher performance.Finer pillars and deeper pillar feature encoding outperform PointPillars configurations with coarser or shallower encoding.
5 Conclusions
The paper proposes PillarNet as a real-time, high-performance one-stage 3D object detector. Its encoder-neck-head design provides scalability and flexibility for pillar size and model complexity.
- PillarNet is a real-time, high-performance one-stage 3D object detector.
- Its encoder-neck-head architecture supports scalability and flexibility across pillar sizes and model complexities.
- The authors expect these findings to stimulate further research into pillar-based point cloud representation learning.