Source-linked AI summary
Pillar-based Object Detection for Autonomous Driving
Yue Wang, Alireza Fathi, Abhijit Kundu, David Ross, Caroline Pantofaru, Thomas Funkhouser, Justin Solomon
TL;DR
3D autonomous-driving detection must handle sparse point clouds, while anchor-based methods require tuning and suffer severe imbalance. The paper introduces an anchor-free pillar-based detector with cylindrical multi-view features and bilinear pillar-to-point projection. On Waymo, it reports state-of-the-art results, including 6.87 3D mAP and 6.71 2D mAP improvements for vehicle detection.
Problem
Anchor-based 3D detectors require dataset-specific hyperparameter tuning and face severe imbalance because fewer than 0.1% of anchors are positive in typical point clouds.
Method
The model uses anchor-free per-pillar box prediction, cylindrical multi-view feature learning, and bilinear interpolation for pillar-to-point projection.
Results
6.87 3D mAP and 6.71 2D mAP improvements are reported for vehicle detection on the Waymo Open Dataset versus top performers.
Takeaways & Limitations
The results suggest that pillar-based detection and selecting appropriate feature views are effective design choices for autonomous-driving 3D detection.
Takeaways & Limitations
The paper identifies hand-designed view projection, 2D rather than 3D sparse convolutions, and the absence of refinement for small objects as future-work opportunities.
Abstract
from arXiv · showhide
We present a simple and flexible object detection framework optimized for autonomous driving. Building on the observation that point clouds in this application are extremely sparse, we propose a practical pillar-based approach to fix the imbalance issue caused by anchors. In particular, our algorithm incorporates a cylindrical projection into multi-view feature learning, predicts bounding box parameters per pillar rather than per point or per anchor, and includes an aligned pillar-to-point projection module to improve the final prediction. Our anchor-free approach avoids hyperparameter search associated with past methods, simplifying 3D object detection while significantly improving upon state-of-the-art.
1 Introduction
The paper proposes a pillar-based, anchor-free detector for autonomous driving that addresses anchor imbalance, view selection, and projection errors. On Waymo, it reports state-of-the-art performance with substantial 3D and 2D mAP improvements.
- 3D object detection supports autonomous-driving perception by identifying pedestrians, vehicles, obstacles, and other environmental features.
- Single-view feature learning can use suboptimal views, while anchor-based prediction introduces many hyperparameters and potentially unstable training.
- Fewer than 0.1% of anchors are positive in a typical point cloud, creating severe class imbalance.
- The anchor-free model predicts the best possible box directly for each bird’s-eye-view pillar, removing anchor assignment.This per-pillar prediction improves performance and is simpler than current state-of-the-art pipelines.
- A cylindrical view complements bird’s-eye view without the perspective distortions associated with spherical projection.Bilinear interpolation is used in pillar-to-point projection to avoid quantization errors.
- 6.87 3D mAP and 6.71 2D mAP improvements are reported for vehicle detection on the Waymo Open Dataset versus top performers.The authors also report that each proposed module outperforms its counterpart in ablation studies.
- The paper presents a pillar-based model and releases code to support reproducibility and future research.
2 Related Work
Related work spans 2D and 3D detection, pillar and point-cloud representations, multi-view learning, and two-stage or one-stage prediction. The paper positions pillar-centric predictions as an alternative to anchor-based 3D detection under sparse, large-scale driving settings.
- 2D object detection: 2D detection commonly combines backbone networks for latent features with detection heads that predict bounding boxes.
- 2D object detection: RCNN uses region proposals followed by CNN-based bounding-box regression, while Fast RCNN shares image features across proposals.
- 2D object detection: Mask RCNN adds mask prediction and uses bilinear interpolation in ROIAlign to avoid quantization error.
- Anchor-based detection: The paper replaces anchors with pillar-centric predictions to alleviate the instability associated with imbalance in 3D detection.
- Learning on point clouds: Point-based approaches learn geometric or flow features directly from points but are constrained by point count and cannot scale to large autonomous-driving settings.
- 3D object detection: VoxelNet voxelizes point clouds for dense 3D convolutions, while PIXOR and PointPillars organize points into vertical pillars.
- 3D object detection: MVF combines bird’s-eye-view pillars with complementary perspective-view information, but related anchor-based methods require task- and dataset-specific tuning.
3 Method
The model combines birds-eye and cylindrical views, aggregates their features, and predicts bounding boxes directly per birds-eye pillar without anchors. It also uses bilinear pillar-to-point projection to reduce spatial misalignment and quantization effects.
- Overall architecture: The architecture projects the point cloud into birds-eye and cylindrical views, learns view-specific features, aggregates them, and reprojects point-wise features to BEV.Final classification and regression networks operate on BEV pillar features.
- Cylindrical view: Cylindrical pillars group points sharing ϕ and z coordinates while avoiding the Z-axis distortion introduced by spherical projection.Spherical-view objects can lose physical scale, making distant cars smaller.
- Pillar-based prediction: The prediction module classifies each BEV pillar and regresses its box position, size, and heading directly, eliminating anchor assignment.The classification target is whether a pillar matches a ground-truth box.
- Pillar-based prediction: Anchor-based detection uses dense anchors with IoU-based positive and negative assignment, while the proposed method predicts directly from each pillar.This changes the prediction unit from an anchor-associated target to the pillar itself.
- Bilinear interpolation: Nearest-neighbor pillar-to-point projection can create quantization and spatial misalignment, so the model uses bilinear interpolation for spatially consistent features.Points in the same pillar otherwise receive identical copied pillar features despite different locations.
- Loss function: The training objective combines pillar classification and regression losses, using focal loss for pillar classification and the specified regression loss for box targets.The focal-loss parameters are α = 0.25 and γ = 2, with σ = 3.0 for the regression loss.
4 Experiments
Experiments on the Waymo Open Dataset evaluate the model against strong baselines and isolate the effects of prediction heads, view projections, and pillar-to-point interpolation. The proposed design improves vehicle and pedestrian detection, with especially large gains for distant vehicles.
- 4.1 Results compared to state-of-the-art: The evaluation uses Waymo vehicle and pedestrian detection, official 3D and BEV mAP metrics, distance breakdowns, and comparisons with PointPillars, MVF, StarNet, and LaserNet.The dataset contains 798 training sequences and 202 validation sequences; vehicle and pedestrian results are reported separately.
- 4.1 Results compared to state-of-the-art: The proposed method achieves significantly better results for both pedestrians and vehicles, with larger improvements for distant vehicles at 30m–Inf.The paper links the distant-vehicle pattern to more serious anchor imbalance because distant anchors are less likely to match ground-truth boxes.
- 4.2 Comparing anchor-based, point-based, and pillar-based prediction: Pillar-based prediction performs best among anchor-, point-, and pillar-based heads, balancing coarse anchor-level and fine point-level prediction.The comparison keeps the overall architecture the same while changing the detection head.
- 4.3 View combinations: The cylindrical view outperforms alternative single views, especially at 50m–Inf, and remains strongest when combined with the bird’s-eye view.Spherical projection distorts the Z-axis, while XZ projection introduces Y-axis occlusions; adding further views does not improve the BEV-plus-cylindrical combination.
- 4.4 Bilinear interpolation: Bilinear interpolation is evaluated against nearest-neighbor interpolation for pillar-to-point projection to address spatial misalignment from assigning identical features within a pillar.Bilinear interpolation samples four neighboring pillars and uses a weighted average rather than copying the closest pillar feature.
5 Discussion
The model achieves state-of-the-art results on a large public 3D detection dataset, while the experiments identify limitations and several directions for future improvement.
- The model achieves state-of-the-art results on the largest publicly available 3D object detection dataset.
- The findings suggest that designs from 2D object detection and visual recognition do not directly transfer to 3D scenarios.
- Correct view selection is important for model performance.
- Future work could learn view transformations from data, use 3D sparse convolutions, or add refinement for small objects.
- Instance segmentation is proposed as a possible extension for fine-grained 3D recognition and robotic manipulation.
A Supplementary Material
The supplementary material specifies the model's three-part structure and directs readers to Figure 5 and Table 7 for architectural and parameter details.
- The model consists of a multi-view feature learning network, a bird's-eye view pillar backbone, and a detection head.
- Table 7 provides convolutional-kernel parameters and feature-map sizes.
- Figure 5 presents the multi-view feature learning module, pillar backbone network, and classification and regression detection networks.