Source-linked AI summary
PillarNeXt: Rethinking Network Designs for 3D Object Detection in LiDAR Point Clouds
Jinyu Li, Chenxu Luo, Xiaodong Yang
TL;DR
LiDAR 3D detection commonly emphasizes fine-grained local geometry, but the paper investigates whether computational resources are better allocated differently. It compares aggregators and modernizes architecture and training, finding that enhanced pillar networks achieve strong accuracy-latency results and state-of-the-art benchmark performance.
Problem
Existing 3D detectors emphasize specialized local point aggregators, while network architecture and computational-budget allocation remain comparatively underexplored.
Method
The paper systematically compares pillar, voxel, and multi-view fusion aggregators and applies minimal 2D-inspired architectural and training adaptations, including receptive-field enlargement.
Results
Enhanced pillar models achieve comparable or better accuracy than voxel models while running faster, and PillarNeXt attains state-of-the-art results on two popular benchmarks.
Takeaways & Limitations
The findings challenge the belief that high-performing LiDAR 3D detection requires fine-grained local geometrical modeling and identify receptive-field enlargement and resolution manipulation as key design choices.
Abstract
from arXiv · showhide
In order to deal with the sparse and unstructured raw point clouds, LiDAR based 3D object detection research mostly focuses on designing dedicated local point aggregators for fine-grained geometrical modeling. In this paper, we revisit the local point aggregators from the perspective of allocating computational resources. We find that the simplest pillar based models perform surprisingly well considering both accuracy and latency. Additionally, we show that minimal adaptions from the success of 2D object detection, such as enlarging receptive field, significantly boost the performance. Extensive experiments reveal that our pillar based networks with modernized designs in terms of architecture and training render the state-of-the-art performance on the two popular benchmarks: Waymo Open Dataset and nuScenes. Our results challenge the common intuition that the detailed geometry modeling is essential to achieve high performance for 3D object detection.
1. Introduction
The paper revisits local point aggregation and network architecture for LiDAR 3D detection, asking how computational resources should be allocated. It finds that enhanced pillar models can match or surpass more geometry-focused alternatives while modernized architectures improve performance.
- LiDAR 3D object detection supports downstream autonomous-driving modules including perception, prediction, and planning.
- Existing work emphasizes specialized local point aggregators, while network architecture remains largely based on original SECOND or PointPillars designs.
- The paper compares pillar, voxel, and multi-view fusion aggregators through computational-budget allocation rather than geometry modeling alone.
- Under comparable budgets and enhanced training, simpler pillar models achieve superior or on-par performance with voxel models, including for pedestrians, while outperforming multi-view fusion.
- Minimal adaptations from 2D detection, especially enlarging receptive fields, outperform many specialized point-cloud architectures.
- PillarNeXt develops scalable pillar networks with accuracy-latency trade-offs and reports state-of-the-art results on two popular benchmarks.
2. Related Work
Related work covers point, grid, and hybrid representations for LiDAR detection, while highlighting that modern feature-fusion, receptive-field, and scaling techniques remain underexplored in 3D detection.
- LiDAR detection methods are categorized as point, grid, or hybrid representations according to their local point aggregators.
- Point-based methods can be expensive on large-scale autonomous-driving point clouds, whereas grid methods enable efficient 2D or 3D convolutional processing.
- Hybrid methods were proposed because grid representations were commonly believed to lose fine-grained information, but this paper instead studies architecture and training.
- Feature fusion and receptive-field techniques widely used in 2D detection have received limited discussion and adoption in 3D detection.
- Prior model-scaling work studies one 3D model type, whereas this paper compares local point aggregators across similar budgets and multiple scales.
3. Network Architecture Overview
The architecture decouples grid encoding, backbone extraction, neck aggregation, and detection output. PillarNeXt combines a simple grid encoder with sparse convolutions, receptive-field expansion, and a center-based head.
- Grid-based networks comprise a grid encoder, backbone, neck, and task-specific detection head, which the paper analyzes separately.
- Pillar encoders arrange points in vertical columns, apply MLPs and max pooling, and represent the resulting features as a pseudo image.
- Voxel encoders preserve height details, while multi-view fusion combines pillar or voxel representations with range-view representations.
- ResNet-18 provides the common backbone, using sparse 2D convolutions for pillar or MVF encoders and sparse 3D convolutions for voxel encoders.
- The neck aggregates features to enlarge receptive fields and incorporate context through designs such as BiFPN or single-level dilated ASPP.
- The center-based detection head represents objects by center points and predicts a centerness heatmap with bounding-box regression at center locations.
4. Experiments
The experiments introduce the setup and then study network components before comparing the resulting models with state-of-the-art methods on two benchmarks.
- The experiments begin with dataset and implementation details, followed by comprehensive studies of each component in a 3D detection model.
- The paper concludes the experimental section with extensive comparisons against state-of-the-art methods on two popular benchmarks.
4.1. Experimental Setup
The study evaluates PillarNeXt on Waymo Open Dataset and reports standard 3D and BEV detection metrics, with experiments spanning neck-module configurations and learning behavior.
- Datasets and metrics: Experiments use the Waymo Open Dataset, comprising 798 training sequences and 202 validation sequences captured with five LiDARs at 10 Hz.The evaluation follows official AP and APH metrics, including L1 and L2 difficulty levels.
- Datasets and metrics: Waymo evaluation reports L1 and L2 3D and BEV AP/APH for vehicle, pedestrian, and cyclist detection.IoU thresholds are 0.7 for vehicles and 0.5 for pedestrians and cyclists.
- Implementation: The implementation uses AdamW with a one-cycle learning-rate schedule, trains for 12 epochs by default, and uses three input frames for Waymo.The pillar size is 0.075 m in the x/y axes.
- Evaluation design: The experimental study compares neck modules using L1 and L2 BEV AP and APH for vehicles and pedestrians on the Waymo validation set.The compared groups include multi-scale and single-scale necks.
- Evaluation design: Figure 3 compares pillar- and voxel-based learning behavior using vehicle L2 3D and BEV APH on the Waymo validation set.The comparison focuses on convergence behavior rather than only final detection scores.
4.2. Network Design Study
The network design study examines grid encoders, necks, resolutions, training, and scaling, showing that resource allocation and modern architectural adaptations substantially improve pillar-based detection.
- Grid encoders: Under comparable computational budgets, pillar models achieve comparable or better results than voxel models while running considerably faster.The study argues that allocating computation to BEV rather than the height dimension is more efficient and effective.
- Grid encoders: Pillar models converge much more slowly than voxel models in 3D APH, but the gap diminishes with sufficient training.Their BEV APH convergence is comparable, indicating that training duration contributes to earlier reported differences.
- Neck modules: Neck adaptations from 2D detection improve vehicle performance, including up to 2.38% improvement with FPN or BiFPN over the PillarNet neck.These experiments compare multi-scale feature aggregation designs on the Waymo validation set.
- Neck modules: Single-scale necks with enlarged receptive fields outperform plain limited-receptive-field designs, with ASPP yielding up to 2.45% vehicle improvement over the PillarNet neck.The results imply that multi-scale features may be unnecessary when the receptive field is sufficiently large.
- Resolutions: Larger grid sizes preserve vehicle performance at fixed output resolution but degrade pedestrian accuracy, while output downsampling harms both categories.Adding an upsampling layer in the detection head substantially improves performance, especially for small objects.
- Overall design: PillarNeXt combines model scaling, an enhanced neck and head, and appropriate training to produce substantial gains over the original baseline.The resulting PillarNeXt-B is compared extensively with point-cloud-specific state-of-the-art methods and can be configured for accuracy-latency trade-offs.
4.3. Comparison with State-of-the-Art on WOD
PillarNeXt-B is compared with state-of-the-art methods on Waymo Open Dataset across 3D and BEV evaluation settings. The results show strong performance without test-time augmentation or model ensembling, including advantages over methods using temporal or fine-grained geometric modeling.
- PillarNeXt-B outperforms many two-stage methods on the Waymo validation set.
- Its pillar-based model achieves better pedestrian results than methods using complex temporal and fine-grained geometric modeling.
- The reported comparisons cover 3D metrics on Waymo test data and BEV metrics relevant to downstream autonomous-driving tasks.
- A 3-frame PillarNeXt model outperforms the sequence-refinement method 3DAL, which uses around 200 frames, on BEV performance.
- PillarNeXt-B outperforms state-of-the-art methods on the Waymo test set without test-time augmentation or model ensembling.
4.4. Comparison with State-of-the-Art on nuScenes
On nuScenes, PillarNeXt-B achieves strong validation performance and generalizes across datasets. Its pillar-based design compares favorably with leading voxel and multi-view methods, while matching or exceeding a voxel model under identical settings.
- PillarNeXt-B achieves 68.8% NDS and 62.5% mAP on the nuScenes validation set.
- PillarNeXt-B outperforms leading voxel and multi-view methods by a large margin in mAP.
- Under exactly the same setting, PillarNeXt-B obtains higher NDS and comparable mAP than the authors’ voxel-based model.
- PillarNeXt-B achieves on-par or superior per-class results compared with the voxel-based model.
5. Conclusions
The paper concludes that high-performing LiDAR 3D object detection does not require fine-grained local geometric modeling. Its experiments instead identify pillar encoding, enlarged receptive fields, and resolution manipulation as central design choices.
- The study finds that a simple pillar encoder with an enhanced strategy performs best in both accuracy and latency.
- The authors report that enlarging the receptive field and manipulating resolutions play key roles in performance.
- These findings challenge the belief that high-performing 3D object detection requires fine-grained local geometric modeling.
A. More Implementation Details
The implementation details specify shared optimizer and augmentation settings, while enhanced training adds faded copy-and-paste augmentation for state-of-the-art comparisons.
- The optimizer uses AdamW with 0.01 weight decay and a one-cycle learning rate of 0.001 for batch size 16.
- All experiments use random flipping, rotation within [−π/4, π/4], scaling between [0.9, 1.1], and translation noise with factor 0.5.
- The enhanced strategy trains Pillar, Voxel, and MVF+ models, while state-of-the-art comparisons additionally use faded copy-and-paste augmentation.