PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking

Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, Jiaya Jia

arXiv:2303.11301v1cs.CV

TL;DR

Existing 3D detectors often use hand-crafted proxies and dense heads, adding computation to sparse voxel processing. VoxelNeXt predicts directly from sparse voxel features with a fully sparse network, achieving leading detection results and first-place LIDAR tracking performance across major benchmarks.

  • Problem

    Hand-crafted anchors or centers require sparse features to be densified for dense prediction heads, causing inefficient computation and complicated pipelines.

  • Method

    VoxelNeXt directly predicts 3D objects and tracks them from sparse voxel features using a fully sparse convolutional network without sparse-to-dense conversion or NMS.

  • Results

    VoxelNeXt achieves leading detection performance with high efficiency on nuScenes, Waymo, and Argoverse2, and ranks first among LIDAR-only methods on the nuScenes tracking benchmark.

  • Takeaways & Limitations

    Direct voxel-based prediction is feasible and effective, making anchors, centers, and dense heads unnecessary within this framework.

  • Takeaways & Limitations

    The gap between theoretical FLOPs and actual inference speed depends strongly on implementation and hardware devices.

Abstract

from arXiv · show

3D object detectors usually rely on hand-crafted proxies, e.g., anchors or centers, and translate well-studied 2D frameworks to 3D. Thus, sparse voxel features need to be densified and processed by dense prediction heads, which inevitably costs extra computation. In this paper, we instead propose VoxelNext for fully sparse 3D object detection. Our core insight is to predict objects directly based on sparse voxel features, without relying on hand-crafted proxies. Our strong sparse convolutional network VoxelNeXt detects and tracks 3D objects through voxel features entirely. It is an elegant and efficient framework, with no need for sparse-to-dense conversion or NMS post-processing. Our method achieves a better speed-accuracy trade-off than other mainframe detectors on the nuScenes dataset. For the first time, we show that a fully sparse voxel-based representation works decently for LIDAR 3D object detection and tracking. Extensive experiments on nuScenes, Waymo, and Argoverse2 benchmarks validate the effectiveness of our approach. Without bells and whistles, our model outperforms all existing LIDAR methods on the nuScenes tracking test benchmark.

1. Introduction

Mainstream 3D detectors use hand-crafted anchors or centers and densify sparse features for dense heads, creating inefficiency and complicated post-processing. VoxelNeXt instead predicts directly from sparse voxel features, enabling efficient detection and tracking without NMS.

  • 3D detectors commonly use anchors or centers as hand-crafted proxies for object prediction.
  • Sparse features are converted into dense representations so dense heads can predict on ordered anchors or centers.
  • Less than 1% of nuScenes Car locations have responses on average, yet dense heads compute over all feature-map positions and require NMS for duplicate predictions.
  • VoxelNeXt directly predicts 3D objects from voxel features with a fully sparse convolutional network and removes anchor proxies, dense conversion, and other complex components.
  • Sparse-only prediction improves inference efficiency by operating only on necessary locations and extends naturally to tracking through query-voxel association.
  • VoxelNeXt achieves leading detection performance with high efficiency on nuScenes, Waymo, and Argoverse2, and ranks first among LIDAR-only nuScenes tracking methods.

2. Related Work

Prior 3D detectors largely adapt dense 2D-style prediction to sparse data, while sparse detectors avoid dense heads through varied complex mechanisms. VoxelNeXt demonstrates that direct prediction with a sparse CNN can suffice.

  • Dense-head detectors: Many LIDAR detectors retain dense convolutional heads despite 3D data sparsity, including anchor-based and region-based methods.
  • Center-based detection: CenterPoint converts sparse backbone output into a dense BEV feature map and predicts dense object-center heatmaps.
  • VoxelNeXt: VoxelNeXt uses a simple sparse CNN instead of dense heads, showing that direct sparse prediction is feasible despite concerns about limited representation capacity.
  • Sparse detectors: Sparse detectors avoid dense heads but introduce additional mechanisms such as range-image segmentation, windowed transformers, clustering, or group correction.
  • 3D object tracking: For tracking, VoxelNeXt associates query voxels, addressing the center-position bias associated with tracking only predicted object centers.

3. Fully Sparse Voxel-based Network

VoxelNeXt adapts sparse CNNs to predict 3D objects directly from voxel features, avoiding hand-crafted centers or anchors and dense processing. Its backbone, sparse head, pruning, and voxel association support efficient detection and tracking.

  • Overview: VoxelNeXt predicts 3D boxes directly from sparse voxel features rather than hand-crafted anchors or centers.The approach targets irregular surface voxels and uses minimal modification to a plain sparse CNN.
  • Sparse CNN Backbone Adaptation: Two additional down-sampling stages expand feature strides from {1, 2, 4, 8} to {1, 2, 4, 8, 16, 32}, enlarging receptive fields with little extra computation.Features from the final three stages are spatially aligned and sparsely concatenated without additional parameterized layers.
  • Sparse Height Compression: Fully sparse height compression places voxels onto the ground plane and sums features sharing 2D positions, costing no more than 1ms.Prediction on compressed 2D sparse features costs less than prediction using 3D sparse features.
  • Sparse CNN Backbone Adaptation: Spatial voxel pruning suppresses dilation for low-magnitude voxels, reducing computation without compromising performance.With a suppression ratio of 0.5, only voxels in the top half of feature magnitudes are dilated.
  • Sparse Prediction Head: Sparse max pooling selects spatially local score maxima, excluding removed voxels from box prediction and avoiding NMS post-processing.The operation works only on non-empty positions and is applied separately for each class.
  • 3D Object Tracking: VoxelNeXt extends naturally to tracking by associating query voxels with boxes, supplementing center-based matching when predicted centers are inaccurate.Query voxels are traced to original input positions and matched using L2 distance, reducing positional bias relative to predicted centers.

4. Experiments

Experiments show that VoxelNeXt maintains strong detection and tracking performance while using sparse, efficient prediction mechanisms across multiple benchmarks and ablations.

  • Ablation Studies: Performance drops sharply with insufficient down-sampling, while additional down-sampling compensates for receptive-field limitations.The largest degradation occurs for large Truck and Bus objects in the D3 variant.
  • Voxel Analysis: Less than 10% of predicted boxes use voxels near object centers, indicating that boundary voxels can also support box prediction.Relative-position analysis uses high-quality predicted boxes and separates near-center, near-boundary, and outside-box voxels.
  • Comparison to CenterPoint: 0.9% mAP and 1.0% NDS improvements over CenterPoint are reported on a reduced-training-set nuScenes comparison.The comparison trains on one-quarter of nuScenes training data and evaluates on the full validation split.
  • Efficiency Analysis: The sparse backbone’s Stage 5 and Stage 6 contribute 6.1G and 2.8G FLOPs, respectively, because their voxel counts are small.These stages require 6 ms and 3 ms and together remain no more than one-third of total backbone computation.
  • Ablation Studies: 56.0% mAP from sparse max-pooling is comparable to 56.2% mAP from NMS.Sparse max-pooling avoids some unnecessary computation on predictions.
  • Ablation Studies: Voxel association improves nuScenes tracking validation performance by 1.1% AMOTA.The association includes voxels that predict boxes in addition to predicted box centers.
  • Main Results: VoxelNeXt achieves leading 3D object-detection performance with high efficiency on nuScenes, Waymo, and Argoverse2.The evaluation compares LIDAR-based methods across all three benchmarks.
  • Main Results: 71.0% AMOTA ranks VoxelNeXt first on the nuScenes 3D LIDAR tracking benchmark.This result uses double-flip 3D detection results.

5. Conclusion and Discussion

The paper concludes that direct prediction from sparse voxel features can support efficient 3D detection and tracking without dense heads or NMS. It also reports a gap between theoretical FLOPs and realized inference speed.

  • Conclusion: VoxelNeXt presents a fully sparse voxel-based framework for 3D object detection and tracking that operates without NMS post-processing.The conclusion describes direct voxel prediction as feasible and effective, making anchors, centers, and dense heads unnecessary in this framework.
  • Conclusion: VoxelNeXt achieves leading 3D detection performance across nuScenes, Waymo, and Argoverse2 and ranks first on the nuScenes LIDAR tracking benchmark.The conclusion pairs these results with high efficiency.
  • Limitations: 38.7G FLOPs versus CenterPoint’s 186.6G does not translate into an equally large latency reduction.Actual latency depends strongly on implementation and hardware devices.

Appendix

The appendix contains implementation details, additional experimental results, visualizations, and discussions.

  • Appendix: The appendix first presents implementation details, then additional experiments, followed by visualizations and discussions.These materials are organized in Sections A through D.

A. Implementation Details

The experiments use established autonomous-driving datasets, sparse voxel configurations, and training procedures aligned with prior work. Implementation choices vary by dataset and include dataset-specific voxelization and augmentation settings.

  • Datasets: nuScenes contains 1,000 sequences, with 700 for training, 150 for validation, and 150 for testing, across 10 annotated classes.The dataset uses a 32-beam synchronized LIDAR and six cameras.
  • Datasets: Waymo contains 1,150 sequences, including 798 for training and 202 for validation, collected using one long-range and four near-range LIDAR sensors.The long-range sensor reaches 75 meters.
  • Datasets: Argoverse2 includes 1,000 sequences, with 700 for training and 150 for validation, and covers a 400m × 400m area.Its perception range is a 200-meter radius, and processing follows FSD.
  • Voxelization: Voxel sizes and point-cloud ranges are specified separately for nuScenes, Waymo, and Argoverse2, with larger vertical voxels in the 2D variants.For example, nuScenes uses 0.075m × 0.075m × 0.2m voxels, while its 2D variant uses an 8m vertical size.
  • Optimization: Training uses Adam and cosine learning-rate decay for nuScenes, while Waymo and Argoverse2 use related Adam-based schedules with 12 and 6 epochs, respectively.NuScenes models train for 20 epochs with batch size 16; Argoverse2 follows Waymo-like settings except for its shorter schedule.
  • Architecture: The network has six stages of residual submanifold sparse-convolution blocks, and its sparse head predicts outputs with 3 × 3 submanifold sparse convolutions.Default stage channels are {16, 32, 64, 128, 128, 128}; prediction layers are shared according to dataset class groupings.

B. Experimental results

Experiments evaluate VoxelNeXt across nuScenes and Waymo settings, including validation comparisons and ablations of training data, prediction resolution, and augmentation. The reported analyses show strong performance patterns for the 3D model and limited benefit from multi-resolution heads.

  • nuScenes validation: The nuScenes validation results provide detailed performance comparisons corresponding to the paper’s main table.The supplied passage identifies the validation split and its role in the comparison but gives no individual metric values.
  • Training-data analysis: The gap between VoxelNeXt and VoxelNeXt-2D is large with one-quarter of the nuScenes training data and narrows as the data amount increases.The 3D network performs much better at small data amounts, while VoxelNeXt-2D has potential with more data.
  • Ablations: Increasing prediction-head resolution from 8 to 32 decreases performance, while multi-head combinations provide no improvement over the single-resolution 8 network.The evaluated multi-head settings are {8, 16, 32} and {2, 4, 8, 16, 32}.
  • Waymo experiments: A Waymo augmentation change decreases pasted instances in ground-truth sampling and increases training by six epochs before producing better results.This protocol follows FSD.
  • Waymo experiments: The Waymo appendix reports vehicle-detection results for VoxelNeXt, but the supplied passage does not include the corresponding metric values.

C. Visualizations

The visualization shows query voxels alongside predicted boxes for adjacent frames, highlighting their correspondence during temporal processing.

  • Adjacent-frame detections: Red squares mark the corresponding query voxels alongside predicted boxes in detections from adjacent frames.The figure is intended for zoomed viewing.

D. Discussions

The discussion contrasts VoxelNeXt with point-based and voxel-based detector traditions while noting that its reliance on sparse 3D data may carry data-collection biases.

  • Related detector paradigms: Point-based detectors are naturally fully sparse, but neighborhood queries remain costly on large-scale benchmarks dominated by voxel-based methods.The discussion contrasts their suitability for limited-point scenes with large-scale autonomous-driving datasets.
  • Broader impacts: VoxelNeXt relies on 3D data and spatial sparsity, which may reflect biases in data collection, including potential negative societal impacts.
Loading 2303.11301v1…