Source-linked AI summary
Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution
Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, Song Han
TL;DR
Autonomous-driving 3D perception must remain accurate and efficient despite constrained hardware and low-resolution voxel representations that impair small-object recognition. The paper introduces SPVConv and 3D-NAS to preserve fine details while searching efficient architectures, producing fast, accurate models across segmentation and detection tasks.
Problem
Constrained hardware and low-resolution voxelization or aggressive downsampling limit existing 3D perception models’ recognition of small instances such as pedestrians and cyclists.
Method
SPVConv adds a low-cost high-resolution point-based branch to Sparse Convolution, while 3D-NAS searches a flexible SPVConv-based architecture space.
Results
The resulting SPVNAS model outperforms MinkowskiNet by 3.3% in mIoU, achieves 8× computation reduction and 3× measured speedup with higher accuracy, and improves KITTI detection over a one-stage baseline.
Takeaways & Limitations
SPVNAS is lightweight, fast, and powerful across outdoor 3D scene segmentation and transferred 3D object detection.
Takeaways & Limitations
The paper keeps all convolution kernel sizes fixed at 3 because larger kernels substantially increase computation and sparse kernel-map construction cost.
Abstract
from arXiv · showhide
Self-driving cars need to understand 3D scenes efficiently and accurately in order to drive safely. Given the limited hardware resources, existing 3D perception models are not able to recognize small instances (e.g., pedestrians, cyclists) very well due to the low-resolution voxelization and aggressive downsampling. To this end, we propose Sparse Point-Voxel Convolution (SPVConv), a lightweight 3D module that equips the vanilla Sparse Convolution with the high-resolution point-based branch. With negligible overhead, this point-based branch is able to preserve the fine details even from large outdoor scenes. To explore the spectrum of efficient 3D models, we first define a flexible architecture design space based on SPVConv, and we then present 3D Neural Architecture Search (3D-NAS) to search the optimal network architecture over this diverse design space efficiently and effectively. Experimental results validate that the resulting SPVNAS model is fast and accurate: it outperforms the state-of-the-art MinkowskiNet by 3.3%, ranking 1st on the competitive SemanticKITTI leaderboard. It also achieves 8x computation reduction and 3x measured speedup over MinkowskiNet with higher accuracy. Finally, we transfer our method to 3D object detection, and it achieves consistent improvements over the one-stage detection baseline on KITTI.
1 Introduction
3D perception for autonomous driving must be accurate and low-latency despite tightly constrained hardware. The paper addresses information loss in voxel-based models by combining a high-resolution point branch with sparse convolution and searching efficient architectures.
- Motivation: 3D perception models must achieve high accuracy and low latency while operating under limited memory and other hardware resources.These constraints arise from vehicle form factor and heat dissipation.
- Problem: Voxel-based methods lose information through memory-limited resolution or aggressive downsampling, merging points and small objects into indistinguishable grids.This particularly disadvantages pedestrians and cyclists relative to large objects.
- Method: SPVConv adds a low-cost, high-resolution point-based branch to vanilla Sparse Convolution to capture fine details.The point branch is designed to address information loss in large outdoor scenes.
- Method: 3D-NAS searches an efficient 3D model over a flexible SPVConv-based design space using fine-grained channel choices and progressive depth shrinking.These design choices increase search-space diversity and accelerate training.
- Results: 3.3% higher mIoU than MinkowskiNet is achieved with lower latency, while the model also delivers 8× computation reduction and 3× measured speedup with higher accuracy.The method also transfers to KITTI 3D object detection with consistent improvements over a one-stage baseline.
- Results: The resulting method ranks 1st on the SemanticKITTI leaderboard and achieves consistent improvements after transfer to 3D object detection.These results cover both outdoor scene segmentation and KITTI detection.
2 Related Work
Related work centers on point-cloud and voxel representations for 3D learning, alongside neural architecture search for automatically designing accurate and efficient networks. Existing approaches motivate the paper’s focus on representation efficiency and lower search cost.
- 3D representations: 3D learning has used volumetric representations and vanilla 3D convolutions, but dense volumetric processing is inefficient and introduces information loss.
- 3D representations: Point-cloud methods directly learn from 3D points using symmetric functions to handle sparse data representations.
- Neural architecture search: Neural architecture search automates network design through approaches including reinforcement learning and evolutionary search.
- Neural architecture search: Efficient-model NAS research targets mobile deployment, while conventional search frameworks require high computation cost and considerable carbon footprint.
3 SPVConv: Designing Effective 3D Modules
SPVConv combines a high-resolution point branch with sparse voxel processing to retain fine details while modeling larger receptive fields. GPU hash-table indexing and lightweight fusion reduce the practical cost of this dual representation.
- Motivation: SPVConv addresses information loss from coarse voxelization and aggressive downsampling in Point-Voxel and Sparse Convolution modules.These losses can make small instances indistinguishable or leave them represented by too few voxel grids.
- SPVConv design: SPVConv adds a high-resolution point branch to sparse voxel processing, preserving point-level details while the voxel branch models different receptive-field sizes.The branches communicate through sparse voxelization and devoxelization.
- SPVConv design: Sparse voxelization aggregates point features into nonzero voxel grids using a normalization factor equal to the number of points in each grid.The representation contains sparse voxel coordinates and features alongside point coordinates and features.
- Efficient implementation: Naive coordinate indexing for sparse voxelization requires O(mn) complexity, making it impractical for typical m,n values around 10^5.A GPU hash table reduces indexing cost by constructing entries in O(n) time and querying points in O(m) total time.
- 3D-NAS: 3D-NAS trains a super network with fine-grained channel numbers and elastic depths, then uses evolutionary search to select a candidate under a computation constraint.The search is built from multiple SPVConv modules and is intended to design efficient 3D models.
- SPVConv design: The point branch applies an MLP per point and fuses its features with the voxel branch by addition, adding only 4% MAC overhead.Sparse devoxelization uses trilinear interpolation over each point’s 8 neighboring voxel grids.
4 3D-NAS: Searching Efficient 3D Architectures
3D-NAS automatically searches efficient 3D architectures within a flexible SPVConv-based design space under resource constraints. Its space varies channels and depths while retaining small kernels, and its training and evolutionary search make this exploration tractable.
- 3D-NAS automatically designs efficient 3D models to meet latency, energy, and accuracy constraints.
- Design Space: The design space uses fine-grained channel numbers and elastic network depths but fixes all convolution kernels to size 3.Kernel sizes are excluded because larger 3D kernels incur higher computation and kernel-map construction costs; depth can provide a large receptive field instead.
- Training Paradigm: A trained super network lets candidate architectures inherit weights, reducing total training cost from O(n) to O(1).
- Training Paradigm: Uniform sampling distributes candidate networks across GPUs, while weight sharing optimizes every candidate at each iteration.
- Training Paradigm: Progressive depth shrinking enlarges the search space gradually so large-depth candidates are sampled more frequently during training.Random depth sampling gives an expected total depth of n×(m+1)/2, below the maximum depth nm, making maximum-depth candidates rarely sampled.
- Architecture Search: Evolutionary search selects high-mIoU candidates under a MAC constraint and generates new architectures through mutations and crossovers.The resource estimate accounts for input sparsity by measuring average kernel-map sizes across the dataset.
5 Experiments
Experiments on SemanticKITTI and KITTI show that the proposed models outperform prior methods in segmentation and detection while using less computation and latency. SPVNAS is especially effective for small instances, and SPVCNN improves cyclist detection.
- SPVNAS consistently outperforms previous state-of-the-art models on 3D semantic segmentation and object detection with lower computation cost and measured latency.
- 3D Scene Segmentation: 3.3% higher mIoU, 1.7× model size reduction, 1.5× computation reduction, and 1.1× measured speedup distinguish SPVNAS from MinkowskiNet.
- 3D Scene Segmentation: SPVNAS recognizes small instances with lower errors than MinkowskiNet, particularly where coarse resolution harms small objects and region boundaries.
- 3D Scene Segmentation: 0.6% higher mIoU, 8.3× model size reduction, 7.6× computation reduction, and 2.7× measured speedup are achieved by the downscaled 15G-MAC SPVNAS over MinkowskiNet.
- 3D Scene Segmentation: SPVNAS exceeds DarkNets by more than 10% in mIoU with 1.2× measured speedup and exceeds KPConv in mIoU with 17× model size and 23× computation reductions.
- 3D Object Detection: SPVCNN significantly improves cyclist detection over SECOND while replacing only its 3D sparse convolutions with SPVConv.The comparison keeps the other detection settings unchanged for a fair comparison.
6 Analysis
Analysis attributes SPVNAS’s gains to high-resolution point-branch attention to small instances and to the complementary contributions of SPVConv and 3D-NAS. The searched architecture also improves efficiency trade-offs and evolutionary-search progress.
- Small-Instance Recognition: SPVNAS gains up to 25% on relatively small objects such as pedestrians and cyclists.
- Small-Instance Recognition: The point-based branch attends to pedestrians, cyclists, trunks, and traffic signs through points with the largest feature norms.The visualization marks the top 5% feature-norm points in the final SPVConv.
- Small-Instance Recognition: More than 80% of bicyclist points are activated in the point-based branch, indicating stronger attention to small instances than the sparse voxel-based branch.
- 3D-NAS Analysis: SPVConv and 3D-NAS are equally important to the final performance boost, with SPVNAS exceeding MinkowskiNet by more than 6% mIoU at 110 ms latency.
- 3D-NAS Analysis: 3D-NAS reduces upsampling’s computation share from 77% to 47–63%, producing a more balanced architecture that emphasizes downsampling-stage feature extraction.
- Search Strategy: Evolutionary search progressively finds better architectures, whereas random search has poor sample efficiency; its final best architecture performs around 3% better.
7 Conclusion
The paper presents SPVConv and 3D-NAS as a lightweight approach for efficient, high-performing 3D scene understanding, especially for small objects and large outdoor scenes.
- SPVConv is a lightweight 3D module specialized for small object recognition.It addresses limitations of Sparse Convolution and Point-Voxel Convolution in preserving resolution and scaling to large 3D scenes.
- 3D-NAS searches architectures built on SPVConv to improve SPVCNN efficiency and performance.The framework is presented as the first architecture search framework for 3D scene understanding.
- The resulting SPVNAS model is lightweight, fast, and powerful on outdoor 3D scene benchmarks.The paper frames these results as evidence supporting efficient 3D deep learning research.
A.1 Implementation Details
The implementation details section explains that the paper provides procedures for constructing SPVCNN, training the super network, and searching for an architecture.
- The implementation details cover SPVCNN backbone construction and the training and search procedures of 3D-NAS.
A.1.1 SPVCNN: Backbone Network
The SPVCNN backbone wraps residual Sparse Convolution blocks with high-resolution point-based branches, using different voxelization and devoxelization placements across the network.
- The main backbone wraps residual Sparse Convolution blocks with a high-resolution point-based branch.Its SPVConv modules are placed around the network’s stemming, downsampling, and upsampling stages.
- The first SPVConv voxelizes before the first layer and devoxelizes before the first downsampling stage.
- The second SPVConv voxelizes after the first module and devoxelizes after all four downsampling stages.
- The final two SPVConv modules each wrap around two upsampling stages.
- A smaller PVCNN-based backbone replaces each volumetric convolution with one convolution layer and two residual Sparse Convolution blocks.Normalization and activation layers follow the replacement convolution.
A.1.2 3D-NAS: Architecture Search
The 3D-NAS procedure trains a super network with fine-grained channels and elastic depth, then uses evolutionary search to extract and fine-tune a candidate architecture.
- The super network is trained for 15 epochs to support fine-grained channel settings.Training starts at learning rate 0.24 with cosine learning-rate decay.
- A further 15 training epochs incorporate elastic network depth.This stage starts at learning rate 0.096 and also uses cosine learning-rate decay.
- Evolutionary architecture search evaluates a population of 50 candidates for 20 generations on sequence 08.
- The best architecture is extracted from the super network and submitted after 10 epochs of fine-tuning.
A.2 More Results
Additional experiments show that SPVConv and 3D-NAS improve MinkowskiNet-based segmentation, while SPVNAS and SPVCNN provide qualitative advantages on small objects, boundaries, and crowded detection scenes.
- 3D Scene Segmentation: Both SPVConv and the 3D-NAS pipeline improve MinkowskiNet on SemanticKITTI test and validation results.
- 3D Object Detection: SPVCNN consistently improves over SECOND across almost all classes on the KITTI validation set.
- 3D Scene Segmentation: SPVNAS corrects errors on very small objects and improves segmentation along sidewalk-building and sidewalk-vegetation boundaries.
- 3D Scene Segmentation: SPVNAS shows advantages over DarkNet53 on both large regions and smaller instances in qualitative comparisons.
- 3D Object Detection: SPVCNN detects a challenging small pedestrian missed by SECOND and avoids duplicate predictions in crowded small-object scenes.