Source-linked AI summary
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation
Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka
TL;DR
Large-scale LiDAR segmentation often converts point clouds into 2D images, but spherical projection makes their feature distributions strongly location-dependent, limiting standard convolutions. The paper introduces Spatially-Adaptive Convolution, which adapts filters efficiently and yields SqueezeSegV3’s reported SemanticKITTI gains with comparable inference speed.
Problem
Spherical projection creates strongly location-dependent LiDAR feature distributions, making standard convolutions poorly matched to large-scale point-cloud segmentation.
Method
Spatially-Adaptive Convolution adapts filters to image location and input content using an efficient factorization into static convolution weights and an attention map.
Results
SqueezeSegV3 outperforms all previously published SemanticKITTI methods by at least 3.7 mIoU with comparable inference speed.
Takeaways & Limitations
SAC provides a general spatially-adaptive framework that includes SE, CBAM, CAM, and PAC as special cases.
Abstract
from arXiv · showhide
LiDAR point-cloud segmentation is an important problem for many applications. For large-scale point cloud segmentation, the \textit{de facto} method is to project a 3D point cloud to get a 2D LiDAR image and use convolutions to process it. Despite the similarity between regular RGB and LiDAR images, we discover that the feature distribution of LiDAR images changes drastically at different image locations. Using standard convolutions to process such LiDAR images is problematic, as convolution filters pick up local features that are only active in specific regions in the image. As a result, the capacity of the network is under-utilized and the segmentation performance decreases. To fix this, we propose Spatially-Adaptive Convolution (SAC) to adopt different filters for different locations according to the input image. SAC can be computed efficiently since it can be implemented as a series of element-wise multiplications, im2col, and standard convolution. It is a general framework such that several previous methods can be seen as special cases of SAC. Using SAC, we build SqueezeSegV3 for LiDAR point-cloud segmentation and outperform all previous published methods by at least 3.7% mIoU on the SemanticKITTI benchmark with comparable inference speed.
1 Introduction
Large-scale LiDAR segmentation commonly projects 3D points into 2D images, but spatially varying LiDAR features make standard convolutions inefficient. The paper proposes efficient Spatially-Adaptive Convolution and builds SqueezeSegV3, which improves SemanticKITTI performance while maintaining comparable inference speed.
- Motivation: 192,000 points per frame make efficient or real-time processing beyond PointNet-based methods for large-scale outdoor LiDAR scenes.A typical Velodyne HDL-64E scan covers approximately 160×160×20 meters.
- Motivation: Spherical projection produces LiDAR images that resemble RGB images, motivating 2D convolutional processing.The projected representation uses point coordinates as features and supports image-based processing.
- Problem: LiDAR feature distributions vary drastically by image location, so standard convolution filters may become active only in specific regions.This spatial variation differs from the largely location-invariant distributions of regular images.
- Contribution: Spatially-Adaptive Convolution adapts filters to different image regions according to the input while factorizing adaptive weights into static weights and an attention map.The attention map is computed by a one-layer convolution, and implementation uses element-wise multiplication, im2col, and reshaping.
- Contribution: SAC generalizes several earlier attention and adaptive-convolution methods, including SE, CBAM, CAM, and PAC, as special cases.Experiments report that the paper’s more general SAC variants outperform previous ones.
- Results: 3.7 mIoU is the minimum improvement SqueezeSegV3 achieves over previously published SemanticKITTI methods with comparable inference speed.The result is reported for SqueezeSegV3’s overall benchmark comparison.
2 Related work
Prior point-cloud segmentation methods are split between small-scale PointNet-style processing and large-scale approaches constrained by annotation difficulty and real-time requirements. The paper situates SAC among adaptive convolution methods and evaluates it using the RangeNet++ backbone for controlled comparison.
- Point-cloud segmentation: Small-scale point-cloud segmentation commonly uses PointNet-based methods, while large-scale processing remains difficult because of speed and annotation constraints.Outdoor LiDAR scans can contain about 200K points, creating real-time latency challenges.
- Adaptive convolutions: Standard convolutions share weights across spatial locations, whereas adaptive convolutions change weights according to input content or image location.Prior approaches adapt channels, spatial positions, kernel dimensions, or depth-aware weights in different ways.
- Efficiency: Many point-cloud segmentation applications require real-time inference, motivating networks optimized for parameters, FLOPs, latency, and power.Efficiency is treated as a property of both the segmentation pipeline and the neural network.
- Experimental setting: The paper uses the same backbone as RangeNet++ to evaluate SAC without changing the backbone architecture.This design supports a more controlled assessment of spatially-adaptive convolution.
3 Spherical Projection of LiDAR Point-Cloud
The method projects each 3D LiDAR point onto a quantized spherical 2D grid and uses its measurements as pixel features. This projection creates strong spatial priors and location-dependent feature distributions that distinguish LiDAR images from regular images.
- Spherical projection: The projection uses 3D coordinates, angular coordinates, image dimensions, sensor field of view, and point range to determine pixel positions.The range is defined as r = x2 + y2 + z2.
- LiDAR representation: Each projected point contributes x, y, z, range, and intensity features, producing a LiDAR image with shape (h, w, 5).Point-cloud segmentation is thereby reduced to image segmentation using ConvNets.
- Spatial variation: LiDAR feature distributions differ drastically across image locations, unlike the more similar location-wise distributions observed in COCO and CIFAR10 images.The spherical projection induces sinusoidal x and y variation across width and higher z-values toward the image top.
4 Spatially-Adaptive Convolution
Standard convolution assumes location-invariant feature distributions, but projected LiDAR images vary spatially; SAC adapts convolution weights to image content and location while retaining efficient computation.
- Standard Convolution: Projected LiDAR images violate standard convolution’s location-invariant feature assumption, leaving filters sparsely active where their learned local features occur.The paper attributes this variation to spherical projection and observes sparse filter activations in pretrained RangeNet21.
- Spatially-Adaptive Convolution: SAC processes different image regions with filters adapted to both spatial location and the raw input.Its general weight function depends on image position and X0, making the operator spatially-adaptive and content-aware.
- Spatially-Adaptive Convolution: SAC factorizes adaptive weights into static convolution weights and an input-dependent attention map, then collapses attention dimensions to obtain tractable variants.The framework includes SAC-OISK, SAC-ISK, SAC-IS, and SAC-S, depending on which output, input, spatial, and kernel dimensions remain adaptive.
- Variants and Efficiency: SE, CBAM, CAM, and PAC can be expressed as special cases of SAC, while retaining spatial attention is sufficient for spatial adaptation.Experiments report that all SAC variants improve SemanticKITTI performance, and SqueezeSegV3 adds only 2.4%–24.8% FLOPs and 1.1%–14.9% parameters.
- Efficient Computation of SAC: SAC variants are computed by combining attention with inputs through element-wise multiplication or im2col, followed by standard convolution.Maps without kernel dimensions multiply directly with the input; kernel-aware maps use unfolding, reshaping, and a 1-by-1 convolution.
5 SqueezeSegV3
SqueezeSegV3 applies spatially-adaptive convolution within a RangeNet-based backbone for LiDAR segmentation, using multi-layer supervision to train semantically stronger intermediate features.
- SqueezeSegV3 uses spatially-adaptive convolution for LiDAR point-cloud segmentation.
- Its RangeNet backbone replaces the first convolution in each block with SAC-ISK and removes the last two downsampling layers while reducing later channels to preserve FLOPs.
- The network adds prediction layers at five stage outputs and supervises them with groundtruth maps downsampled by 1x, 2x, 4x, 8x and 8x.
- Intermediate supervision encourages semantically meaningful features and helps mitigate vanishing gradients during training.
6 Experiments
Experiments on SemanticKITTI compare SqueezeSegV3 with prior methods and isolate the effects of SAC variants, kernel sizes, downsampling removal, and multi-layer loss.
- 6.1 Dataset and Setup: SemanticKITTI provides 43,442 densely annotated scans and 4549 million points across 21 sequences, with mIoU over 19 categories as the accuracy metric.
- 6.1 Dataset and Setup: SqueezeSegV3 projects scans to 64 × 2048 LiDAR images, predicts 2D labels, and restores them to 3D point predictions.
- 6.3 Comparing with Prior Methods: SqueezeSegV3-53* exceeds RangeNet53* by 3.7 mIoU and achieves the best accuracy in 14 out of 19 categories.
- 6.3 Comparing with Prior Methods: SSGV3-21 runs at 16 FPS versus RangeNet21 at 20 FPS, while SSGV3-53 runs at 7 FPS versus RangeNet53 at 12 FPS.
- 6.4 Ablation Study: SAC-ISK significantly outperforms other spatially-adaptive settings in mIoU, while spatial adaptation improves over non-spatial SE and CBAM baselines.
- 6.4 Ablation Study: A 7x7 coordinate-map convolution performs best among tested kernel sizes, although 1x1 also produces a strong result.
- 6.4 Ablation Study: Removing downsampling improves mIoU by 3.9 mIoU, multi-layer loss adds 1.5%, and SAC-ISK further boosts mIoU by 2.3%.
7 Appendix
The appendix supplies pseudocode for SAC variants, showing their attention-map construction, feature adaptation, convolutional processing, and residual output formation.
- The appendix covers SAC-S, SAC-IS, SAC-SK and SAC-ISK implementations.
- Each SAC function takes an input feature and coordinate map and returns an output feature with the same spatial and channel dimensions.
- SAC-S and SAC-IS compute attention maps, multiply them element-wise with the input feature, apply a 3x3 convolution, and add a residual connection.
- SAC-SK and SAC-ISK additionally unfold local features and produce kernel-aware attention maps before feature projection, convolution, and residual addition.