Source-linked AI summary
Spherical Transformer for LiDAR-based 3D Recognition
Xin Lai, Yukang Chen, Fanbin Lu, Jianhui Liu, Jiaya Jia
TL;DR
Existing LiDAR recognition methods often leave sparse distant points with limited receptive fields and disconnected information because they do not account for varying point sparsity. SphereFormer uses radial-window self-attention with supporting position encoding and dynamic feature selection to aggregate information from dense close regions. It ranks first on nuScenes and SemanticKITTI semantic segmentation and third on nuScenes object detection.
Problem
Existing methods do not specially consider LiDAR’s varying-sparsity distribution, leaving sparse distant points with limited receptive fields and information disconnection.
Method
SphereFormer directly aggregates long-range information using narrow, long radial windows in spherical coordinates, with exponential splitting and dynamic feature selection.
Results
SphereFormer ranks 1st on nuScenes and SemanticKITTI semantic segmentation and 3rd on nuScenes object detection.
Takeaways & Limitations
Direct long-range aggregation enables sparse distant points to use information from dense close regions and improves distant-point performance.
Abstract
from arXiv · showhide
LiDAR-based 3D point cloud recognition has benefited various applications. Without specially considering the LiDAR point distribution, most current methods suffer from information disconnection and limited receptive field, especially for the sparse distant points. In this work, we study the varying-sparsity distribution of LiDAR points and present SphereFormer to directly aggregate information from dense close points to the sparse distant ones. We design radial window self-attention that partitions the space into multiple non-overlapping narrow and long windows. It overcomes the disconnection issue and enlarges the receptive field smoothly and dramatically, which significantly boosts the performance of sparse distant points. Moreover, to fit the narrow and long windows, we propose exponential splitting to yield fine-grained position encoding and dynamic feature selection to increase model representation ability. Notably, our method ranks 1st on both nuScenes and SemanticKITTI semantic segmentation benchmarks with 81.9% and 74.8% mIoU, respectively. Also, we achieve the 3rd place on nuScenes object detection benchmark with 72.8% NDS and 68.5% mAP. Code is available at https://github.com/dvlab-research/SphereFormer.git.
1. Introduction
SphereFormer addresses LiDAR’s varying sparsity by directly connecting sparse distant points with dense close regions through long-range radial attention. Its position encoding and feature selection designs support this operator, yielding strong benchmark performance and improved distant-point segmentation.
- Motivation: LiDAR points are dense near the sensor but much sparser at farther distances, creating a distribution that standard uniformly local operators do not specifically address.Existing methods perform decently on dense close points but struggle with sparse distant points.
- Motivation: Limited receptive fields leave sparse distant points with few neighbors, causing inconclusive features and information disconnection that prevents effective expansion.The Effective Receptive Field visualization attributes this disconnection to the extreme sparsity of distant cars.
- Method: SphereFormer directly aggregates long-range information in one operator, using spherical coordinates and narrow, long radial windows partitioned along θ and ϕ.The module is designed to gather information from points more than 50 meters away and can be inserted into existing backbones.
- Method: The radial design avoids relying on stacked local operators, allowing sparse distant points to aggregate semantically rich information from dense regions.The paper reports a +17.1% mIoU improvement for distant points.
- Method: Exponential splitting provides fine-grained relative position encoding for long radial windows, while dynamic feature selection differentiates local and global information across locations.Long windows can exceed 50 meters, making ordinary discretization intervals large and position encoding coarse.
- Results: SphereFormer achieves new state-of-the-art results across semantic segmentation and object detection benchmarks.The introduction identifies this as a multi-benchmark outcome, while the conclusion reports first place on nuScenes and SemanticKITTI segmentation and third place on nuScenes detection.
2. Related Work
Prior LiDAR recognition methods commonly use view-, point-, or voxel-based representations and emphasize local information aggregation. SphereFormer instead targets direct long-range aggregation and is designed as a plug-in enhancement for existing recognition frameworks.
- Semantic Segmentation: LiDAR semantic segmentation methods are grouped into view-based, point-based, and voxel-based approaches with different representations and feature operators.View-based methods use range or bird’s-eye views, point-based methods aggregate neighbor features, and voxel-based methods apply sparse convolutions.
- Semantic Segmentation: These semantic segmentation approaches primarily focus on capturing local information, whereas SphereFormer directly aggregates long-range information.The paper positions its approach as a continuation of this research line with a different aggregation range.
- Object Detection: Object detection frameworks include single-stage and two-stage designs, and the experiments use CenterPoint as a widely used anchor-free baseline.SphereFormer is intended to enhance sparse distant-point features within existing detection frameworks.
- Transformer-based Methods: Transformer-based methods introduced window attention and related mechanisms in 2D vision, while LiDAR methods include local window and cubic-window attention variants.The paper distinguishes its radial-window design from these existing local attention approaches.
3. Our Method
SphereFormer models LiDAR point clouds with radial windows that aggregate long-range information for sparse distant points, supplemented by exponential position splitting and dynamic feature selection.
- 3.1. Spherical Transformer: SphereFormer represents LiDAR scenes in spherical coordinates and partitions them into non-overlapping, long, narrow radial windows along θ and ϕ.The resulting radial windows have a pyramid shape and can be inserted as a plugin module into existing backbones.
- 3.1. Spherical Transformer: Radial windows let sparse distant points directly aggregate information from dense-point regions, overcoming information disconnection and increasing the effective receptive field.The design targets long-range feature extraction for distant points while preserving a smooth receptive-field expansion.
- 3.1. Spherical Transformer: Unlike cubic-window approaches, SphereFormer addresses LiDAR sparsity directly rather than relying on stacked local operators that leave distant-point receptive fields limited.The paper contrasts its objective with methods that merely replace cubic voxels or enlarge receptive fields without resolving disconnection.
- 3.2. Position Encoding: Exponential splitting uses smaller radial intervals near the query and larger intervals farther away, enabling finer near-pair position encoding with the same number of intervals.Uniform splitting can assign substantially separated key points to the same index, whereas exponential splitting distinguishes them.
4. Experiments
Experiments evaluate SphereFormer on LiDAR semantic segmentation and object detection, then analyze its radial windows, exponential splitting, dynamic feature selection, and window size.
- Experimental Setting: Experiments evaluate semantic segmentation on nuScenes, SemanticKITTI, and Waymo Open Dataset, plus object detection on nuScenes.
- Semantic Segmentation Results: 74.8% mIoU on SemanticKITTI establishes a new state-of-the-art result and exceeds 2DPASS by 1.9% mIoU.
- Semantic Segmentation Results: SphereFormer ranks first on nuScenes semantic segmentation and performs better than approaches using additional 2D information.
- Semantic Segmentation Results: 9.3% mIoU is gained for far points on Waymo, alongside a 3.3% mIoU improvement over the baseline.
- Object Detection Results: SphereFormer ranks third on the nuScenes LiDAR-only object detection benchmark and generalizes to instance-level tasks.
- Ablation Study: The radial window shape yields 5.67% and 13.39% mIoU gains for medium and far points, respectively, over the baseline comparison.
- Ablation Study: Window size affects information capture and optimization difficulty: overly small windows may miss meaningful information, while overly large windows may hinder optimization.
- Ablation Study: Exponential splitting adds 1.36% mIoU overall, while dynamic feature selection adds 0.8% mIoU mainly through improved close-point handling.
5. Conclusion
SphereFormer addresses varying-sparsity LiDAR distributions by enabling sparse distant points to aggregate information from dense close points through radial attention and complementary feature mechanisms.
- SphereFormer directly aggregates information from close dense points for sparse distant points using radial window self-attention.
- Exponential splitting provides more detailed position encoding, while dynamic selection of local or global features further improves the model.
- The method ranks first on nuScenes and SemanticKITTI semantic segmentation benchmarks and third on the nuScenes object detection benchmark.