Source-linked AI summary
SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds
Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, Dragomir Anguelov
TL;DR
Sparse point-cloud detection must cope with irregular occupancy and sparse features that complicate accurate object localization. SWFormer uses hierarchical sparse-window Transformers with bucketing, multi-scale context, and voxel diffusion, and reports state-of-the-art Waymo performance.
Problem
Sparse point occupancy and missing voxel features near object centers make accurate 3D object detection challenging.
Method
SWFormer processes dynamically voxelized point clouds with hierarchical sparse-window Transformers, bucketing, multi-scale fusion, and voxel diffusion.
Results
SWFormer achieves state-of-the-art 3D object-detection results on the Waymo Open Dataset.
Takeaways & Limitations
The model provides a pure Transformer-based approach for learning sparse 3D representations and detecting objects from sparse features.
Abstract
from arXiv · showhide
3D object detection in point clouds is a core component for modern robotics and autonomous driving systems. A key challenge in 3D object detection comes from the inherent sparse nature of point occupancy within the 3D scene. In this paper, we propose Sparse Window Transformer (SWFormer ), a scalable and accurate model for 3D object detection, which can take full advantage of the sparsity of point clouds. Built upon the idea of window-based Transformers, SWFormer converts 3D points into sparse voxels and windows, and then processes these variable-length sparse windows efficiently using a bucketing scheme. In addition to self-attention within each spatial window, our SWFormer also captures cross-window correlation with multi-scale feature fusion and window shifting operations. To further address the unique challenge of detecting 3D objects accurately from sparse features, we propose a new voxel diffusion technique. Experimental results on the Waymo Open Dataset show our SWFormer achieves state-of-the-art 73.36 L2 mAPH on vehicle and pedestrian for 3D object detection on the official test set, outperforming all previous single-stage and two-stage models, while being much more efficient.
1 Introduction
3D point-cloud detection must handle sparse occupancy and growing sensing ranges. SWFormer adapts hierarchical window Transformers with sparse-window bucketing, limited shifting, multi-scale fusion, voxel diffusion, and a new detection-head design, achieving state-of-the-art Waymo results.
- Challenges: Sparse point occupancy and expanding LiDAR sensing regions make 3D point-cloud learning distinct from dense language and image models.Some commercial LiDARs sense up to 250m and 300m in all directions.
- SWFormer: SWFormer groups variable-length sparse windows into buckets, enabling efficient Transformer processing despite differing numbers of non-empty voxels.A spatial window can have a fixed grid size but substantially different effective sequence lengths.
- SWFormer: One shifting operation plus multi-scale features and fusion provides cross-window context while avoiding repeated sparse-feature reordering.The fusion network uses custom sparse-feature downsampling and upsampling algorithms.
- Detection head: Voxel diffusion addresses inaccurate anchor placement by improving detection from sparse features.The contribution is presented as a generic module for the unique challenge of sparse-feature object detection.
- Results: SWFormer’s Waymo experiments demonstrate state-of-the-art 3D object-detection performance.The supplied passages identify extensive Waymo experiments and a state-of-the-art result, without giving the headline score.
2 Related Work
Prior 3D detectors use point-based, dense-grid, sparse-convolution, or multi-view approaches, each with scalability or representation limitations. SWFormer instead uses a hierarchical Transformer without dense or sparse convolutions to learn sparse point-cloud representations.
- Point-based methods: PointNet-style methods encode unordered points with MLPs and max pooling but are difficult to scale to large point clouds with good accuracy.Hierarchical structure improves local modeling, but these methods generally have lower representation capacity than newer approaches.
- Sparse convolutions: Sparse submanifold convolutions process sparse inputs efficiently but struggle to connect disconnected features and require heavily optimized custom operations.Their small kernels limit representation capacity, while custom operations are incompatible with matmul-optimized accelerators such as TPUs.
- Transformer methods: SWFormer learns 3D representations with a hierarchical Transformer without dense or sparse convolutions.The approach is inspired by hierarchical window-based Transformers and adds improved blocks, multi-scale fusion, and voxel diffusion.
- Transformer methods: Transformers suit sparse point clouds because they accept variable-length sequences without requiring dense 2D or 3D image representations.Prior 3D Transformer methods include Voxel Transformer and SST, but SST’s single stride limits receptive field for large objects.
3 Sparse Window Transformer
SWFormer adapts hierarchical window-based Transformers to sparse voxelized point clouds, using bucketing, limited window shifting, and multi-scale fusion to process variable-length sparse windows. Its voxel diffusion module expands foreground features near object centers to support detection from sparse representations.
- Hierarchical encoder: SWFormer voxelizes point clouds into sparse 2D BEV voxels and processes them with hierarchical Transformer blocks across five scales.The architecture uses strides {1, 2, 4, 16, 32} and combines the resulting features with multi-scale fusion before detection.
- Sparse window partition: Bucketing groups sparse windows by sequence length so variable-length non-empty voxel sequences can be padded, batched, and processed efficiently.Windows are partitioned into at most k buckets, with padded tokens masked during Transformer processing.
- Sparse Window Transformer block: Each SWFormer block applies self-attention within windows, performs one shifted window repartition, and then processes the shifted windows with additional Transformer layers.A block contains N + M Transformer layers but only one window-shift operation, reducing repeated shift overhead while connecting neighboring windows.
- Multi-scale fusion: Hierarchical scales and multi-scale feature fusion compensate for the limited receptive field caused by restricting window shifts.The encoder progressively increases strides, while fusion combines features from large-stride and small-stride representations.
- Voxel diffusion: Voxel diffusion first filters foreground voxels, then expands their features to neighboring locations through k × k max pooling on a dense BEV grid.The module addresses cases where no valid voxel lies near an object center, while filtering background voxels to preserve sparsity.
4 Experiments
Experiments on Waymo Open Dataset evaluate SWFormer’s accuracy, efficiency, and design choices. SWFormer achieves leading detection results while voxel diffusion, multi-scale features, and window shifting contribute substantially to performance.
- Experimental Setup: Waymo Open Dataset experiments use 1,150 scenes split into 798 training, 202 validation, and 150 test scenes.Each scene contains about 200 full-360-degree frames, and SWFormer uses all five LiDARs.
- Experimental Setup: Official evaluation reports BEV and 3D AP, heading-weighted APH, and L1 and L2 difficulty levels for vehicles and pedestrians.Leaderboard metrics use IoU cutoffs of 0.7 for vehicles and 0.5 for pedestrians; additional high-IoU and large-vehicle results are also reported.
- Main Results: 1.5 APH/L2 higher than RSN establishes SWFormer as the leading single-stage vehicle detector on the Waymo validation set.It also exceeds PVRCNN++ by 0.42 APH/L2 and improves large-vehicle AP/L2 over RSN by 6.35.
- Efficiency: 43ms inference latency on an Nvidia T4 is faster than PointPillars at about 100ms, with optimized kernels reducing latency to 20ms.The measurement uses a frame containing 68 vehicles and 69 pedestrians.
- Main Results: SWFormer outperforms all previous single-stage and two-stage methods on the official WOD test-set mAPH/L2 ranking metric.The comparison covers vehicle and pedestrian detection results reported in Table 3.
- Ablation Study: 6.37 and 3.22 3D AP drops occur for vehicle and pedestrian detection, respectively, when voxel diffusion is disabled.The comparison uses diffusion window size k = 1 versus k = 9; multi-scale features and window shifting also improve accuracy.
5 Conclusion
SWFormer is a scalable, accurate transformer-only model for learning 3D point-cloud representations from sparse features. It uses sparse-window processing and voxel diffusion, achieving state-of-the-art results on Waymo.
- SWFormer uses pure Transformer layers to process sparse point-cloud windows without convolutions.Its bucketing-based multi-scale architecture is designed to exploit point-cloud sparsity.
- Voxel diffusion further detects 3D objects from sparse features.
- SWFormer achieves state-of-the-art 3D object-detection results on the challenging Waymo Open Dataset.
A Window Shift
Window shifting partitions voxelized point clouds using shifted coordinates, increasing the number of non-empty sparse windows. Adding further shifts slowed training and slightly reduced accuracy.
- Half-window coordinate offsets rerun the partitioning algorithm to implement window shifting.
- 10% slower training and slightly lower accuracy followed one additional shift at scales 1 and 2.The authors hypothesize that extra shifts make training harder when receptive-field expansion is unnecessary.
- Four non-empty 4 × 4 windows become five after shifting an 8 × 8 BEV voxel grid.
B Qualitative Results
Qualitative visualizations show predictions nearly overlapping ground truth while attention patterns differ across layers and scales. Foreground-point attention scores are mostly binary-like, near 0 or 1.
- Predicted vehicle and pedestrian boxes almost perfectly overlap their ground-truth boxes in the selected Waymo validation frame.
- Different layers and scales capture different information, reflected in distinct attention-score patterns.
- Most attention scores for foreground query points are either 0 or 1.
C Future Work: More Tasks
The authors extend SWFormer to joint semantic segmentation and detection using an additional segmentation head and fused point- and voxel-level features. Preliminary results are reasonable, but the extension remains future work.
- The joint model predicts segmentation logits from concatenated per-point and fused voxel features.
- Reasonable joint segmentation results are reported on Waymo validation and test sets.
- An additional segmentation head extends the detection architecture to joint semantic segmentation and detection.
- The qualitative visualization distinguishes vehicles, pedestrians, buildings, roads, sidewalks, poles, signs, tree trunks, and vegetation by color.