Source-linked AI summary
Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds
Chenhang He, Ruihuang Li, Shuai Li, Lei Zhang
TL;DR
Large, sparse point clouds make global self-attention difficult, while grouped and convolutional alternatives respectively risk point handling issues or limited attention fields. VoxSeT introduces voxel-based set attention using latent codes and two cross-attentions, and achieves competitive detection performance on KITTI and Waymo. The approach provides a transformer-based alternative to convolutional and point-based backbones, though its explored linear-attention formulation limits expressive power.
Problem
Self-attention is difficult to compute on large, uneven point clouds, while grouped methods may drop points and convolutional attention typically has narrow fields.
Method
VoxSeT uses voxel-based set attention that applies two cross-attentions through latent-code-induced hidden spaces to process arbitrary-sized voxel clusters.
Results
VoxSeT achieves competitive performance on the Waymo and KITTI 3D detection benchmarks.
Takeaways & Limitations
VoxSeT provides a transformer-based alternative for modeling 3D point clouds alongside convolutional and point-based backbones.
Takeaways & Limitations
The explored induced latent-code linear-attention formulation limits VoxSeT's expressive power for representing different point-cloud structures and correlations.
Abstract
from arXiv · showhide
Transformer has demonstrated promising performance in many 2D vision tasks. However, it is cumbersome to compute the self-attention on large-scale point cloud data because point cloud is a long sequence and unevenly distributed in 3D space. To solve this issue, existing methods usually compute self-attention locally by grouping the points into clusters of the same size, or perform convolutional self-attention on a discretized representation. However, the former results in stochastic point dropout, while the latter typically has narrow attention fields. In this paper, we propose a novel voxel-based architecture, namely Voxel Set Transformer (VoxSeT), to detect 3D objects from point clouds by means of set-to-set translation. VoxSeT is built upon a voxel-based set attention (VSA) module, which reduces the self-attention in each voxel by two cross-attentions and models features in a hidden space induced by a group of latent codes. With the VSA module, VoxSeT can manage voxelized point clusters with arbitrary size in a wide range, and process them in parallel with linear complexity. The proposed VoxSeT integrates the high performance of transformer with the efficiency of voxel-based model, which can be used as a good alternative to the convolutional and point-based backbones. VoxSeT reports competitive results on the KITTI and Waymo detection benchmarks. The source codes can be found at \url{https://github.com/skyhehe123/VoxSeT}.
1. Introduction
The paper addresses inefficient or limited self-attention for sparse, unevenly distributed point clouds by introducing VSA and the VoxSeT architecture. VoxSeT uses voxel-based set attention to model arbitrary-sized token clusters and offers a transformer-based alternative for 3D point-cloud detection.
- Motivation: Self-attention is costly on long point-cloud sequences because its quadratic computation requires updating each token with all others.Point Transformer-style local grouping reduces scope, but does not remove the underlying efficiency challenge.
- Limitations of existing methods: Uneven point distributions force grouped methods to drop or pad points, causing unstable detection results and redundant computation.Grouping n points into m clusters also costs O(nm).
- Limitations of existing methods: Convolutional attention has O(n) complexity but typically uses small attention fields, hindering long-range dependency modeling.This limitation arises because convolutional attention is point-wise and memory-sensitive.
- Proposed method: VSA assigns trainable latent codes to each voxel and decomposes self-attention into two cross-attentions through a fixed-length hidden bottleneck.For n d-dimensional features and k latent codes, VSA has O(nkd) complexity and can use general matrix multiplications.
- Proposed method: VoxSeT combines VSA modules, MLP layers, and a shallow CNN for BEV extraction to learn point-cloud features through set-to-set translation.The architecture targets 3D object detection and can also be adopted in point-based detectors.
- Evaluation: Experiments on KITTI and Waymo show competitive performance, positioning VoxSeT as an alternative to convolutional and point-based backbones.The paper reports benchmark evaluation on both datasets.
2. Related work
Prior point-cloud detectors use compact voxel, image-view, or point-based representations, while transformer methods offer permutation- and cardinality-invariant processing. VoxSeT instead performs induced set attention locally within voxels, enabling arbitrary-sized token clusters with voxel-based efficiency.
- Voxel-based approaches: Early 3D detectors transform point clouds into BEV images, range images, volumetric features, or sparse tensors before feature extraction.Voxel features can also be stacked into pillars and processed with 2D CNNs.
- Voxel-based approaches: Voxel-based processing partitions point clouds into a grid and applies local self-attention, providing inductive bias and computational efficiency.The architecture is motivated by this representation choice.
- Transformer-based approaches: Transformer methods suit point clouds because self-attention is invariant to permutation and input cardinality, but standard self-attention has quadratic complexity.Point-cloud transformer approaches include operators and sequential processing strategies.
- VoxSeT: VoxSeT decomposes self-attention into two cross-attentions, allowing attention over token clusters of arbitrary size rather than fixed-size groups.This design follows the induced set transformer idea.
3. Methodology
VoxSeT introduces voxel-based set attention (VSA), which replaces expensive point-cloud self-attention with latent-code cross-attention in locally partitioned voxels. Its set-to-set backbone combines VSA with convolutional feature interaction and supports arbitrary voxel population sizes with linear complexity.
- Voxel-based Set Attention: Two cross-attentions approximate self-attention: one maps input points into latent hidden features, and the other reconstructs an output set of the original length.With a fixed number k of latent codes, each cross-attention costs O(nkd), linear in the number of points n.
- Voxel-based Set Attention: VSA partitions the scene into voxels and assigns latent codes to each voxel, locally compressing point features into a fixed-length hidden space.The design adapts induced set attention to point clouds, whose local regions contain stronger structural detail than the scene level.
- Voxel-based Set Attention: VSA avoids stochastic dropping or padding within voxels and gives the model linear complexity while processing variable-sized voxel clusters in parallel.Scatter kernels assign point features to voxel segments and support vectorized reduction across voxels.
- Complexity: The VSA encoder uses two GEMMs and two scatter operations, with overall complexity O(2n(k + 1)d).The decoder has complexity O(2nkd), and the encoder-decoder formulation treats point-cloud processing as set-to-set translation.
- Convolutional Feed-Forward Network: A ConvFFN refines voxel hidden features through depth-wise convolutions, enabling information exchange across voxels and adding spatial context.The enriched hidden features are scattered into a sparse 3D tensor before convolutional processing.
- Voxel Set Transformer: VoxSeT connects MLP and VSA modules in residual blocks, then uses a shallow CNN to encode point-wise features into a BEV representation for detection.Larger VSA voxels empirically provide richer context and better understanding of sparse pedestrian and cyclist instances.
4. Experiments
Experiments on Waymo and KITTI show that VoxSeT delivers strong detection performance across datasets and settings, while ablations and efficiency tests support its architectural choices and practical utility.
- Waymo results: VoxSeT outperforms PV-RCNN by 5% LEVEL 1 mAP and VoxelRCNN by 2.4% LEVEL 2 mAP on Waymo vehicle detection.
- Waymo results: VoxSeT improves over VoTR-TSD by 0.9% LEVEL 1 mAP and 1.4% LEVEL 2 mAP on Waymo.
- KITTI results: On KITTI single-stage detection, VoxSeT matches SECOND on vehicles and performs much better on pedestrian and cyclist classes.The authors attribute this to a wider effective receptive field from VSA conditioned on large voxels.
- KITTI results: As a two-stage detector, VoxSeT exceeds CT3D by 0.7% Easy, 0.4% Moderate and 0.3% Hard mAP on KITTI test.
- Generalization: VoxSeT shows consistently superior performance on KITTI and Waymo, whereas CT3D and VoTR vary in performance across the two datasets.
- Ablation study: Replacing ConvFFN with conventional FFN significantly degrades accuracy, while more latent codes enhance context encoding and modeling capacity.
- Backbone and efficiency: Replacing PointNet++ with VoxSeT in PointRCNN produces obvious performance improvements, and VoxSeT is faster with lower runtime memory than SECOND.The reported runtime profile and acceptable cost suggest suitability as a PointPillars alternative for real-time applications.
- Attention analysis: Attention visualizations show that VSA focuses on object regions and that different latent codes encode different object contexts.
5. Conclusion and discussions
VoxSeT is a transformer-based framework that models LiDAR point-cloud processing as set-to-set translation and uses voxel-based set attention for 3D detection. Its explored linear-attention formulation remains limited in expressive power, leaving stronger attention mechanisms as future work.
- VoxSeT models point-cloud feature learning as set-to-set translation while preserving the full resolution of raw point clouds during extraction.
- VSA performs self-attention on voxel clusters of arbitrary size and captures discriminative context from a large receptive field.
- VoxSeT achieves competitive performance on the Waymo and KITTI datasets, positioning it as an alternative for point-cloud modeling.
- VoxSeT explores only one latent-code-based formulation of linear attention, limiting its expressive power for representing diverse point-cloud structures and correlations.
- The authors identify stronger attention mechanisms as a future direction that could further improve performance.