Source-linked AI summary
Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection
Guowen Zhang, Lue Fan, Chenhang He, Zhen Lei, Zhaoxiang Zhang, Lei Zhang
TL;DR
Voxel serialization sacrifices spatial proximity, and Transformer complexity limits the size of groups used to address that loss. Voxel Mamba processes all voxels as one sequence with SSMs and adds hierarchical and positional mechanisms. It achieves state-of-the-art results on Waymo and nuScenes, with lower memory use than group-based Voxel Transformer methods reported without elaborated optimization.
Problem
Voxel serialization loses spatial proximity, while quadratic Transformer complexity limits group size and makes that loss difficult to address.
Source: §1 · pp. 1–2Method
Voxel Mamba uses linear-complexity SSMs to serialize all voxels as one sequence, with a Dual-scale SSM Block and positional encoding to enhance proximity.
Source: §1 · p. 2Results
Voxel Mamba achieves state-of-the-art results on Waymo and nuScenes, and consumes less memory than group-based Voxel Transformer methods without elaborated optimization.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
The group-free design models the entire voxel sequence with global context and avoids padding tokens.
Source: §1 · p. 2Hilbert Input Layer curve templates use GPU memory that becomes more substantial as voxel resolution increases.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Serialization-based methods, which serialize the 3D voxels and group them into multiple sequences before inputting to Transformers, have demonstrated their effectiveness in 3D object detection. However, serializing 3D voxels into 1D sequences will inevitably sacrifice the voxel spatial proximity. Such an issue is hard to be addressed by enlarging the group size with existing serialization-based methods due to the quadratic complexity of Transformers with feature sizes. Inspired by the recent advances of state space models (SSMs), we present a Voxel SSM, termed as Voxel Mamba, which employs a group-free strategy to serialize the whole space of voxels into a single sequence. The linear complexity of SSMs encourages our group-free design, alleviating the loss of spatial proximity of voxels. To further enhance the spatial proximity, we propose a Dual-scale SSM Block to establish a hierarchical structure, enabling a larger receptive field in the 1D serialization curve, as well as more complete local regions in 3D space. Moreover, we implicitly apply window partition under the group-free framework by positional encoding, which further enhances spatial proximity by encoding voxel positional information. Our experiments on Waymo Open Dataset and nuScenes dataset show that Voxel Mamba not only achieves higher accuracy than state-of-the-art methods, but also demonstrates significant advantages in computational efficiency.
1 Introduction
Voxel Mamba addresses the spatial-proximity loss caused by voxel serialization with a group-free SSM design. It adds two modules to improve proximity preservation and reports superior performance on Waymo Open and nuScenes.
1 Introduction
- Serialization can separate neighboring voxels, while Transformer complexity limits group size and makes that proximity loss difficult to address.Existing approaches group non-empty voxels into short sequences, sometimes separating neighboring voxels across groups.Source: §1 · pp. 1–2
- The Dual-scale SSM Block combines high-resolution forward and low-resolution backward branches, while Implicit Window Partition encodes voxel positions without explicit window partitioning.The hierarchy enlarges the sequence’s effective receptive field; positional embeddings encode positions inside and across windows.Source: §1 · p. 2
- Voxel Mamba serializes all voxels into one sequence, using linear-complexity SSMs to support group-free modeling without padding tokens.The authors describe this design as more efficient and deployment-friendly than previous methods.Source: §1 · p. 2
- Voxel Mamba achieves superior performance to previous state-of-the-art methods on Waymo Open and nuScenes datasets.Source: §1 · p. 2
2 Related Work
Related work covers point- and voxel-based detection, SSMs, space-filling curves, and grouping strategies. Voxel Mamba instead processes the entire voxel sequence to capture global context.
3D Object Detection from Point Clouds
- Point-based detectors extract geometry from local raw-point regions but have low inference efficiency and limited context; voxel-based approaches are described as the current mainstream.Voxel methods convert points into grids and use sparse convolutions or Transformers.Source: §2 · p. 3
State Space Models
- SSMs are presented as alternatives to CNNs and Transformers, with prior work applying them to language, vision, and point-cloud processing.The paper investigates SSMs as a baseline for LiDAR-based 3D detection in driving scenes.Source: §2 · p. 3
Space-filling Curve
- Space-filling curves reduce dimensionality while maintaining spatial topology and locality; the paper employs the Hilbert curve for locality preservation.Prior point-cloud work includes Hilbert, Z-order, Morton-order, and window-sweep serialization approaches.Source: §2 · p. 3
Point Cloud Grouping
- Grouping supports parallel computation and reduces complexity, but restricts information flow and effective receptive field; Voxel Mamba models all voxels as one sequence for global context.Prior methods use KNN, hash-table neighborhoods, windows, or space-filling curves to form groups.Source: §2 · p. 3
3 Methods
Voxel Mamba is a group-free voxel backbone that serializes a scene into one sequence, then combines hierarchical SSM processing and coordinate embeddings to preserve spatial proximity.
3.1 Preliminaries
- SSM maps 1D input to output through a hidden state, with learnable parameters and a residual term; discretization supports recurrent and parallel-convolutional forms.Source: §3.1 · p. 4
3.2 Overall Architecture
- The architecture serializes an entire scene into one sequence, processes it with a dual-scale SSM block, and uses implicit window partitioning to encode 3D position before producing BEV features.Source: §3.2 · p. 5
3.3 Hilbert Input Layer
- The Hilbert Input Layer maps voxel coordinates to traversal positions and sorts voxels into one sequence, using a space-filling curve to improve serialized spatial proximity.Source: §3.3 · p. 5
3.4 Dual-scale SSM Block
- The DSB combines a high-resolution forward path with a downsampled backward path, building a larger receptive field to mitigate proximity loss without extra parameters.Source: §3.4 · pp. 5–6
3.5 Implicit Window Partition
- IWP encodes voxel coordinates inside and across windows rather than explicitly partitioning tokens, so serialized features retain position and 3D proximity information.Source: §3.5 · p. 6
3.6 The Voxel Mamba Backbone
- The backbone organizes N DSBs into stages, scatters sparse tokens into dense BEV features, and integrates with different detection frameworks on Waymo and nuScenes.On Waymo it uses the Centerpoint-Pillar BEV backbone; on nuScenes it replaces DSVT's 3D backbone.Source: §3.6 · p. 6
4 Experiments
Across Waymo and nuScenes, Voxel Mamba outperforms prior detectors; experiments also examine speed and memory, component choices, downsampling, and receptive fields.
4.1 Datasets and Evaluation Metrics
- Waymo has 230k annotated samples and uses mAP/mAPH at L1 and L2; nuScenes has 40k samples and uses mAP/NDS.Waymo splits are 160k/40k/30k for train/validation/test; nuScenes splits are 28k/6k/6k.Source: §4.1 · p. 6
4.2 Implementation Details
- The implementation stacks six DSB blocks across three stages, uses 128-channel voxel features, and trains on eight RTX A6000 GPUs.Voxel sizes are (0.32m, 0.32m, 0.1875m) on Waymo and (0.3m, 0.3m, 0.2m) on nuScenes.Source: §4.2 · pp. 6–7
4.3 Comparison with State-of-the-art Methods
- 79.6/73.4 L1/L2 mAPH on Waymo validation and 79.6/74.3 on test; nuScenes validation reaches 71.9 NDS and 67.5 mAP, with best test mAP and NDS.Waymo validation is +1.4/+1.5 over DSVT-Voxel; nuScenes validation is +0.5 NDS and +0.8 mAP over the previous best.Source: Table 1 · p. 7, Table 2 · p. 7, §4.3 · p. 7
4.4 Ablation Studies
- Random serialization lowers performance, while Z-order and window partition improve it and Hilbert improves further; group-free SSM, voxelization, DSB, and IWE add gains over baseline.The ablations use the nuScenes validation set; the component baseline is CenterPoint-Pillar.Source: §4.4 · pp. 8–9, Table 6 · p. 9
4.3 Comparison with State-of-the-art Methods
- Voxel Mamba is at least +1.5 higher in detection accuracy and faster than DSVT and PV-RCNN++; CenterPoint-Pillar comparison adds only 0.5 GB memory for +9.0 L2 mAPH.Some methods are faster than Voxel Mamba, but their accuracy is substantially lower.Source: §4.3 · p. 8, Figure 3 · p. 9, Table 5 · p. 9
4.4 Ablation Studies
- Strides {1,2,4} balance ERF and detail: larger late-stage rates help, but {2,2,2} and {4,4,4} lose fine details; IWE improves spatial proximity.IWE is reported to add rich 3D positional and proximate information.Source: §4.4 · p. 9
4.5 Effective Receptive Field of Voxel Mamba
- Voxel Mamba shows a wider ERF than DSVT and group-based bidirectional Mamba under pillar inputs, covering a more complete local region.The figure visualizes merged ERFs, with wider activation indicating a larger receptive field.Source: Figure 4 · p. 10, §4.5 · p. 10
5 Conclusion
Voxel Mamba is a group-free SSM-based backbone for point-cloud 3D detection that uses DSB and IWP to improve receptive fields and spatial proximity. Experiments report state-of-the-art results on Waymo and nuScene, alongside efficiency advantages and limitations tied to voxel resolution and sampling design.
- Experiments show state-of-the-art results on Waymo and nuScene, with less memory than group-based Voxel Transformers and a more efficient, deployment-friendly group-free strategy.Source: §5 · p. 10
- Voxel Mamba uses a group-free, linear-complexity SSM strategy, with DSB and IWP to enlarge effective receptive fields and improve voxel spatial proximity.Source: §5 · p. 10
- Hilbert Input Layer curve templates consume GPU memory that grows with voxel resolution, while redesigned downsampling and upsampling may further improve efficiency.Source: §5 · p. 10