PaperlayerView exact revision on arXiv

Source-linked AI summary

Point Cloud Mamba: Point Cloud Learning via State Space Model

Tao Zhang, Haobo Yuan, Lu Qi, Jiangning Zhang, Qianyu Zhou, Shunping Ji, Shuicheng Yan, Xiangtai Li

arXiv:2403.00762v4cs.CV

TL;DR

Point cloud analysis needs efficient global modeling, while existing Mamba approaches still lag prior point-based methods. Point Cloud Mamba combines local and global modeling with Mamba, and outperforms prior methods across four datasets, reaching 79.6 mIoU on S3DIS with enhanced local features.

  • Problem

    Transformers incur high computation and memory costs, while PointMamba still has performance gaps against previous point-based methods.

    Source: Introduction · pp. 1–2

  • Method

    Point Cloud Mamba combines local modeling with global Mamba features, using Consistent Traverse Serialization and its six coordinate-order variants to sequence spatially adjacent points.

    Source: Introduction · p. 2

  • Results

    79.6 mIoU on S3DIS with enhanced local feature extraction surpasses PTv3 by 4.9 mIoU, while PCM outperforms PointNeXt and PTV3 on four datasets.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    PCM demonstrates that a Mamba-based framework combining local and global modeling can outperform point-based and transformer-based methods on point cloud benchmarks.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Recently, state space models have exhibited strong global modeling capabilities and linear computational complexity in contrast to transformers. This research focuses on applying such architecture to more efficiently and effectively model point cloud data globally with linear computational complexity. In particular, for the first time, we demonstrate that Mamba-based point cloud methods can outperform previous methods based on transformer or multi-layer perceptrons (MLPs). To enable Mamba to process 3-D point cloud data more effectively, we propose a novel Consistent Traverse Serialization method to convert point clouds into 1-D point sequences while ensuring that neighboring points in the sequence are also spatially adjacent. Consistent Traverse Serialization yields six variants by permuting the order of \textit{x}, \textit{y}, and \textit{z} coordinates, and the synergistic use of these variants aids Mamba in comprehensively observing point cloud data. Furthermore, to assist Mamba in handling point sequences with different orders more effectively, we introduce point prompts to inform Mamba of the sequence's arrangement rules. Finally, we propose positional encoding based on spatial coordinate mapping to inject positional information into point cloud sequences more effectively. Point Cloud Mamba surpasses the state-of-the-art (SOTA) point-based method PointNeXt and achieves new SOTA performance on the ScanObjectNN, ModelNet40, ShapeNetPart, and S3DIS datasets. It is worth mentioning that when using a more powerful local feature extraction module, our PCM achieves 79.6 mIoU on S3DIS, significantly surpassing the previous SOTA models, DeLA and PTv3, by 5.5 mIoU and 4.9 mIoU, respectively.

Introduction

Point Cloud Mamba (PCM) combines local point-cloud modeling with Mamba-based global modeling at linear computational complexity. It uses Consistent Traverse Serialization, order prompts, and spatial coordinate mapping to adapt Mamba to point-cloud sequences.

Introduction

  • Point clouds are unordered and irregular; prior point-based methods emphasize local geometry, transformers add global modeling at high computational and memory cost, and PointMamba had performance gaps.The introduction motivates Mamba as a way to model long-range dependencies while retaining linear computational complexity.Source: Introduction · pp. 1–2
  • PCM combines local and global modeling, using Mamba to model point-cloud global features with linear computational complexity.The paper presents PCM as its framework for applying Mamba to point-cloud analysis.Source: Introduction · p. 2
  • Consistent Traverse Serialization (CTS) creates spatially adjacent point sequences and six coordinate-order variants; order prompts convey sequence rules, while spatial coordinate mapping supplies positional embeddings.The paper says the coordinate-mapping embedding is more suitable for irregular point clouds than RoPE and learnable embeddings.Source: Introduction · p. 2
  • 79.6 mIoU on S3DIS with enhanced local feature extraction exceeds PTv3 by 4.9 mIoU; PCM also outperforms PointNeXt on ScanObjectNN, ModelNet40, and ShapeNetPart.The introduction also reports that PCM outperforms PointNeXt and PTV3 on ScanObjectNN, ModelNet40, ShapeNetPart, and S3DIS.Source: Introduction · p. 2

Related Work

Related work progresses from MLP- and graph-based point-cloud processing to transformers and state space models. The paper positions Point Cloud Mamba as an exploration of the architecture, serialization, and positional encoding needed to apply Mamba effectively to point clouds.

3D Point Cloud Classification

  • PointNet and PointNet++ pioneered direct MLP-based point-cloud processing, while later work explored graph-based geometry, other architectures, pretraining, and in-context abilities.The cited architectural work includes MLPs and transformers; the paper contrasts its results with a concurrent point-cloud SSM work that had performance gaps versus prior methods.Source: Related Work · p. 2

3D Visual Transformers

  • PCM combines local point processing with a new traverse serialization strategy, which the paper says yields better results than direct SSM traversal.Earlier point-cloud transformers used global attention, which had memory and computational-complexity issues; Point Transformer introduced local attention to address them.Source: Related Work · p. 2
  • Transformer-based point-cloud methods explore serialization and positional embeddings, but Mamba requires tailored architecture, serialization, and positional-embedding strategies rather than a simple layer replacement.This is the stated objective of the paper.Source: Related Work · p. 2

State Space Models

  • State space models can capture long-range dependencies; S4 reduces computation and memory costs, while Mamba adds a selection mechanism, motivating their exploration for point clouds.The paper presents PCM as further demonstrating SSM potential in 3D point clouds, with results that can exceed previous architectures.Source: Related Work · p. 3

Method

Point Cloud Mamba adapts Mamba’s global, linear-complexity sequence modeling to non-causal 3-D point clouds through ordered serialization, sequence-order prompts, spatial positional encoding, and a multi-stage architecture.

Preliminaries

  • Mamba (Gu and Dao 2023) uses input-dependent system parameters to model long sequences with linear inference complexity, but its causal 1-D design complicates direct use on 3-D point clouds.The method aims to combine Mamba’s global modeling with local point-cloud modeling.Source: Preliminaries · p. 3, Method · p. 3

A Naive Mamba-based Point Cloud Network

  • 84.1 OA on ScanObjectNN was achieved by the naive z-order, bidirectional Mamba network, but single-order serialization loses spatial relations and its PointMLP (Ma et al. 2022) design may not suit Mamba.The naive network converts points to a sequence with z-order serialization and uses bidirectional Mamba so each point can receive features from other points.Source: A Naive Mamba-based Point Cloud Network · pp. 3–4

Point Cloud Mamba

  • Consistent Traverse Serialization (CTS) preserves spatial adjacency in point sequences and offers six axis-order variants that provide complementary point-cloud observations.PCM uses different serialization strategies across Mamba layers so the model can observe the point cloud more comprehensively.Source: Point Cloud Mamba · p. 4, Figure 3 · p. 4
  • Learnable order prompts identify each serialization to Mamba, with embeddings added to both sequence ends and mapped to each stage’s feature size.A shared Linear layer within each stage maps the prompts to the required channel size.Source: Point Cloud Mamba · p. 4
  • A shared coordinate-to-embedding mapping encodes spatial position because sequence positions poorly represent gaps between adjacent points in sparse, irregular clouds.The positional mapping is stage-specific, while Mamba layers within a stage share its function.Source: Point Cloud Mamba · p. 5
  • PCM uses four stages with geometric affine modules and Mamba layers; its segmentation decoder interpolates points, concatenates multi-stage features, and applies MLP channel transformation.PCM-Tiny’s stage settings are in Table 1; PCM increases layer counts and channel sizes and adopts additional serialization methods.Source: Point Cloud Mamba · p. 5

Experiments

Experiments evaluate PCM on ScanObjectNN and ModelNet40 classification, ShapeNetPart part segmentation, and S3DIS semantic segmentation, with ablations examining serialization, channel size, positional encoding, order prompts, and local features.

Main Results

  • On ScanObjectNN PB_T50_RS, PCM achieved 88.1 OA and 86.6 mAcc, with OA exceeding PointNeXt by 0.4 and PTv3 by 1.7.PCM-Tiny reached 86.9 OA and 85.0 mAcc using 20% of PCM’s parameters, while outperforming PointMLP by 1.5 OA and 1.1 mAcc.Source: Main Results · pp. 5–6
  • PCM achieved 93.4 OA and 90.7 mAcc on ModelNet40, and 87.0 Ins. mIoU and 85.3 Cls. mIoU on ShapeNetPart, exceeding PointNeXt by 0.5 Ins. mIoU.On ShapeNetPart, PCM used no extra test augmentation; PCM-Tiny reached 86.9 Ins. mIoU and 85.0 Cls. mIoU, while PCM outperformed PointMLP by 1.0 Ins. mIoU and 1.0 Cls. mIoU.Source: Main Results · p. 6
  • With four DeLA local-feature blocks, PCM-Tiny reached 79.6 mIoU on S3DIS, exceeding DeLA by 5.5 mIoU and PTv3 by 4.9 mIoU.Without that extractor, PCM achieved 74.1 mIoU and 92.9 OA, surpassing PointNext by 3.0 mIoU and 1.9 OA and PTv2 by 1.4 mIoU and 1.3 OA; PCM-Tiny with the extractor also reached 95.1 OA and 82.8 mAcc.Source: Main Results · p. 6, Table 4 · p. 6

Ablation Analysis and Visualization

  • Combining serialization variants improved classification over a single xyz order: xyz, yzx, and zxy gained 0.17 OA and 0.11 mAcc, while all six variants gained 0.39 OA and 0.51 mAcc.Adding Hilbert, z, and z-trans to all six variants yielded gains of 0.49 OA and 0.54 mAcc; different orders let Mamba layers observe point clouds from different perspectives.Source: Ablation Analysis and Visualization · pp. 6–7, Table 5 · p. 6
  • Reducing channel size from 768 to 384 lowered performance by 1.12 OA and 1.41 mAcc, while reducing it to 96 lowered performance by 2.56 OA and 3.76 mAcc.The final configuration uses 384 channels in the first two stages and 768 in the last two; reducing the first two stages to 384 caused only minor decay while saving computation.Source: Table 6 · p. 6, Ablation Analysis and Visualization · p. 7
  • Spatial-coordinate mapping with a Linear layer improved performance over RoPE by 0.37 OA and 0.7 mAcc; replacing it with an MLP did not improve results.RoPE and learnable positional embeddings encode sequence position and performed poorly or comparably, while separate Linear layers per Mamba layer caused overfitting.Source: Ablation Analysis and Visualization · p. 7, Table 7 · p. 7
  • Six order prompts improved performance by 0.83 OA and 1.47 mAcc over none, while using 12 neighboring points for local features produced the highest performance in that ablation.Four neighbors raised PCM to 84.66 OA and 82.72 mAcc, gains of 5.34 OA and 6.47 mAcc over no local features; more than 12 neighbors reduced performance.Source: Ablation Analysis and Visualization · p. 7, Table 8 · p. 7, Table 9 · p. 7

Conclusion

Point Cloud Mamba applies Mamba to point-cloud learning using sequence and positional techniques, and reports state-of-the-art performance across four datasets.

  • Point Cloud Mamba achieves SOTA on ScanObjectNN, ModelNet40, ShapeNetPart, and S3DIS, and for the first time outperforms PointNeXt and PTv3.Source: Conclusion · p. 7
  • Consistent Traverse Serialization converts 3D point clouds into 1D sequences while keeping neighboring sequence points spatially adjacent.Source: Conclusion · p. 7
  • Order prompts provide sequence arrangement rules to help Mamba handle point sequences serialized in different orders.Source: Conclusion · p. 7
  • Spatial coordinate mapping supplies a simple, effective positional encoding for point cloud sequences.Source: Conclusion · p. 7
  • With a stronger local feature extractor, Point Cloud Mamba also outperforms previous SOTA methods by a large margin on S3DIS.Source: Conclusion · p. 8

Implementation Details

Implementation adds cascaded DeLA blocks before PCM for segmentation and uses dataset-specific training schedules and input sampling.

Additional local feature extractor

  • PCM adds four cascaded DeLA (Chen et al. 2023a) blocks before PCM; their features feed PCM for local and global modeling, and both components train from scratch on segmentation datasets.The added DeLA blocks provide local features before PCM performs local and global modeling.Source: Implementation Details · p. 11

Experiment Setup

  • Training uses AdamW (Loshchilov and Hutter 2017), an initial learning rate of 1e-4, cosine decay, and weight decay of 1e-4; warmup is 5 epochs, or 5% of iterations on S3DIS.Batch sizes are 32 for ScanObjectNN, ModelNet40, and ShapeNetPart, and 16 for S3DIS.Source: Implementation Details · p. 11
  • Training lasts 250 epochs on ScanObjectNN and ModelNet40, 300 on ShapeNetPart, and 3000 on S3DIS; ShapeNetPart learning-rate decay follows PointNeXt (Qian et al. 2022).ShapeNetPart decay occurs at epochs 210 and 270, with a decay rate of 0.5.Source: Implementation Details · p. 11
  • Inputs use 1,024 points on ScanObjectNN and ModelNet40 (randomly sampled for training, farthest-point sampled for testing), 2,048 ShapeNetPart points with normals, and 30,000 randomly sampled S3DIS training points.PCM-Tiny uses the same experimental settings as PCM on all datasets.Source: Implementation Details · p. 11
  • Ablations use default PCM on ScanObjectNN for 125 epochs, with all other settings matching the main experiment.The ablation setup is shortened from the main experiment; the table caption identifies an ablation on points downsampling but gives no results.Source: Implementation Details · p. 11

More Experiment Results.

Additional experiments report PCM’s benchmark comparisons, gains from multiple serialization, and trade-offs involving model efficiency and point downsampling.

3D semantic segmentation in ScanNet dataset

  • 75.5 mIoU on ScanNet; PCM exceeded DeLA by 0.8 mIoU, 1.4 mAcc, and 0.2 OA, and PointNeXt-XL by 4.0 mIoU.The ScanNet comparison is attributed to Figure 10 in the passage.Source: More Experiment Results · p. 11

Comparison with naive mamba-based architecture

  • 0.4 Ins. mIoU and 0.5 Cls. mIoU are PCM’s gains over the naive architecture on ShapeNetPart.The naive architecture also surpasses PointMamba by 0.8 Ins. mIoU and 0.9 Cls. mIoU.Source: More Experiment Results · p. 11

Point downsampling

  • 0.15 OA and 0.17 mAcc were the losses from downsampling by a factor of 2 at all stages except the first, which reduced computation.Excessive downsampling to 64 points at the last stage reduced performance by 0.67 OA and 0.59 mAcc; PCM ultimately used the strategy designated line 2.Source: More Experiment Results · p. 11

Comparison with naive mamba-based architecture

Comparison of the multiple serialization strategy with the single serialization strategy

  • 0.42 OA and 0.87 mAcc gains over single serialization were reported for classification; on ShapeNetPart, multiple serialization improved Ins. mIoU by 0.4 and Cls. mIoU by 0.7.Source: More Experiment Results · p. 12

Comparison of the Parameters, FLOPs, and Throughput

  • PCM-Tiny exceeded PointMLP by 1.5 OA and 1.1 mAcc with 52% of its parameters and 35% of its computational complexity, but had lower throughput.PCM exceeded PointMLP by 2.7 OA and 2.7 mAcc with more parameters and computational complexity.Source: More Experiment Results · p. 12

Visualization Results

Visualizations show that PCM handles elongated and flattened objects well, but can misclassify small parts and is vulnerable to point clouds with many missing points.

  • PCM performs poorly on some smaller object parts, often confusing them with similar parts of larger objects.Source: Visualization Results · p. 12
  • PCM is susceptible to errors when point clouds contain numerous missing points, as illustrated by a car example.Source: Visualization Results · p. 12
  • PCM achieves good results on elongated and flattened objects, supporting the feasibility of point-sequence serialization and Mamba for modeling global features in irregularly distributed point clouds.Source: Visualization Results · p. 12

Further works.

Future work will address PCM’s difficulty with global modeling of large-scale point clouds and explore scalable serialization, stronger local-feature integration, and adaptation to complex outdoor scenes.

Further works.

  • Future work will explore scalable serialization methods to bridge the gap between cropped training and whole-cloud testing.The paper identifies this training–testing mismatch as a limitation of Mamba-based methods.Source: Further works · pp. 12–13
  • For point clouds with ≥100k points, scan-based training incurs quadratic complexity and requires cropping, creating a training–inference gap for global modeling.At testing, the whole cloud is often processed at once, which the paper says limits Mamba-based performance.Source: Further works · pp. 12–13
  • Further directions include better combining local feature extractors with PCM and adapting it to huge, more complex outdoor point-cloud scenes.The paper identifies both as challenges for future work.Source: Further works · p. 13
Loading 2403.00762v4…