PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral-Spatial Mamba for Hyperspectral Image Classification

Lingbo Huang, Yushi Chen, Xin He

arXiv:2404.18401v3cs.CV

TL;DR

HSI classification needs models that capture rich spatial-spectral information without the quadratic cost of Transformer self-attention. SS-Mamba addresses this by processing spatial and spectral token sequences with stacked Mamba blocks and feature enhancement, achieving competitive results on widely used HSI datasets. Its token partitioning can disrupt semantic structure and overlook object orientation and shape.

  • Problem

    Transformer self-attention models long-range HSI dependencies but has quadratic computational complexity, making it computationally heavy for high-dimensional HSI data.

  • Method

    SS-Mamba converts HSI cubes into spatial and spectral token sequences, processes them separately with stacked Mamba blocks, and fuses them using center-region feature enhancement.

  • Results

    SS-Mamba achieves competitive classification results compared with state-of-the-art methods on four widely used HSI datasets.

  • Takeaways & Limitations

    Mamba provides a sequence-modeling framework for HSI classification that combines spectral-spatial information while leveraging computational efficiency and long-range feature extraction.

  • Takeaways & Limitations

    The token partitioning may disrupt HSI semantic structure because it does not account for object orientation and shape, potentially separating pixels from the same object across patches.

Abstract

from arXiv · show

Recently, deep learning models have achieved excellent performance in hyperspectral image (HSI) classification. Among the many deep models, Transformer has gradually attracted interest for its excellence in modeling the long-range dependencies of spatial-spectral features in HSI. However, Transformer has the problem of quadratic computational complexity due to the self-attention mechanism, which is heavier than other models and thus has limited adoption in HSI processing. Fortunately, the recently emerging state space model-based Mamba shows great computational efficiency while achieving the modeling power of Transformers. Therefore, in this paper, we make a preliminary attempt to apply the Mamba to HSI classification, leading to the proposed spectral-spatial Mamba (SS-Mamba). Specifically, the proposed SS-Mamba mainly consists of spectral-spatial token generation module and several stacked spectral-spatial Mamba blocks. Firstly, the token generation module converts any given HSI cube to spatial and spectral tokens as sequences. And then these tokens are sent to stacked spectral-spatial mamba blocks (SS-MB). Each SS-MB block consists of two basic mamba blocks and a spectral-spatial feature enhancement module. The spatial and spectral tokens are processed separately by the two basic mamba blocks, respectively. Besides, the feature enhancement module modulates spatial and spectral tokens using HSI sample's center region information. In this way, the spectral and spatial tokens cooperate with each other and achieve information fusion within each block. The experimental results conducted on widely used HSI datasets reveal that the proposed model achieves competitive results compared with the state-of-the-art methods. The Mamba-based method opens a new window for HSI classification.

1. Introduction

HSI classification benefits from rich spatial-spectral information but traditional methods rely on handcrafted features, while Transformers capture long-range dependencies at prohibitive quadratic cost. SS-Mamba applies Mamba to HSI classification through spectral-spatial tokenization, stacked Mamba blocks, and feature enhancement for token fusion.

  • HSIs capture hundreds of narrow spectral bands alongside spatial information, supporting pixel-level land-cover classification across applications.
  • Traditional classifiers require manually crafted, scenario-specific features and have limited ability to extract high-level image representations.
  • Deep learning learns high-level discriminative features from HSI data and includes models such as autoencoders, belief networks, CNNs, and RNNs.
  • Transformers model long-range spatial-spectral dependencies, but self-attention causes quadratic computational complexity that is prohibitive for high-dimensional HSI data.
  • SS-Mamba converts HSI cubes into spatial and spectral token sequences, processes them with stacked dual-branch Mamba blocks, and fuses them through center-region feature enhancement.

2. Methodology

SS-Mamba converts HSI cubes into spatial and spectral token sequences, processes them with separate Mamba branches, and fuses their information through center-region-guided enhancement.

  • 2. Methodology: SS-Mamba combines a token-generation module with stacked spectral-spatial Mamba blocks for deep HSI feature extraction.The overall design also performs spectral-spatial interaction during token generation, within Mamba blocks, and during mean-token addition.
  • 2.1. Overview of the State Space Models: The state-space formulation maps input sequences to outputs through hidden states and can be reformulated as convolution for efficient parallel training.The convolution uses structured kernels K̅, whose length depends on the input sequence length N.
  • 2.2. Spectral-Spatial Tokens Generation: Spatial tokens are generated by spectral mapping, spatial partitioning, and patch embedding, while spectral tokens use spatial mapping, spectral partitioning, and patch embedding.Spatial processing uses the full HSI sample, whereas spectral processing begins from a small center region and partitions along the spectral dimension.
  • 2.2. Spectral-Spatial Tokens Generation: The spectral branch extracts tokens from a center region of size S×S, with S set to 3, before spectral partitioning and patch embedding.The center region is used instead of only the center pixel to make spectral features more robust.
  • 2.3. Spectral-Spatial Mamba Block: Each spectral-spatial Mamba block separately processes spatial and spectral tokens, then enhances them using center-region information from the opposite token type.The enhancement derives a central spatial token and an averaged spectral feature, fuses them, applies an MLP and sigmoid scaling, and multiplies the resulting weights with the tokens.

3. Results

SS-Mamba is evaluated on four widely used HSI datasets against diverse traditional, CNN-based, Transformer-based, attention-based, and mixture methods. Across these datasets and limited-training settings, it achieves superior or competitive classification results, while spectral-spatial learning and Mamba-based sequence modeling outperform corresponding alternatives.

  • Ablation Experiment with Basic Sequence Model: Spectral-spatial models outperform spatial and spectral models using the same sequence model, while spectral models are least effective.On Pavia University, spectral-spatial Mamba exceeds spatial Mamba by 2.77 OA points, 5.14 AA points, and 0.0364 K.
  • Ablation Experiment with Basic Sequence Model: Mamba-based models outperform LSTM, GRU, and Transformer counterparts within the evaluated learning frameworks.On Pavia University, spectral-spatial Mamba exceeds spectral-spatial GRU by 0.45 OA points, 0.52 AA points, and 0.0046 K; on Houston, it exceeds Transformer by 0.92 OA points.
  • Ablation Experiment with Basic Sequence Model: Spectral models are harder to train, requiring larger learning rates and more epochs, especially on Indian Pines.The authors associate their poorer Indian Pines performance with the dataset’s low quality.
  • Classification results: SS-Mamba achieves superior classification performance over comparison methods across all four datasets when using twenty training samples per class.The evaluated datasets are Indian Pines, Pavia University, Houston, and Chikusei.
  • Classification results: 0.53 percentage points, 3.68 percentage points, and 0.0073 are SS-Mamba’s reported improvements over DBDA in OA, AA, and K, respectively.These values are reported for the comparison described in Table 9.
  • Classification results: 0.38 percentage points, 0.63 percentage points, and 4.12 percentage points are SS-Mamba’s OA improvements over MSSG, DBDA, and SSFTT on Houston, respectively.The comparison is reported for the Houston dataset.

4. Discussion

The discussion highlights SS-Mamba’s efficiency, feature-map behavior, end-to-end design, and tokenization limitation. Mamba is faster than Transformer, while sequential models trade longer inference for higher accuracy than CNNs.

  • Complexity Analysis: Mamba achieves faster inference than Transformer, while sequential models generally take longer than CNNs but attain higher classification accuracies.The complexity analysis evaluates batches of 100 Pavia University samples; Mamba-2 is noted as a possible future efficiency improvement.
  • Features Maps: Spatial feature maps retain image details across blocks, although semantic structure gradually becomes blurred with depth.Different channels emphasize different image information, while depth-related changes remain limited because the model is shallow.
  • Comparison with Spectral Unmixing: Unlike spectral-unmixing pipelines, SS-Mamba performs end-to-end feature extraction and classification without intermediate processing or expert mixing-model knowledge.The authors identify this streamlined design as more efficient and user-friendly, while suggesting future combinations with spectral unmixing.
  • Limitations and Feature Work: Token partitioning can disrupt semantic structure because object orientation and shape are not considered, potentially separating pixels from the same object across patches.Future work proposes combining SS-Mamba with CNNs to improve local feature extraction and token generation.

5. Conclusions

The paper introduces SS-Mamba as a Mamba-based spectral-spatial model for HSI classification and evaluates it on four widely used datasets. Its experiments report superior or competitive classification performance and effective spectral-spatial learning.

  • Conclusions: SS-Mamba converts HSI cubes into spectral and spatial token sequences, then uses stacked Mamba blocks to model and fuse their relationships.The model enhances both token types to produce more discriminant features.
  • Conclusions: Across Indian Pines, Pavia University, Houston, and Chikusei, SS-Mamba achieves superior HSI-classification performance by using spatial-spectral information.The conclusion summarizes comparative experiments across four widely used datasets.
  • Conclusions: Ablation experiments show that Mamba provides competitive classification performance against Transformer, LSTM, and GRU sequence models.The conclusion also reports that the designed spectral-spatial learning components contribute to the model’s performance.
Loading 2404.18401v3…