PaperlayerView exact revision on arXiv

Source-linked AI summary

MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification

Yapeng Li, Yong Luo, Lefei Zhang, Zengmao Wang, Bo Du

arXiv:2501.04944v1cs.CV

TL;DR

HSI classification needs fine-grained pixel representations, long-range dependencies, and integrated spatial-spectral modeling, while CNN locality and Transformer quadratic complexity limit existing approaches. MambaHSI uses whole-image spatial and spectral Mamba blocks with adaptive fusion, and experiments on four diverse datasets report superior performance over CNN- and Transformer-based models.

  • Problem

    Existing HSI classification methods have limited long-range modeling because CNNs are local and Transformers have quadratic computational complexity, while HSI requires refined pixel-level spatial-spectral representations.

  • Method

    MambaHSI is a pure-SSM, image-level framework that uses spatial and spectral Mamba blocks plus adaptive spatial-spectral fusion for whole-image classification.

  • Results

    MambaHSI achieves superior performance over state-of-the-art CNN-based and Transformer-based HSI classification models across experiments on four diverse real-world datasets.

  • Takeaways & Limitations

    MambaHSI demonstrates the potential of Mamba as a backbone for HSI models that combine long-range interaction with spatial and spectral information.

Abstract

from arXiv · show

Transformer has been extensively explored for hyperspectral image (HSI) classification. However, transformer poses challenges in terms of speed and memory usage because of its quadratic computational complexity. Recently, the Mamba model has emerged as a promising approach, which has strong long-distance modeling capabilities while maintaining a linear computational complexity. However, representing the HSI is challenging for the Mamba due to the requirement for an integrated spatial and spectral understanding. To remedy these drawbacks, we propose a novel HSI classification model based on a Mamba model, named MambaHSI, which can simultaneously model long-range interaction of the whole image and integrate spatial and spectral information in an adaptive manner. Specifically, we design a spatial Mamba block (SpaMB) to model the long-range interaction of the whole image at the pixel-level. Then, we propose a spectral Mamba block (SpeMB) to split the spectral vector into multiple groups, mine the relations across different spectral groups, and extract spectral features. Finally, we propose a spatial-spectral fusion module (SSFM) to adaptively integrate spatial and spectral features of a HSI. To our best knowledge, this is the first image-level HSI classification model based on the Mamba. We conduct extensive experiments on four diverse HSI datasets. The results demonstrate the effectiveness and superiority of the proposed model for HSI classification. This reveals the great potential of Mamba to be the next-generation backbone for HSI models. Codes are available at https://github.com/li-yapeng/MambaHSI .

I. INTRODUCTION

HSI classification benefits from rich spectral information but requires fine-grained pixel representations and long-range modeling. MambaHSI addresses these needs with spatial and spectral Mamba blocks, adaptive fusion, and linear computational complexity.

  • Motivation: HSI captures dozens or hundreds of continuous wavebands, enabling identification of objects difficult to distinguish in RGB images.HSI supports applications including urban mapping, resource exploration, and environmental monitoring.
  • Motivation: CNN locality and Transformer quadratic complexity limit existing HSI methods’ ability to model long-range dependencies.These limitations are especially relevant to pixel-level classification, which requires refined representations that distinguish subtle pixel differences.
  • Proposed Framework: MambaHSI is an image-level, pure-SSM framework that takes the whole HSI as input and models long-range dependencies with linear complexity.The model is trained end-to-end and produces predictions for all pixels in one forward pass.
  • Proposed Framework: Spatial and spectral Mamba blocks separately extract spatial and spectral information while modeling long-range interaction of the whole image.The blocks are designed to capture discriminative features for classification.
  • Proposed Framework: The spatial-spectral fusion module adaptively estimates the importance of spatial and spectral information to guide their fusion.Residual learning is introduced to assist module training.
  • Results: MambaHSI achieves superior performance to state-of-the-art CNN-based and Transformer-based HSI classification models.The paper reports extensive experiments on four diverse real-world HSI datasets.

II. RELATED WORK

Prior HSI classification methods include ML, GCN, CNN, Transformer, and SSM approaches, each addressing spatial, spectral, or long-range modeling with different limitations. MambaHSI combines spatial and spectral Mamba blocks with adaptive fusion to provide whole-image long-range modeling at linear complexity.

  • Hyperspectral Image Classification: ML-based methods classify spectral features but can lack surrounding spatial information, leading to suboptimal performance.These methods may use only the spectral information of a single pixel.
  • Hyperspectral Image Classification: GCNs model long-range spatial relations and can jointly represent spatial and spectral neighborhoods or superpixel information.The cited approaches extend GCNs, use superpixel segmentation, or exploit information inside and outside superpixels.
  • Hyperspectral Image Classification: CNNs extract hierarchical features and support spatial-spectral or image-level processing, but their locality prevents modeling long-range dependencies.Image-level FCN methods were introduced to reduce running time while separately or jointly extracting spatial and spectral features.
  • Hyperspectral Image Classification: Transformers model long-range dependencies and global spatial features, but some methods ignore spatial location or incur quadratic computational complexity.Spatial-spectral Transformers were proposed to capture both types of information, while patch tokens are often used because of computational cost.
  • State Space Models: SSMs capture long-range dependencies with linear complexity and can efficiently parallelize calculations, motivating their use as alternatives to CNNs and Transformers.Mamba improves S4 with selective, input-dependent information selection and hardware-aware efficiency.
  • State Space Models: Prior visual SSM work extends one-dimensional S4 to images and videos, combines S4 with attention, or applies Mamba to biomedical segmentation.These examples establish SSM use beyond sequence modeling.
  • State Space Models: Existing Mamba HSI methods include spectral, spectral-spatial, and three-dimensional approaches, while MambaHSI uses whole-image input and adaptive spatial-spectral fusion.The proposed spatial and spectral Mamba blocks capture the two information types, and SSFM fuses them linearly.

III. PRELIMINARIES

HSI classification requires fine-grained pixel representations and long-range modeling, but CNN locality and Transformer quadratic complexity constrain existing approaches. Mamba-based state space models offer linear-complexity sequence modeling through discretized continuous systems.

  • State Space Models: SSMs map a one-dimensional input sequence to an output through a hidden state governed by a linear ordinary differential equation.The state matrix and projection parameters define the continuous system.
  • State Space Models: S4 and Mamba discretize the continuous system using a timescale parameter and zero-order hold before computation.The discretized model can be evaluated through linear recurrence or global convolution.
  • Motivation: HSI classification needs long-range dependencies to capture discriminative pixel-level features, while CNNs are local and Transformers have quadratic complexity.Transformer patch tokenization can hinder finer pixel-level representations.

B. Overview

MambaHSI processes the whole hyperspectral image through pixel-level embedding, an encoder, and a segmentation head. Its encoder combines spatial modeling, spectral modeling, and spatial-spectral fusion for dense classification.

  • Overview: The embedding layer projects each pixel’s spectral vector into an embedding space rather than using patch-based embeddings.The resulting embedding has spatial dimensions H × W and embedding dimension D.
  • Overview: The encoder backbone extracts discriminative spatial-spectral features using spatial Mamba, spectral Mamba, and spatial-spectral fusion components.The segmentation head applies a 1 × 1 convolution to produce final logits.
  • Spatial Mamba Block: SpaMB models long-range dependencies over pixel-level embeddings with linear computational complexity.The block uses a flattened image sequence, group normalization, a Mamba block, and residual learning.
  • Spatial Mamba Block: The spatial Mamba block reshapes learned residual spatial features back to image dimensions and adds them to the input embeddings.Its sequence length is L1 = H × W, and experiments set D to 128.

D. Spectral Mamba Block

SpeMB addresses hyperspectral spectral relationships by dividing spectral features into groups, modeling relations across groups, and updating features using those relationships.

  • Spectral Grouping: SpeMB divides spectral features into G groups to model relationships across different spectral groups.This design targets discriminative spectral feature extraction from contiguous hyperspectral information.
  • Feature Extraction: The grouped spectral features are flattened and processed to obtain residual spectral features and output spectral features.HGspe, HFspe, HRspe, and Ho_spe denote divided, flattened, residual, and output spectral features, respectively.
  • Feature Extraction: The output spectral features retain the original embedding dimension D, while each group has dimension M = D/G.The standard Mamba block is used to process the grouped spectral representation.

E. Spatial-Spectral Fusion Module

The SSFM adaptively fuses spatial and spectral features using learnable weights, while residual learning helps address overfitting with few labeled samples.

  • SSFM adaptively estimates spatial and spectral importance to guide feature fusion.The spatial and spectral fusion weights are randomly initialized and updated through backpropagation.
  • Residual learning is introduced in SSFM to alleviate overfitting during training with limited labeled samples.
  • MambaHSI analyzes pixel-dependent complexity as O(L^2) for Transformers versus O(L) for Mamba with sequence length L.The comparison is motivated by the whole-image setting, where the number of pixels L is large.

G. Training and Inference

MambaHSI processes the whole hyperspectral image in an end-to-end pipeline, producing all pixel predictions in one forward pass and evaluating the approach across diverse datasets.

  • MambaHSI uses the whole image as input and obtains all pixel predictions in one forward pass.This distinguishes it from patch-level HSI classification methods.
  • The encoder extracts spatial-spectral features through SpaMB, SpeMB, and SSFM modules.
  • Training computes logits and loss from the whole image and training labels, then updates network parameters with Adam.
  • Experiments evaluate MambaHSI on four diverse hyperspectral datasets: Pavia University, Houston, HanChuan, and HongHu.

3) HanChuan:

The experiments include HanChuan and other HSI datasets with varied acquisition settings, while comparisons span machine-learning, GCN, CNN, and Transformer methods.

  • HanChuan dataset: HanChuan contains seven crop species in a rural-urban fringe scene and has imagery sized 1217 × 303 pixels with 274 bands from 400 to 1000 nm.
  • Comparison methods: The evaluation compares MambaHSI with SVM and RF machine-learning baselines, DMSGer and GiGCN GCNs, and CNN and Transformer methods.The listed CNN and Transformer comparisons include FullyContNet, CLOLN, Spectralformer, and GSC-ViT.
  • Implementation: The implementation randomly selects 30 training and 10 validation samples from images in each of ten trials.

B. Comparison with State-of-the-Arts

MambaHSI is compared with state-of-the-art methods across four datasets and achieves the reported strongest accuracy and visualization quality, with SSFM improving complementary feature integration.

  • Quantitative comparison: MambaHSI achieves SOTA performance on all datasets against ML-based, GCN-based, CNN-based, and Transformer-based methods.The authors report that backbone superiority varies by dataset: GSC-ViT leads CLOLN by 1.54% OA on Houston, while CLOLN leads GSC-ViT by about 2.32% OA on Pavia University.
  • Qualitative comparison: MambaHSI produces classification maps with less misclassification, smoother objects, and better-maintained boundaries than the compared methods.On HongHu, several methods show small-area noise, while others show large misclassification areas and fuzzy boundaries.
  • Qualitative comparison: The qualitative comparisons cover Pavia University, Houston, HanChuan, and HongHu classification maps.
  • Ablation study: Ablation results show that SSFM improves performance over spatial features alone and that direct summation can degrade performance.On HongHu, adding SSFM improves OA by 5.47% over SpaMB.

D. Complexity Analysis

MambaHSI reduces the computational burden associated with full-image hyperspectral processing by using image-level modeling and Mamba-based linear-complexity operations. Experiments compare runtime and complexity against patch-level methods and self-attention.

  • Full-image complexity: Patch-level methods require substantially higher full-image computational complexity than image-level methods because adjacent patches involve redundant computations.The comparison uses FLOPs for whole-image testing and distinguishes patch-level from image-level approaches.
  • Runtime comparison: MambaHSI achieved the fastest testing and training times among the evaluated image-level methods.Running-time comparisons were conducted on the Pavia University dataset.
  • Mamba versus self-attention: The complexity analysis evaluates an encoder block comprising spatial Mamba, spectral Mamba, and spatial-spectral fusion modules against self-attention.The encoder block was tested on images of varying sizes to quantify computational complexity.
  • Mamba versus self-attention: When image size increased from 100 × 100 to 200 × 200, self-attention complexity increased approximately sixteenfold as sequence length quadrupled.This result quantitatively demonstrates the quadratic complexity of self-attention layers.

E. Effect of Pixel-wise Spatial Features

Experiments show that pixel-wise spatial features are important for MambaHSI, while spectral sequence information and suitable hyper-parameters materially affect classification performance. Performance degrades without spectral ordering and can decline when hidden dimensions become too large.

  • Effect of Pixel-wise Spatial Features: Increasing patch size significantly degrades MambaHSI performance, demonstrating the importance of pixel-wise spatial features.The comparison replaces pixel-wise features with patch features of different sizes on Pavia University.
  • Effect of Spectral Sequence Information: Removing spectral sequence information reduces OA by 5.57%, AA by 3.75%, and Kappa by 5.50%.The comparison sets the spectral Mamba group number to G = 1, eliminating sequence-order relationships between spectra.
  • Hyper-Parameter Analysis: As hidden dimension increases, performance initially improves and then declines because the dataset’s sample size becomes insufficient, leading to overfitting.The trend is attributed first to increased learning capacity and later to excessive capacity relative to the available samples.
  • Overall conclusion: The conclusion identifies integrating spatial and spectral information as vital for hyperspectral image classification.This conclusion follows the reported design and experiments across multiple datasets.
Loading 2501.04944v1…