PaperlayerView exact revision on arXiv

Source-linked AI summary

Token Clustering and Semantic Sequence Mamba for Hyperspectral Image Classification

Yimin Zhu, Mahmood Elahi, Lincoln Linlin Xu

arXiv:2609.28580v1cs.CV

TL;DR

Hyperspectral image classification is challenged by spectral–spatial heterogeneity, while existing Mamba approaches typically sequence tokens by predefined spatial neighborhoods. STMamba builds sparse, semantically coherent sequences through token clustering and models spatial and spectral dependencies; experiments on three large-scale HSI datasets report competitive classification performance and improved fine-structure preservation.

  • Problem

    HSI classification remains challenging amid spectral–spatial heterogeneity, and existing Mamba sequence construction typically does not explicitly account for semantic similarity or spatial non-stationarity.

    Source: §I · p. 1

  • Method

    STMamba uses hierarchical token clustering and quadtree-based sparse selection to form semantic-token sequences, with parallel spatial and spectral SWSM modules and CNA upsampling.

    Source: §I · p. 2

  • Results

    Experiments on three large-scale HSI datasets show competitive classification performance and improved preservation of fine structures.

    Source: §VI · p. 12

  • Takeaways & Limitations

    STMamba combines sparse semantic sequencing and spatial–spectral modeling in a hierarchical framework for HSI classification.

    Source: §VI · p. 11

    The method uses a fixed cluster number and incurs additional cost from quadtree selection.

    Source: §VI · p. 12

Abstract

from arXiv · show

Although hyperspectral images (HSIs) provide rich spectral-spatial information, accurate pixel-level classification remains challenging because of spectral-spatial heterogeneity and complex spatial structures. Existing vision state space models (Mamba) typically construct sequences according to predefined spatial neighborhoods, without explicitly accounting for semantic similarity or spatial non-stationarity. To address this limitation, we propose Token Clustering and Semantic Sequence Mamba (STMamba), which organizes sparse tokens into semantically coherent sequences for hyperspectral image classification with the following features. First, at the macro level, a hierarchical encoder decoder progressively selects semantic tokens with the Token Clustering Module (TCM) and restores dense features using a parameter-free Cross-scale Neighborhood Attention (CNA) Upsampler. Second, at the micro level, TCM first identifies representative cluster centers through density-aware clustering and estimates soft memberships based on feature similarity. A quadtree-based dynamic selection strategy then retains sparse and spatially distributed tokens from each semantic cluster, forming coherent semantic-token sequences while reducing redundant pixel-wise representations. Third, parallel Spatial and Spectral Semantic-wise Sequencing Mamba (SWSM) modules capture complementary long-range spatial and spectral dependencies within homogeneous semantic token sequences while suppressing irrelevant interactions across heterogeneous regions. Experimental results on three large-scale benchmark datasets demonstrate that STMamba outperforms the SOTA methods with respect to quantitative and qualitative results.

I. INTRODUCTION

HSI classification must handle spectral–spatial heterogeneity, while many existing approaches use dense or regular tokens that do not explicitly reflect semantic structure. STMamba constructs sparse semantic-token sequences and models spatial and spectral dependencies within them.

I. INTRODUCTION

  • HSI classification is challenged by heterogeneity, mixed pixels, low signal-to-noise ratio, and illumination variation, which produce spectral variability and complex spatial structures.The paper identifies representative feature extraction, spatial–spectral dependency modeling, and efficient long-range context as requirements for effective classification.Source: §I · p. 1

A. Patch-free Network for HSIC

  • Many patch-free and Mamba-based methods still use dense or regular-grid tokens, without explicitly organizing pixels by semantic similarity.Regular scanning can place heterogeneous regions in one sequence, while assigning a supertoken prediction to all constituent pixels can suppress boundary variation.Source: §I · p. 1, A. Patch-free Network for HSIC · p. 3

I. INTRODUCTION

  • STMamba reformulates sequence construction as sparse, semantic-membership-defined sequencing rather than geometry-defined scanning.The proposed approach targets coherent sequences in which representative tokens from homogeneous regions can interact over long distances.Source: §I · p. 2, Figure 1 · p. 2
  • A hierarchical encoder–decoder progressively extracts semantic tokens with TCM and restores dense representations with a parameter-free CNA Upsampler.At the micro level, TCM uses density-aware representative centers and feature-similarity soft memberships, then quadtree selection retains sparse, spatially distributed tokens and reduces redundant pixel-wise representations.Source: §I · p. 2
  • Parallel spatial and spectral SWSM modules model long-range dependencies within homogeneous semantic-token sequences while reducing interactions across heterogeneous regions.This separates complementary spatial and spectral dependency modeling within the proposed sequences.Source: §I · p. 2

B. Token Definition in Mamba Model

Mamba requires multidimensional visual features to be converted into one-dimensional sequences, and token construction affects which relationships state propagation can model. STMamba groups deep features semantically before selecting sparse tokens for sequencing.

B. Token Definition in Mamba Model

  • Visual Mamba models must convert multidimensional features into 1D sequences, and existing strategies include directional scanning, domain decomposition, multiscale serialization, and adaptive tokenization.Examples include corner-based directional sequences, separate spatial and spectral sequences, and multiscale feature serialization.Source: B. Token Definition in Mamba Model · p. 3
  • Geometric locality or predefined reference pixels can mix semantic regions in remote-sensing imagery, making meaningful token relationships important for Mamba state propagation.The paper notes that historical information may decay as sequence distance increases.Source: B. Token Definition in Mamba Model · p. 3, A. Preliminaries · p. 3
  • STMamba clusters deep features into coherent semantic groups, then dynamically selects sparse tokens so each selected token represents a semantic concept rather than a fixed grid cell.This creates more homogeneous sequences for modeling long-range dependencies.Source: B. Token Definition in Mamba Model · p. 3

B. Overview Architecture

STMamba uses a four-stage hierarchical encoder–decoder to construct semantic sequences at each stage and restore dense predictions. Its TCM identifies semantic centers from sampled anchors using density-peak clustering.

B. Overview Architecture

  • STMamba processes the full hyperspectral image through four hierarchical stages, parallel spatial and spectral SWSM modules, and a feed-forward layer at each stage.Downsampling builds hierarchical features; a bottleneck models global context, and the decoder uses parameter-free CNA upsampling under guidance from the preceding stage’s feature map to produce prediction logits.Source: B. Overview Architecture · pp. 3–4, Figure 2 · p. 4

C. Token Clustering Module (TCM)

  • TCM determines representative semantic centers through density-peak clustering and generates soft memberships by cosine similarity between centers and available tokens.These memberships support subsequent semantic sequence modeling.Source: C. Token Clustering Module (TCM) · p. 4

1) Clustering Center Determination:

  • To reduce the cost of applying DPC-KNN to all valid tokens, the method first sparsely samples feature tokens on grids with random spatial offsets.Local density is estimated for sampled anchors, and anchors with the largest density-peaks scores are selected as candidate centers; multiple independently offset samplings are aggregated before a second DPC-KNN operation yields the final centers.Source: 1) Clustering Center Determination · p. 4
  • The number of clusters K is set to the number of semantic classes as a unified semantic-capacity setting, without assuming one-to-one correspondence between clusters and classes.The paper states that this clustering complexity is substantially lower than directly clustering all N tokens.Source: 1) Clustering Center Determination · p. 5

2) Soft Clustering Membership:

The soft membership step relates tokens to cluster centers by feature similarity, producing token-to-cluster relationships for semantic sequence modeling.

2) Soft Clustering Membership:

  • Soft clustering membership is obtained using cosine similarity.The passage identifies the similarity measure but does not provide further operational detail.Source: 2) Soft Clustering Membership · p. 5
  • The soft semantic membership is calculated as part of the clustering process.The supplied passage states the calculation but does not specify its full expression in text.Source: 2) Soft Clustering Membership · p. 5
  • The resulting membership P ∈ R^{Ni×K} provides dense token-to-cluster relationships for subsequent semantic sequence modeling.Source: 2) Soft Clustering Membership · p. 5

3) Clustering Center Diversity Loss:

The section regularizes cluster centers with a diversity loss and selects sparse, spatially distributed tokens within semantic clusters using a quadtree strategy.

3) Clustering Center Diversity Loss:

  • Center diversity loss penalizes average squared cosine similarity between distinct cluster centers to improve their discriminability.Across the four stages, the diversity loss is combined with cross-entropy as the total loss.Source: 3) Clustering Center Diversity Loss · p. 5

D. Quadtree-based Adaptive Sparse Token Selection

  • The quadtree recursively partitions each cluster’s membership map and selects high-membership tokens from different spatial regions.This avoids concentrating selections in a small area; the global sparse ratio controls the number retained, and selected tokens are ordered by membership score for SWSM.Source: D. Quadtree-based Adaptive Sparse… · p. 5

E. Positional Embedding

Positional descriptors preserve location information for dynamically selected tokens, while SWSM sequences and gates tokens according to semantic-cluster membership.

E. Positional Embedding

  • Each feature-map location receives normalized horizontal and vertical coordinates plus sine and cosine coordinate features, projected to the feature dimension and scaled by a learnable parameter.The scale is initialized small to avoid strongly perturbing original features early in training.Source: E. Positional Embedding · p. 5

F. Semantic-wise Sequencing Mamba (SWSM)

  • Within each semantic cluster, selected tokens are sorted by membership score, placing semantically similar tokens together and reducing abrupt feature transitions during state propagation..Source: F. Semantic-wise Sequencing Mamba (SWSM) · p. 5
  • A relative membership gate modulates the Mamba input and readout parameters, so strongly associated tokens contribute more to state evolution and readout while weakly associated tokens are suppressed.Source: F. Semantic-wise Sequencing Mamba (SWSM) · pp. 5–6

1) Spa-SWSM:

Parallel spatial and spectral SWSM branches process semantically coherent sparse tokens, and the CNA decoder progressively restores dense spatial features.

1) Spa-SWSM:

  • Spa-SWSM models long-range spatial dependencies within each semantic cluster, allowing spatially distant but semantically similar tokens to interact in one sequence.Processed sparse tokens are mapped back to their original spatial locations using quadtree-generated indices.Source: 1) Spa-SWSM · p. 6

2) Spe-SWSM:

  • Spe-SWSM reorganizes the same semantic tokens along the spectral-feature dimension to model complementary cross-band dependencies at multiple scales.The spectral outputs are scattered back to the original token locations and fused with stage features alongside the spatial branch.Source: 2) Spe-SWSM · p. 6

G. Cross-scale Neighborhood Attention (CNA) Upsampler

  • CNA uses high-resolution encoder features as queries and keys, with low-resolution features supplying attention values, to guide neighborhood-based upsampling.Applied progressively across decoder stages, CNA restores dense spatial features before pixel-wise classification.Source: G. Cross-scale Neighborhood Attention… · p. 6

IV. EXPERIMENTAL RESULTS AND ANALYSIS

The evaluation uses PU, HU13, and HU18 with limited per-class training samples, compares STMamba against varied model families, and reports standard classification metrics.

A. Datasets and Evaluation Metrics

1) Pavia University:

  • PU was acquired by ROSIS in Pavia, Italy, and contains 103 bands, 610 × 340 pixels, and nine land-cover categories.Source: 1) Pavia University · p. 6

2) Houston2013:

  • HU13 contains 144 bands, measures 349 × 1905 pixels, and covers 15 land-cover categories around the University of Houston.Source: 2) Houston2013 · p. 6

3) Houston2018:

  • HU18 contains 48 bands across 380–1050 nm, has a 601 × 2384 spatial dimension and 1 m ground sampling distance, and includes 20 land-cover classes.Source: 3) Houston2018 · p. 6

A. Datasets and Evaluation Metrics

  • Evaluation metrics are Overall Classification Accuracy, Average Classification Accuracy, per-class accuracy, and Kappa Coefficient.Source: 3) Houston2018 · p. 7

2) Comparing with Other Methods:

C. Experimental Results

STMamba leads the reported quantitative comparisons on PU, HU13, and HU18, while visual results and clustering visualizations describe its spatial detail and semantic-token organization.

1) Numerical Evaluation:

  • On PU, STMamba improves OA, AA, and κ over MambaHSI+ by 5.88%, 5.90%, and 7.17%, respectively.These comparisons are reported in Table II; the supplied passages do not include the table itself.Source: 1) Numerical Evaluation · p. 8
  • On HU13, STMamba outperforms second-best MambaLG by 0.97%, 0.86%, and 1.05% in OA, AA, and κ, respectively.MambaLG uses both within-patch and cross-patch scanning and outperforms most other Mamba-based baselines.Source: 1) Numerical Evaluation · p. 8
  • On HU18, STMamba exceeds MambaLG by 4.93, 3.79, and 5.55 percentage points in OA, AA, and κ, respectively.STMamba achieves the highest reported OA, AA, and κ on this larger scene.Source: 1) Numerical Evaluation · p. 8

2) Visual Evaluation:

  • On PU, STMamba preserves finer spatial details, object boundaries, and small objects, including trees, more clearly than comparison methods.The reported examples include a road intersection, shadowed areas beneath tree canopies, building footprints, and a square grassland area.Source: 2) Visual Evaluation · p. 8, Figure 3 · p. 8
  • On HU13 and HU18, STMamba follows object boundaries more closely and preserves structural features more completely than comparison methods.HU13 examples include linear, rectangular, and circular structures; HU18 examples include buildings and linear features. S2Mamba misclassifies some non-residential buildings as cars, while DMSGer fragments predictions within continuous objects.Source: 2) Visual Evaluation · p. 9, Figure 4 · p. 9

3) Visualization of Clustering Center and Semantic Sequences:

  • Across stages, token selection becomes more concentrated as deeper features form clearer membership regions, and later stages select fewer tokens under sparsity control.Multi-sampling and diversity loss produce more dispersed, separable feature-space centers that remain sparse in image space and lie in coherent membership regions.Source: 3) Visualization of Clustering Center… · p. 9

A. Ablation Study

Ablations assess STMamba’s components, hyperparameter choices, and learned semantic clusters; results show trade-offs in token sparsity and less distinct clustering in some larger Houston regions.

A. Ablation Study

  • Replacing CNA with bilinear interpolation reduces HU18 OA from 71.34% to 66.64%, the largest reported degradation among the major component ablations.Removing positional encoding, quadtree selection, or cluster-diversity loss also degrades performance.Source: A. Ablation Study · p. 9
  • Removing the complete SWSM pathway reduces HU13 OA from 95.97% to 92.42%, while Spa-SWSM removal and membership-score ordering replacement also degrade results.Spe-SWSM provides complementary spectral modeling, with the largest OA drop of 2.98 percentage points on PU; removing membership gating particularly degrades HU18.Source: A. Ablation Study · p. 9

B. Hyperparameter Analysis

  • Best-performing sparsity differs by dataset: PU and HU13 perform best around λs = 0.10, while HU18 performs best at λs = 0.05.The highly sparse λs = 0.01 setting remains competitive; increasing λs from 0.01 to 0.30 raises PU peak GPU memory from 2.69 to 5.04 GB, and D = 64 gives the most consistent performance across datasets.Source: B. Hyperparameter Analysis · p. 10

C. Analysis of Semantic Clustering

  • Across hierarchical stages on HU18, membership maps become more concentrated and semantically structured, with deeper hard-clustering maps showing increasingly discriminative regions.Roads and buildings appear as linear and block-shaped structures across multiple memberships.Source: C. Analysis of Semantic Clustering · p. 10
  • Some larger Houston regions show less distinct cluster separation, potentially because of class spectral similarity or imperfect anchor sampling.Despite this, learned memberships provide spatially coherent semantic groups for sparse token selection and sequence ordering rather than predefined raster topology.Source: C. Analysis of Semantic Clustering · p. 10

D. Effect of Training Samples

STMamba reports competitive classification performance and improved fine-structure preservation across three large-scale HSI datasets, while balancing model capacity and computational cost. Training-sample trends vary by dataset, and visualized SWSM influence generally decreases with sequence distance.

D. Effect of Training Samples

E. Computational Complexity

  • STMamba has approximately 0.6M parameters across three datasets and remains less computationally expensive than MambaLG on large Houston scenes.Its GFLOPs exceed MambaHSI and MambaHSI+ because of hierarchical feature processing and FFNs, while its parameter count is substantially below DSCC and MambaHSI+ and comparable to lightweight Mamba baselines.Source: E. Computational Complexity · pp. 10–11, Table VIII · p. 11

F. Visualization of Hidden Influence Matrices in Mamba

VI. CONCLUSION

  • Experiments on three large-scale HSI datasets show competitive classification performance and improved preservation of fine structures; fixed cluster counts and quadtree-selection cost remain limitations.The conclusion identifies adaptive clustering and more efficient dynamic sequence construction as future work.Source: §VI · pp. 11–12

Related papers

Loading 2609.28580v1…