Mamba-in-Mamba: Centralized Mamba-Cross-Scan in Tokenized Mamba Model for Hyperspectral Image Classification

• MiM explores Mamba architecture for HSI classification with a focus on competitive performance and efficiency.Source: Highlights · p. 2
• The Tokenized Mamba encoder combines Gaussian Decay Mask, Semantic Token Learner, and Semantic Token Fuser to improve feature generation and concentration.Source: Highlights · p. 2
• Centralized Mamba-Cross-Scan converts HSI patches into diverse sequences while emphasizing the center pixel for patch-wise classification.Source: Highlights · p. 2
• The study reports competitive and sometimes state-of-the-art performance on four HSI datasets with fixed and disjoint training-testing samples.Source: Highlights · p. 2

1. Introduction

• RNNs can favor late sequence steps over the central pixel, while larger patches expose them to interfering peripheral pixels and limit parallel computation.Source: §1 · pp. 4–5
• Transformers incur quadratic self-attention costs and risk overfitting with limited HSI training data, while lacking strong spatial inductive bias.Source: §1 · p. 5
• Existing image scans can create boundary discontinuities, while sequentially aggregating unidirectional scans may neglect the center pixel and increase redundancy and computation.Source: §1 · p. 5
• Mamba scales linearly with sequence length, but its potential for HSI classification had not been fully explored, motivating a vision-based model designed for this task.Source: §1 · p. 6
• MiM combines centralized MCS, which scans patches into diverse sequences, with a T-Mamba encoder containing GDM, STL, and STF.Source: §1 · pp. 6–7
• The proposed method reports competitive and sometimes state-of-the-art performance on four datasets with fixed and disjoint training-testing samples.Source: §1 · p. 7

2. Preliminary

• The section introduces SSMs and Mamba as foundational sequence-model concepts for the study.It frames the preliminary material as background for what follows.Source: §2 · p. 7, §2.1 · p. 7

2.1. Structured State Space Sequence Model (S4)

• S4 maps input signals through latent states and uses fixed, input-independent matrices, yielding a static representation.Its continuous system is discretized with a timescale parameter, then expressed as a sequence-to-sequence model with a global convolution.Source: §2.1 · p. 8

2.2. Selective Scan-Based S4 Model (S6)

• Mamba’s S6 adds selective scan, making B, C, and the step size input-dependent so sequence inputs can be processed selectively.The section describes this processing as a parallel scan operation.Source: §2.2 · p. 9

2.3. Mamba-Based Works

• Although Mamba-based methods have been applied to vision and HSI tasks, tailored ViM with multidirectional scanning for HSI remains underexplored.The authors also identify refined semantic tokens and HSI patch multi-scale features as challenges for directly using ViM.Source: §2.3 · p. 10

3. Methodology

• The methodology is organized around an overview, centralized MCS, T-Mamba encoder, WMF module, and multi-scale loss design.The section points readers to Figure 3 and Algorithm 1 for the overall structure and pseudocode.Source: §3 · p. 10

3.1. Methodological Overview

• MCS produces four types of paired directional sequences from an HSI patch for processing by T-Mamba encoders.The encoders use GDM, a downsample-driven STL, and an attention-driven STF to learn representative tokens.Source: §3.1 · p. 10
• Four T-Mamba encoders process the sequences, and WMF dynamically weights and combines their outputs.The fused feature at each scale is recorded for later decoding.Source: Figure 3 · p. 11, §3.1 · p. 11
• MiM repeats tokenization and downsampling until spatial size reaches 1, then supplies accumulated multi-scale features to the decoder.Each accumulated feature is linked to a specific loss function during training.Source: Figure 3 · p. 11, §3.1 · p. 11

3.2. Centralized Mamba-Cross-Scan (MCS)

• MCS replaces raster flattening with continuous snake scans from multiple directions, addressing boundary jumps and using multiple image perspectives.The scans converge at the center pixel, and the centralized design is intended for patch-wise classification where the center pixel supplies the label.Source: §3.2 · pp. 12–13

3.2.1. MCS Type-1

• Type-1 forms forward and backward subsequences that both end at the patch center; for a 5 × 5 patch, each has length 13.The center step can iterate features from the preceding steps, and the two outputs are merged to represent the patch.Source: §3.2.1 · p. 15

3.2.2. MCS Type-2

• Type-2 uses vertical scans from two opposing positions, with subsequence lengths matching the Type-1 description..Source: §3.2.2 · p. 15

3.2.3. MCS Type-3

• Type-3 is presented as a cross-scan variant with a different starting position from Types 1 and 2.The supplied description identifies its scan as horizontal.Source: §3.2.3 · p. 16

3.2.4. MCS Type-4

• Type-4 adds the fourth MCS variant; together, the four types produce eight subsequences for the bidirectional T-Mamba encoders.The authors describe the centralized two-subsequence representation as reducing computational burden and complexity relative to using two complete sequences.Source: §3.2.4 · p. 16

3.3.1. Whole Flowchart of T-Mamba

• T-Mamba processes each MCS-generated forward and backward sub-sequence independently through a 1D convolution and S6 model before merging their features.The example uses sub-sequences of length 13 and preserves an odd patch size to retain the center pixel.Source: §3.3.1 · pp. 17–18
• A residual connection from the original input is added to the fused features before output to the next stage.The resulting features are described as semantic tokens representing the entire image patch.Source: Figure 6 · p. 17, §3.3.1 · p. 18

3.3.2. Gaussian Decay Mask (GDM)

• The Gaussian Decay Mask softly weights sequence features by spectral and index distances, focusing the weighting around the central pixel.For the spatial mask, the sub-sequence’s last step represents the center pixel.Source: §3.3.2 · pp. 18–19, §3.3.1 · p. 18

3.3.3. Cross Scan Merge

• Cross Scan Merge combines the two half-directional feature sequences into one sequence while restoring the original sequence length.It averages the final-step features and places that combined feature between the initial segments of the two sequences.Source: §3.3.3 · p. 20

3.3.4. Semantic Token Learner (STL)

• The Semantic Token Learner applies sequential attention and learns representative semantic tokens that condense the sequence for downstream tasks.It extracts length-level and dimension-level semantic features and combines them to derive the tokens.Source: §3.3.4 · pp. 20–22

3.3.5. Semantic Token Fuser (STF)

• The Semantic Token Fuser uses influence scores from z to blend token information into the sequence and combines it with attention-modified features.The fused feature is produced by summing the token-influenced and attention-modified data.Source: §3.3.5 · pp. 22–23

3.4. Weighted MCS Fusion (WMF)

• Weighted MCS Fusion assigns attention weights to feature sets from different scan orders according to their relevance and importance.The weights are learned within the network and sum to one across the four encoded features.Source: §3.4 · pp. 23–24
• The weighted fused feature is passed to an MLP decoder to refine and interpret it for model training.Each encoded feature receives a weight before fusion.Source: §3.4 · p. 24
• The model is trained with cross-entropy loss measuring the discrepancy between predicted and actual labels.The loss is defined over the training samples.Source: §3.4 · p. 24

3.5. Multi-Scale Loss Design

• The MiM model iteratively downsamples patch features, passing each scale’s features to the next MiM stage until the patch size reaches one.Patch sizes remain odd; the example reduces 11 to 9, 7, 5, 3, and finally 1.Source: §3.5 · pp. 24–25
• Each scale feature is independently integrated into the loss, and the total loss combines the scale-specific losses.The design uses output-level decoding to account for multiple scales in an image patch.Source: §3.5 · pp. 24–25

4.1. Datasets Description

• The experiments cover Indian Pines, Pavia University, Houston 2013, and WHU-Hi-HongHu, using manually selected fixed and disjoint training and testing samples.Indian Pines has 200 retained spectral bands and 16 land-cover classes; WHU-Hi-HongHu has 22 ground-cover classes.Source: §4.1 · pp. 25–27

4.2. Experimental Evaluation Indicators

• Performance is assessed with overall accuracy, average accuracy, kappa, and accuracy within each class.Higher values indicate superior classification results.Source: §4.2 · p. 27

4.3. Brief Description and Settings of Compared Methods

• Comparisons include eight patch-wise baseline and state-of-the-art models, using the same number of training samples and no data augmentation.The listed methods are vanilla ViT, SpeFormer, MAEST, HiT, SSFTT, 3DViT, HUSST, and ViM.Source: §4.3 · p. 27

4. Experimental results and analysis

• 92.0794% OA for MiM on Indian Pines, above every other method listed in the classification-map comparison.The listed comparison includes vanilla ViT, SpeFormer, MAEST, HiT, SSFTT, 3DViT, HUSST, and ViM.Source: Figure 10 · p. 28

4.4.1. Quantitative Results and Classification Maps

• MiM has the highest listed OA among methods on PU (91.5756%), HU2013 (92.8917%), and HongHu (89.5749%), with smoother maps and clearer land-cover boundaries.The prose also reports generally competitive or superior class-wise accuracies and a lighter model than HiT, SSFTT, and HUSST.Source: Figure 11 · p. 29, Figure 12 · p. 30, §4.4.1 · p. 30
• Vanilla ViT maps show sharper artifacts, while CNN-Transformer hybrids provide clearer boundaries but require larger models and more computation than MiM.SpeFormer and MAEST also improve on vanilla ViT; 3DViT links spatial and spectral features in a 3D sequential model.Source: §4.4.1 · pp. 29–30

4.4.2. Learned Feature Visualizations by T-SNE

• MiM features form more discriminative, smoother same-color clusters, but T-SNE cluster proximity does not establish similarity and same-color clusters may appear far apart.The paper attributes the latter behavior to T-SNE’s focus on local structure and possible local minima.Source: §4.4.2 · pp. 31–32, Figure 14 · p. 32

4.4.3. Comparison of Computational Costs

• MiM balances running time and parameter count: it is more efficient than several advanced models and uses fewer parameters than other leading methods, though it is not the fastest.The comparison reports training time, testing time, and model size for the IP dataset.Source: §4.4.3 · p. 32

4.5.1. Ablation on Proposed Different Components

• STL outperforms average pooling, STF outperforms simple summation, and combining STL with STF yields the most substantial improvements.GDM reweights encoded sequence features by proximity to the central pixel, reducing the influence of interfering pixels and yielding modest improvements.Source: §4.5.1 · p. 33

4.5.2. Analysis on Initial Patch Size

• The best initial patch sizes are 7 × 7 for IP, 9 × 9 for HU2013 and HongHu, and 11 × 11 for PU.An appropriately larger patch captures more spatial context, but excessive size can introduce noise from interfering pixels.Source: §4.5.2 · pp. 33–34

4.5.3. Investigation and Comparison on Scan Design

• The Mamba-scan with four directional types performs best; four scan types also improve each design, and more continuous arrangements outperform Raster-scan.Zig-zag scanning slightly improves on Diagonal scanning, consistent with its partial continuity at image boundaries.Source: §4.5.3 · p. 34, Figure 15 · p. 34

4.5.4. Exploring the Feature Fusion Strategy in T-Mamba

• Pre-Merge is more effective than Post-Merge because STL analyzes merged image features and STF uses a pooled representation of the whole image.In Post-Merge, each branch is processed independently without integrating semantic information across branches.Source: §4.5.4 · p. 35, Figure 16 · p. 35

4.5.5. Comparison with Other Sequential Networks

• T-Mamba improves classification over basic Mamba and balances model size with efficacy; RT performs competitively but requires a larger parameter set.The comparison includes RNN, GRU, LSTM, Transformer encoder, RT, Mamba, and T-Mamba under identical test conditions.Source: §4.5.5 · pp. 35–36, Figure 17 · p. 36

4.5.6. Further Discussion on Mamba and Transformer under Data Augmentation

• With sixfold augmentation to 4170 samples, MiM is no longer the top performer; MAEST leads, while MiM excels with limited, disjoint, fixed samples.Transformer-based methods tend to perform better with abundant training data, whereas increasing the training sample size improves all models and narrows accuracy gaps.Source: §4.5.6 · pp. 37–38

5. Conclusion

• MiM combines centralized MCS, a T-Mamba encoder with GDM, STL, and STF, and WMF with multi-scale loss for HSI classification.The conclusion reports that MiM outperforms baselines and state-of-the-art methods with limited, disjoint samples, while Transformer-based models tend to do better with augmented samples.Source: §5 · p. 38