SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers

• Hyperspectral images contain densely sampled wavelength bands whose spectral differences support fine-grained material identification.Source: §I · p. 1
• Existing backbones obtain promising results but remain inadequate at characterizing spectral series information and subtle discrepancies along the spectral dimension.Source: §I · pp. 1–2
• CNNs model spatial and local context effectively but struggle with middle- and long-term spectral dependencies and may distort sequential information.Source: §I · p. 2
• RNNs model spectral bands sequentially but depend strongly on band order, can generate vanishing gradients, struggle with long-term dependencies, and cannot train samples in parallel.Source: §I · p. 2
• SpectralFormer uses transformers to learn group-wise embeddings from neighboring bands rather than band-wise representations, targeting locally detailed spectral information.Source: §I · pp. 2–3
• Its cross-layer adaptive fusion learns to combine soft residuals, conveying memory-like components from shallow to deep layers.Source: §I · pp. 2–3
• Experiments on Indian Pines, Pavia University, and University of Houston report approximately 10% higher OA than classic transformers and at least 2% higher OA than other state-of-the-art backbones.Source: §I · p. 3

A. A Brief Review of Transformers

• Transformers use self-attention to capture internal correlations and global information across sequence positions.Self-attention computes attention representations from sequence inputs, while multi-head attention assembles multiple self-attention layers.Source: A. A Brief Review of Transformers · p. 3
• Self-attention inputs a sequence, embeds each element, derives queries, keys, and values, and computes normalized attention scores.The attention scores use scaled inner products followed by Softmax normalization before generating attention representations.Source: A. A Brief Review of Transformers · p. 3

II. SPECTRALFORMER

• SpectralFormer is introduced as a transformer-based backbone with GSE and CAF modules for hyperspectral image classification.The overview identifies these two modules as making the backbone more applicable to hyperspectral images.Source: §II · p. 3, Figure 3 · p. 4

A. A Brief Review of Transformers

• Multi-head attention concatenates multiple attention representations and applies a linear transformation to match the input feature dimension.The reviewed formulation uses multiple self-attention representations before output projection.Source: A. A Brief Review of Transformers · p. 4
• Because self-attention contains no position information, positional vectors are added to feature embeddings to encode sequence order.The resulting embedding is formulated as ai+ei, where ei is a manually given positional vector.Source: A. A Brief Review of Transformers · p. 4

B. Overview of SpectralFormer

• SpectralFormer is a generic ViT-based backbone focused on spectrometric characteristics for fine hyperspectral image classification.The design is intended to improve applicability to highly accurate and fine-grained material classification.Source: B. Overview of SpectralFormer · p. 4
• GSE improves detail capture for subtle spectral discrepancies by learning from local spectral profiles rather than isolated bands.The module is integrated into the transformer framework as part of the backbone design.Source: B. Overview of SpectralFormer · p. 4
• CAF enhances information transitivity between layers by reducing information loss as the network becomes deeper.It addresses connectivity across layer representations within the transformer framework.Source: B. Overview of SpectralFormer · p. 4
• SpectralFormer supports both pixel-wise hyperspectral classification and spatial-spectral classification with batch-wise inputs.The latter is described as the spatial-spectral SpectralFormer version.Source: B. Overview of SpectralFormer · p. 4

C. Group-wise Spectral Embedding (GSE)

• Hyperspectral channels are densely sampled at small wavelength intervals, producing approximately continuous spectral signatures unlike discrete transformer sequences.Neighboring spectral positions reflect different absorption characteristics associated with material properties.Source: C. Group-wise Spectral Embedding (GSE) · p. 4
• GSE learns group-wise spectral embeddings from locally neighboring bands instead of using classic band-wise representations.The goal is to capture locally detailed absorption changes important for accurate material classification.Source: C. Group-wise Spectral Embedding (GSE) · p. 4
• GSE forms grouped representations with W and X, where n denotes the number of neighboring bands.W is optimized as a network layer, while X contains grouped spectral representations.Source: C. Group-wise Spectral Embedding (GSE) · p. 4
• The grouping operation is overlapping, constructing each local spectral profile from neighboring entries around a central band.The local vector uses rounded offsets on both sides of the central spectral position.Source: C. Group-wise Spectral Embedding (GSE) · pp. 4–5
• The section contrasts band-wise and group-wise spectral embeddings in transformer-based backbone networks.The comparison is illustrated as BSE versus GSE.Source: C. Group-wise Spectral Embedding (GSE) · p. 5

D. Cross-layer Adaptive Fusion (CAF)

• CAF uses a middle-range skip connection to adaptively fuse features across transformer layers.The module skips one encoder, connecting z^(l−2) to z^l.Source: D. Cross-layer Adaptive Fusion (CAF) · p. 5
• The design addresses limited memory in short skips and insufficient fusion from long skips spanning shallow and deep features.The paper motivates CAF by contrasting short and long skip connections.Source: D. Cross-layer Adaptive Fusion (CAF) · p. 5

E. Spatial-Spectral SpectralFormer

• The backbone also supports patch-wise inputs by unfolding each band’s spatial patch into a one-dimensional vector before spectral processing.This input formulation preserves spectral sequence information while incorporating spatial context.Source: E. Spatial-Spectral SpectralFormer · pp. 5–6

III. EXPERIMENTS

• The experimental section evaluates SpectralFormer quantitatively and qualitatively, including ablation analysis.Classification maps provide qualitative comparisons alongside quantitative experiments.Source: §III · p. 6

1) Indian Pines Data

• Indian Pines contains 200 retained spectral bands, 16 investigated categories, and 145 × 145 pixels after removing noisy and water absorption bands.The scene was collected over North-Western Indiana using AVIRIS.Source: 1) Indian Pines Data · p. 6

2) Pavia University Data

• Pavia University contains 103 spectral bands, 610 × 340 pixels, and 9 land-cover classes.The scene was acquired over Pavia, Italy, using ROSIS at a 1.3m ground sampling distance.Source: 2) Pavia University Data · p. 6

3) Houston2013 Data

• Houston2013 contains 144 wavelength bands and 349 × 1905 pixels, using a cloud-free version processed to recover missing data or remove occlusions.The dataset covers the University of Houston campus and neighboring rural areas.Source: 3) Houston2013 Data · p. 6

B. Experimental Setup

• Models are evaluated using Overall Accuracy, Average Accuracy, Kappa Coefficient, and visualized classification maps.These metrics support quantitative comparison, while maps provide qualitative comparison.Source: 1) Evaluation Metrics · p. 7
• The comparison includes KNN, SVM, RF, 1-D CNN, 2-D CNN, RNN, miniGCN, transformers, and SpectralFormer.The selected methods span classical classifiers and multiple backbone architectures.Source: 2) Comparison with State-of-the-art… · p. 7
• KNN uses K = 10, RF uses 200 decision trees, and SVM uses an RBF kernel with hyperparameters selected by five-fold cross-validation.The SVM search ranges are σ = [2^-3, 2^-2, ..., 2^4] and λ = [10^-2, 10^-1, ..., 10^4].Source: 2) Comparison with State-of-the-art… · p. 7
• The CNN, RNN, and miniGCN baselines use the specified convolutional, recurrent, and graph-convolutional blocks and neuron configurations.The RNN has two GRU layers with 128 units each, while miniGCN uses a KNN-based graph with K = 10.Source: 2) Comparison with State-of-the-art… · p. 7
• The transformer baseline uses five ViT-based encoder blocks containing only transformer encoders.This establishes the classic-transformer comparison architecture.Source: 2) Comparison with State-of-the-art… · p. 7
• SpectralFormer uses the same backbone architecture for fair comparison, with 64-unit embedded spectra and five cascaded transformer encoder blocks.Each block uses four-head self-attention, an MLP with 8 hidden dimensions, and GELU activation.Source: 2) Comparison with State-of-the-art… · p. 7

3) Implementation Details

• SpectralFormer is implemented in PyTorch with Adam, mini-batches of 64, an initial learning rate of 5e-4, and dataset-specific training epochs.The learning rate is multiplied by 0.9 after each one-tenth of the total epochs.Source: 3) Implementation Details · p. 7
• The CAF module can converge using about 300 epochs instead of the roughly 1000 epochs initially set for the three datasets.The paper reports this as an efficiency improvement observed in practice.Source: 3) Implementation Details · p. 7
• SpectralFormer has per-layer complexity O(m^2d + md^2), dominated by self-attention and multi-head operations.Here m is spectral length and d is the hidden-feature size.Source: 4) Computational Complexity Analysis · pp. 7–8

Ablation Analysis

• The ablation study adds GSE and CAF stepwise on Indian Pines to assess their contributions to classification performance.The corresponding results are reported in Table V.Source: Table V · p. 8

Sensitivity Analysis

• Sensitivity analysis varies the number of grouped neighboring bands and reports OA, AA, and κ for GSE and GSE + CAF.The analysis is conducted on Indian Pines and summarized in Table VI.Source: Table VI · p. 8

Skip-Connection Comparison

• Table VII compares different skip-connection designs in transformers on the Indian Pines dataset.The comparison isolates skip-connection choices within the transformer setting.Source: Table VII · p. 8

Ablation Study

• ViT without GSE or CAF produces the lowest classification accuracies on Indian Pines.Source: 1) Ablation Study · p. 8
• Adding GSE or CAF improves pixel-wise SpectralFormer over ViT by around 4% OA and 3% OA, respectively.Source: 1) Ablation Study · p. 8
• Jointly using GSE and CAF yields more than 4% additional OA improvement over the individual-module settings.Source: 1) Ablation Study · p. 8
• Patch-wise SpectralFormer improves OA by at least 3% over the second-best pixel-wise record of 78.55%.Source: 1) Ablation Study · p. 8, 2) Parameter Sensitivity Analysis · p. 9

Parameter Sensitivity Analysis

• CAF outperforms short- and long-range skip connections by exchanging cross-layer information more effectively and reducing information loss.Source: 2) Parameter Sensitivity Analysis · p. 8
• With 80%, 90%, or 100% of training samples, OA tends toward stability, indicating robustness to additional samples in this evaluation.Source: 2) Parameter Sensitivity Analysis · p. 8

Quantitative Results

• Quantitative evaluation reports OA, AA, κ, and per-class accuracies on Indian Pines, Pavia University, and Houston2013.Source: D. Quantitative Results and Analysis · p. 9
• Deep-learning backbones generally outperform KNN, RF, and SVM, while transformers achieve performance comparable to CNN-, RNN-, and GCN-based models.Source: D. Quantitative Results and Analysis · p. 9
• SpectralFormer improves classification by extracting local spectral information from neighboring bands, addressing transformers’ limited modeling of local spectral discrepancies.Source: D. Quantitative Results and Analysis · p. 9
• Patch-wise SpectralFormer achieves higher accuracies than competing methods because spatial-contextual information is jointly considered during sequential feature extraction.Source: D. Quantitative Results and Analysis · p. 10
• SpectralFormer performs better on challenging or imbalanced Indian Pines classes by focusing on particular absorption positions of spectral profiles.Source: D. Quantitative Results and Analysis · p. 10

Classification Maps

• Conventional classifiers tend to produce salt-and-pepper noise in maps across the three datasets, indicating less accurate material identification.Source: E. Visual Evaluation · p. 10
• Classification maps are provided for Indian Pines, Pavia University, and Houston2013, each with a 2× zoomed region for detailed observation.Source: Figure 7 · p. 11, Figure 8 · p. 11, Figure 10 · p. 12

Feature Visualization

• CAF-based encoder features have finer object edges, outlines, and textural structures than features from the original ViT without CAF.Source: Figure 9 · p. 11, E. Visual Evaluation · p. 12

Interpretation

• Transformers characterize global sequential properties, whereas SpectralFormer focuses more specifically on local spectral information and cross-layer memory-like components.Source: §IV · p. 12

Limitations and Future Work

• The authors identify future work in lightweight architectures, physical priors, interpretability, and selecting the number of CAF-connected encoders.Source: §IV · pp. 12–13