PaperlayerView exact revision on arXiv

Source-linked AI summary

SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot

arXiv:2107.02988v2cs.CVcs.AI

TL;DR

Hyperspectral classification requires modeling locally detailed spectral sequences, which classic CNN and transformer backbones handle imperfectly. SpectralFormer addresses this with group-wise spectral embeddings and cross-layer adaptive fusion, achieving state-of-the-art classification results and stronger patch-wise performance.

  • Problem

    CNNs and classic transformers inadequately model the sequential and locally detailed spectral properties of hyperspectral signatures.

  • Method

    SpectralFormer uses group-wise spectral embedding to model neighboring bands and cross-layer adaptive fusion to convey memory-like components across layers.

  • Results

    Patch-wise SpectralFormer improves accuracy by at least 3% OA over the 78.55% second record and achieves state-of-the-art hyperspectral classification results.

  • Takeaways & Limitations

    SpectralFormer supports both pixel-wise and patch-wise inputs, with patch-wise processing jointly incorporating spatial-contextual information during sequential feature extraction.

  • Takeaways & Limitations

    Future work should incorporate physical spectral-band characteristics and prior hyperspectral knowledge, while studying how CAF's skipped-encoder count affects performance.

Abstract

from arXiv · show

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Owing to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers, and propose a novel backbone network called \ul{SpectralFormer}. Beyond band-wise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding group-wise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layer-wise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse "soft" residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixel- and patch-wise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of reproducibility.

I. INTRODUCTION

Hyperspectral image classification requires models that capture subtle, sequential spectral differences while preserving useful spatial context. SpectralFormer addresses limitations of existing backbones with group-wise spectral embeddings and cross-layer adaptive fusion, and improves classification across three datasets.

  • Hyperspectral images contain densely sampled wavelength bands whose spectral differences support fine-grained material identification.
  • Existing backbones obtain promising results but remain inadequate at characterizing spectral series information and subtle discrepancies along the spectral dimension.
  • CNNs model spatial and local context effectively but struggle with middle- and long-term spectral dependencies and may distort sequential information.
  • RNNs model spectral bands sequentially but depend strongly on band order, can generate vanishing gradients, struggle with long-term dependencies, and cannot train samples in parallel.
  • SpectralFormer uses transformers to learn group-wise embeddings from neighboring bands rather than band-wise representations, targeting locally detailed spectral information.
  • Its cross-layer adaptive fusion learns to combine soft residuals, conveying memory-like components from shallow to deep layers.
  • Experiments on Indian Pines, Pavia University, and University of Houston report approximately 10% higher OA than classic transformers and at least 2% higher OA than other state-of-the-art backbones.

II. SPECTRALFORMER

Transformers process sequential data through self-attention and multi-head attention, while positional embeddings restore sequence information absent from self-attention. SpectralFormer adapts this transformer framework for hyperspectral image classification.

  • A. A Brief Review of Transformers: Transformers use self-attention to capture internal correlations and global information across sequence positions.Self-attention computes attention representations from sequence inputs, while multi-head attention assembles multiple self-attention layers.
  • A. A Brief Review of Transformers: Self-attention inputs a sequence, embeds each element, derives queries, keys, and values, and computes normalized attention scores.The attention scores use scaled inner products followed by Softmax normalization before generating attention representations.
  • II. SPECTRALFORMER: SpectralFormer is introduced as a transformer-based backbone with GSE and CAF modules for hyperspectral image classification.The overview identifies these two modules as making the backbone more applicable to hyperspectral images.
  • A. A Brief Review of Transformers: Multi-head attention concatenates multiple attention representations and applies a linear transformation to match the input feature dimension.The reviewed formulation uses multiple self-attention representations before output projection.
  • A. A Brief Review of Transformers: Because self-attention contains no position information, positional vectors are added to feature embeddings to encode sequence order.The resulting embedding is formulated as ai+ei, where ei is a manually given positional vector.

B. Overview of SpectralFormer

SpectralFormer is a generic ViT-based backbone designed around hyperspectral spectrometric characteristics. Its GSE and CAF modules target subtle spectral discrepancies and information transfer across layers, while supporting pixel-wise and patch-wise inputs.

  • B. Overview of SpectralFormer: SpectralFormer is a generic ViT-based backbone focused on spectrometric characteristics for fine hyperspectral image classification.The design is intended to improve applicability to highly accurate and fine-grained material classification.
  • B. Overview of SpectralFormer: GSE improves detail capture for subtle spectral discrepancies by learning from local spectral profiles rather than isolated bands.The module is integrated into the transformer framework as part of the backbone design.
  • B. Overview of SpectralFormer: CAF enhances information transitivity between layers by reducing information loss as the network becomes deeper.It addresses connectivity across layer representations within the transformer framework.
  • B. Overview of SpectralFormer: SpectralFormer supports both pixel-wise hyperspectral classification and spatial-spectral classification with batch-wise inputs.The latter is described as the spatial-spectral SpectralFormer version.

C. Group-wise Spectral Embedding (GSE)

GSE replaces band-wise transformer embeddings with group-wise embeddings learned from neighboring spectral bands. This design reflects the dense, approximately continuous sampling of hyperspectral signatures and captures local spectral detail.

  • C. Group-wise Spectral Embedding (GSE): Hyperspectral channels are densely sampled at small wavelength intervals, producing approximately continuous spectral signatures unlike discrete transformer sequences.Neighboring spectral positions reflect different absorption characteristics associated with material properties.
  • C. Group-wise Spectral Embedding (GSE): GSE learns group-wise spectral embeddings from locally neighboring bands instead of using classic band-wise representations.The goal is to capture locally detailed absorption changes important for accurate material classification.
  • C. Group-wise Spectral Embedding (GSE): GSE forms grouped representations with W and X, where n denotes the number of neighboring bands.W is optimized as a network layer, while X contains grouped spectral representations.
  • C. Group-wise Spectral Embedding (GSE): The grouping operation is overlapping, constructing each local spectral profile from neighboring entries around a central band.The local vector uses rounded offsets on both sides of the central spectral position.
  • C. Group-wise Spectral Embedding (GSE): The section contrasts band-wise and group-wise spectral embeddings in transformer-based backbone networks.The comparison is illustrated as BSE versus GSE.

D. Cross-layer Adaptive Fusion (CAF)

CAF adaptively fuses representations across nearby transformer layers to preserve memory-like information while avoiding the semantic gap associated with longer skip connections.

  • D. Cross-layer Adaptive Fusion (CAF): CAF uses a middle-range skip connection to adaptively fuse features across transformer layers.The module skips one encoder, connecting z^(l−2) to z^l.
  • D. Cross-layer Adaptive Fusion (CAF): The design addresses limited memory in short skips and insufficient fusion from long skips spanning shallow and deep features.The paper motivates CAF by contrasting short and long skip connections.
  • E. Spatial-Spectral SpectralFormer: The backbone also supports patch-wise inputs by unfolding each band’s spatial patch into a one-dimensional vector before spectral processing.This input formulation preserves spectral sequence information while incorporating spatial context.

III. EXPERIMENTS

The experiments evaluate SpectralFormer on three representative hyperspectral datasets, using dataset-specific scenes and standard land-cover training and testing sets.

  • III. EXPERIMENTS: The experimental section evaluates SpectralFormer quantitatively and qualitatively, including ablation analysis.Classification maps provide qualitative comparisons alongside quantitative experiments.
  • 1) Indian Pines Data: Indian Pines contains 200 retained spectral bands, 16 investigated categories, and 145 × 145 pixels after removing noisy and water absorption bands.The scene was collected over North-Western Indiana using AVIRIS.
  • 2) Pavia University Data: Pavia University contains 103 spectral bands, 610 × 340 pixels, and 9 land-cover classes.The scene was acquired over Pavia, Italy, using ROSIS at a 1.3m ground sampling distance.
  • 3) Houston2013 Data: Houston2013 contains 144 wavelength bands and 349 × 1905 pixels, using a cloud-free version processed to recover missing data or remove occlusions.The dataset covers the University of Houston campus and neighboring rural areas.

B. Experimental Setup

The experimental setup compares SpectralFormer with classical classifiers and representative deep backbones under specified implementation configurations.

  • B. Experimental Setup: Models are evaluated using Overall Accuracy, Average Accuracy, Kappa Coefficient, and visualized classification maps.These metrics support quantitative comparison, while maps provide qualitative comparison.
  • B. Experimental Setup: The comparison includes KNN, SVM, RF, 1-D CNN, 2-D CNN, RNN, miniGCN, transformers, and SpectralFormer.The selected methods span classical classifiers and multiple backbone architectures.
  • B. Experimental Setup: KNN uses K = 10, RF uses 200 decision trees, and SVM uses an RBF kernel with hyperparameters selected by five-fold cross-validation.The SVM search ranges are σ = [2^-3, 2^-2, ..., 2^4] and λ = [10^-2, 10^-1, ..., 10^4].
  • B. Experimental Setup: The CNN, RNN, and miniGCN baselines use the specified convolutional, recurrent, and graph-convolutional blocks and neuron configurations.The RNN has two GRU layers with 128 units each, while miniGCN uses a KNN-based graph with K = 10.
  • B. Experimental Setup: The transformer baseline uses five ViT-based encoder blocks containing only transformer encoders.This establishes the classic-transformer comparison architecture.
  • B. Experimental Setup: SpectralFormer uses the same backbone architecture for fair comparison, with 64-unit embedded spectra and five cascaded transformer encoder blocks.Each block uses four-head self-attention, an MLP with 8 hidden dimensions, and GELU activation.

3) Implementation Details:

Implementation uses Adam training, reports computational complexity, and includes ablation and sensitivity analyses for SpectralFormer’s modules and grouped-band parameter.

  • 3) Implementation Details: SpectralFormer is implemented in PyTorch with Adam, mini-batches of 64, an initial learning rate of 5e-4, and dataset-specific training epochs.The learning rate is multiplied by 0.9 after each one-tenth of the total epochs.
  • 3) Implementation Details: The CAF module can converge using about 300 epochs instead of the roughly 1000 epochs initially set for the three datasets.The paper reports this as an efficiency improvement observed in practice.
  • 3) Implementation Details: SpectralFormer has per-layer complexity O(m^2d + md^2), dominated by self-attention and multi-head operations.Here m is spectral length and d is the hidden-feature size.
  • Ablation Analysis: The ablation study adds GSE and CAF stepwise on Indian Pines to assess their contributions to classification performance.The corresponding results are reported in Table V.
  • Sensitivity Analysis: Sensitivity analysis varies the number of grouped neighboring bands and reports OA, AA, and κ for GSE and GSE + CAF.The analysis is conducted on Indian Pines and summarized in Table VI.
  • Skip-Connection Comparison: Table VII compares different skip-connection designs in transformers on the Indian Pines dataset.The comparison isolates skip-connection choices within the transformer setting.

C. Model Analysis

Ablation studies show that GSE and CAF each improve ViT-based classification, while their combination and patch-wise input provide further gains. CAF also supports effective cross-layer information exchange, and performance remains stable with many training samples.

  • Ablation Study: ViT without GSE or CAF produces the lowest classification accuracies on Indian Pines.
  • Ablation Study: Adding GSE or CAF improves pixel-wise SpectralFormer over ViT by around 4% OA and 3% OA, respectively.
  • Ablation Study: Jointly using GSE and CAF yields more than 4% additional OA improvement over the individual-module settings.
  • Ablation Study: Patch-wise SpectralFormer improves OA by at least 3% over the second-best pixel-wise record of 78.55%.
  • Parameter Sensitivity Analysis: CAF outperforms short- and long-range skip connections by exchanging cross-layer information more effectively and reducing information loss.
  • Parameter Sensitivity Analysis: With 80%, 90%, or 100% of training samples, OA tends toward stability, indicating robustness to additional samples in this evaluation.

D. Quantitative Results and Analysis

Across three hyperspectral datasets, deep-learning backbones outperform conventional classifiers, while SpectralFormer addresses transformers’ limited local spectral modeling. Its patch-wise form achieves the strongest reported classification performance by incorporating spatial context.

  • Quantitative Results: Quantitative evaluation reports OA, AA, κ, and per-class accuracies on Indian Pines, Pavia University, and Houston2013.
  • Quantitative Results: Deep-learning backbones generally outperform KNN, RF, and SVM, while transformers achieve performance comparable to CNN-, RNN-, and GCN-based models.
  • Quantitative Results: SpectralFormer improves classification by extracting local spectral information from neighboring bands, addressing transformers’ limited modeling of local spectral discrepancies.
  • Quantitative Results: Patch-wise SpectralFormer achieves higher accuracies than competing methods because spatial-contextual information is jointly considered during sequential feature extraction.
  • Quantitative Results: SpectralFormer performs better on challenging or imbalanced Indian Pines classes by focusing on particular absorption positions of spectral profiles.

E. Visual Evaluation

Visual evaluations compare classification maps across three datasets and encoder features with or without CAF. The reported visualizations indicate reduced noise and finer object structure when stronger spatial or cross-layer representations are used.

  • Classification Maps: Conventional classifiers tend to produce salt-and-pepper noise in maps across the three datasets, indicating less accurate material identification.
  • Classification Maps: Classification maps are provided for Indian Pines, Pavia University, and Houston2013, each with a 2× zoomed region for detailed observation.
  • Feature Visualization: CAF-based encoder features have finer object edges, outlines, and textural structures than features from the original ViT without CAF.
  • Interpretation: Transformers characterize global sequential properties, whereas SpectralFormer focuses more specifically on local spectral information and cross-layer memory-like components.
  • Limitations and Future Work: The authors identify future work in lightweight architectures, physical priors, interpretability, and selecting the number of CAF-connected encoders.
Loading 2107.02988v2…