Source-linked AI summary
Dual Selective Fusion Transformer Network for Hyperspectral Image Classification
Yichu Xu, Di Wang, Lefei Zhang, Liangpei Zhang
TL;DR
HSI Transformers face fixed receptive fields that can miss land-cover-specific context scales and self-attention that includes irrelevant tokens. DSFormer jointly selects spatial-spectral receptive fields and relevant tokens; ablations report performance gains from both proposed blocks.
Problem
Existing HSI Transformers often use fixed receptive fields despite differing land-cover context needs, and dense attention can involve redundant or irrelevant tokens.
Source: §1 · p. 1, §3.3 · p. 5Method
DSFormer combines adaptive multiscale spatial-spectral feature selection in KSFTB with relevant-token selection and fusion in TSFTB.
Source: §1 · p. 2Results
Ablations report that both KSFTB and TSFTB significantly enhance classification performance.
Source: §4.4 · p. 9Takeaways & Limitations
DSFormer provides joint spatial-spectral context modeling through adaptive receptive-field and token selection.
Source: §5 · p. 14
Abstract
from arXiv · showhide
Transformer has achieved satisfactory results in the field of hyperspectral image (HSI) classification. However, existing Transformer models face two key challenges when dealing with HSI scenes characterized by diverse land cover types and rich spectral information: (1) A fixed receptive field overlooks the effective contextual scales required by various HSI objects; (2) invalid self-attention features in context fusion affect model performance. To address these limitations, we propose a novel Dual Selective Fusion Transformer Network (DSFormer) for HSI classification. DSFormer achieves joint spatial and spectral contextual modeling by flexibly selecting and fusing features across different receptive fields, effectively reducing unnecessary information interference by focusing on the most relevant spatial-spectral tokens. Specifically, we design a Kernel Selective Fusion Transformer Block (KSFTB) to learn an optimal receptive field by adaptively fusing spatial and spectral features across different scales, enhancing the model's ability to accurately identify diverse HSI objects. Additionally, we introduce a Token Selective Fusion Transformer Block (TSFTB), which strategically selects and combines essential tokens during the spatial-spectral self-attention fusion process to capture the most crucial contexts. Extensive experiments conducted on four benchmark HSI datasets demonstrate that the proposed DSFormer significantly improves land cover classification accuracy, outperforming existing state-of-the-art methods. Specifically, DSFormer achieves overall accuracies of 96.59%, 97.66%, 95.17%, and 94.59% in the Pavia University, Houston, Indian Pines, and Whu-HongHu datasets, respectively, reflecting improvements of 3.19%, 1.14%, 0.91%, and 2.80% over the previous model. The code will be available online at https://github.com/YichuXu/DSFormer.
1. Introduction
HSI classification assigns labels to image pixels, while Transformer methods must accommodate varying land-cover context scales and avoid noise from uninformative tokens. DSFormer addresses these challenges by selecting and fusing spatial-spectral features and relevant tokens.
1. Introduction
- CNNs capture hierarchical features but local convolutions limit long-range dependencies, while self-attention captures non-local context at high computational cost.The cited cost comes from extensive similarity calculations.Source: §1 · p. 1
- Fixed receptive fields overlook differing land-cover context needs, and vanilla self-attention can introduce noise from uninformative tokens while spatial and spectral contexts remain separately modeled.The cited example describes meadows and bitumen as unnecessary tokens relative to an asphalt query.Source: §1 · pp. 1–2
- KSFTB adaptively selects and integrates spatial and spectral features across receptive fields to acquire context suited to varied HSI objects.The block is designed to learn an optimal receptive field.Source: §1 · p. 2
- TSFTB selects relevant tokens during context fusion, integrating essential information while reducing irrelevant-object interference in spatial-spectral features.DSFormer combines these selective mechanisms to model spatial-spectral context.Source: §1 · p. 2
- DSFormer outperforms other state-of-the-art methods across four benchmark HSI datasets.The datasets are Pavia University, Houston, Indian Pines, and Whu-HongHu.Source: §1 · p. 2
2. Related Works
The related-work section traces HSI classification from convolutional feature extraction toward Transformer-based context modeling. It positions DSFormer as combining adaptive receptive-field selection with selective token fusion.
2.1. CNN-based Methods
- CNN-based methods progressed from 1D spectral and 2D spatial-spectral extraction to 3D joint features and multiscale convolution, but local perception limits captured information.Attention-based approaches subsequently emerged, particularly self-attention methods.Source: §2.1 · p. 2, §2.2 · p. 2
2.2. Transformer-based Methods
- Transformer-based HSI methods use self-attention for context modeling and include spectral, grouped-token, multiscale, and morphological approaches; DSFormer adds adaptive receptive-field and relevant-token fusion.Its DSFTG comprises a KSFTB and three consecutive TSFTBs, followed by a fully connected classification head.Source: §2.2 · p. 3, Figure 2 · p. 3
3. Methodology
DSFormer processes overlapping HSI patch cubes through shallow convolution, two dual selective fusion groups, and a classification head. Its KSFTB selects spatial-spectral features across receptive fields, while TSFTB selects relevant tokens for self-attention.
3.1. Overview of DSFormer
- Overlapping HSI patch cubes are classified by their central pixels after shallow convolution, two DSFTGs, layer normalization, and a fully connected classification head.The overview first applies PCA for dimensionality reduction and extracts overlapping patch cubes.Source: §3.1 · pp. 3–4, §3 · p. 3
3.2. Kernel Selective Fusion Transformer Block
- KSFTB uses dilated depth-wise convolutions with different receptive fields and selects an appropriate scale for each land-cover context.The resulting features are integrated across spatial and spectral perspectives to form effective spatial-spectral features.Source: §3.2 · p. 4
- KSFA concatenates features, uses channel-wise average and max pooling for spatial descriptors, and uses global average pooling and a fully connected layer to form spectral attention features; softmax generates spectral masks, and matrix multiplication with corresponding masks yields selection weights.Source: §3.2 · p. 4
3.3. Token Selective Fusion Transformer Block
- TSFA groups input features and applies 3D point-wise and depth-wise convolutions to generate spatial-spectral queries, keys, and values for attention.It retains the top k percent of attention entries per row and masks the rest, focusing attention on selected tokens rather than all token interactions.Source: §3.3 · p. 5
- The multi-head TSFA outputs are concatenated and aggregated by a 1×1 convolution, with the attention output added to the input through a residual connection.The resulting feature is projected as the TSFA output.Source: §3.3 · pp. 5–6
4. Experiment
Experiments evaluate DSFormer on four HSI datasets through parameter analysis, ablations, comparisons with other methods, visualizations, and efficiency analysis. The results examine accuracy, robustness, and inference efficiency.
4.1. Data Descriptions
- The four benchmarks span Pavia University, Houston, Indian Pines, and Whu-HongHu, with imagery dimensions, spectral bands, and land-cover categories differing across datasets.The datasets range from 9 categories and 103 bands for Pavia University to 17 crop types and 270 bands for Whu-HongHu.Source: §4.1 · p. 6
4.3. Parameter Analysis
- Across four datasets, the selected token fractions were k=0.4, 0.6, 0.8, and 0.8; accuracy was lower with insufficient tokens and declined when selection exceeded the optimum.Most tested k configurations outperformed full self-attention at k=1.Source: Figure 5 · p. 6, §4.3 · p. 8
- Grouping improved classification over g=1, and g=4 achieved the best accuracy across all four datasets while retaining relatively efficient computational complexity.Larger group counts expand the attention matrix and increase computational complexity.Source: Figure 6 · p. 6, §4.3 · p. 9
4.2. Experimental Settings
- Experiments used 30 training samples per class for Pavia University and generally 50 for the other datasets, repeated ten times with different random seeds.Indian Pines classes 1, 7, and 9 used 15 training samples; reported results are means and standard deviations.Source: §4.2 · p. 7
4.4. Ablation Study
- Both KSFTB and TSFTB improved classification, and their combination achieved the best OAs across the four datasets.On Pavia University, the single-convolution baseline reached 87.67% OA, nine stacked convolutional layers reached 86.33%, KSFTB reached 93.44%, TSFTB reached 95.81%, and both modules together reached 96.59%. The single spatial and spectral branches reached 96.09% and 96.01% OA, respectively, compared with 96.59% when both receptive-field fusion mechanisms were applied.Source: §4.4 · pp. 9–10
4.5. Performance Comparison and Analysis
- 96.59% OA on Pavia University surpassed the second-best method by 3.2%, and DSFormer achieved the best results in nearly all per-class accuracy metrics.MorphFormer slightly outperformed DSFormer on the ninth class, shadows.Source: §4.5 · p. 12
- DSFormer remained the most accurate method across training-sample configurations, produced classification maps closer to ground truth, and balanced precision with computational efficiency.Across four datasets, accuracy steadily increased with training samples; DSFormer also had fewer parameters than GAHT and GSCViT and was faster than SpectralFormer while achieving higher OAs.Source: §4.5 · p. 13
4.6. Visualization
- For Asphalt, the larger receptive field received 54.07% weight; smaller receptive fields were sufficient for spatially continuous Meadows and Bare Soil.In the illustrated attention matrix, token selection retained 40% of values and set the other 60% to zero.Source: Figure 12 · p. 12, §4.6 · pp. 13–14, Figure 13 · p. 13
5. Conclusion
DSFormer combines adaptive receptive-field selection with selective spatial-spectral token fusion for HSI classification. Experiments on four benchmark datasets report competitive accuracy, robustness, and inference efficiency.
5. Conclusion
- DSFormer uses KSFTB to select and fuse multiscale features through multiple dilated convolutions, adapting receptive fields to objects' spatial-spectral context needs.The network also uses TSFTB for adaptive selection of relevant spatial-spectral tokens under a spectral grouping strategy.Source: §5 · p. 14
- TSFTB selects valuable spatial-spectral tokens during context fusion to reduce the influence of redundant and irrelevant regions and targets.The conclusion presents KSFTB and TSFTB as complementary selection-and-fusion mechanisms.Source: §5 · p. 14
- Experiments on four benchmark HSI datasets report competitive DSFormer performance in accuracy, robustness, and inference efficiency versus advanced methods.Source: §5 · p. 14