PaperlayerView exact revision on arXiv

Source-linked AI summary

AGSA-Net: Abundance-Guided Self-Attention Network for Spectral Unmixing-Aware Hyperspectral Remote Sensing Image Classification

Nafisa Anjum, Satavisa Dey Borno, Ananna Saha, Mir Faiyaz Hossain, Sifat Momen, Nabeel Mohammed, Shafin Rahman

arXiv:2609.06359v1cs.CVcs.AIcs.LG

TL;DR

HSI classification remains difficult under mixed pixels, spectral redundancy, noise, and limited labels, while existing methods often use abundance information only as auxiliary features. AGSA-Net couples spectral unmixing with transformer classification by injecting abundance-derived material affinity into self-attention and fusing abundance with transformer features, demonstrating competitive performance across three benchmark datasets, particularly complex urban scenes.

  • Problem

    Mixed pixels, spectral redundancy, sensor noise, and limited labeled samples challenge HSI classification, while existing methods often fail to integrate material composition into contextual reasoning.

  • Method

    AGSA-Net uses a dual-branch unmixing-transformer framework that converts abundance maps into material affinity for additive self-attention guidance and fuses abundance with transformer representations.

  • Results

    Experiments on Indian Pines, Augsburg, and Berlin demonstrate competitive performance, particularly in complex urban environments.

  • Takeaways & Limitations

    Physically informed attention can incorporate subpixel material composition into contextual HSI representation learning for mixed-pixel and heterogeneous scenes.

  • Takeaways & Limitations

    AGSA-Net introduces additional computational cost due to transformer-based attention, and its performance depends on the available setting described by the authors.

Abstract

from arXiv · show

Hyperspectral image (HSI) classification plays a vital role in remote sensing applications, including agriculture, environmental monitoring, and urban analysis. However, its performance remains challenged by high spectral redundancy, noise sensitivity, and the difficulty of jointly modeling local material composition and long-range spectral dependencies. To address this, we propose AGSA-Net, an abundance-guided self-attention network that explicitly integrates spectral unmixing priors into the classification process. AGSA Net first estimates physically meaningful subpixel abundance maps subject to non-negativity and sum-to-one constraints, regularized by hybrid linear-nonlinear reconstruction decoder. The learned abundances are then used to construct an abundance affinity prior that guides a spectral transformer to emphasize class-discriminative interactions, and the resulting transformer features are fused with compact abundance descriptors for final prediction; in contrast to existing approaches that use abundance as auxiliary or concatenated features. Experiments on Indian Pines, Augsburg, and Berlin demonstrate the benefit of incorporating abundance- guided contextual modeling, particularly in heterogeneous urban scenes. The source code and trained models are available at: https://github.com/nnuvi/AGSA-Net

I. INTRODUCTION

HSI classification is challenged by mixed pixels, spectral redundancy, noise, and limited labels, while existing methods often fail to incorporate material composition into contextual reasoning. AGSA-Net couples spectral unmixing with transformer classification, using abundance-guided attention and feature fusion, and reports consistent benchmark improvements.

  • Mixed pixels, sensor noise, and limited labeled samples increase inter-class similarity and intra-class variability, making robust HSI classification difficult.
  • Existing approaches often treat abundance information as auxiliary or concatenated features, while transformer attention primarily uses spectral or learned feature similarity.
  • The method converts abundance representations into a pixel-wise material-affinity matrix and injects it as an additive bias into multi-head self-attention.
  • AGSA-Net couples spectral unmixing with transformer-based classification through a dual-branch framework rather than using unmixing only as an auxiliary input.
  • Abundance-guided attention and joint feature fusion support subpixel-aware spectral-spatial modeling and contextual interactions based on material composition.
  • Experiments on Indian Pines, Augsburg, and Berlin demonstrate consistent improvements over several recent state-of-the-art methods.

II. RELATED WORKS

Related HSI classification research progressed from handcrafted and classical spectral-spatial methods to CNNs, recurrent networks, graph models, and transformers. Recent work emphasizes global dependencies, hybrid feature extraction, multimodal flexibility, and robustness under scarce supervision, while often incurring computational costs.

  • Early HSI classification used SVMs, extended morphological profiles, sparse representation, local binary patterns, and other handcrafted spectral-spatial features.
  • CNNs enabled hierarchical spectral-spatial learning, while recurrent networks modeled sequential spectral relationships and unified architectures improved contextual modeling.
  • Graph-based methods modeled non-Euclidean spatial structures and improved spatial consistency and robustness, often at the cost of increased computational burden.
  • Transformers introduced self-attention for global dependencies, with later models targeting spectral long-range modeling, multimodal inputs, and spectral-spatial tokenization.
  • Recent transformer and hybrid approaches seek to balance local detail preservation with global context modeling and address limited labels through self-supervised and contrastive learning.
  • Multifeature hybrids and survey work connect classical efficiency with deep expressivity while highlighting feature fusion and scarce-supervision robustness.

D. Limitations of Existing Works and Our Approach

Existing HSI classifiers struggle to combine local material composition with long-range context: CNNs are local, whereas transformer methods often rely on unreliable feature similarity in mixed and redundant spectra. AGSA-Net addresses this by coupling unmixing-derived abundance information with contextual attention.

  • CNN-based models primarily capture local spectral-spatial patterns and struggle with long-range contextual dependencies.
  • Transformer methods improve global context modeling but typically rely on feature similarity alone, which is unreliable under mixed pixels and spectral redundancy.
  • Unmixing-assisted classifiers estimate material abundances but often treat them as auxiliary inputs instead of integrating them into contextual reasoning.
  • AGSA-Net integrates abundance-based material affinity into self-attention to combine subpixel composition information with spectral features.
  • The framework extracts patches, estimates abundance maps with an unmixing autoencoder, and uses abundance-derived affinity to guide contextual feature interactions.

C. Unmixing Autoencoder

The unmixing autoencoder estimates physically meaningful pixel-wise abundance maps and reconstructs input spectra with a hybrid mixing decoder. Non-negativity, sum-to-one normalization, and spectral-angle supervision support stable abundance representations for classification.

  • Abundance estimation: The encoder maps each pixel spectrum to a K-dimensional abundance vector representing fractional contributions of latent endmembers.K is set equal to the number of semantic classes for a stable and interpretable representation.
  • Abundance estimation: 1 × 1 convolutions perform pixel-wise abundance estimation while preserving the patch’s spatial structure.
  • Abundance constraints: Non-negativity and sum-to-one constraints make abundances physically interpretable at each spatial location.Absolute values enforce non-negativity, while normalization along the endmember dimension enforces the sum-to-one constraint.
  • Abundance constraints: The normalized abundance vectors preserve relative material proportions needed to construct the attention-guiding affinity matrix.The formulation is described as maintaining stable gradients during normalization.
  • Spectral reconstruction: A lightweight decoder reconstructs input spectra using linear and nonlinear mixing interactions under spectral angle distance supervision.The unmixing formulation and training objective follow the cited prior framework.

D. Abundance-Guided Spectral Transformer

The abundance-guided spectral transformer converts normalized abundance maps into material-affinity relationships and injects them into self-attention. This supplies composition-based contextual guidance alongside learned spectral interactions.

  • Transformer input: Each pixel spectrum becomes a transformer token, and N = p^2 tokens are projected into a d-dimensional latent space with a prepended classification token.
  • Material affinity: Flattened abundance vectors produce an affinity matrix that measures pixel-to-pixel similarity in latent material composition.High affinity indicates similar material composition, whereas low affinity indicates compositionally distinct pixels.
  • Material affinity: The affinity matrix is low-rank with rank bounded by K, providing a compact, physically meaningful representation of pixel relationships.Here K equals the number of semantic classes, imposing a structured inductive bias on attention.
  • Abundance-guided attention: After padding to the token sequence length, the affinity matrix guides transformer attention through abundance-guided multi-head self-attention.
  • Abundance-guided attention: Adding λAF to attention logits encourages attention toward pixels with similar sub-pixel composition while retaining query-key similarity.λ is learnable, allowing flexible balancing between spectral interactions and composition-based guidance.

E. Feature Fusion and Classification

AGSA-Net combines the transformer’s global contextual representation with a compact abundance descriptor before classification. Training jointly uses classification and abundance-reconstruction objectives within the proposed experimental framework.

  • Feature fusion: The final transformer classification token is normalized to form a global contextual representation, while a lightweight convolutional module produces a global abundance descriptor.
  • Feature fusion: Concatenating transformer and abundance representations preserves their distinct contextual and physically interpretable information before the linear classification head.The fusion is intended to combine complementary information without premature mixing.
  • Training objective: Cross-entropy classification loss and spectral angle distance loss supervise prediction and abundance reconstruction, respectively.The weighting coefficient α is fixed to 1 in all experiments.
  • Training process: Algorithm 1 summarizes training through abundance estimation, patch reconstruction, and abundance-guided transformer classification.
  • Experimental framework: The model is evaluated on three benchmark hyperspectral image datasets: Indian Pines, Augsburg, and Berlin.

Berlin.

The experiments use Indian Pines, Augsburg, and Berlin HSI datasets with distinct scene characteristics, while training uses HSI data only and follows specified optimization protocols.

  • Datasets: The evaluated datasets include Indian Pines, Augsburg, and Berlin, whose spectral and spatial properties support diverse performance assessment.Their characteristics are summarized in Table I.
  • Datasets: Indian Pines contains 145 × 145 pixels, 224 spectral bands reduced to 200 after water-band removal, and 16 land-cover classes.The scene covers agricultural fields, forests, and grasslands.
  • Datasets: Augsburg contains 332 × 485 pixels, 180 spectral bands, and seven urban-material classes at 30 m spatial resolution.The dataset includes vegetation, water, roads, bare soil, and buildings.
  • Datasets: Berlin contains 1723 × 476 pixels, 244 spectral bands, and eight land-cover classes at 30 m spatial resolution.Its classes include vegetation, tough surfaces, soil, and multiple building types.
  • Experimental scope: Although Augsburg and Berlin provide SAR and DSM metadata, experiments use only hyperspectral images for training and evaluation.The additional modalities are listed to describe dataset characteristics but are not used experimentally.
  • Implementation: Training uses AdamW with learning rate η = 1×10^-3, weight decay 1×10^-4, batch size 64, and dataset-specific patch sizes.Patch sizes are 7 × 7 for Indian Pines and Augsburg and 5 × 5 for Berlin.

B. Quantitative Performance Comparison

AGSA-Net performs competitively across the three datasets, with stronger overall results on heterogeneous Augsburg and Berlin scenes and improvements for several overlap-affected classes. Its gains come with moderate computational overhead and do not extend uniformly to every class.

  • Overall comparison: AGSA-Net achieves competitive performance across all datasets and provides better overall results on heterogeneous Augsburg and Berlin scenes.On Indian Pines, some recent methods obtain higher overall accuracy, but AGSA-Net outperforms DSNet.
  • Class-wise performance: AGSA-Net improves several classes over DSNet, including overlap-affected agricultural classes on Indian Pines and urban classes on Augsburg and Berlin.Reported improvements include Corn-notill, Soybean-mintill, Woods, Industrial Area, Residential Area, Soil, and Forest.
  • Class-wise performance: AGSA-Net does not improve all classes uniformly, with lower performance than DSNet for several urban categories on Berlin and persistent difficulty for Commercial Area and Water on Augsburg.The cited errors are associated with severe class overlap, limited training samples, and shared materials among urban categories.
  • Qualitative analysis: Qualitative maps show more spatially coherent urban predictions and fewer fragmented boundary errors in Augsburg and Berlin, while residual errors remain for similar urban surfaces and minority classes.These observations support improved contextual discrimination without eliminating difficult cases.
  • Computational cost: AGSA-Net introduces a moderate increase in FLOPs and inference time relative to lightweight models such as DSNet and SSFTT.The paper characterizes the model as a trade-off between improved representation capability and moderate computational cost.

C. Quantitative Analysis of Abundance Affinity

The analysis evaluates abundance representations as material-oriented similarity priors and tests whether abundance-guided attention contributes beyond spectral or disrupted guidance. Abundance similarity produces larger class-separation gaps, and the proposed guidance performs best in the controlled intervention.

  • Similarity construction: Patch-level abundance similarity is computed from normalized, spatially averaged abundance vectors, while spectral similarity uses cosine similarity between mean spectral signatures.The abundance comparison follows the inner-product formulation associated with Eq. 3.
  • Similarity comparison: Abundance representations consistently produce larger same-class versus different-class similarity gaps than raw spectral representations across all datasets.This supports their use as a stronger material-oriented prior for attention guidance.
  • Guidance intervention: The controlled intervention compares spectral similarity guidance, shuffled abundance guidance, and the proposed abundance guidance while isolating the attention-bias mechanism.The shuffled condition preserves affinity statistics but disrupts material-relationship structure.
  • Guidance intervention: Abundance guidance achieves the best performance across all datasets, whereas spectral and shuffled guidance reduce performance.The result links the intervention outcome to the structure of material relationships rather than affinity statistics alone.

D. Impact of Training Sample Ratio

AGSA-Net remains competitive as labeled-data availability decreases and shows a more stable accuracy trend than several comparison methods. The full configuration performs strongly overall, while ablation results characterize abundance-guided attention as complementary rather than uniformly dominant.

  • Training protocol: Training uses stratified class-wise ratios of 20%, 40%, 60%, 80%, and 100%, with the test set unchanged across settings.The protocol reduces labeled samples proportionally within each class.
  • Low-label performance: At a 20% training ratio on Indian Pines, AGSA-Net reaches an overall accuracy of 0.5994, versus 0.4994 for DSNet and 0.4279 for MambaHSI+.It also exceeds S2Mamba at 0.5263 while remaining below GraphGST at 0.6764.
  • Cross-dataset trends: Across Augsburg and Berlin, AGSA-Net maintains higher or comparable performance at low training ratios and remains competitive as the ratio increases.At 80% and 100%, performance gaps narrow, but AGSA-Net avoids degradation.
  • Robustness: AGSA-Net shows smaller accuracy reductions when moving from higher to lower training ratios than DSNet, MambaHSI+, and S2Mamba.The paper interprets this as stronger robustness under limited labeled-data conditions.
  • Ablation: The full AGSA-Net configuration achieves the strongest overall performance on Indian Pines and Augsburg and remains competitive on Berlin.The ablation integrates unmixing, abundance-guided self-attention, subpixel fusion, and spectral-transformer reasoning.
  • Ablation: Abundance-guided self-attention contributes complementary contextual information, although its OA effect is modest and can be offset by abundance fusion on Berlin.Removing it slightly increases Berlin OA and κ despite lower performance elsewhere.

F. Visualization of Abundance-Guided Attention Behavior

Abundance guidance reshapes transformer attention toward regions with similar material composition, with dataset-dependent strength and complementary gains alongside abundance feature fusion.

  • The attention analysis compares unguided and guided maps, abundance similarity, and their difference for representative queries in Augsburg and Indian Pines.Query tokens are selected by the largest alignment increase while excluding boundary tokens.
  • Guided attention redistributes attention toward regions with similar abundance characteristics, with stronger restructuring on Augsburg than Indian Pines.Indian Pines shows a more moderate but consistent refinement behavior.
  • Abundance guidance provides complementary gains to abundance feature fusion rather than replacing it.This interpretation is reported alongside the attention redistribution analysis and the core-component comparison.
  • The learnable guidance weight λ controls how strongly abundance affinity contributes to attention logits and adapts to each dataset during training.Training analysis tracks λ alongside validation accuracy on Indian Pines, Augsburg, and Berlin.

H. Sensitivity Analysis of the Number of Endmembers (K)

AGSA-Net treats the endmember count as a compact latent-representation parameter rather than a direct estimate of physical materials, while learned features show improved class separation.

  • Sensitivity Analysis of the Number of Endmembers (K): AGSA-Net sets K = U, where U is the number of semantic classes, to obtain a compact latent abundance representation consistent with prior frameworks.The representation is used for structural priors and feature fusion, not explicit material identification.
  • Sensitivity Analysis of the Number of Endmembers (K): The selected endmember count is not assumed to equal the number of physical materials because classes may contain multiple materials and materials may be shared across classes.This motivates evaluating K as a representation-design choice rather than a physical-material count.
  • Sensitivity Analysis of the Number of Endmembers (K): The sensitivity analysis evaluates K values from 0.5U to 2U using OA, AA, and Kappa across the three datasets.Table IX reports the best values for each dataset with highlighted entries.
  • Feature Representation Visualization: AGSA-Net produces the most distinct and well-separated Indian Pines t-SNE clusters, with higher intra-class compactness and reduced overlap than the compared representations.Raw spectral features overlap strongly, Transformer features improve partially, and DSNet produces more compact clusters.
  • Conclusion: The method is reported to achieve competitive performance across multiple benchmark datasets, particularly in complex urban environments.The conclusion highlights physically informed attention for HSI classification.
Loading 2609.06359v1…