AGSA-Net: Abundance-Guided Self-Attention Network for Spectral Unmixing-Aware Hyperspectral Remote Sensing Image Classification

• Mixed pixels, sensor noise, and limited labeled samples increase inter-class similarity and intra-class variability, making robust HSI classification difficult.Source: §I · p. 1
• Existing approaches often treat abundance information as auxiliary or concatenated features, while transformer attention primarily uses spectral or learned feature similarity.Source: §I · p. 1
• The method converts abundance representations into a pixel-wise material-affinity matrix and injects it as an additive bias into multi-head self-attention.Source: §I · p. 2, Figure 1 · p. 2
• AGSA-Net couples spectral unmixing with transformer-based classification through a dual-branch framework rather than using unmixing only as an auxiliary input.Source: §I · pp. 1–2
• Abundance-guided attention and joint feature fusion support subpixel-aware spectral-spatial modeling and contextual interactions based on material composition.Source: §I · p. 2
• Experiments on Indian Pines, Augsburg, and Berlin demonstrate consistent improvements over several recent state-of-the-art methods.Source: §I · p. 2

• Early HSI classification used SVMs, extended morphological profiles, sparse representation, local binary patterns, and other handcrafted spectral-spatial features.Source: A. Machine Learning Approaches in HSI… · p. 2
• CNNs enabled hierarchical spectral-spatial learning, while recurrent networks modeled sequential spectral relationships and unified architectures improved contextual modeling.Source: CNN-based models · p. 2
• Graph-based methods modeled non-Euclidean spatial structures and improved spatial consistency and robustness, often at the cost of increased computational burden.Source: Graph-based and Hybrid Models · p. 3
• Transformers introduced self-attention for global dependencies, with later models targeting spectral long-range modeling, multimodal inputs, and spectral-spatial tokenization.Source: Transformer-based Models · p. 3
• Recent transformer and hybrid approaches seek to balance local detail preservation with global context modeling and address limited labels through self-supervised and contrastive learning.Source: C. Emerging Trends and Hybrid Techniques · p. 3
• Multifeature hybrids and survey work connect classical efficiency with deep expressivity while highlighting feature fusion and scarce-supervision robustness.Source: C. Emerging Trends and Hybrid Techniques · p. 3

• CNN-based models primarily capture local spectral-spatial patterns and struggle with long-range contextual dependencies.Source: D. Limitations of Existing Works and… · p. 3
• Transformer methods improve global context modeling but typically rely on feature similarity alone, which is unreliable under mixed pixels and spectral redundancy.Source: D. Limitations of Existing Works and… · p. 3
• Unmixing-assisted classifiers estimate material abundances but often treat them as auxiliary inputs instead of integrating them into contextual reasoning.Source: D. Limitations of Existing Works and… · p. 3
• AGSA-Net integrates abundance-based material affinity into self-attention to combine subpixel composition information with spectral features.Source: D. Limitations of Existing Works and… · p. 3, A. Problem Formulation · p. 4
• The framework extracts patches, estimates abundance maps with an unmixing autoencoder, and uses abundance-derived affinity to guide contextual feature interactions.Source: A. Problem Formulation · pp. 3–4, B. Model Overview · p. 4

Abundance estimation

• The encoder maps each pixel spectrum to a K-dimensional abundance vector representing fractional contributions of latent endmembers.K is set equal to the number of semantic classes for a stable and interpretable representation.Source: C. Unmixing Autoencoder · p. 4
• 1 × 1 convolutions perform pixel-wise abundance estimation while preserving the patch’s spatial structure.Source: C. Unmixing Autoencoder · p. 4

Abundance constraints

• Non-negativity and sum-to-one constraints make abundances physically interpretable at each spatial location.Absolute values enforce non-negativity, while normalization along the endmember dimension enforces the sum-to-one constraint.Source: C. Unmixing Autoencoder · p. 4
• The normalized abundance vectors preserve relative material proportions needed to construct the attention-guiding affinity matrix.The formulation is described as maintaining stable gradients during normalization.Source: C. Unmixing Autoencoder · p. 4

Spectral reconstruction

• A lightweight decoder reconstructs input spectra using linear and nonlinear mixing interactions under spectral angle distance supervision.The unmixing formulation and training objective follow the cited prior framework.Source: C. Unmixing Autoencoder · p. 4

Transformer input

• Each pixel spectrum becomes a transformer token, and N = p^2 tokens are projected into a d-dimensional latent space with a prepended classification token.Source: D. Abundance-Guided Spectral Transformer · p. 4

Material affinity

• Flattened abundance vectors produce an affinity matrix that measures pixel-to-pixel similarity in latent material composition.High affinity indicates similar material composition, whereas low affinity indicates compositionally distinct pixels.Source: D. Abundance-Guided Spectral Transformer · p. 4
• The affinity matrix is low-rank with rank bounded by K, providing a compact, physically meaningful representation of pixel relationships.Here K equals the number of semantic classes, imposing a structured inductive bias on attention.Source: D. Abundance-Guided Spectral Transformer · p. 5

Abundance-guided attention

• After padding to the token sequence length, the affinity matrix guides transformer attention through abundance-guided multi-head self-attention.Source: D. Abundance-Guided Spectral Transformer · p. 5
• Adding λAF to attention logits encourages attention toward pixels with similar sub-pixel composition while retaining query-key similarity.λ is learnable, allowing flexible balancing between spectral interactions and composition-based guidance.Source: D. Abundance-Guided Spectral Transformer · p. 5

Feature fusion

• The final transformer classification token is normalized to form a global contextual representation, while a lightweight convolutional module produces a global abundance descriptor.Source: E. Feature Fusion and Classification · p. 6
• Concatenating transformer and abundance representations preserves their distinct contextual and physically interpretable information before the linear classification head.The fusion is intended to combine complementary information without premature mixing.Source: E. Feature Fusion and Classification · p. 6

Training objective

• Cross-entropy classification loss and spectral angle distance loss supervise prediction and abundance reconstruction, respectively.The weighting coefficient α is fixed to 1 in all experiments.Source: E. Feature Fusion and Classification · p. 6

Training process

• Algorithm 1 summarizes training through abundance estimation, patch reconstruction, and abundance-guided transformer classification.Source: E. Feature Fusion and Classification · p. 6

Experimental framework

• The model is evaluated on three benchmark hyperspectral image datasets: Indian Pines, Augsburg, and Berlin.Source: §IV · p. 6, A. Dataset Details and Experimental… · p. 6

Datasets

• The evaluated datasets include Indian Pines, Augsburg, and Berlin, whose spectral and spatial properties support diverse performance assessment.Their characteristics are summarized in Table I.Source: Berlin · p. 6, Indian Pines · p. 6, Augsburg · p. 6
• Indian Pines contains 145 × 145 pixels, 224 spectral bands reduced to 200 after water-band removal, and 16 land-cover classes.The scene covers agricultural fields, forests, and grasslands.Source: Indian Pines · p. 6
• Augsburg contains 332 × 485 pixels, 180 spectral bands, and seven urban-material classes at 30 m spatial resolution.The dataset includes vegetation, water, roads, bare soil, and buildings.Source: Augsburg · p. 6
• Berlin contains 1723 × 476 pixels, 244 spectral bands, and eight land-cover classes at 30 m spatial resolution.Its classes include vegetation, tough surfaces, soil, and multiple building types.Source: Augsburg · p. 6

Experimental scope

• Although Augsburg and Berlin provide SAR and DSM metadata, experiments use only hyperspectral images for training and evaluation.The additional modalities are listed to describe dataset characteristics but are not used experimentally.Source: Augsburg · p. 6

Implementation

• Training uses AdamW with learning rate η = 1×10^-3, weight decay 1×10^-4, batch size 64, and dataset-specific patch sizes.Patch sizes are 7 × 7 for Indian Pines and Augsburg and 5 × 5 for Berlin.Source: Augsburg · p. 7

Overall comparison

• AGSA-Net achieves competitive performance across all datasets and provides better overall results on heterogeneous Augsburg and Berlin scenes.On Indian Pines, some recent methods obtain higher overall accuracy, but AGSA-Net outperforms DSNet.Source: B. Quantitative Performance Comparison · p. 7

Class-wise performance

• AGSA-Net improves several classes over DSNet, including overlap-affected agricultural classes on Indian Pines and urban classes on Augsburg and Berlin.Reported improvements include Corn-notill, Soybean-mintill, Woods, Industrial Area, Residential Area, Soil, and Forest.Source: B. Quantitative Performance Comparison · p. 7
• AGSA-Net does not improve all classes uniformly, with lower performance than DSNet for several urban categories on Berlin and persistent difficulty for Commercial Area and Water on Augsburg.The cited errors are associated with severe class overlap, limited training samples, and shared materials among urban categories.Source: B. Quantitative Performance Comparison · p. 7

Qualitative analysis

• Qualitative maps show more spatially coherent urban predictions and fewer fragmented boundary errors in Augsburg and Berlin, while residual errors remain for similar urban surfaces and minority classes.These observations support improved contextual discrimination without eliminating difficult cases.Source: B. Quantitative Performance Comparison · p. 7

Computational cost

• AGSA-Net introduces a moderate increase in FLOPs and inference time relative to lightweight models such as DSNet and SSFTT.The paper characterizes the model as a trade-off between improved representation capability and moderate computational cost.Source: B. Quantitative Performance Comparison · p. 7, Table V · p. 8

Similarity construction

• Patch-level abundance similarity is computed from normalized, spatially averaged abundance vectors, while spectral similarity uses cosine similarity between mean spectral signatures.The abundance comparison follows the inner-product formulation associated with Eq. 3.Source: C. Quantitative Analysis of Abundance… · p. 8

Similarity comparison

• Abundance representations consistently produce larger same-class versus different-class similarity gaps than raw spectral representations across all datasets.This supports their use as a stronger material-oriented prior for attention guidance.Source: C. Quantitative Analysis of Abundance… · p. 9, Table VI · p. 9

Guidance intervention

• The controlled intervention compares spectral similarity guidance, shuffled abundance guidance, and the proposed abundance guidance while isolating the attention-bias mechanism.The shuffled condition preserves affinity statistics but disrupts material-relationship structure.Source: C. Quantitative Analysis of Abundance… · p. 9
• Abundance guidance achieves the best performance across all datasets, whereas spectral and shuffled guidance reduce performance.The result links the intervention outcome to the structure of material relationships rather than affinity statistics alone.Source: C. Quantitative Analysis of Abundance… · p. 9

Training protocol

• Training uses stratified class-wise ratios of 20%, 40%, 60%, 80%, and 100%, with the test set unchanged across settings.The protocol reduces labeled samples proportionally within each class.Source: D. Impact of Training Sample Ratio · p. 9

Low-label performance

• At a 20% training ratio on Indian Pines, AGSA-Net reaches an overall accuracy of 0.5994, versus 0.4994 for DSNet and 0.4279 for MambaHSI+.It also exceeds S2Mamba at 0.5263 while remaining below GraphGST at 0.6764.Source: D. Impact of Training Sample Ratio · p. 10

Cross-dataset trends

• Across Augsburg and Berlin, AGSA-Net maintains higher or comparable performance at low training ratios and remains competitive as the ratio increases.At 80% and 100%, performance gaps narrow, but AGSA-Net avoids degradation.Source: D. Impact of Training Sample Ratio · p. 10

Robustness

• AGSA-Net shows smaller accuracy reductions when moving from higher to lower training ratios than DSNet, MambaHSI+, and S2Mamba.The paper interprets this as stronger robustness under limited labeled-data conditions.Source: D. Impact of Training Sample Ratio · pp. 10–11

Ablation

• The full AGSA-Net configuration achieves the strongest overall performance on Indian Pines and Augsburg and remains competitive on Berlin.The ablation integrates unmixing, abundance-guided self-attention, subpixel fusion, and spectral-transformer reasoning.Source: E. Ablation on Component Contribution · p. 11
• Abundance-guided self-attention contributes complementary contextual information, although its OA effect is modest and can be offset by abundance fusion on Berlin.Removing it slightly increases Berlin OA and κ despite lower performance elsewhere.Source: E. Ablation on Component Contribution · p. 11

• The attention analysis compares unguided and guided maps, abundance similarity, and their difference for representative queries in Augsburg and Indian Pines.Query tokens are selected by the largest alignment increase while excluding boundary tokens.Source: F. Visualization of Abundance-Guided… · p. 11, Figure 8 · p. 12
• Guided attention redistributes attention toward regions with similar abundance characteristics, with stronger restructuring on Augsburg than Indian Pines.Indian Pines shows a more moderate but consistent refinement behavior.Source: F. Visualization of Abundance-Guided… · p. 11
• Abundance guidance provides complementary gains to abundance feature fusion rather than replacing it.This interpretation is reported alongside the attention redistribution analysis and the core-component comparison.Source: F. Visualization of Abundance-Guided… · p. 11, Table VIII · p. 12
• The learnable guidance weight λ controls how strongly abundance affinity contributes to attention logits and adapts to each dataset during training.Training analysis tracks λ alongside validation accuracy on Indian Pines, Augsburg, and Berlin.Source: G. Analysis of the Abundance Guidance… · pp. 11–12, Figure 9 · p. 12

Sensitivity Analysis of the Number of Endmembers (K)

• AGSA-Net sets K = U, where U is the number of semantic classes, to obtain a compact latent abundance representation consistent with prior frameworks.The representation is used for structural priors and feature fusion, not explicit material identification.Source: H. Sensitivity Analysis of the Number… · p. 12
• The selected endmember count is not assumed to equal the number of physical materials because classes may contain multiple materials and materials may be shared across classes.This motivates evaluating K as a representation-design choice rather than a physical-material count.Source: H. Sensitivity Analysis of the Number… · p. 12
• The sensitivity analysis evaluates K values from 0.5U to 2U using OA, AA, and Kappa across the three datasets.Table IX reports the best values for each dataset with highlighted entries.Source: H. Sensitivity Analysis of the Number… · p. 12, Table IX · p. 13

Feature Representation Visualization

• AGSA-Net produces the most distinct and well-separated Indian Pines t-SNE clusters, with higher intra-class compactness and reduced overlap than the compared representations.Raw spectral features overlap strongly, Transformer features improve partially, and DSNet produces more compact clusters.Source: §I · pp. 12–13, Figure 10 · p. 13

Conclusion

• The method is reported to achieve competitive performance across multiple benchmark datasets, particularly in complex urban environments.The conclusion highlights physically informed attention for HSI classification.Source: §V · p. 13