PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Hyperspectral Unmixing using Transformer Network

Preetam Ghosh, Swalpa Kumar Roy, Bikram Koirala, Behnood Rasti, Paul Scheunders

arXiv:2203.17076v1cs.CVeess.IV

TL;DR

Hyperspectral unmixing must recover abundance maps and endmember spectra despite CNNs' limited access to global contextual information. The proposed convolutional-autoencoder–transformer model captures long-range dependencies among image patches and achieves superior results across three real datasets.

  • Problem

    CNN-based hyperspectral unmixing can lose contextual information because convolution is limited to local features and the latent representation has fewer bands than the original data.

  • Method

    The model combines a convolutional autoencoder with transformer encoders using Multihead Self-Patch Attention to exchange information among patch tokens and capture long-range context.

  • Results

    The proposed model consistently outperforms competing methods across Samson, Apex, and Washington DC Mall, including mean RMSE 0.0783 and mean SAD 0.0608 on Samson.

  • Takeaways & Limitations

    Capturing long-range feature dependencies enables superior hyperspectral unmixing results across datasets with differing endmember challenges.

Abstract

from arXiv · show

Currently, this paper is under review in IEEE. Transformers have intrigued the vision research community with their state-of-the-art performance in natural language processing. With their superior performance, transformers have found their way in the field of hyperspectral image classification and achieved promising results. In this article, we harness the power of transformers to conquer the task of hyperspectral unmixing and propose a novel deep unmixing model with transformers. We aim to utilize the ability of transformers to better capture the global feature dependencies in order to enhance the quality of the endmember spectra and the abundance maps. The proposed model is a combination of a convolutional autoencoder and a transformer. The hyperspectral data is encoded by the convolutional encoder. The transformer captures long-range dependencies between the representations derived from the encoder. The data are reconstructed using a convolutional decoder. We applied the proposed unmixing model to three widely used unmixing datasets, i.e., Samson, Apex, and Washington DC mall and compared it with the state-of-the-art in terms of root mean squared error and spectral angle distance. The source code for the proposed model will be made publicly available at \url{https://github.com/preetam22n/DeepTrans-HSU}.

I. INTRODUCTION

Hyperspectral unmixing estimates endmembers and abundances from mixed pixels, but highly mixed data and limited CNN context motivate a transformer-enhanced deep model.

  • Hyperspectral pixels may contain several pure materials because spatial resolution is limited.
  • Unmixing estimates endmembers and fractional abundances using only the observed hyperspectral image, often requiring spectral libraries or pure-pixel extraction.
  • Linear unmixing commonly imposes abundance non-negativity and sum-to-one constraints, with FCLSU satisfying both.
  • Blind unmixing jointly estimates endmembers and abundances as a nonconvex optimization problem.
  • Highly mixed pixels can yield unsatisfactory endmembers and poor abundance maps, while sparse unmixing depends on a rich spectral library.
  • The proposed method combines a convolutional autoencoder with a transformer and introduces Multihead Self-Patch Attention to capture long-range patch dependencies.

A. Problem formulation

The formulation models hyperspectral observations and physical abundance constraints, then uses a convolutional autoencoder and transformer to reconstruct abundance maps and endmember spectra.

  • A. Problem formulation: The linear mixing model represents observed spectra using endmembers, abundances, and additive noise under non-negativity and sum-to-one constraints.
  • A. Problem formulation: The vector 1_n denotes an n-component column vector of ones in the sum-to-one constraint.
  • A. Problem formulation: Because unmixing reconstructs abundance maps from hyperspectral inputs, autoencoders provide a suitable reconstruction framework.
  • A. Problem formulation: The model complements the autoencoder with a transformer to improve abundance maps and extracted endmember spectra.
  • B. Hyperspectral feature extraction using AE: The CNN encoder reduces spectral bands and extracts discriminative high-level features for the subsequent transformer.
  • B. Hyperspectral feature extraction using AE: The encoder uses three convolutional layers with batch normalization, dropout in the first layer, and Leaky ReLU in the first two layers.
  • B. Hyperspectral feature extraction using AE: The encoder table summarizes layers using B for input spectral bands and C for output bands.
  • B. Hyperspectral feature extraction using AE: The encoder transforms the input HSI into a lower-dimensional feature cube, with layer functions, weights, biases, and transposition defined explicitly.

C. Patch and Position Embeddings

The encoded hyperspectral feature cube is partitioned into patch embeddings, augmented with class and positional tokens, and processed by a transformer encoder.

  • The autoencoder output is grouped into spatial patches and reshaped into a matrix of patch embeddings.
  • Each patch embedding has dimension p · p · C, while N′ denotes the total number of patches.
  • A learnable class token is added so the transformer can capture long-range semantic information from patch tokens.
  • Learned positional tokens are added to retain patch positional information and are randomly initialized.
  • The transformer encoder uses Multihead Self-Patch Attention to exchange information among patch tokens and pass long-range context to the class token.

D. Transformer Encoder with Multihead Self-Patch Attention

The transformer encoder applies Multihead Self-Patch Attention to exchange information among patch tokens and capture long-range contextual dependencies. Its output is normalized, processed through an MLP with a residual connection, and passed onward using two sequential encoders.

  • Transformer attention: Multihead Self-Patch Attention exchanges information among patch tokens to capture long-range contextual dependencies.The resulting contextual information is fed into the class token while preserving the patch structure.
  • Transformer attention: The attention block computes class-token queries and patch-wide keys and values before applying pairwise similarity and softmax weighting.The query is derived from the class token, while keys and values are computed from the entire patch matrix.
  • Multihead attention: Multiple attention heads reshape q, k, and v into head-specific matrices to enhance relationships among different patches.The number of heads is denoted by hn, and each head uses dimensionality D/hn.
  • Attention output: The multihead output is reshaped, projected through a linear layer, and added to the original class token.This produces the updated class token while retaining the residual connection.
  • Encoder block: The updated class token is concatenated with normalized patch tokens, then passed through an MLP and residual connection to form the encoder output.Two transformer encoders are applied sequentially in the proposed model.
  • Implementation: The paper provides pseudocode describing the transformer encoder and its Multihead Self-Patch Attention operations.The algorithm includes the attention projections, softmax weighting, multihead attention, and class-token update.

E. Unmixing with decoder

The decoder converts the transformer-derived class token into abundance maps and reconstructs the hyperspectral image to estimate endmember spectra. Upsampling and convolutional processing produce the abundance cube before a single convolutional decoder restores the spectral bands.

  • Decoder input: Only the transformer's class token is forwarded to the upsampling block for unmixing.The class token is the first row of the transformer output and has size 1 × D.
  • Abundance estimation: The class token is reshaped and upscaled to spatial dimensions, then convolved and reshaped into an abundance cube of size R×H×W.A 3×3 convolution with stride 1 and padding 1 reduces noise introduced during upsampling.
  • Spectral reconstruction: The abundance matrix is passed through a single convolutional decoder layer that increases its bands from R to B, producing the reconstructed hyperspectral image.The decoder weights are initialized with VCA endmembers and optimized during training to estimate the final endmember spectra.

F. Losses and Optimization functions

Training combines reconstruction error and spectral angle distance losses to preserve essential hyperspectral features while addressing the scale sensitivity of mean squared error. The total objective is a weighted sum of these losses.

  • Loss design: The model is trained with a combination of Reconstruction Error loss and Spectral Angle Distance loss.These two objectives jointly guide optimization of the unmixing model.
  • Reconstruction loss: Mean squared error reconstruction loss encourages the encoder to retain essential input features while discarding nonessential details.The reconstruction objective operates on the difference between the input and reconstructed hyperspectral data.
  • Spectral angle loss: Spectral Angle Distance provides a scale-invariant objective that counteracts the undesirable absolute-magnitude sensitivity of mean squared error in hyperspectral unmixing.Including SAD loss is reported to make the overall model converge much faster.
  • Optimization objective: The total training loss is computed as a weighted sum of the reconstruction and spectral-angle losses using regularization parameters β and γ.The parameters β and γ control the contributions of the loss components.

III. EXPERIMENTAL RESULTS

The experiments use three hyperspectral unmixing datasets—Samson, Apex, and Washington DC Mall—with dataset-specific image sizes, spectral bands, endmembers, and FCLSU-derived ground truth abundances. The accompanying figures show true-color images and endmember spectra.

  • Dataset selection: The experiments evaluate the proposed model on three datasets: Samson, Apex, and Washington DC Mall.The paper introduces the dataset descriptions before presenting experimental results.
  • Dataset visualizations: The Samson figure pairs a true-color image using 571.01, 539.53, and 432.48 nm channels with corresponding endmember spectra.The red, green, and blue channels are specified in that order.
  • Samson: Samson contains 95×95 pixels across 156 bands, with Soil, Tree, and Water as endmembers.Its ground truth endmembers were manually selected, and abundance maps were produced using FCLSU.
  • Dataset visualizations: The Apex and Washington DC Mall figures pair true-color images with endmember spectra using their respective listed red, green, and blue wavelengths.Apex uses 572.2, 532.3, and 426.5 nm, while Washington DC Mall uses 572.7, 530.1, and 425.0 nm.
  • Apex: Apex contains 110×110 pixels across 285 bands, with Water, Tree, Road, and Roof as endmembers.Its ground truth endmembers were manually selected, and abundance maps were produced using FCLSU.
  • Washington DC Mall: Washington DC Mall contains 290×290 pixels across 191 bands and six endmembers: Grass, Tree, Roof, Road, Water, and Trail.Its ground truth endmembers were manually selected, and abundance maps were produced using FCLSU.

B. Experimental Setup

The proposed model is evaluated against six unmixing techniques using abundance RMSE and endmember SAD, with dataset-specific training settings.

  • Comparison methods: Six techniques from geometrical, blind, sparse, and deep unmixing categories are used as comparison methods.The baselines include FCLSU with VCA, NMF-QMV, Collaborative LASSO, uDAS, UnDIP, and CyCUNet.
  • Training settings: Samson training uses 5×5 patches, transformer dimensionality C=24, 200 epochs, and learning-rate decay every 15 epochs.The regularization parameters are β=4 × 10^3 and γ=5 × 10^-3, with initial learning rate 6 × 10^-3.
  • Training settings: Apex and Washington DC Mall use 5×5 and 10×10 patches, respectively, with 200 and 150 training epochs.Their transformer dimensionalities are C=32 and C=24, respectively.
  • Performance measures: Quantitative evaluation uses RMSE between estimated and ground-truth abundance fractions.
  • Performance measures: Endmember estimation is evaluated using SAD in degrees between estimated and ground-truth endmembers.

E. Unmixing Experiments: Quantitative Results

Across Samson, Apex, and Washington DC Mall, the proposed model reports stronger abundance and endmember estimation than competing methods, including on highly similar signatures.

  • Samson dataset: 0.0783 mean RMSE on Samson represents a 48.02% improvement over the next best method.The proposed model also reports a mean SAD of 0.0608, a 35.93% improvement.
  • Apex dataset: 0.1264 mean RMSE and 0.0867 mean SAD are reported on Apex, with best SAD estimates for Road and Water.The Road endmember is described as challenging for other methods but satisfactorily estimated by the proposed model.
  • Washington DC Mall dataset: 0.1661 Grass RMSE and 0.0963 Tree RMSE are reported on Washington DC Mall, where the model separates nearly identical signatures.Overall RMSE and SAD improve over the closest competitor by 43.71% and 52.11%, respectively.
  • Overall observations: The proposed method beats competing methods by a significant margin in overall RMSE, while baseline performance varies across datasets and endmembers.UnDIP and uDAS are consistent across datasets but do not beat the other methods for any given class; CyCU has mixed within-dataset results.
  • Overall observations: Endmember spectral signatures are harder to estimate than abundance maps, yet the proposed model obtains Apex and Washington DC Mall SAD values of 0.0867 and 0.1537.These values are described as about half of the next best method.
  • Metric interpretation: SAD alone does not guarantee accurate abundance maps because it removes endmember spectral norms and ignores scaling factors affecting abundance estimation.

F. Visual Analysis of Abundance Maps and Endmembers

Visual comparisons indicate that the proposed model produces abundance maps closest to ground truth and modifies VCA-initialized endmembers toward the reference spectra.

  • Abundance maps: The proposed method’s abundance maps are visually most similar to the ground-truth maps across the displayed experiments.UnDIP and uDAS fail to properly represent the Water endmember across the experimental datasets described.
  • Abundance maps: Self-patch attention is associated with finding long-distance feature dependencies that are absent from convolutional-network abundance maps.
  • Endmembers: Unlike methods initialized with VCA that make limited subsequent improvements, the proposed method modifies VCA-initialized spectra toward the ground-truth endmembers with lower SAD errors.

G. Sensitivity Analysis to Hyperparameters

Sensitivity analysis identifies practical settings for γ, learning rate, and weight decay. The model is most sensitive to weight decay, whose increase rapidly degrades abundance-map quality.

  • γ sensitivity: γ can be set between 1 × 10−4 and 1 × 10−2, with RMSE and SAD generally changing in similar ways.β was fixed at 5 × 10^3 across datasets.
  • Learning rate and weight decay: Learning rates from 0.001 to 0.009 were tested, with the best results generally occurring between 0.006 and 0.009.Images with lower spatial dimensions preferred a slightly lower learning rate.
  • Learning rate and weight decay: Weight decay around 3 × 10−5 was optimal, while increasing weight decay quickly deteriorated abundance-map quality.The tested weight-decay range was 1 × 10−5 to 9 × 10−5.
  • Parameter selection: The reported results used parameter combinations selected by grid search to minimize the loss function in Eq. (12).
Loading 2203.17076v1…