PaperlayerView exact revision on arXiv

Source-linked AI summary

Endmember-Guided Unmixing Network (EGU-Net): A General Deep Learning Framework for Self-Supervised Hyperspectral Unmixing

Danfeng Hong, Lianru Gao, Jing Yao, Naoto Yokoya, Jocelyn Chanussot, Uta Heiden, Bing Zhang

arXiv:2105.10194v1eess.IVcs.CV

TL;DR

Hyperspectral unmixing methods struggle to model spectral variability while producing physically meaningful endmembers and accurate abundances. EGU-Net addresses this with a self-supervised, endmember-guided two-stream architecture, and experiments report stronger unmixing performance than competing methods on real hyperspectral datasets.

  • Problem

    Existing autoencoder-like deep unmixing methods can reconstruct data but often produce physically meaningless endmembers, impairing abundance estimation and increasing sensitivity to spectral variability.

  • Method

    EGU-Net jointly trains an endmember network and an unmixing-reconstruction network, transferring endmember information through partial parameter sharing and embedding non-negativity and sum-to-one constraints.

  • Results

    EGU-Net outperforms competing methods on abundance estimation or endmember extraction, improving aRMSE by nearly 0.005 and 2.5% over CNNAEU on two real hyperspectral datasets.

  • Takeaways & Limitations

    The framework supports both pixel-wise spectral unmixing and spatial-spectral unmixing, with EGU-Net-ss exceeding the pixel-wise EGU-Net-pw variant.

Abstract

from arXiv · show

Over the past decades, enormous efforts have been made to improve the performance of linear or nonlinear mixing models for hyperspectral unmixing, yet their ability to simultaneously generalize various spectral variabilities and extract physically meaningful endmembers still remains limited due to the poor ability in data fitting and reconstruction and the sensitivity to various spectral variabilities. Inspired by the powerful learning ability of deep learning, we attempt to develop a general deep learning approach for hyperspectral unmixing, by fully considering the properties of endmembers extracted from the hyperspectral imagery, called endmember-guided unmixing network (EGU-Net). Beyond the alone autoencoder-like architecture, EGU-Net is a two-stream Siamese deep network, which learns an additional network from the pure or nearly-pure endmembers to correct the weights of another unmixing network by sharing network parameters and adding spectrally meaningful constraints (e.g., non-negativity and sum-to-one) towards a more accurate and interpretable unmixing solution. Furthermore, the resulting general framework is not only limited to pixel-wise spectral unmixing but also applicable to spatial information modeling with convolutional operators for spatial-spectral unmixing. Experimental results conducted on three different datasets with the ground-truth of abundance maps corresponding to each material demonstrate the effectiveness and superiority of the EGU-Net over state-of-the-art unmixing algorithms. The codes will be available from the website: https://github.com/danfenghong/IEEE_TNNLS_EGU-Net.

I. INTRODUCTION

Hyperspectral unmixing must estimate material components and abundances from mixed pixels despite spectral variability that limits linear and conventional deep-learning approaches. EGU-Net addresses this with endmember guidance, self-supervised two-stream learning, physically meaningful constraints, and pixel-wise or spatial-spectral modeling.

  • Motivation: Hyperspectral imagery contains many mixed pixels because its low spatial resolution degrades spectral discrimination.
  • Motivation: Spectral unmixing estimates endmembers and their corresponding fractional abundances, but spectral variability makes accurate unmixing difficult.Variability includes illumination and topography changes, environmental or sensor noise, atmospheric effects, and nonlinear material mixing.
  • Motivation: Autoencoder-like deep unmixing methods can reconstruct data but may produce physically meaningless endmembers and abundance estimates sensitive to spectral variability.
  • Proposed framework: EGU-Net uses a two-stream self-supervised architecture that learns endmember properties and transfers them into an unmixing network through parameter sharing.The framework embeds non-negativity and sum-to-one constraints through ReLU and softmax layers.
  • Proposed framework: The framework provides pixel-level and spatial-spectral versions, using DNN- and CNN-dominated architectures respectively.
  • Evaluation: The paper introduces a hyperspectral scene over Munich with functional ground truth to enable quantitative evaluation of unmixing algorithms.

II. RELATED WORK

Prior unmixing work spans linear, nonlinear, and deep-learning approaches, but existing methods remain constrained by assumptions about spectral variability or limited physical interpretability. Deep-learning methods improve representation and reconstruction, yet autoencoder-based approaches can produce physically meaningless endmembers.

  • Linear and nonlinear models: Dictionary-adjusted, perturbed, and extended linear mixing models represent spectral variability through dictionary mismatch, additive perturbations, or scaling factors.Perturbed models assume Gaussian-distributed variation, while scaling factors are coherent with endmembers and are not well explained by additive errors alone.
  • Linear and nonlinear models: Bilinear and generalized bilinear models address nonlinear material interactions, but bilinear approaches have limited generalization when pixels contain interactions among more than two materials.
  • Deep-learning methods: Deep-learning unmixing methods use neural networks for blind hyperspectral unmixing, including autoencoders with nonlinear unmixing and linear reconstruction.
  • Deep-learning methods: Autoencoder-based methods can reconstruct data finely but tend to generate physically meaningless endmembers without effective endmember guidance.

III. ENDMEMBER-GUIDED UNMIXING NETWORK

EGU-Net is a self-supervised, two-stream framework that learns endmember properties from pseudo-pure spectra and transfers them to an unmixing-reconstruction network. Spectral bundles and shared parameters help incorporate spectral variability and endmember information into abundance estimation.

  • Framework: EGU-Net combines an endmember network with an encoder-decoder unmixing-reconstruction network in a self-supervised, end-to-end architecture.The endmember network maps pseudo-pure endmembers to approximate one-hot abundances, while the unmixing-reconstruction network processes hyperspectral data.
  • Framework: Parameter sharing transfers hierarchical endmember representations into UR-Net, while feedback from global pixels guides E-Net learning.
  • Endmember Network: EGU-Net trains E-Net using relatively pure endmembers extracted from the HSI and approximate one-hot abundances supplied by existing unmixing models.This self-supervised strategy requires no manual labeling.
  • Endmember Network: Spectral bundles are built by blockwise VCA extraction, HySime-based endmember-number estimation, and clustering to remove duplicates and aggregate endmembers.The bundles include material variation and noise, supporting spectral-variability modeling during E-Net training.
  • Endmember Network: The E-Net represents extracted endmembers through successive nonlinear encoder layers with learned weights and biases.

zBN

The zBN block applies batch normalization and dropout-related processing before nonlinear activation. Dropout is restricted to the first block to reduce the influence of outliers and spectral variability.

  • zBN: Batch normalization produces z-score representations using learnable scale and shift parameters γ and β.
  • zBN: A dropout layer can remove possible outliers and spectral variability before activation, but EGU-Net activates dropout only in the first block.
  • zBN: The final two blocks impose abundance constraints using ReLU for non-negativity and softmax for the sum-to-one constraint.

aReLU

The UR-Net uses nonlinear reconstruction and shared parameters with E-Net to improve modeling of spectral variability and transfer endmember information into unmixing. Its reconstruction loss contributes to the overall EGU-Net objective.

  • Unmixing-Reconstruction Network: UR-Net contains unmixing and reconstruction modules within an autoencoder-like architecture.
  • Unmixing-Reconstruction Network: Nonlinear reconstruction is preferred because linearized operations cannot finely reconstruct signatures affected by complex spectral variability.
  • Unmixing-Reconstruction Network: Partial parameter sharing hierarchically transfers endmember information from E-Net into UR-Net during joint end-to-end training.
  • Optimization and output: The UR-Net reconstruction loss is optimized using mapping functions for unmixing and reconstruction, after which abundances are directly inferred from learned parameters.
  • Spatial-spectral extension: EGU-Net extends from pixel-wise to spatial-spectral unmixing by replacing fully connected encoders with convolutional operations and using the whole HSI as input.

C. Spatial-Spectral Unmixing with CNNs: An Extension

EGU-Net-ss extends the pixel-wise EGU-Net with convolutional operators to model spatial context during hyperspectral unmixing, including global processing of the full image.

  • C. Spatial-Spectral Unmixing with CNNs: An Extension: EGU-Net-ss uses 1 × 1 convolution kernels instead of a fully connected encoder and shares the last two convolutional blocks with UR-Net.This preserves the basic EGU-Net architecture while incorporating spatial processing.
  • C. Spatial-Spectral Unmixing with CNNs: An Extension: The spatial-spectral model reconstructs spectral signatures pixel by pixel while also modeling neighboring pixels through receptive fields of different sizes.
  • C. Spatial-Spectral Unmixing with CNNs: An Extension: EGU-Net-ss can feed the whole hyperspectral image into UR-Net, producing a global unmixing that gives equivalent attention to spatial information.

D. Clarifying Details on Our Network Architecture

This section details EGU-Net’s layer design, evaluates it on synthetic, AVIRIS, and EnMAP data, and describes a processing chain for generating functional ground truth.

  • D. Clarifying Details on Our Network Architecture: EGU-Net begins with encoder, batch-normalization, dropout, and activation layers, using dropout to remove some outliers associated with spectral variability.
  • D. Clarifying Details on Our Network Architecture: ReLU enforces the abundance non-negativity constraint, while softmax imposes the abundance sum-to-one constraint.
  • D. Clarifying Details on Our Network Architecture: The experiments use synthetic, AVIRIS Jasper Ridge, and EnMAP Munich hyperspectral datasets with differing image sizes, spectral bands, and investigated materials.The synthetic data include scaling-factor and non-Gaussian spectral variabilities; the real and simulated scenes contain four and five main endmembers, respectively.
  • D. Clarifying Details on Our Network Architecture: The ground-truth processing chain downsamples high-resolution imagery, produces classification maps, computes low-resolution abundances, and selects potential endmembers from single-material pixels.

B. Experimental Setup

The experimental setup specifies training procedures, dataset-processing context, and metrics for evaluating abundance and endmember estimates.

  • B. Experimental Setup: The network uses Adam optimization with a poly learning-rate policy whose base learning rate is 0.1 and power is 0.99.
  • B. Experimental Setup: EGU-Net uses the number of extracted endmembers as its minibatch size because UR-Net has more samples than E-Net.
  • 2) Evaluation metrics:: The abundance overall root mean square error, aRMSE, aggregates pixel-wise abundance errors for quantitative evaluation.
  • 2) Evaluation metrics:: Spectral angle distance, SAD, measures similarity between estimated and real endmembers, while average SAD summarizes the individual SAD values.
  • B. Experimental Setup: Table II compares linear and nonlinear reconstruction, endmember-based E-Net addition, and fully connected versus convolutional operations in a synthetic-data ablation.
  • B. Experimental Setup: The nonlinear UR-Net reconstruction indirectly extracts endmembers, which can alternatively be estimated by solving a linear system when abundances are given.The linear model is used for visualization and endmember-quality evaluation, not as the network’s reconstruction mechanism.

C. Ablation Study on the Synthetic Data

The synthetic-data ablation shows that nonlinear reconstruction and endmember guidance improve EGU-Net’s unmixing performance over the tested baselines.

  • C. Ablation Study on the Synthetic Data: Nonlinear UR-Net-pw and UR-Net-ss outperform their corresponding linear-reconstruction baselines DAEU and CNNAEU on the synthetic dataset.
  • C. Ablation Study on the Synthetic Data: Adding the self-supervised E-Net improves aRMSE by approximately 0.0040 over UR-Net-pw despite using only a limited number of pseudo-endmembers.
  • C. Ablation Study on the Synthetic Data: EGU-Net-pw and EGU-Net-ss improve aRMSE by at least 0.01 compared with the baselines.
  • C. Ablation Study on the Synthetic Data: The state-of-the-art comparison evaluates FCLSU, PCLSU, SUnSAL, SULoRA, ALMM, and deep-learning models using abundance and endmember measures.
  • C. Ablation Study on the Synthetic Data: EGU-Net-pw and EGU-Net-ss generally achieve the best estimates across indices, with EGU-Net-ss more often producing optimal results.

E. Results and Analysis on the Real Data

On the Jasper Ridge and EnMAP Munich datasets, EGU-Net outperforms traditional and competing deep-learning unmixing methods in abundance estimation and endmember extraction, with spatial-spectral modeling providing further gains.

  • Real-data evaluation: The real-data evaluation uses Jasper Ridge and Munich datasets, reporting aRMSE, aSAD, and per-endmember SAD with mean values in Tables IV and V.The comparison includes the same methods used in the broader evaluation.
  • Traditional methods: Traditional LMM-based methods remain sensitive to spectral variability: relaxing the ASC improves PCLSU over FCLSU, while SULoRA and ALMM improve over SUnSAL by modeling variability.PCLSU improves by more than 0.01 and 9% aRMSE on the two datasets, respectively; SULoRA and ALMM also yield reported gains.
  • Deep-learning methods: Deep-learning unmixing networks generally outperform traditional LMM-based methods on aRMSE, SAD, and aSAD across both real datasets.The advantage is reported as more pronounced on the real data.
  • EGU-Net performance: EGU-Net improves nearly 0.005 and 2.5% in aRMSE over CNNAEU on the two real datasets, respectively, while surpassing other competitors in abundance estimation or endmember extraction.These gains are reported for the two real hyperspectral images.
  • Spatial-spectral modeling: EGU-Net-ss observably exceeds EGU-Net-pw, supporting convolutional spatial-spectral modeling over the pixel-wise strategy in these blind unmixing tasks.EGU-Net-pw uses a pixel-wise strategy, whereas EGU-Net-ss emphasizes simultaneous spatial-spectral modeling.

F. Visual Evaluation

Visual evaluations show that EGU-Net produces abundance maps close to ground truth and extracts endmembers with strong shape agreement, although complex urban scenes leave larger endmember gaps.

  • Visual evaluation: The visual evaluation compares abundance-map and endmember estimates across the unmixing methods.Abundance maps are shown for Jasper Ridge and Munich, while endmembers are compared between EGU-Net-ss and ground truth.
  • Abundance Visualization: On Jasper Ridge, EGU-Net produces abundance maps basically identical to ground truth, while EGU-Net-ss yields more robust and smooth maps.Traditional methods show larger errors, especially for Water; some other networks confuse Road with Water.
  • Abundance Visualization: On the more complex Munich urban scene, EGU-Net still visualizes nearly identical material shapes compared with ground truth despite noisier and more easily confused abundance maps.The passage describes the urban scene as presenting greater challenges.
  • Endmember Comparison: EGU-Net-ss estimates Jasper Ridge endmembers relatively accurately, whereas Munich retains a gap from ground truth but matches shapes well for Asphalt and Vegetation.The Munich gap is associated with a complex urban scene possibly involving many unknown spectral variabilities.
  • Endmember-guided framework: The framework combines an endmember-learning stream with an autoencoder-based unmixing stream, using parameter transfer and feedback to guide more reasonable unmixing.The proposed EGU-Net includes pixel-wise and spatial-spectral variants.
Loading 2105.10194v1…