PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Inspired Autoencoder for Unsupervised Hyperspectral Image Super-Resolution

Jianjun Liu, Zebin Wu, Liang Xiao, Xiao-Jun Wu

arXiv:2110.11591v1eess.IVcs.CV

TL;DR

HSI super-resolution fuses LR-HSI and HR-MSI to recover HR-HSI, while existing deep methods are mostly supervised and model-based methods rely on hand-crafted priors. MIAE integrates target-image NMF into an unsupervised, pixel-wise autoencoder. Experiments report close visual agreement with HR-RGB on real data and little performance degradation from blind estimation on the stated datasets.

  • Problem

    HSI super-resolution must fuse LR-HSI with HR-MSI, but deep learning approaches are mostly supervised and model-based approaches rely on hand-crafted priors.

    Source: §I · pp. 1–2

  • Method

    MIAE integrates target HR-HSI NMF into an implicit autoencoder and uses a pixel-wise fusion model to estimate hidden outputs and form the encoder.

    Source: §I · p. 2, B. Pixel-Wise Fusion Model Inspired… · p. 3

  • Results

    On real data, MIAE’s result is visually much closer to the HR-RGB image; blind estimation causes very small performance degradation on PaviaU and KSC and not significant degradation on DC.

    Source: 1) Blind and Nonblind · p. 5, D. Experiment Results on Real Data · p. 9

  • Takeaways & Limitations

    The pixel-wise architecture is described as a manifold prior-based model that can be trained patch by patch rather than on the entire image.

    Source: §I · p. 2, §IV · p. 9

Abstract

from arXiv · show

This paper focuses on hyperspectral image (HSI) super-resolution that aims to fuse a low-spatial-resolution HSI and a high-spatial-resolution multispectral image to form a high-spatial-resolution HSI (HR-HSI). Existing deep learning-based approaches are mostly supervised that rely on a large number of labeled training samples, which is unrealistic. The commonly used model-based approaches are unsupervised and flexible but rely on hand-craft priors. Inspired by the specific properties of model, we make the first attempt to design a model inspired deep network for HSI super-resolution in an unsupervised manner. This approach consists of an implicit autoencoder network built on the target HR-HSI that treats each pixel as an individual sample. The nonnegative matrix factorization (NMF) of the target HR-HSI is integrated into the autoencoder network, where the two NMF parts, spectral and spatial matrices, are treated as decoder parameters and hidden outputs respectively. In the encoding stage, we present a pixel-wise fusion model to estimate hidden outputs directly, and then reformulate and unfold the model's algorithm to form the encoder network. With the specific architecture, the proposed network is similar to a manifold prior-based model, and can be trained patch by patch rather than the entire image. Moreover, we propose an additional unsupervised network to estimate the point spread function and spectral response function. Experimental results conducted on both synthetic and real datasets demonstrate the effectiveness of the proposed approach.

I. INTRODUCTION

HSI super-resolution fuses a low-spatial-resolution HSI with a high-spatial-resolution MSI to recover a high-spatial-resolution HSI. MIAE addresses the reliance of supervised deep methods on labeled samples and model-based methods on hand-crafted priors through an unsupervised, model-inspired autoencoder.

I. INTRODUCTION

  • HSI super-resolution fuses LR-HSI with HR-MSI to form HR-HSI, addressing the usual sacrifice of HSI spatial resolution under sensor trade-offs.The passage describes limited incident energy as requiring trade-offs among spectral resolution, spatial resolution, and signal-to-noise ratio.Source: §I · p. 1
  • HSI fusion approaches include component substitution, multiresolution analysis, model-based methods, and deep learning, with richer spectral information making fusion more complex than pansharpening.The paper also outlines model-based recovery and decomposition strategies, including subspace, endmember-abundance, dictionary, and tensor approaches.Source: §I · p. 1
  • Deep learning approaches are mostly supervised and require many labeled samples, while model-based methods are unsupervised but rely on hand-crafted priors and empirical parameters.The introduction also notes that many unsupervised deep fusion networks rely heavily on human experience for their construction.Source: §I · p. 2
  • MIAE integrates the target HR-HSI’s NMF into an implicit autoencoder, treating each pixel as a sample and enabling patch-by-patch training.The network is described as a manifold prior-based model.Source: §I · p. 2
  • A pixel-wise fusion model estimates hidden outputs from the two observed images, and its gradient-descent algorithm is unfolded into the encoder network.The loss is built on the spectral and spatial degradation mechanisms, and an additional unsupervised network estimates the PSF and SRF.Source: §I · p. 2

II. PROPOSED APPROACH

The approach integrates a nonnegative factorization of the target HR-HSI into an autoencoder. Its spectral matrix serves as decoder weights, while the spatial matrix supplies hidden outputs.

A. NMF Inspired Autoencoder

  • NMF is applied to the target HR-HSI to capture its intrinsic structure and represent it sparsely, with a rank-J factorization where J is below the smaller matrix dimension.The paper motivates this factorization as facilitating the inference process for super-resolution.Source: A. NMF Inspired Autoencoder · p. 2
  • The autoencoder treats each hyperspectral pixel as an individual sample, with the spectral matrix A as decoder weights and spatial vectors s_i as hidden outputs.The decoder reconstructs each spectrum as the clamped product of A and s_i; the encoder maps input pixels to hidden outputs.Source: A. NMF Inspired Autoencoder · p. 3
  • The network constrains A and s_i to [0,1] and trains by feeding the HR-HSI pixels individually, without imposing the sum-to-one constraint used in unmixing.The paper says it does not aim to perform fusion and unmixing simultaneously.Source: A. NMF Inspired Autoencoder · p. 3

B. Pixel-Wise Fusion Model Inspired Encoder Network

The encoder estimates each pixel’s hidden output directly from the observed HR-MSI and interpolated LR-HSI through a pixel-wise fusion model. The authors unfold its gradient-descent algorithm into a K-stage network.

B. Pixel-Wise Fusion Model Inspired Encoder Network

  • The observations are modeled as spatially and spectrally degraded versions of the unavailable target HR-HSI, with a resolution ratio r greater than 1.The spatial degradation uses blur B and downsampling D; the spectral response is R.Source: B. Pixel-Wise Fusion Model Inspired… · p. 3
  • Because spatial blur and downsampling couple pixels, the authors bilinearly interpolate the LR-HSI and use each pixel’s HR-MSI and interpolated LR-HSI directly.This avoids first estimating the implicit target pixel, which the authors say would lead to error accumulation.Source: B. Pixel-Wise Fusion Model Inspired… · p. 3
  • A pixel-wise fusion model estimates hidden output s_i, and its gradient-descent algorithm is unfolded into the encoder network.The iteration index k runs from 1 to K, and η is the step size.Source: B. Pixel-Wise Fusion Model Inspired… · p. 3
  • The encoder has K stages, using fully connected layers and Leaky ReLUs to implement matrix multiplications and combine modules.For K = 3, Fig. 2 depicts the encoder; the final hidden output is obtained after applying the stages.Source: B. Pixel-Wise Fusion Model Inspired… · p. 3, Figure 2 · p. 4

C. Loss Function and Training Strategy

The loss trains the autoencoder against the observed LR-HSI and HR-MSI because the target HR-HSI is implicit. Overlapping patches can accelerate training while accounting for spatial blur at patch boundaries.

C. Loss Function and Training Strategy

  • The network’s outputs are spatially and spectrally degraded to predict the observed LR-HSI and HR-MSI, and their differences are measured with an L1 loss.The authors choose L1 because it is more robust to outliers than L2.Source: C. Loss Function and Training Strategy · p. 4
  • The loss cannot use the target HR-HSI directly because its pixels are implicit, and spatial degradation couples the reconstructed pixels.Consequently, the observations must be combined into images when computing the loss.Source: C. Loss Function and Training Strategy · p. 4
  • Training can use overlapping patches that cover all pixels, discarding boundary pixels affected by spatial blur when computing the loss.The authors present this patch strategy as a way to accelerate training.Source: C. Loss Function and Training Strategy · p. 4

D. Blind Estimation Network

The paper estimates the spatial point spread function and spectral response function with an additional network, using observed images as inputs and the functions as trainable parameters.

D. Blind Estimation Network

  • The autoencoder requires the spatial PSF B and spectral response function R, which are obtained under physical constraints.The paper formulates their estimation as problem (15).Source: D. Blind Estimation Network · p. 4
  • The blind estimation network takes Z and Y as inputs and treats B and R as trainable parameters.The network is trained using a loss on its output data.Source: D. Blind Estimation Network · p. 4
  • Fig. 3 shows the blind estimation network, whose output data are used in its loss function.The text identifies the network’s output data but does not specify their full expressions in these passages.Source: D. Blind Estimation Network · p. 4

E. Relationship to Model-Based Approaches

The paper relates MIAE to a fusion model through the constraint that maps each pixel’s observed inputs to a low-dimensional hidden output. It describes that constraint as both manifold regularization and a self-supervised deep prior.

E. Relationship to Model-Based Approaches

  • The pixel-wise mapping acts as manifold regularization by embedding the combined inputs into a low-dimensional space.This interpretation connects the encoder constraint to a manifold-prior model.Source: E. Relationship to Model-Based… · p. 4
  • Each hidden output s_i depends only on the HR-MSI and interpolated LR-HSI inputs at the corresponding spatial position.The paper characterizes f as a nonlinear mapping into the low-dimensional space R^J.Source: E. Relationship to Model-Based… · p. 4
  • The same mapping acts as a self-supervised deep prior that uses the input data themselves as training data.The paper presents this as another role of the constraint s_i = f(z_i, y↑_i; θ).Source: E. Relationship to Model-Based… · p. 4

III. EXPERIMENTAL RESULTS AND ANALYSIS

The experiments assess fused-image quality on synthetic datasets using five quality measures.

  • All datasets are scaled to the range [0, 1] before the experiments.Source: §III · p. 4
  • Synthetic fused images are assessed with RMSE, PSNR, SAM, ERGAS, and UIQI.Source: §III · p. 4

A. Synthetic Datasets and Implementation Details

Experiments simulate paired observations from three HSI datasets and tune rank and stage settings using PSNR trends, with large settings prone to overfitting.

A. Synthetic Datasets and Implementation Details

  • PaviaU, KSC, and DC reference images use selected regions of 512 × 256, 512 × 512, and 512 × 256 pixels, respectively.The selected regions come from ROSIS, AVIRIS, and HYDICE imagery, respectively.Source: A. Synthetic Datasets and… · p. 5
  • Synthetic LR-HSI is blurred and downsampled by 8×, while four-band HR-MSI is generated from IKONOS RGB and NIR responses; noise is added to both.The LR-HSI noise is SNR = 30 dB and HR-MSI noise is SNR = 40 dB.Source: A. Synthetic Datasets and… · p. 5
  • Training uses overlapping 40 × 40 patches with stride 24, Adam for 10,000 iterations, and a separate blind-estimation network trained for 5,000 iterations.The blind-estimation network uses a learning rate of 5 × 10^-5.Source: A. Synthetic Datasets and… · p. 5

B. Influence of Parameters

  • PSNR improves as rank J increases, but large J can cause overfitting; K = 3, and J = 80 for PaviaU and KSC and 30 for DC.A moderate K performs better than small K, while a too-large K is prone to overfitting.Source: B. Influence of Parameters · p. 5

C. Experiment Results on Synthetic Datasets

Across the three synthetic datasets, MIAE performs strongly against the compared methods, while blind estimation causes little degradation and interpolation choices usually make little difference.

1) Blind and Nonblind:

  • Blind PSF and SRF estimation causes very small performance degradation on PaviaU and KSC and no significant degradation on DC.The comparison evaluates MIAE with exact versus estimated degradation parameters.Source: 1) Blind and Nonblind · p. 5

2) Influence of LR-HSI Interpolation:

  • Interpolation choice makes no obvious difference in most cases; nearest interpolation is slightly worse on KSC, and bicubic interpolation is slightly worse on DC.The experiment compares bilinear, nearest, bicubic, and cubic spline interpolation.Source: 2) Influence of LR-HSI Interpolation · p. 5, Table II · p. 6

3) Comparison With the State of the Art:

4) Computational Efficiency:

MIAE trains faster than the other two autoencoder-based methods and uses fewer trainable parameters than HyCoNet.

4) Computational Efficiency:

  • MIAE takes less time to train than uSDN and HyCoNet and has much fewer trainable parameters than HyCoNet, while its runtime is comparable to model-based approaches when platform is ignored.The comparison used MATLAB for the first seven methods and PyTorch for the autoencoder-based methods.Source: 4) Computational Efficiency · p. 8

D. Experiment Results on Real Data

On the real UH data, MIAE gives color and brightness results described as good, and its result is much closer to the HR-RGB reference. The conclusion describes an unsupervised autoencoder that integrates target-HR-HSI NMF and estimates hidden outputs through a pixel-wise fusion model.

D. Experiment Results on Real Data

IV. CONCLUSION

  • MIAE integrates target-HR-HSI NMF into an implicit autoencoder, using the spectral and spatial matrices as decoder parameters and hidden outputs, respectively.Source: §IV · p. 9
  • Because the encoder input pixel is unknown, a pixel-wise fusion model estimates hidden vectors from the two observed images, and its gradient-descent algorithm is unfolded into the encoder.The loss is built on the relationship between the target HR-HSI and the two observed images.Source: §IV · p. 9
  • MIAE's pixel-wise architecture is manifold-prior-like, supports patch-by-patch training, and adds an unsupervised network to estimate the PSF and SRF.Source: §IV · p. 9
Loading 2110.11591v1…