Source-linked AI summary
Coupled Convolutional Neural Network with Adaptive Response Function Learning for Unsupervised Hyperspectral Super-Resolution
Ke Zheng, Lianru Gao, Wenzhi Liao, Danfeng Hong, Bing Zhang, Ximin Cui, Jocelyn Chanussot
TL;DR
HSI-MSI fusion seeks high spatial and spectral resolution, but existing methods often require unavailable high-resolution HSI or known PSF and SRF information. HyCoNet uses unsupervised coupled autoencoders with adaptively learned degradation functions and produces stable fusion results for arbitrary PSFs and SRFs.
Problem
HSI has limited spatial resolution, while fusion methods may require unavailable high-resolution HSI or prior PSF and SRF information.
Method
HyCoNet couples three autoencoders under linear unmixing and learns PSF and SRF parameters adaptively through a joint-loss, end-to-end training framework.
Results
The experiments show stable and robust fusion results for arbitrary PSFs and SRFs without prior PSF or SRF information.
Takeaways & Limitations
HyCoNet provides an unsupervised fusion strategy that requires only the spectral coverage of the MSI and HSI.
Abstract
from arXiv · showhide
Due to the limitations of hyperspectral imaging systems, hyperspectral imagery (HSI) often suffers from poor spatial resolution, thus hampering many applications of the imagery. Hyperspectral super-resolution refers to fusing HSI and MSI to generate an image with both high spatial and high spectral resolutions. Recently, several new methods have been proposed to solve this fusion problem, and most of these methods assume that the prior information of the Point Spread Function (PSF) and Spectral Response Function (SRF) are known. However, in practice, this information is often limited or unavailable. In this work, an unsupervised deep learning-based fusion method - HyCoNet - that can solve the problems in HSI-MSI fusion without the prior PSF and SRF information is proposed. HyCoNet consists of three coupled autoencoder nets in which the HSI and MSI are unmixed into endmembers and abundances based on the linear unmixing model. Two special convolutional layers are designed to act as a bridge that coordinates with the three autoencoder nets, and the PSF and SRF parameters are learned adaptively in the two convolution layers during the training process. Furthermore, driven by the joint loss function, the proposed method is straightforward and easily implemented in an end-to-end training manner. The experiments performed in the study demonstrate that the proposed method performs well and produces robust results for different datasets and arbitrary PSFs and SRFs.
I. INTRODUCTION
HSI has high spectral resolution but typically poor spatial resolution, motivating fusion with higher-resolution MSI. HyCoNet addresses the practical challenge of unknown PSF and SRF information through an unsupervised coupled-network strategy.
- Motivation: HSI's hardware trade-off between spectral and spatial resolution usually produces coarse spatial resolution that restricts applications.Fusion with higher-resolution MSI is presented as a natural solution.
- Related work: Existing HSI-MSI fusion methods include pansharpening, model-based optimization, matrix factorization, and deep-learning approaches.The literature spans wavelet and component-substitution methods, Bayesian and variational formulations, unmixing, and CNNs.
- Research gap: Many deep-learning fusion methods are supervised and require high-resolution HSI unavailable in practice, while some fusion methods assume known PSF or SRF information.The paper targets both the lack of training ground truth and the difficulty of obtaining degradation-function priors.
- Contribution: HyCoNet is an unsupervised network designed to handle arbitrary PSFs and SRFs, using only the spectral coverage of the MSI and HSI.The method is evaluated against nine state-of-the-art methods on four remote sensing datasets.
- Contribution: The method couples three autoencoders under linear unmixing, sharing endmember parameters and using a learned PSF layer to connect low- and high-resolution abundances.A joint loss supports unsupervised, end-to-end training.
III. PROPOSED METHOD
HyCoNet estimates the high-resolution abundance matrix and spectral endmembers by coupling autoencoders that exploit complementary spatial and spectral information from HrMSI and LrHSI.
- III. PROPOSED METHOD: HyCoNet uses three coupled autoencoders to estimate high-resolution abundances and spectral endmembers from HrMSI and LrHSI.HrMSI provides spatial contextual information correlated with abundances, while LrHSI preserves spectral information consistent with endmembers.
A. Coupled Autoencoder Network for Image Fusion
HyCoNet couples three autoencoders to reconstruct and fuse low-resolution HSI, high-resolution MSI, and estimated target HSI. Shared endmember parameters and learned SRF and PSF operations connect spectral and spatial representations.
- HyCoNet contains LrHSI, HrMSI, and LrMSI autoencoders for jointly modeling the input and target image relationships.The estimated target HrHSI is embedded in the HrMSI autoencoder.
- The LrHSI autoencoder learns abundances and endmembers while reconstructing the low-resolution HSI through an encoder-decoder structure.Its latent variables and decoder parameters reconstruct the input through matrix multiplication optimized by gradient descent.
- A shared 1×1 convolution layer represents the endmembers and is used by both the LrHSI and HrMSI autoencoders.The layer has no bias and preserves the shared spectral basis across the two autoencoders.
- The SRF module uses a convolution layer followed by normalization to learn spectral resampling from the estimated target HSI to reconstructed MSI.The convolution weights cover the known spectral ranges of the MSI bands, while normalization represents the denominator of the resampling operation.
- The PSF convolution layer bridges low- and high-resolution abundances by modeling local spatial contributions and enforcing spectral consistency in the reconstructed target.Its kernel size matches the ratio of the low- and high-resolution GSDs, and its stride equals the kernel size.
- The LrMSI autoencoder represents the equivalence between spatial degradation of HrMSI and spectral degradation of LrHSI.This relationship produces an estimated low-resolution MSI alongside the other reconstruction paths.
B. Joint Loss Function
HyCoNet trains its coupled autoencoders with a joint loss that balances reconstruction errors while enforcing abundance constraints. The implementation also uses sum-to-one, non-negativity, and sparsity regularization.
- Abundance estimates are constrained to satisfy the sum-to-one property across the channel dimension.The authors report that their proposed constraint converges more accurately than a softmax implementation.
- Clamping constrains abundance and reconstructed-image outputs to [0,1] and keeps shared convolution weights non-negative after updates.The clamp is applied to relevant outputs and weights during training.
- A KL-divergence sparsity term encourages most abundance elements to remain near zero.The sparsity parameter is set to 0.0001 in the network.
- The resulting optimization problem combines reconstruction objectives with abundance constraints and sparsity regularization.These components jointly define the training objective for the fusion network.
- The joint loss combines reconstruction errors with trade-off parameters that balance their contributions during optimization.The loss can be directly provided to the optimizer for end-to-end training.
IV. EXPERIMENTS
The experiments evaluate HyCoNet through simulation studies covering parameter sensitivity, abundance constraints, endmember counts, and visual and quantitative fusion quality.
- Simulation experiments assess the sensitivity of trade-off parameters α, β, γ, µ, and ν.The study also investigates how abundance constraints affect the method.
- The experiments examine the effectiveness of different numbers of endmembers and explore the character of the estimated abundances.
- Fused images from different methods are evaluated using both visual and quantitative measures.
A. Experimental Dataset
HyCoNet is evaluated on four widely used HSI datasets: Pavia University, Indian Pines, Washington DC, and University of Houston. The datasets provide varied airborne hyperspectral imagery for the experiments.
- The evaluation uses Pavia University, Indian Pines, Washington DC, and University of Houston hyperspectral datasets.
- Pavia University imagery was acquired by the ROSIS-3 airborne sensor and contains 610 × 340 pixels with 1.3 m GSD and 115 spectral bands before band removal.The passage states that 12 bands were removed because of noise and water-vapor absorption.
B. Implementation Details
The implementation evaluates HyCoNet through simulated downsampling experiments, parameter studies, abundance constraints, endmember counts, and learned-kernel analysis. Training uses Adam with a scheduled learning rate, while quality is assessed with multiple fusion metrics.
- Experimental setup: Gaussian filters model spatial downsampling, with filter dimensions set by the high- to low-resolution GSD ratio.The standard deviation is generally 0.5, with other values tested for robustness.
- Experimental setup: The experiments investigate trade-off parameters, abundance constraints, endmember counts, and learned PSF kernels across four simulated datasets.The datasets are Pavia University, Indian Pines, Washington DC, and University of Houston.
- Experimental setup: Simulation experiments downsample original HrHSI data to create LrHSI and evaluate estimated HrHSI against the known target.The study uses spatial and spectral downsampling on original HrHSI, which serves as the truth target.
- Evaluation: Fusion quality is evaluated using RMSE, PSNR, SAM, and additional image and hyperspectral-cube quality measures.These measures assess reconstruction and spectral fidelity of the fusion results.
- Training: The model is trained with Adam, an initial learning rate of 5×10^-3, linear decay, and 10000 epochs.The experiment uses one input image, so one epoch equals one iteration, with batch size one.
C. Parameters Discussion
The parameter study examines reconstruction, sum-to-one, and sparsity loss weights, as well as network components and endmember counts. Results identify settings that improve or stabilize fusion quality and reveal the importance of the reconstructed-HSI branch.
- Loss parameters: α, β, γ, µ, and ν balance reconstruction, sum-to-one, and sparsity losses in the joint objective.α, β, and γ weight different reconstruction errors, while µ and ν weight abundance constraints.
- Loss parameters: The best Pavia University result occurs at α = β = 10, with performance more sensitive to β than α.β controls reconstructed HrMSI weighting and affects spatial quality; α weights the reconstructed LrHSI-related error.
- Loss parameters: γ has only a slight effect on fusion quality, but reconstruction is higher and more stable at γ = 100.The parameter γ controls the reconstruction weight for LrMSI images.
- Loss parameters: Fusion performance is sensitive to µ but not to ν when the number of endmembers is 100.µ weights the sum-to-one loss, whereas ν weights the sparsity loss.
- Ablation study: Removing the reconstructed-HSI branch causes a sharp performance drop, whereas removing other branches has a smaller effect.The ablation study evaluates the impact of deleting network components or losses.
D. Constraint Function for the Abundance
The abundance constraints compare clamp and softmax functions while enforcing nonnegativity and sum-to-one behavior. Clamp produces better convergence and reconstruction accuracy in the reported experiments.
- Constraint functions: The loss restricts the sum-to-one property, while a clamp function enforces nonnegative abundances.The clamp function is applied before the abundance output.
- Constraint functions: Softmax also enforces nonnegative, sum-to-one abundances, but it produces lower reconstruction accuracy than clamp.The comparison is reported for the fusion accuracy experiment.
- Convergence behavior: Softmax convergence is nonuniform and can become arbitrarily slow, reducing gradients for points with smaller values.Clamp updates gradients faster within the range [0, 1].
E. Number of Endmembers
The number of endmembers affects fusion quality across four datasets and also determines abundance feature size and shared-convolution kernel size. Increasing this number generally improves reconstruction, while learned PSF kernels remain similar to the originals.
- Number of Endmembers: Increasing the endmember number p generally improves reconstruction accuracy, as measured by PSNR and SAM across four datasets.Each p value is evaluated three times, and the reported curves use average PSNR and SAM.
- Number of Endmembers: The endmember number p controls both abundance feature size and the kernel size of the shared convolution layer.Larger p allows the model to be more representative and may exceed the actual number of pure spectral bases.
- Learned PSF kernels: For Pavia University and Washington DC, estimated PSF kernels are similar to the original Gaussian kernels.The tested Gaussian standard deviations are 0.5, 1, and 2, with kernel sizes determined by GSD ratios of 4 and 8.
- Fusion evaluation: The fusion visualizations report color composites, RMSE errors for color-composite images, and MARE and SAM errors for HSI cubes.These layouts are shown for Pavia University, Indian Pines, and Washington DC.
- Abundance behavior: Estimated abundances are sparse, while some edge regions of one abundance estimate do not completely satisfy the sum-to-one constraint.Other abundance estimates are better constrained because they are jointly constrained by the LrHSI and HrMSI autoencoders.
H. Comparison with the State of the Art
The comparison evaluates unsupervised fusion methods under limited prior information about the SRF and unknown PSF, using visualizations, heatmaps, and bandwise PSNR across multiple datasets. HyCoNet is reported to produce the best visual results while remaining stable across datasets and PSF deviations.
- Evaluation setting: The comparison uses unsupervised fusion methods because supervised methods require unavailable high-resolution HSI training samples.The evaluation assumes MSI spectral coverage is known, while PSF and SRF prior information is unavailable to the proposed model.
- Quantitative Performance: Figure 14 compares bandwise PSNR across Pavia University, Indian Pines, Washington DC, and University of Houston.The plots represent reconstructed spatial quality for each HSI band.
- Quantitative Performance: HyCoNet significantly outperforms the tested methods in bandwise PSNR, and its results are not greatly affected by dataset choice.HySure, ICCV15, and GSOMP perform well on some datasets, whereas most compared methods perform poorly on Washington DC.
- Visual Performance: The visualizations encode color-composite images, color-image RMSE, HSI-cube MRAE, and per-pixel SAM errors.The four columns respectively show fused RGB appearance, spatial error, reconstructed-cube error, and spectral consistency.
- Visual Performance: HyCoNet achieves the best visual results, although object-edge errors remain unavoidable because low-resolution-image mixing complicates heterogeneous regions.The comparison also reports block or locally irregular error patterns for several competing methods.
2) Quantitative Performance:
The quantitative evaluation measures spectral consistency, spatial quality, and global error across datasets and PSF deviations. HyCoNet remains stable under these variations, while other methods are more dataset-sensitive, and it significantly outperforms them in bandwise PSNR.
- Quantitative Performance: mSAM measures mean spectral consistency, mPSNR measures mean spatial quality, and ERGAS measures global dimensionless fusion error.These metrics summarize spectral fidelity, spatial reconstruction quality, and overall error.
- Quantitative Performance: HyCoNet produces stable results across different datasets and PSF deviations, whereas competing methods are especially unstable on Washington DC.The Washington DC imagery is described as complex, making local relationships between low- and high-resolution data difficult to model.
- Quantitative Performance: CNMF, FUSE, GSA, HySure, and ICCV15 show better spectral consistency than the other compared methods.This comparison concerns spectral consistency rather than the overall stability pattern reported for HyCoNet.
- Quantitative Performance: HyCoNet significantly outperforms the tested methods in PSNR across HSI bands, with performance relatively unaffected by dataset choice.The comparison covers Pavia University, Indian Pines, Washington DC, and University of Houston at σ = 0.5.
- Method and conclusion: HyCoNet learns PSF and SRF adaptively through coupled autoencoder-based fusion, enabling unsupervised end-to-end training without prior PSF and SRF information.The PSF is learned from high- and low-resolution abundance correlation, while the SRF is learned through autoencoder reconstruction.
- Method and conclusion: The experiments indicate stable and robust fusion results for arbitrary PSFs and SRFs.The method is presented as a straightforward end-to-end training strategy.