Source-linked AI summary
Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution
Jing Yao, Danfeng Hong, Jocelyn Chanussot, Deyu Meng, Xiaoxiang Zhu, Zongben Xu
TL;DR
HSI-SR seeks to improve hyperspectral spatial resolution using higher-resolution multispectral data, but unsupervised deep networks face limitations from training requirements and sensor modeling. CUCaNet combines coupled spectral unmixing, cross-attention, adaptive response-function learning, and consistency constraints. Experiments on indoor and airborne datasets report evident quantitative and perceptual improvements over competitive methods.
Problem
Unsupervised HSI-SR remains challenging because supervised methods require cross-sensor image pairs, while existing unsupervised methods use complicated optimization or external datasets.
Method
CUCaNet uses coupled hyperspectral and multispectral autoencoders with cross-attention, learned PSFs and SRFs, and spatial-spectral consistency constraints.
Results
CUCaNet achieves evident quantitative and perceptual performance improvements over competitive HSI-SR methods on indoor and airborne datasets.
Takeaways & Limitations
Jointly combining cross-attention with spatial-spectral consistency yields the best results in the reported ablation study and supports CUCaNet's superiority across evaluated HSI-SR settings.
Takeaways & Limitations
Estimating degradation functions introduces additional freedoms that can limit further performance improvement, and the formulation remains highly ill-posed despite physical constraints.
Abstract
from arXiv · showhide
The recent advancement of deep learning techniques has made great progress on hyperspectral image super-resolution (HSI-SR). Yet the development of unsupervised deep networks remains challenging for this task. To this end, we propose a novel coupled unmixing network with a cross-attention mechanism, CUCaNet for short, to enhance the spatial resolution of HSI by means of higher-spatial-resolution multispectral image (MSI). Inspired by coupled spectral unmixing, a two-stream convolutional autoencoder framework is taken as backbone to jointly decompose MS and HS data into a spectrally meaningful basis and corresponding coefficients. CUCaNet is capable of adaptively learning spectral and spatial response functions from HS-MS correspondences by enforcing reasonable consistency assumptions on the networks. Moreover, a cross-attention module is devised to yield more effective spatial-spectral information transfer in networks. Extensive experiments are conducted on three widely-used HS-MS datasets in comparison with state-of-the-art HSI-SR models, demonstrating the superiority of the CUCaNet in the HSI-SR application. Furthermore, the codes and datasets will be available at: https://github.com/danfenghong/ECCV2020_CUCaNet.
1 Introduction
HSI-SR combines low-spatial-resolution hyperspectral data with higher-spatial-resolution multispectral data, but unsupervised deep learning remains difficult because existing approaches require paired training data, complex optimization, or external sensor datasets. CUCaNet addresses these challenges with coupled unmixing, cross-attention, adaptive sensor-response learning, and spatial-spectral consistency regularization.
- HSI instruments provide rich spectral information but insufficient spatial resolution and mixing effects limit their practical usefulness.
- HSI-SR enhances a low-resolution HSI using a higher-resolution MSI that offers finer spatial information but reduced spectral resolution.
- Optimization-based HSI-SR methods rely on hand-crafted spatial-spectral priors and sensor characteristics such as SRFs and PSFs.
- Supervised deep HSI-SR requires sufficient image pairs from different sensors, while unsupervised methods rely on complicated alternating optimization or external CSR datasets.
- CUCaNet is an unsupervised coupled convolutional autoencoder that models physical mixing to transfer MSI spatial information while preserving HSI spectral resolution.
- Cross-attention transfers spectral or spatial information between HSI and MSI branches, while closed-loop consistency regularization constrains both spatial and spectral domains.
2 Related Work
HSI-SR research spans pansharpening adaptations, optimization-based spectral and statistical models, and increasingly deep learning methods. Unsupervised deep approaches remain comparatively limited, while sensor-related freedoms can constrain performance.
- Pansharpening-based HSI-SR methods may fail to capture global spectral-profile continuity, causing performance degradation.
- Optimization-based HSI-SR methods include Bayesian models and spectral-unmixing approaches that exploit subspace and intimate-mixing assumptions.
- Deep HSI-SR methods include supervised CNN and unfolding approaches, while unsupervised methods have received comparatively less study.
- Estimating two degradation functions introduces additional freedoms that can limit further performance improvement.
3 Coupled Unmixing Nets with Cross-Attention
CUCaNet couples two-stream unmixing of hyperspectral and multispectral data with learned spatial-spectral consistency and cross-attention. The network transfers complementary information between modalities while estimating sensor-related responses from paired observations.
- Coupled unmixing: CUCaNet uses a two-stream convolutional autoencoder to jointly decompose HS and MS data into spectral bases and corresponding coefficients.The fused high-resolution HSI is obtained by decoding MSI encoder maps through the HSI decoder.
- Problem formulation: The HSI-SR objective recovers a latent high-resolution HSI from low-resolution HSI and high-resolution MSI observations.The formulation represents the latent image using endmembers and abundance coefficients under a linear mixing model.
- Network training: The network incorporates physically meaningful abundance constraints, including sum-to-one and non-negativity, with non-negativity enforced through encoder activation.These constraints are combined with reconstruction and sparsity-related training terms.
- Spatial-spectral consistency: CUCaNet estimates spatial and spectral degradation responses without requiring prior PSF or SRF knowledge by imposing consistency between observed and reconstructed modalities.The consistency design uses convolutional layers to simulate spatial and spectral downsampling and supports closed-loop training.
- Cross-attention: Cross-attention transfers spatial and spectral guidance between the MSI and HSI branches through attention maps computed from the two modalities.The refined cross-modal features are concatenated with the original features before the next encoder layer.
4 Experimental Results
Experiments evaluate CUCaNet on three HSI-MSI datasets using complementary image-quality metrics, ablations, quantitative comparisons, and visual assessments. Across indoor and remotely sensed data, the results support the effectiveness of its coupled modules, learned degradation functions, and spectral-spatial reconstruction.
- Dataset and Experimental Setting: Experiments use CAVE, Pavia University, and Chikusei datasets with sensor-specific SRFs, Gaussian-filtered low-resolution HS images, and scale ratios of 16 or 32.CAVE uses 16 selected scenes; Nikon D700 and Landsat-8 SRFs model the multispectral observations.
- Evaluation Metrics: Five complementary metrics—PSNR, SAM, ERGAS, SSIM, and UIQI—quantify spatial, spectral, and overall HSI-SR quality.SAM measures spectral similarity through the average angle between estimated and reference spectra.
- Ablation Study: Ablations show that CUCaNet outperforms CNMF across all metrics, clamp activation improves performance, and jointly using SSC and CA achieves the best results.SSC performs better than CA except in SAM, where CA tends to favor spectral consistency.
- Comparative Experiments: CUCaNet is the first reported unsupervised method to learn SRFs and PSFs simultaneously in an end-to-end fashion.Among compared methods, only HySure and supervised MHFnet also learn both unknown functions, but through different training schemes.
- Indoor Dataset: On CAVE, CUCaNet achieves the best results on four major metrics and maintains high visual fidelity, while MHFnet is slightly better in SSIM.Its SAM values are lower on the two illustrated scenes, consistent with stronger spectral-characteristic capture.
- Remotely Sensed Dataset: On remotely sensed datasets, CUCaNet performs better in most cases, especially in SAM, while residual comparisons show smaller errors and more reliable fused results than competing methods.The paper attributes uSDN’s comparable performance to multi-stage training becoming trapped in local minima and notes stronger MHFnet performance on Chikusei than Pavia University.
5 Conclusion
CUCaNet integrates coupled spectral unmixing with deep learning for unsupervised HSI-SR and introduces learned sensor responses, cross-attention, and consistency enforcement. Experiments on indoor and airborne datasets show quantitative and perceptual improvements over competing methods, while theoretical explanation remains future work.
- CUCaNet integrates coupled spectral unmixing and deep learning for HSI-SR.
- The network learns unknown SRFs and PSFs across MS-HS sensors and uses cross-attention with consistency enforcement.
- Experiments on indoor and airborne HS datasets show evident quantitative and perceptual improvements over competitive methods.
- The paper identifies theoretical explanation of the network's effectiveness as future work.