Source-linked AI summary
Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution
Ying Qu, Hairong Qi, Chiman Kwan
TL;DR
HSI-SR seeks HR hyperspectral images by fusing LR HSI with HR MSI, but available deep learning solutions require supervised HR-HSI data. The paper proposes uSDN, an unsupervised shared-decoder architecture with sparse Dirichlet representations and angular matching, and reports superiority over state-of-the-art methods on two benchmark datasets.
Problem
HSI-SR must recover high spatial and spectral resolution from complementary LR HSI and HR MSI without relying on unavailable HR-HSI ground truth.
Method
uSDN uses two encoder-decoder networks with shared decoder weights, sparse Dirichlet representations, and minimized angular differences between modalities.
Results
uSDN consistently outperforms competing methods on RMSE and SAM, with especially improved SAM performance on the more challenging CAVE dataset.
Takeaways & Limitations
The unsupervised architecture preserves spectral information while extracting spatial information from multimodal inputs for HSI-SR.
Takeaways & Limitations
The formulation assumes representation vectors are non-negative, sum to one, and are sparse because each pixel contains only a few spectral bases.
Abstract
from arXiv · showhide
In many computer vision applications, obtaining images of high resolution in both the spatial and spectral domains are equally important. However, due to hardware limitations, one can only expect to acquire images of high resolution in either the spatial or spectral domains. This paper focuses on hyperspectral image super-resolution (HSI-SR), where a hyperspectral image (HSI) with low spatial resolution (LR) but high spectral resolution is fused with a multispectral image (MSI) with high spatial resolution (HR) but low spectral resolution to obtain HR HSI. Existing deep learning-based solutions are all supervised that would need a large training set and the availability of HR HSI, which is unrealistic. Here, we make the first attempt to solving the HSI-SR problem using an unsupervised encoder-decoder architecture that carries the following uniquenesses. First, it is composed of two encoder-decoder networks, coupled through a shared decoder, in order to preserve the rich spectral information from the HSI network. Second, the network encourages the representations from both modalities to follow a sparse Dirichlet distribution which naturally incorporates the two physical constraints of HSI and MSI. Third, the angular difference between representations are minimized in order to reduce the spectral distortion. We refer to the proposed architecture as unsupervised Sparse Dirichlet-Net, or uSDN. Extensive experimental results demonstrate the superior performance of uSDN as compared to the state-of-the-art.
1. Introduction
HSI-SR fuses low-spatial-resolution HSI with high-spatial-resolution MSI to recover images with both high spectral and spatial resolution. uSDN addresses this fusion problem unsupervisedly through shared-decoder networks, sparse Dirichlet representations, and angular-similarity constraints.
- HSI provides hundreds of contiguous spectral bands but has lower spatial resolution than MSI or RGB because of hardware limitations.
- HSI-SR fuses HSI with MSI or conventional color images to generate images with high spectral and spatial resolution.
- Existing deep learning approaches are supervised and require training data and HR HSI, while sensor- and band-specific mappings may differ across images.
- uSDN uses two networks with shared decoder weights to extract spectral and spatial information from LR HSI and HR MSI.
- uSDN encourages sparse Dirichlet representations and minimizes their angular difference to incorporate physical constraints and reduce spectral distortion.
2. Problem Formulation
HSI-SR estimates an HR HSI from paired LR HSI and HR MSI by sharing spectral-basis information across modalities. The formulation imposes non-negativity, sum-to-one, sparsity, and low spectral distortion requirements while operating without HR-HSI ground truth.
- The goal is to estimate HR HSI X from LR HSI Yh and corresponding HR MSI Ym, whose spatial and spectral resolutions are complementary.
- Each HSI pixel is modeled as a linear combination of spectral bases Φh, with coefficients Sh preserving spatial structure.
- MSI uses spectral bases Φm and spatial representations Sm, linked to HSI bases through the sensor-provided transformation matrix R.
- Because HR-HSI ground truth is unavailable, the problem is solved unsupervisedly by exploiting the shared HSI spectral basis Φh across modalities.
- Representation vectors must be non-negative, sum to one, and sparse because pixels contain only a few spectral bases; spectral distortion must also be reduced.
3. Proposed Approach
The proposed unsupervised architecture uses coupled HSI and MSI networks with a shared decoder, sparse Dirichlet representations, and angular similarity regularization. These design choices extract spectral information from LR HSI and spatial information from HR MSI while addressing spectral distortion.
- Network Architecture: Two encoder-decoder networks learn representations from LR HSI and HR MSI while sharing decoder weights.The shared decoder supports coupled reconstruction of both modalities and preserves spectral information from HSI.
- Network Architecture: Decoder weights correspond to HSI spectral bases, while HSI representation variables preserve spatial structure.Under identity activations and no decoder bias, the decoder weights equal the spectral basis in the linear mixture model.
- Network Architecture: The MSI encoder extracts HR spatial information, and the desired HR HSI is generated by combining it with the HSI spectral basis.The MSI decoder shares HSI decoder weights, yielding the reconstruction X = SmΦh.
- Sparse Dirichlet-Net: Dirichlet-distributed representations impose the sum-to-one constraint, while stick-breaking produces positive representation variables.The representations correspond to proportional coefficients for HSI and MSI data.
- Sparse Dirichlet-Net: A generalized Shannon entropy function reinforces sparsity even when representations satisfy the sum-to-one constraint.Unlike standard Shannon entropy, the proposed function decreases monotonically as data become sparse and has minima at quadrant boundaries.
- Angle Similarity: Angular similarity between HSI and MSI representations is minimized to reduce spectral distortion without assuming a known down-sampling function.Spectral angle mapper measures this difference, with lower scores indicating more similar representations.
4. Experiments and Results
uSDN is evaluated on CAVE and Harvard using RMSE and SAM against state-of-the-art methods, with experiments covering quantitative, visual, ablation, convergence, and representation analyses. Across these evaluations, uSDN consistently preserves spatial and spectral information while outperforming competing approaches in both reconstruction error and spectral distortion.
- Experimental setup: The evaluation uses CAVE and Harvard benchmark datasets, comparing uSDN with seven state-of-the-art methods using RMSE and SAM.RMSE measures reconstruction error, while SAM measures spectral distortion.
- Quantitative results: uSDN outperforms the other approaches on both RMSE and SAM across the benchmarked experiments.Traditional CS and MRA methods suffer spectral distortion, while different Bayesian and matrix-factorization methods have dataset- or metric-specific weaknesses.
- Ablation study: Reconstruction loss combined with angle similarity produces the lowest SAM and fastest convergence, while reconstruction loss stabilizes the optimization process.The ablation study compares unconstrained, sparse, Dirichlet, and sparse Dirichlet architectures and several loss combinations.
- Convergence and representation analysis: Both networks converge smoothly, and the learned representations are sparse while the estimated bases from uSDN more effectively cover the data than those from CSU and BSR.The MSI network shows slight fluctuation because angular difference is minimized every 10 iterations; RMSE decreases as λ increases from 2 × 10^-7 to 1 × 10^-6.
5. Conclusion
uSDN addresses HSI-SR without supervision by combining spectral information from LR HSI and spatial information from HR MSI through a shared decoder. Its sparse Dirichlet representations and angular-difference objective support strong performance on benchmark datasets.
- uSDN performs HSI-SR in an unsupervised fashion, extracting spectral bases from LR HSI and spatial representations from HR MSI through a shared decoder.
- Sparse Dirichlet regularization encourages representations from both modalities to incorporate their physical constraints.
- Minimizing angular differences between the two representations reduces spectral distortion during optimization.
- Experiments on two benchmark datasets demonstrate uSDN's superiority over state-of-the-art approaches.