Source-linked AI summary
Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net
Qi Xie, Minghao Zhou, Qian Zhao, Deyu Meng, Wangmeng Zuo, Zongben Xu
TL;DR
Video-rate systems provide either high-resolution multispectral or low-resolution hyperspectral images, motivating reconstruction of a high-resolution hyperspectral image. The paper builds an observation-model- and low-rankness-aware fusion model, unfolds its proximal-gradient solver into MS/HS Fusion Net, and reports visual and quantitative superiority on simulated and real data.
Problem
Video-rate optical systems generally cannot simultaneously provide high spatial and high spectral resolution, creating the MS/HS fusion problem of reconstructing HrHS from HrMS and LrHS images.
Method
The method combines HrMS and LrHS observation models with spectral low-rankness, then unfolds a proximal-gradient algorithm into an end-to-end trainable MS/HS Fusion Net.
Results
Experiments on simulated and real MS/HS fusion cases substantiate MHF-net’s visual and quantitative superiority over state-of-the-art methods.
Takeaways & Limitations
MHF-net is specifically interpretable to MS/HS fusion and can discover spatial and spectral response operators end-to-end from training data.
Takeaways & Limitations
The low-rank assumption is imperfect for noisy HrMS observations, so the network adds a final ResNet to adjust the resulting gap.
Abstract
from arXiv · showhide
Hyperspectral imaging can help better understand the characteristics of different materials, compared with traditional image systems. However, only high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images can generally be captured at video rate in practice. In this paper, we propose a model-based deep learning approach for merging an HrMS and LrHS images to generate a high-resolution hyperspectral (HrHS) image. In specific, we construct a novel MS/HS fusion model which takes the observation models of low-resolution images and the low-rankness knowledge along the spectral mode of HrHS image into consideration. Then we design an iterative algorithm to solve the model by exploiting the proximal gradient method. And then, by unfolding the designed algorithm, we construct a deep network, called MS/HS Fusion Net, with learning the proximal operators and model parameters by convolutional neural networks. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively as compared with state-of-the-art methods along this line of research.
1. Introduction
Hyperspectral images provide rich spectral information, but video-rate optical systems trade spatial resolution against spectral resolution. The paper formulates and unfolds a task-specific MS/HS fusion method that uses observation models and spectral low-rankness to reconstruct HrHS images.
- Hyperspectral images contain many spectral bands that improve scene characterization and performance in recognition, classification, tracking, and segmentation.
- Video-rate optical systems generally capture either high-spatial-resolution multispectral data or high-spectral-resolution data with reduced spatial resolution.
- MS/HS fusion seeks to generate a high-resolution hyperspectral image from high-resolution multispectral and low-resolution hyperspectral observations.
- The HrMS and LrHS observations are modeled as Y = XR + Ny and Z = CX + Nz, where R describes spectral response and C describes spatial degradation.The target X is the HrHS image; C is often composed of convolution and down-sampling.
- Because HrHS recovery is ill-posed, prior-based methods use spatial sparsity, spatial smoothness, or spectral low-rankness, but fixed assumptions may not adapt to diverse real scenes.
- The proposed method integrates observation models and image-prior learning, constructs a low-rank-aware fusion model, unfolds its proximal-gradient algorithm into MHFnet, and learns parameters end-to-end.The model represents X using Y and estimated bases, while the network learns the response operators rather than estimating them beforehand.
- Experiments on synthetic and real problems substantiate the proposed method’s quantitative and visual superiority over state-of-the-art methods.
2. Related work
Related work includes pansharpening, machine-learning fusion, and deep-learning approaches adapted to MS/HS fusion. Existing approaches can perform well, but pansharpening risks spectral distortion and generic deep networks lack task-specific interpretability.
- Pansharpening-related approaches fuse multispectral and panchromatic imagery, and can be heuristically adapted to MS/HS fusion by treating HrMS bands as panchromatic images.
- Component-substitution and multiresolution-analysis pansharpening methods suffer high spectral distortion because one panchromatic image contains limited spectral information.
- Machine-learning MS/HS methods use sparse coding or sparse matrix factorization to learn spatial or spectral representations for reconstructing HrHS images.
- Deep-learning methods adapted from pansharpening employ structures such as 3D CNNs, but typically use off-the-shelf components rather than models designed for MS/HS fusion.
- Existing deep-learning methods neglect the MS/HS observation model and intrinsic hyperspectral prior structures such as spectral correlation, limiting interpretability for this task.
3. MS/HS fusion model
The model reformulates MS/HS fusion using observation-model equivalences and represents the HrHS image with known and estimated bases. A proximal-gradient solver then updates the estimated bases, whose operations are later mapped into a tensor-form network.
- 3.1. Model formulation: Theorem 1 establishes equivalent conditions linking the observation model to representing X with known bases and an estimated basis matrix.The theorem assumes rank(X)=r>s and rank(˜Y)=s, with coefficient matrices A and B and estimated bases ˆY.
- 3.1. Model formulation: The HrMS image is usually full rank along its spectral mode because its band number is small, such as three bands for RGB.This assumption lets ˜Y = Y − N_y satisfy the theorem’s rank conditions.
- 3.1. Model formulation: The fusion model incorporates the observation models and regularizes the estimated bases ˆY rather than X to preserve spatial details in the known HrMS image.The regularization uses a trade-off parameter λ and function f(·); the paper notes that some regularizers can remove sharp edges, lines, and highlights.
- 3.2. Model optimization: A proximal-gradient algorithm iteratively updates ˆY by minimizing a quadratic approximation with a stepsize parameter η.The update uses Q(ˆY, ˆY^(k)) and proceeds across iterations k=1,2,…,K.
- 3.2. Model optimization: The algorithm is later unfolded into a deep network that performs the operations in tensor form while preserving the original image structure.Figure 2 illustrates the relationship between the matrix-form algorithm and tensor-form network structure.
4. MS/HS fusion net
MS/HS Fusion Net unfolds an iterative optimization algorithm into K network stages, combining tensor-form observation operations with learned proximal operators and end-to-end training.
- Network architecture: Each stage takes HrMS image Y, LrHS image Z, and the previous estimated ˆY as inputs, then outputs an updated ˆY for the next stage.The first stage initializes ˆY(1) = 0, while subsequent stages use the preceding estimate.
- Tensor implementation: Tensor-form operations preserve spatial, spectral, and channel structure while approximating the model through mode-3 multiplications and convolutional operators.The implementation uses tensor inputs instead of matrix forms; mode-3 multiplication realizes spectral-channel transformations, while downsampling and transposed convolution approximate C and C^T.
- Network architecture: The network implements K iterations through K stages, with separate first, intermediate, and final-stage designs.Setting ˆY(k) = 0 makes stage Sk equivalent to S1.
- Learned operators: A ResNet learns the proximal operator in each stage, with its parameters denoted θ_p^(k).The proximal operator is embedded in the unfolded stage rather than fixed analytically.
- Final reconstruction: The final stage approximately generates the HrHS image from X^(K) and adds a ResNet adjustment to address the gap caused by noisy HrMS data and imperfect low-rankness.The final output is denoted ˆX, and the adjustment is applied to X^(K).
- Network training: Training uses final-output, per-stage, and observation-model residual losses, with paired data when HrHS ground truth exists and Wald-protocol data creation otherwise.The network is trained with Adam for 50000 iterations, batch size 10, and learning rate 0.0001.
5. Experimental results
Experiments on simulated CAVE and Chikusei data, plus real WorldView-2 imagery, evaluate MHF-net against competing methods using quantitative and visual comparisons. MHF-net consistently achieves stronger reconstruction quality, while its interpretable stage structure improves performance over a comparable ResNet.
- Model verification with CAVE data: MHF-net with more stages outperforms settings with fewer stages, even when the total number of network levels is similar.The comparison adjusts ResNet levels to keep total network depth comparable across settings.
- Model verification with CAVE data: MHF-net achieves about 5db higher PSNR than ResNet, with the stage structure identified as the main architectural difference.The authors attribute the improvement to the problem-specific stage structure that introduces interpretability for MS/HS fusion.
- Experiments with simulated data: On CAVE data, MHF-net significantly outperforms competing methods across all five evaluation measures and better recovers fine textures and coarse structures.The comparison includes traditional methods and an implemented ResNet baseline; visual results use the 10th band at 490nm.
- Experiments with simulated data: On Chikusei data, MHF-net significantly outperforms all competing methods on every evaluation measure and produces composite images closest to the ground truth.The visual comparison uses bands 70-100-36 as R-G-B and includes a four-times zoomed region.
- Experiments with real data: On real WorldView-2 imagery without HrHS ground truth, MHF-net gives clearer results whose color and brightness are closer to the LrHS image than ResNet.The data contain an HrMS image with three bands and an LrHS image with eight bands.
6. Conclusion
The paper concludes that MHF-net combines end-to-end learning with the MS/HS data-generation mechanism through an observation-model-based formulation and an optimization-inspired network. Experiments on simulated and real cases substantiate its superiority over state-of-the-art methods.
- Conclusion: MHF-net learns parameters from training data while incorporating the underlying generation mechanism of MS/HS fusion data.The model is built from observation models and its optimization algorithm is unfolded into a deep network.
- Conclusion: The optimization-inspired architecture is specifically interpretable to MS/HS fusion and can discover spatial and spectral response operators end to end.The conclusion identifies this task-specific interpretability as a central property of the network.
- Conclusion: Experiments on simulated and real MS/HS fusion cases substantiate MHF-net’s superiority over state-of-the-art methods.The conclusion reports both simulated and real-data evidence without specifying individual metrics.