Source-linked AI summary
Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network
Frosti Palsson, Johannes R. Sveinsson, Magnus O. Ulfarsson
TL;DR
The paper tackles fusion of high-spatial-resolution MS and low-spatial-resolution HS images to obtain a high-resolution HS image. It uses a 3D-CNN with PCA-based dimensionality reduction and reports better quality than MAP-based comparisons, including when HS data contain additive noise. The method also shows reduced sensitivity to decimation-filter choice, although it requires more computation.
Problem
MS/HS fusion seeks to combine high spatial resolution with the rich spectral information of HS images, but existing approaches provide a comparison point for a new deep-learning method.
Method
A supervised 3D-CNN learns fusion filters from spatially decimated training data, while PCA reduces HS dimensionality before fusion.
Results
The proposed method significantly outperforms MAP1 and MAP2 according to ERGAS, SAM, and SSIM, including with additive Gaussian noise, while its noise tolerance is similar or slightly below MAP2.
Takeaways & Limitations
The method provides good fusion results and noise tolerance, automatically learns the decimation filter, and produces images free of halos and ringing artifacts.
Abstract
from arXiv · showhide
In this paper, we propose a method using a three dimensional convolutional neural network (3-D-CNN) to fuse together multispectral (MS) and hyperspectral (HS) images to obtain a high resolution hyperspectral image. Dimensionality reduction of the hyperspectral image is performed prior to fusion in order to significantly reduce the computational time and make the method more robust to noise. Experiments are performed on a data set simulated using a real hyperspectral image. The results obtained show that the proposed approach is very promising when compared to conventional methods. This is especially true when the hyperspectral image is corrupted by additive noise.
I. INTRODUCTION
The paper addresses multispectral/hyperspectral fusion using deep learning, motivated by the rich spectral information in HS images and the need to combine complementary spatial and spectral data. It proposes a 3D-CNN method with PCA-based dimensionality reduction and reports stronger quality results than conventional methods, including under noise.
- Motivation: MS/HS fusion combines high-spatial-resolution MS data with low-spatial-resolution HS data containing hundreds of material-discriminating spectral bands.HS spectral signatures support applications such as land-cover classification.
- Related work: Existing MS/HS fusion approaches include sparse coding, spectral unmixing, coupled matrix factorization, 3D-wavelet transforms, and pansharpening-style sub-problems.Related methods also include component-substitution, multiresolution, model-based, and statistical-inference approaches.
- Proposed direction: The paper proposes fusing MS and HS images with a supervised 3D-CNN that learns filters from spatially decimated training data.Decimation creates a lower-resolution training relationship that is assumed to apply at the higher-resolution scale.
- Proposed direction: PCA reduces HS dimensionality before fusion, improving computational efficiency while assuming lower- and higher-resolution HS spectral singular vectors are identical.The approach is reported to outperform conventional methods according to three quantitative quality metrics.
- Reported advantages: The 3D-CNN automatically learns the decimation filter and produces images without halos or ringing artifacts often observed with conventional methods.This makes the method relatively insensitive to the decimation filter used to prepare training samples.
II. CONVOLUTIONAL NEURAL NETWORKS
The paper introduces CNN concepts relevant to its 3D-CNN, including local receptive fields, shared filters, feature maps, and pooling. The trained network processes full-resolution input to produce spatial loadings for HS-image reconstruction.
- CNN structure: CNN neurons receive input from local rectangular receptive fields, which tile the input image as the field slides across it.Locally connected neurons share weights and bias, allowing the same feature to be detected at different locations.
- CNN structure: A convolutional layer produces feature maps using shared filters, while pooling can subsample maps and reduce network parameters.Pooling may use functions such as maximum value selection.
- CNN advantages: CNN weight sharing and local connectivity require fewer learned parameters than fully connected networks, supporting deeper networks that learn faster without sacrificing performance.The paper attributes this benefit to shift invariance.
- 3D-CNN operation: A 3D-CNN uses 3D filters and receptive fields to compute feature-map outputs from the input through 3D convolution, shared biases, filters, and nonlinear activation.The notation identifies x as the input and σ as the nonlinear activation function.
- Estimation pipeline: During estimation, the trained CNN receives the entire full-resolution input and outputs high-resolution spatial loadings used with an inverse PCA transform to reconstruct the HS image.This full-image processing avoids breaking the estimation input into patches.
III. PROPOSED METHOD
This section presents the proposed method and then discusses the selected 3D-CNN architecture.
- The proposed-method section first describes the fusion approach and then discusses the chosen 3D-CNN architecture.
- The method description and network-architecture discussion are treated as separate parts of the section.
- The section establishes the 3D-CNN architecture as part of the proposed fusion method.
A. General Outline of the Method
The method reduces the HS image dimensionality with PCA, trains a 3D-CNN on decimated MS/HS data, and reconstructs a high-resolution HS estimate from sharpened spatial loadings.
- A. General Outline of the Method: Supervised training uses spatially decimated input data because the target high-resolution HS image is unavailable.The approach assumes the learned relationship at lower resolution also applies at higher resolution.
- A. General Outline of the Method: PCA reduces the HS image to its first r spatial loadings before fusion, using the spectral singular vectors of the observed HS image.The method assumes these spectral singular vectors are identical at low and high spatial resolutions.
- A. General Outline of the Method: The MS image and interpolated reduced HS loadings are stacked into an m×n×(r+P) input and divided into matching 7×7 training patches.The trained network can later process the entire input at once.
- A. General Outline of the Method: The trained 3D-CNN outputs estimated high-resolution spatial loadings, which replace the corresponding interpolated loadings during HS reconstruction.For noisy HS data, reconstruction can instead retain only the first r principal components.
B. CNN Architecture
The network uses 3D convolutions to learn spectral-spatial features while preserving image dimensions through zero-padding and reducing memory overhead through PCA.
- B. CNN Architecture: A 3D-CNN is used because HS images have two spatial dimensions and one spectral dimension, enabling spectral-spatial feature learning.The architecture applies three-dimensional filters and receptive fields.
- B. CNN Architecture: Zero-padding preserves input dimensions through convolutional layers and avoids boundary artifacts.For filter size i×j×k, padding is (i−1)/2, (j−1)/2, and (k−1)/2 at the respective dimension ends.
- B. CNN Architecture: The network has three convolutional layers with 32, 64, and r filters, using 3×3×3 filters in the first two layers and 1×1×1 in the output layer.The first two layers use ReLU activation and the output layer uses linear activation.
- B. CNN Architecture: PCA dimensionality reduction reduces memory overhead because the flexible-input CNN can estimate an entire large image at once.The full-image operation can otherwise be very memory consuming.
A. Simulated Data
Experiments use a simulated MS/HS pair derived from the ROSIS Pavia center hyperspectral dataset, with Fig. 3 evaluating ERGAS across principal-component counts.
- A. Simulated Data: The experiments use the ROSIS Pavia center hyperspectral dataset, originally sized 512×512 pixels with 102 spectral bands.A blank strip reduces the used row dimension to 480 pixels.
- A. Simulated Data: The simulated MS image has 512×480 pixels and 4 bands, while the spatially decimated HS image has 128×120 pixels and 102 bands.The MS image is generated using IKONOS R, G, B, and NIR spectral response profiles; the HS image is decimated by factor 4 with bicubic filtering.
- A. Simulated Data: The original HS image serves as the reference for quantitative quality evaluation.This reference is paired with the simulated lower-resolution HS and MS inputs.
B. Results
Experiments compare the 3D-CNN fusion method with MAP1 and MAP2 across dimensionality reduction, noise, SNR, and decimation-filter settings. The proposed method generally achieves the strongest quality results and is less sensitive to decimation-filter choice, although MAP2 shows similar or slightly better noise tolerance.
- Experimental setup: 8192 matched 7 × 7-pixel patches train the 3D-CNN using simulated MS and HS spatial loadings as inputs and the first r HS spatial loadings as targets.The network minimizes mean squared error for 50 epochs using ADAM, with batch size 5 and Gaussian noise regularization variance 0.5.
- Compared methods: MAP1 and MAP2 are maximum a posteriori wavelet-coefficient methods, with MAP2 additionally applying PCA dimensionality reduction.The proposed method is compared against both baselines.
- Dimensionality reduction: 10 sharpened principal components produce optimal ERGAS results for both the proposed method and MAP2.The tested alternatives were 2, 6, 10, 15, 20, 25, and 30 principal components.
- Quality evaluation: Without added noise, the proposed method significantly outperforms MAP1 and MAP2 on ERGAS, SAM, and SSIM, while visual inspection also favors the proposed method.The comparison uses additive zero-mean Gaussian noise at SNR=20dB for the noisy condition; the proposed method requires more computation time.
- Noise robustness: With additive Gaussian noise, the proposed method again performs significantly better on the quality metrics, but its noise tolerance is similar or slightly lower than MAP2's.MAP1 performs significantly worse in the presence of noise, and the proposed method's training time could be reduced by up to an order of magnitude 2 using a GPU.
- Noise robustness: Across SNR values from 10 to 30 dB in 5 dB increments, the proposed method performs best, whereas MAP1 performs significantly worse.Figure 4 reports means and standard deviations from six trials at each SNR value.
- Decimation-filter sensitivity: Bicubic decimation gives the best results for all methods, while nearest-neighbor decimation degrades MAP1 and MAP2 more than the proposed method.Bilinear decimation also degrades ERGAS and SAM, but affects the proposed method and MAP2 less than MAP1.
- Visual evaluation: Figure 5 compares interpolated, reference, MAP1, MAP2, and proposed images using a subset of HS band 102.The figure provides the visual comparison supporting the reported image-quality assessment.
V. CONCLUSIONS
The paper concludes that PCA-assisted 3D-CNN fusion provides good MS/HS fusion results on a simulated dataset and remains tolerant to HS-image noise. PCA reduces computational cost and can improve results in noisy conditions.
- Conclusion: The proposed method fuses MS and HS images using a 3D-CNN with PCA dimensionality reduction before fusion.The method is compared with two MAP-estimation methods.
- Conclusion: PCA reduces computational cost significantly without affecting fused-image quality and can improve results when noise is present.These conclusions are reported from experiments on a dataset simulated using a real hyperspectral image.
- Conclusion: The experiments demonstrate good results and tolerance to noise in the HS image.The evidence is limited to the simulated dataset used in the study.