PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances and New Guidelines on Hyperspectral and Multispectral Image Fusion

Renwei Dian, Shutao Li, Bin Sun, Anjing Guo

arXiv:2008.03426v1eess.IVcs.CV

TL;DR

HSI offers high spectral resolution but limited spatial resolution, while existing HSI–MSI fusion literature lacks a full-scale review. This paper reviews and classifies fusion methods, compares their reported performance, and identifies alignment challenges affecting large spatial-resolution differences.

  • Problem

    Imaging cameras face a tradeoff between spectral and spatial resolution, and the growing HSI–MSI fusion literature lacks a comprehensive review.

  • Method

    The paper comprehensively reviews HSI–MSI fusion methods and classifies them into pan-sharpening, matrix factorization, tensor representation, and CNN-based families.

  • Results

    Fused HSI improves detection accuracy, while Fuse-S and CNN-Fus achieve leading performance in reported evaluations.

  • Takeaways & Limitations

    The review provides a framework for comparing HSI–MSI fusion approaches and highlights image registration as an important challenge when spatial-resolution differences are large.

  • Takeaways & Limitations

    Existing fusion methods scarcely consider difficult image alignment, particularly under severe spatial distortions.

Abstract

from arXiv · show

Hyperspectral image (HSI) with high spectral resolution often suffers from low spatial resolution owing to the limitations of imaging sensors. Image fusion is an effective and economical way to enhance the spatial resolution of HSI, which combines HSI with higher spatial resolution multispectral image (MSI) of the same scenario. In the past years, many HSI and MSI fusion algorithms are introduced to obtain high-resolution HSI. However, it lacks a full-scale review for the newly proposed HSI and MSI fusion approaches. To tackle this problem,this work gives a comprehensive review and new guidelines for HSI-MSI fusion. According to the characteristics of HSI-MSI fusion methods, they are categorized as four categories, including pan-sharpening based approaches, matrix factorization based approaches, tensor representation based approaches, and deep convolution neural network based approaches. We make a detailed introduction, discussions, and comparison for the fusion methods in each category. Additionally, the existing challenges and possible future directions for the HSI-MSI fusion are presented.

1. Introduction

HSI provides rich spectral information but typically has low spatial resolution because imaging sensors trade spectral and spatial resolution. The paper reviews HSI-MSI fusion methods, organizes them into four families, and discusses their characteristics, limitations, challenges, and research directions.

  • HSI captures many spectral bands, but its spatial resolution is often only dozens of meters, limiting applications.
  • HSI-MSI fusion combines same-scenario HSI and higher-spatial-resolution MSI to produce imagery with high spectral and spatial resolution.
  • Fused HSI can support material recognition and tasks including object classification, anomaly detection, and change detection.
  • Existing reviews covered pan-sharpening and matrix-factorization approaches, but did not consider tensor-based and deep CNN-based fusion methods.
  • This paper comprehensively reviews HSI-MSI fusion and categorizes methods into pan-sharpening, matrix factorization, tensor representation, and deep CNN families.
  • The review introduces method characteristics, advantages, and limitations, and presents challenges and guidelines with particular attention to tensor and deep CNN approaches.

2. Notations

The paper represents high-resolution HSI, low-resolution HSI, and high-resolution MSI as tensors with spatial modes followed by a spectral mode. It also introduces tensor–matrix mode products and factor-matrix notation.

  • High-resolution HSI, low-resolution HSI, and high-resolution MSI are denoted by X ∈ R^(W×H×S), Y ∈ R^(w×h×S), and Z ∈ R^(W×H×s), respectively.
  • The first two tensor modes represent spatial dimensions, while the third mode represents the spectral dimension.
  • The n-mode product combines a tensor with a matrix along the tensor's n-th mode.
  • Factor matrices D_n correspond to tensor modes, and N denotes the resulting tensor in the notation for tensor products.

3. HSI-MSI Fusion Methods

HSI-MSI fusion approaches are organized into four families, with pan-sharpening methods emphasizing efficient spatial-detail injection and MF methods modeling spectral structure through factorization. These families differ in computational cost, spectral preservation, and assumptions about spectral representation.

  • Overview: The review classifies HSI-MSI fusion methods into pan-sharpening, matrix factorization, tensor representation, and deep CNN approaches.Each family is introduced and compared according to its characteristics.
  • Pan-sharpening-based approaches: Pan-sharpening methods use component substitution or multiresolution analysis to inject high-resolution PAN spatial structures into upsampled MSI.CS methods replace transformed spatial information, whereas MRA methods extract spatial structures through wavelets, pyramids, contourlets, or curvelets.
  • Pan-sharpening-based approaches: Pan-sharpening-based HSI-MSI methods are computationally efficient, but large HSI-MSI spatial-resolution differences can produce remarkable distortions.Their HSI adaptation may group HSI bands or synthesize high-resolution images from MSI; synthesized images yielded better fusion than a selected MSI band.
  • Matrix factorization-based approaches: Low-rank MF methods reduce spectral-mode dimensionality and achieve faster fusion than sparse-representation approaches while preserving spectral properties.The review also reports that MF approaches generally outperform pan-sharpening methods, but often require costly iterative optimization.
  • Matrix factorization-based approaches: MF methods estimate spectral bases and coefficients through optimization, using sparse dictionaries or low-dimensional subspaces to model spectral signatures.Sparse methods assume signatures combine a few dictionary atoms, while low-rank methods use a basis with L << S.
  • Tensor and deep CNN-based approaches: Tensor-representation methods report excellent fusion results on simulated data, while deep CNN methods learn nonlinear mappings and can flexibly handle different data types.CNN approaches require additional training data, and their mapping is parameterized by Θ.

4.1. Testing Approaches

The evaluation compares nine representative methods spanning pan-sharpening, matrix factorization, tensor representation, and deep CNN categories.

  • Compared approaches: Nine methods are compared: GSA, GLP-HS, NSSR, CNMF, CSU, Fuse-S, CSTF, LTMR, and CNN-Fus.GSA represents component-substitution pan-sharpening, GLP-HS represents multi-resolution analysis, and the remaining methods cover MF, TR, and deep CNN approaches.
  • Method categories: GSA and GLP-HS are pan-sharpening representatives, while NSSR, CNMF, CSU, and Fuse-S use matrix factorization.NSSR is based on spectral sparse representation; CNMF, CSU, and Fuse-S use low-rank representation.
  • Method categories: CSTF and LTMR are tensor-representation methods, whereas CNN-Fus belongs to the deep CNN category.CSTF uses sparse Tucker decomposition, and LTMR uses low tensor multi-rank representation.

4.2. Quality Metrics

Because the ideal high-resolution HSI is unknown, evaluation simulates reference data and uses multiple quality metrics plus visual inspection. The metrics assess reconstruction quality, spectral distortion, and computational efficiency.

  • Evaluation protocol: The unknown ideal high-resolution HSI motivates simulating high-resolution MSI and low-resolution HSI from an available reference HSI.Fusion is then applied to the simulated pair, allowing quality metrics to compare the fused HSI with the reference.
  • Evaluation protocol: The spatial down-sampling filter should match the HSI sensor PSF, while the spectral filter should match the MSI sensor spectral response function.The spatial procedure commonly applies a zero-mean Gaussian filter before uniform subsampling.
  • Evaluation protocol: Simulated-data performance may mismatch real-fusion performance, so visual inspection is also used to identify spatial and spectral distortions.The visual assessment complements quantitative evaluation of the fused images.
  • Quality metrics: PSNR increases with better fusion, whereas ERGAS and SAM decrease as reconstruction error and spectral distortion decrease.PSNR is averaged across bands; SAM specifically evaluates spectral distortions.
  • Quality metrics: UIQI and SSIM measure image similarity, with larger UIQI values indicating better fusion results and SSIM averaged across spectral bands.UIQI averages patch-level values, while SSIM is computed per band before averaging.
  • Efficiency: Computational efficiency is assessed using running time T and time complexity, with all compared approaches implemented under the same computer configuration.The reported running time is measured in seconds.

4.3. Experimental Dataset

Experiments use simulated fusion inputs derived from the Pavia University and Cuprite Mine hyperspectral datasets, with sensor-inspired spatial, spectral, and noise degradations.

  • Pavia University: Pavia University contains 610 × 340 pixels, 115 bands, and 1.3m spatial resolution, with 93 bands retained after low-SNR removal.The data were acquired by the ROSIS-3 airborne optical sensor over the University of Pavia, Italy.
  • Pavia University: Pavia’s low-resolution HSI is simulated with a 7 × 7 Gaussian blur, standard deviation 3, and subsampling factor 5.The four-band MSI uses an IKONOS-like spectral response, with Gaussian noise added at 35dB for MSI and 32dB for HSI.
  • Cuprite Mine: Cuprite Mine contains 512×614 pixels, 224 bands, and 17m spatial resolution; experiments use the top-left 512×512 pixels and retain 188 bands.The dataset was acquired by AVIRIS over Nevada’s Cuprite mining district in 1995.
  • Cuprite Mine: Cuprite’s low-resolution HSI is generated with the same 7 × 7 Gaussian blur and subsampling factor 4.Its six-band MSI selects wavelengths 480, 560, 660, 830, 1650, and 2220nm; simulated MSI noise is 35dB.

4.4. Quantitative Evaluation

The experiments reveal category-specific trade-offs: pan-sharpening is fastest but less accurate, MF and TR methods perform well with different efficiency profiles, and CNN-Fus performs best on most metrics.

  • Pan-sharpening methods: Pan-sharpening methods GSA and GLP-HS have low computational burden but comparatively poor quality metrics.The authors attribute this to not considering the MSI and HSI observational model, which is crucial for fusion.
  • Matrix factorization methods: MF methods perform well on both datasets; Fuse-S achieves the best MF performance but is very time-consuming.Fuse-S’s computational burden mainly comes from patch-based dictionary learning and sparse coding, while CNMF combines good performance with high efficiency.
  • Tensor representation methods: CSTF and LTMR produce better quality indexes among the TR methods on Cuprite Mine and Pavia University.The discussion relates LTMR’s performance to its use of spatial similarities in the fused HSI.
  • Deep CNN methods: CNN-Fus achieves the best fusion results on most quality metrics for both datasets.Its design combines spectral subspace representation with CNN-based coefficient estimation; the former maintains spectral characteristics and the latter preserves spatial structures.

4.5. Visual Evaluation

The visual evaluation compares false-color and spectral error images to assess spatial structures and spectral distortions in fused HSIs. Across Cuprite Mine and Pavia University, several methods show visibly lower spectral distortion than GSA and GLP-HS.

  • Figure 2 uses false-color images from bands 30, 18, and 5 alongside spectral error images for Cuprite Mine.Spectral error images reflect SAM, while false-color images mainly reflect spatial structures.
  • GSA, GLP-HS, and NSSR produce severe spectral distortions on Cuprite Mine.
  • CNMF, Fuse-S, CSTF, and CNN-Fus produce much less spectral distortion on Cuprite Mine.CSTF, CNMF, and CSU are also associated with relatively less spectral distortion through low-rank spectral-mode modeling.
  • Figure 3 uses false-color images from bands 60, 29, and 7 with spectral error images for Pavia University.
  • GSA and GLP-HS show strong spectral errors on Pavia University, whereas Fuse-s and CNN-Fus show less spectral distortion.The stated advantage of Fuse-s and CNN-Fus mainly comes from spectral subspace representation.

5. Challenges and New Guidelines for HSI-MSI fusion

The review identifies unresolved challenges in HSI-MSI fusion involving registration, temporal changes, large resolution gaps, sensor-model uncertainty, deep-learning generalization, and computational cost. It outlines corresponding directions such as improved degradation modeling, changed-area handling, zero-shot learning, subspace representations, and GPU-accelerated CNNs.

  • Image Registration: Most fusion methods assume perfect HSI-MSI alignment and do not account for distortions caused by non-rigid registration.Non-rigid-registration-aware fusion is identified as an important future research topic.
  • 5.2. HSI-MSI Fusion for Multi-temporal Images: Multi-temporal fusion is difficult because images acquired at different times may contain different ground objects, complicating observation models and alignment.
  • 5.2. HSI-MSI Fusion for Multi-temporal Images: Existing methods scarcely address multi-temporal fusion, and changed areas can produce obvious flaws and blur in the fused image.One proposed direction is to identify changed areas and assign them less weight in the MSI.
  • 5.3. HSI-MSI Fusion for Big Spatial Resolution Differences: Large HSI-MSI resolution differences create a severely ill-posed problem because most spatial information is lost, potentially causing severe spatial distortions.Accurate estimation of the spatial degradation model is identified as key to addressing this problem.
  • Observation Models: Methods relying on known PSF and SRF may perform well on simulated data but poorly on real data when those sensor responses are not perfectly known.Estimating PSF and SRF in advance is therefore necessary for applying these assumptions to real fusion data.
  • Deep CNN Based Methods: Deep CNN methods offer promising fused-HSI quality and computational efficiency but face insufficient training data and limited generalization across differing observation models and spectral settings.Zero-shot learning is proposed as a possible way to train from the data being fused.
  • Computational Efficiency: MF and TR methods can have high computational complexity from iterative optimization, whereas low-dimensional subspaces and end-to-end CNNs can reduce computation.CNN approaches can also be accelerated using GPUs.

6. Conclusions

The paper provides a comprehensive review of HSI-MSI fusion methods by organizing them into four main families and discussing each in detail. It also analyzes current challenges and proposes guidelines and potential research directions.

  • The review classifies HSI-MSI fusion methods into pan-sharpening, matrix factorization, tensor representation, and deep CNN families.
  • Each category receives detailed introductions and discussions of its approaches.
  • The paper analyzes existing HSI-MSI fusion challenges and presents new guidelines and potential research directions.
Loading 2008.03426v1…