PaperlayerView exact revision on arXiv

Source-linked AI summary

Super-Resolution for Hyperspectral and Multispectral Image Fusion Accounting for Seasonal Spectral Variability

Ricardo Augusto Borsoi, Tales Imbiriba, José Carlos Moreira Bermudez

arXiv:1808.10072v2cs.CV

TL;DR

Existing HS–MS fusion methods often neglect spectral differences between images acquired under different conditions or seasons, despite their relevance to fusion. FuVar combines unmixing with a parametric spectral-variability model, and experiments report strong performance under variability and competitive performance otherwise.

  • Problem

    Many HS–MS fusion methods assume identical acquisition conditions, although seasonal and acquisition differences can change material spectra.

    Source: §I · p. 1, §II · p. 2

  • Method

    FuVar combines an unmixing-based fusion formulation with a parametric model of spectral variability between HS and MS images.

    Source: §I · p. 2, A. The proposed model · p. 3

  • Results

    Experiments with synthetic and real data show FuVar improves performance under spectral variability and remains competitive with state-of-the-art methods without it.

    Source: 1) Discussion · p. 7, a) Discussion · p. 8, 2) HS and MS images with a large… · p. 8

  • Takeaways & Limitations

    The proposed strategy is reported to be relatively insensitive to hyperparameter choice and competitive even when spectral variability is absent.

    Source: §V · p. 9

Abstract

from arXiv · show

Image fusion combines data from different heterogeneous sources to obtain more precise information about an underlying scene. Hyperspectral-multispectral (HS-MS) image fusion is currently attracting great interest in remote sensing since it allows the generation of high spatial resolution HS images, circumventing the main limitation of this imaging modality. Existing HS-MS fusion algorithms, however, neglect the spectral variability often existing between images acquired at different time instants. This time difference causes variations in spectral signatures of the underlying constituent materials due to different acquisition and seasonal conditions. This paper introduces a novel HS-MS image fusion strategy that combines an unmixing-based formulation with an explicit parametric model for typical spectral variability between the two images. Simulations with synthetic and real data show that the proposed strategy leads to a significant performance improvement under spectral variability and state-of-the-art performance otherwise.

I. INTRODUCTION

HS–MS fusion combines the spatial detail of multispectral images with the spectral detail of hyperspectral images, but existing methods often assume matching acquisition conditions. FuVar models differences between the images’ spectral signatures using distinct endmember sets linked by parametric scaling factors.

I. INTRODUCTION

  • HS images provide many contiguous spectral bands but have low spatial resolution, while MS images offer higher spatial resolution with fewer bands.Combining them can produce an image with both high spatial and spectral resolution.Source: §I · p. 1

A. The proposed model

  • FuVar addresses acquisition- and season-related spectral variability by allowing distinct high-resolution images and endmember sets, linked through a parametric variability model.The model uses positive scaling factors and a shared abundance matrix to represent the relationship between the images.Source: §I · p. 2, §II · p. 2, A. The proposed model · pp. 2–3

II. SPECTRAL VARIABILITY IN IMAGE FUSION

  • Different acquisition conditions and seasonal changes can alter material signatures, challenging fusion methods that assume both images represent one underlying scene under identical conditions.Such variability may arise from illumination, atmospheric conditions, or seasonal effects, including vegetation changes.Source: §II · p. 2, §I · p. 2

III. THE IMAGE FUSION PROBLEM

The fusion objective estimates variability and abundance matrices from the observed HS and MS images. It regularizes spectral variability and spatial abundance structure, then alternates between the variables to seek a local stationary point.

III. THE IMAGE FUSION PROBLEM

  • The objective estimates the variability matrix Ψ and abundance matrix A, with regularizers constraining Ψ near unity and spectrally smooth while promoting spatial smoothness in A.The optimization is non-convex jointly but convex in each variable separately, motivating alternating minimization.Source: §III · p. 3
  • The method alternates updates of Ψ and A; each subproblem is handled with ADMM when the objective’s structure requires splitting.For the A subproblem, the L2,1 norm makes the objective non-differentiable and non-separable across pixels.Source: §III · p. 3, A. Optimizing w.r.t. A · p. 4
  • The algorithm estimates endmembers from the observed HSI and initializes abundances using bicubic interpolation of an FCLS estimate.Its outputs are the estimated high-resolution HS and MS images.Source: §III · p. 3

B. Optimizing w.r.t. Ψ

The Ψ subproblem is split into smaller problems for iterative solution with ADMM.

B. Optimizing w.r.t. Ψ

  • The Ψ update is reformulated for fixed A and split into smaller problems solved iteratively using ADMM.The resulting formulation is expressed as an ADMM problem with a scaled augmented Lagrangian.Source: B. Optimizing w.r.t. Ψ · pp. 4–5

IV. EXPERIMENTAL RESULTS

The experiments assess FuVar using synthetic data, same-time HS–MS images, and real images acquired at different times.

IV. EXPERIMENTAL RESULTS

  • Three experiments test controlled synthetic variability, fusion without spectral variability, and real HS–MS fusion across different acquisition times.The synthetic experiment evaluates estimation performance; the other settings compare methods under matching and variable spectral conditions.Source: §IV · p. 5

A. Experimental Setup

The experiments compare FuVar with three fusion methods under common observation conditions, and evaluate reconstructions using four quality metrics. The setup fixes key model parameters and uses both synthetic and real images.

A. Experimental Setup

  • FuVar is compared with HySure, CNMF, and GLP-HS, with the number of endmembers fixed at P = 30 for CNMF, HySure, and FuVar.The blur kernel is assumed known for HySure and FuVar, while the HSI endmember matrix is estimated using VCA.Source: A. Experimental Setup · p. 5
  • Real-image preprocessing removes low-SNR or water-absorption bands, normalizes intensities, and denoises the HSI to form a high-SNR reference.Observed HSIs are blurred, decimated, and given 30dB SNR; observed MSIs are given 40dB SNR.Source: A. Experimental Setup · p. 5

B. Quality measures:

  • PSNR is computed bandwise, and larger values indicate better spatial reconstruction, approaching infinity as bandwise reconstruction error approaches zero.The four metrics compare reconstructed images with the original high-resolution images.Source: B. Quality measures · p. 5
  • The other reported measures include Spectral Angle Mapper (SAM) and ERGAS, alongside PSNR and a fourth quality metric.The supplied passages name SAM and ERGAS but do not specify the fourth metric.Source: B. Quality measures · p. 5

C. Example 1

The experiments evaluate fusion on synthetic images and real HS-MS pairs, including pairs acquired at different times. FuVar improves all reported metrics in the synthetic variability experiment and remains close to state of the art without variability.

C. Example 1

  • FuVar gains more than 5dB for Zh and 9dB for Zm in PSNR, and improves all other reported metrics in the synthetic experiment.Visual inspection also identifies FuVar as the most accurate, with a good balance between smooth regions and sharp edges and without the artifacts or color distortions seen in competing results.Source: 1) Discussion · p. 6, Table I · p. 6, Figure 1 · p. 6

D. Example 2

  • Without spectral variability, FuVar is slightly better than competing methods on all metrics except SAM, where it is second best and close to the best.The reconstructed images are visually similar across methods.Source: 1) Discussion · p. 7, Table II · p. 7, Figure 3 · p. 7

E. Example 3

  • Example 3 evaluates real HS-MS image pairs acquired at different times, with examples spanning small and large acquisition-time differences.The supplied passages describe the experiment’s image-pair conditions but do not report its reconstruction results.Source: E. Example 3 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7

1) HS and MS images with a small acquisition time difference:

In real-image tests, FuVar generally outperformed competing methods for both small and large acquisition-time differences; it also showed modest sensitivity to regularization choices and higher execution times. The conclusion reports performance gains with spectral variability and competitiveness without it.

1) HS and MS images with a small acquisition time difference:

  • Across Lake Isabella and Lockwood, FuVar generally outperformed competitors, with gains in remaining metrics and recovered images closer to references across visible and infrared bands.The images had a small acquisition-time difference.Source: a) Discussion · pp. 7–8, Table III · p. 8, Figure 6 · p. 8

2) HS and MS images with a large acquisition time difference:

F. Execution Times

  • FuVar's execution times were higher than those of methods that do not model spectral variability, which solve simpler optimization problems.Table V reports execution times in seconds.Source: F. Execution Times · p. 9, Table V · p. 9

G. Parameter Sensitivity

  • Even parameter variations across several orders of magnitude had relatively little effect on FuVar's PSNR, which remained significantly above the competing methods.The sensitivity test varied λA, λ1, and λ2 individually in Example 1; Figure 10 plots normalized parameter variation.Source: G. Parameter Sensitivity · p. 9, Figure 10 · p. 9

V. CONCLUSIONS

  • The conclusion reports that FuVar improves performance when spectral variability is present, remains competitive without it, and is quite insensitive to hyperparameter choice.The paper links the modeled mismatches to illumination and seasonal changes.Source: §V · p. 9

APPENDIX A OPTIMIZATION OF THE ADMM FOR THE A SUBPROBLEM

The appendix derives the ADMM subproblem for w by isolating terms in the scaled augmented Lagrangian and expressing the optimization per pixel.

APPENDIX A OPTIMIZATION OF THE ADMM FOR THE A SUBPROBLEM

  • The appendix optimizes the scaled augmented Lagrangian with respect to the variables in Θ, including w and B1 through B6 and u.This section begins the derivation for the A subproblem.Source: Appendix A · p. 9

A. Optimizing w.r.t. w

  • The w subproblem is formed by selecting the terms of the Lagrangian that depend on w and can be rewritten per pixel.w is the vectorization of the abundance matrix A.Source: A. Optimizing w.r.t. w · p. 9
  • The derivation sets the gradient with respect to each a_n to zero and then gives the solution for the subproblem.The supplied passages do not state the solution's explicit form.Source: A. Optimizing w.r.t. w · p. 9

B. Optimizing w.r.t. B1

The B1 subproblem is reformulated in matrix form and then expressed separately for each pixel before deriving its stationary condition.

B. Optimizing w.r.t. B1

C. Optimizing w.r.t. B2

The appendix derives updates for the ADMM variables, including block soft-thresholding for one subproblem and efficient Fourier-domain computation for differential-operator terms.

C. Optimizing w.r.t. B2

  • Sparse differential operators permit iterative solution, while assuming block-circulant operators allows related computations in the Fourier domain.The appendix describes Fourier-domain computation independently for each endmember and gives the transform-based solution form.Source: C. Optimizing w.r.t. B2 · p. 10, D. Optimizing w.r.t. B3 · p. 10
  • The B2 subproblem is solved using the proximal operator of the L2 norm, expressed through block soft thresholding.The passage identifies the soft operator by reference to.Source: E. Optimizing w.r.t. B4 · p. 10
  • A related optimization problem is identified as identical to the preceding problem, and its solution is given using the same strategy.The supplied passages do not specify the solution's explicit form.Source: F. Optimizing w.r.t. B5 · p. 11
  • The dual variable u6 is updated by adding vec(B6) − w to its current value.Source: H. Dual update u · p. 11

APPENDIX B OPTIMIZATION OF THE ADMM FOR THE Ψ SUBPROBLEM

The appendix describes variable-wise optimization of the scaled augmented Lagrangian over B, Ψ, and U. For Ψ, it derives a sparse linear system and then imposes positivity on its solution.

APPENDIX B OPTIMIZATION OF THE ADMM FOR THE Ψ SUBPROBLEM

  • For B, setting the derivative to zero leads to the Sylvester equation AXB − X + C = 0, which MATLAB dlyap solves.Source: A. Optimizing w.r.t. B · p. 11
  • For Ψ, dropping nonnegativity first yields stationarity condition (57), then vectorization recasts it as a sparse linear system (58).The system is described as efficient to solve because its left-hand-side matrices are very sparse.Source: A. Optimizing w.r.t. B · p. 11
  • Positivity is introduced in the solution obtained by solving the sparse system.Source: A. Optimizing w.r.t. B · p. 11
Loading 1808.10072v2…