PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Superresolution of Multispectral Imagery with Joint Sparse and Low-Rank Learning

Lianru Gao, Danfeng Hong, Jing Yao, Bing Zhang, Paolo Gamba, Jocelyn Chanussot

arXiv:2007.14006v1eess.IVcs.CV

TL;DR

Large-scale HS acquisition is limited, motivating spectral superresolution of MS imagery from partially overlapped HS-MS data. J-SLoL jointly learns low-rank HS-MS dictionary pairs with consistent sparse representations, and experiments across reconstruction, classification, and unmixing show its effectiveness and superiority over baselines.

  • Problem

    Limited HS acquisition makes conventional HS-MS fusion difficult for large-scale scenes, motivating SSR of MS images using partially overlapped HS-MS imagery.

  • Method

    J-SLoL jointly learns low-rank HS-MS dictionary pairs and consistent sparse representations, then uses shared sparse coefficients to reconstruct unknown HS images.

  • Results

    Across reconstruction, classification, and unmixing evaluations, J-SLoL demonstrates superiority over existing baselines and produces unmixing results comparable to real HS imagery.

  • Takeaways & Limitations

    SSR with J-SLoL can exploit partially overlapped HS-MS data to recover HS imagery over larger coverage while using the correspondences between the two modalities.

Abstract

from arXiv · show

Extensive attention has been widely paid to enhance the spatial resolution of hyperspectral (HS) images with the aid of multispectral (MS) images in remote sensing. However, the ability in the fusion of HS and MS images remains to be improved, particularly in large-scale scenes, due to the limited acquisition of HS images. Alternatively, we super-resolve MS images in the spectral domain by the means of partially overlapped HS images, yielding a novel and promising topic: spectral superresolution (SSR) of MS imagery. This is challenging and less investigated task due to its high ill-posedness in inverse imaging. To this end, we develop a simple but effective method, called joint sparse and low-rank learning (J-SLoL), to spectrally enhance MS images by jointly learning low-rank HS-MS dictionary pairs from overlapped regions. J-SLoL infers and recovers the unknown hyperspectral signals over a larger coverage by sparse coding on the learned dictionary pair. Furthermore, we validate the SSR performance on three HS-MS datasets (two for classification and one for unmixing) in terms of reconstruction, classification, and unmixing by comparing with several existing state-of-the-art baselines, showing the effectiveness and superiority of the proposed J-SLoL algorithm. Furthermore, the codes and datasets will be available at: https://github.com/danfenghong/IEEE\_TGRS\_J-SLoL, contributing to the RS community.

I. INTRODUCTION

The paper reframes hyperspectral superresolution as spectral superresolution of multispectral imagery, using partially overlapped HS-MS data to recover HS signals over larger coverage. It proposes J-SLoL to jointly learn paired dictionaries and sparse representations, then evaluates the approach through reconstruction, classification, and unmixing.

  • Motivation: HS-SR traditionally fuses corresponding HS and MS images, but limited HS acquisition makes same-size pairs difficult for large-covered areas.MS imagery is more readily available at large and even global scales, although its poorer spectral information limits material recognition.
  • Limitations of prior work: Prior regression approaches are highly ill-conditioned without priors, while existing sparse recovery transfers coefficients from MS dictionaries without fully exploiting HS-MS correspondences.These limitations leave potential in the overlapped HS-MS data underused.
  • Proposed method: J-SLoL jointly learns low-rank HS-MS dictionary pairs and consistent sparse representations from overlapped images, then reconstructs unknown HS images using shared sparse coefficients.The learned dictionaries are consistent in the activated locations of sparse coefficients for each pixel.
  • Evaluation: The paper evaluates reconstruction, classification, and unmixing on three HS-MS datasets and reports superiority of J-SLoL in the SSR setting.The evaluations cover two classification datasets and one unmixing dataset according to the paper context.
  • Problem and opportunity: SSR enhances MS spectral resolution using only partially overlapped HS-MS images, avoiding the need for fully overlapping image pairs.The paper presents SSR as an alternative to HS-SR for obtaining high-spatial- and high-spectral-resolution HS products.

B. Problem Formulation

The SSR formulation assumes that spectra outside the overlapped region can be represented by the same sparse combinations of atoms learned from paired HS and MS data. J-SLoL therefore learns low-rank dictionaries with consistent sparse codes and uses a two-step dictionary-learning and sparse-recovery pipeline.

  • B. Problem Formulation: J-SLoL assumes each spectrum outside the overlap can be reconstructed by an identical sparse combination of atoms from the learned HS or MS dictionary.The assumption applies to both HS and MS images.
  • B. Problem Formulation: The overlapped HS data are represented as Hin with P spectral bands across N pixels.The passage introduces Hin as the HS spectral signatures in the overlapped part.
  • B. Problem Formulation: The overlapped MS data are represented as Min with Q channels across the same N pixels.The MS signatures are defined alongside the HS signatures for the overlapped region.
  • B. Problem Formulation: Pixels outside the overlap are separately defined for HS and MS images, enabling reconstruction beyond the observed paired region.The formulation distinguishes out-of-overlap pixels before describing the recovery procedure.
  • B. Problem Formulation: The SSR pipeline has two steps: low-rank dictionary learning followed by sparse recovery.This two-step structure is explicitly stated for the J-SLoL model.
  • 1) Low-Rank Dictionary Learning (D-Step):: The dictionary-learning step solves a constrained optimization problem for paired HS and MS dictionaries and shared sparse representations.Dh and Dm are learned for HS and MS data, while X denotes consistent sparse representations on both dictionaries.
  • 1) Low-Rank Dictionary Learning (D-Step):: The objective uses nuclear norms to approximate dictionary rank and an ℓ1-based term to promote sparsity, balanced by penalty parameters α, β, and γ.The constraints include nonnegative dictionaries and column-sum normalization of X.

2) Sparse Recovery (S-Step):

Sparse recovery first codes the unknown MS region using the learned MS dictionary, then reconstructs the corresponding HS signal with the learned HS dictionary. The S-Step is solved iteratively through constrained updates within the J-SLoL framework.

  • Sparse recovery: Sparse recovery has two stages: sparse coding on Dm followed by HS reconstruction using Dh.The reconstruction is formed from the sparse coefficients obtained from the MS data.
  • Sparse coding: The MS sparse-coding problem estimates coefficients Y for the unknown MS region Mout.Y denotes the sparse coefficients associated with Mout.
  • HS reconstruction: Given Y, the SSR output is reconstructed as Ĥout = DhY.The learned HS dictionary maps the MS-derived coefficients into the reconstructed HS signal.
  • J-SLoL structure: J-SLoL comprises low-rank dictionary learning and sparse coding, with the latter treated as a special case of the former.The solver jointly uses the learned HS and MS dictionaries and their consistent sparse representations.
  • S-Step solver: The S-Step takes Mout, Dh, and Dm as inputs and outputs Y through iterative updates until convergence or the maximum iteration count.The listed updates successively modify Y, O, and the convergence state.
  • Optimization updates: The S-Step updates Y by constrained least-squares regression, O by the corresponding auxiliary rule, and the Lagrange multiplier by its update rule.The auxiliary-variable formulation supports the ADMM-style optimization procedure.
  • Dictionary updates: The Dm update uses a closed-form matrix expression, while J and K use singular value thresholding to enforce low-rank structure.The low-rank updates apply SVD-based thresholding to the relevant variables.
  • Proximal updates: The Z update uses an element-wise soft-threshold operator, and the algorithm checks convergence after updating the optimization variables.The positive-part operator is applied element-wise where specified.

2) S-Step Solver:

The S-Step solves sparse coding for the unknown MS region using an ADMM-based optimizer. Auxiliary variables and multiplier updates support the constrained least-squares solution.

  • S-Step optimization: The S-Step applies an ADMM-based optimizer to solve the sparse-coding problem.The procedure introduces an auxiliary variable O and updates the associated multiplier.
  • Coefficient update: Sparse coefficients are estimated through constrained least-squares regression with a closed-form solution.The update is expressed using the matrices Ã, B̃, and C̃ defined for the S-Step.
  • Auxiliary updates: The auxiliary variable O and Lagrange multiplier Δ are updated using rules analogous to earlier optimization steps.Algorithm 2 summarizes the resulting S-Step optimization details.

D. Convergence Analysis

The paper solves both optimization problems with ADMM and relates the procedure to inexact ALM. It notes that multi-block ADMM lacks a strict mathematical proof here, although convergence is supported by practical precedent.

  • Convergence analysis: Both optimization problems are solved using an ADMM solver, which the paper relates to inexact ALM.The paper states that inexact ALM has theoretically guaranteed convergence under a block-count condition.
  • Convergence limitation: The multi-block ADMM formulation used for problem (3) lacks a strict mathematical convergence proof in the paper.The authors nevertheless state that convergence has been maintained across many practical cases.

III. EXPERIMENTS

The experiments evaluate SSR products from reconstruction, classification, and unmixing perspectives using three HS-MS datasets. The cited figures and table cover qualitative visualization and reconstruction comparison.

  • Evaluation criteria: SSR product quality is evaluated through direct reconstruction differences, downstream classification, and unmixing performance.Classification assesses spectral properties for later applications, while unmixing reflects material-mixture information.
  • Qualitative evaluation: Selective-band visualizations compare real HS images with J-SLoL SSR products on Indian Pines and Pavia University.Figures 4 and 5 provide the corresponding qualitative views.
  • Quantitative reconstruction: Table I compares reconstruction performance of different methods on the HS-MS Indian Pines dataset using five indices.The caption states that the best results are shown in bold.
  • Datasets: Three HS-MS datasets support the experiments, with the first two used for reconstruction and classification and the last for unmixing.The dataset allocation follows the paper’s stated evaluation design.

1) Indian Pines Data:

Indian Pines is evaluated using an AVIRIS HS scene, a simulated Sentinel-2 MS image, and an overlapped HS-MS region. Reconstruction and classification comparisons are reported against multiple methods, with the best classification results identified in Table III.

  • The Indian Pines scene was collected by AVIRIS over Indiana, USA.
  • The scene contains 145 × 145 pixels and 220 spectral channels, with a 145 × 45 partially overlapped HS-MS region.
  • Table III compares methods using OA, AA, κ, and per-class classification accuracy, with R-HS denoting the real HS image.

2) Pavia University Data:

Pavia University is evaluated using ROSIS hyperspectral data covering a large urban scene with 103 spectral bands and nine land-cover classes.

  • The Pavia University scene contains 610×340 pixels at 1.3m GSD and 103 spectral bands spanning 430nm to 860nm.
  • The dataset includes nine classes used for land-cover classification.

3) Jasper Ridge Data:

Jasper Ridge provides a 100 × 100-pixel AVIRIS scene for reconstruction and unmixing evaluation, while related experiments examine classification, parameter sensitivity, and visual comparisons. Across the reported evaluations, J-SLoL is compared with several SSR baselines and approaches the real-HS reference in classification quality.

  • Jasper Ridge Data: Jasper Ridge uses a 100 × 100-pixel ROI with 198 spectral bands and a 100 × 30 overlapped HS-MS region.
  • Reconstruction Evaluation: Reconstruction is evaluated with RMSE, PSNR, SAD, SSIM, and ERGAS against PwC, CRISP, Sun’s, Yokoya’s, and Arad’s methods.
  • Reconstruction Evaluation: J-SLoL achieves a comprehensive increase over competing methods on Indian Pines reconstruction results.
  • Cross-dataset Comparison: Indian Pines and Pavia University show basically identical quantitative and qualitative trends, although Pavia University is more challenging because of its larger image size and fewer spectral bands.
  • Parameter Sensitivity Analysis: The parameter combination (α, β, γ, η) = (1, 0.001, 0.1, 0.0001) is selected from Indian Pines sensitivity analysis and reused for the remaining datasets.
  • Classification Evaluation: J-SLoL outperforms competing SSR methods in classification, with results very close to those obtained from the real HS image.

D. Unmixing-based Evaluation

Unmixing evaluation uses FCLSU and three error criteria on Jasper Ridge, alongside reconstruction and computational-cost analyses. J-SLoL performs strongly against competing methods, while the conclusion identifies linearized sparse representation as a limitation for large-scale and complex cases.

  • Unmixing Evaluation: FCLSU unmixing is assessed with aRMSE, rRMSE, and aSAM after reconstruction evaluation.
  • Unmixing Evaluation: J-SLoL exceeds the compared methods particularly in reconstruction RMSE and SSIM and unmixing aRMSE on Jasper Ridge.
  • Unmixing Evaluation: J-SLoL unmixing results are comparable to those from the real HS image under all three measures.
  • Abundance Maps: J-SLoL produces abundance maps more similar to the real-HS reference for materials including Water and Soil than PwC and CRISP.
  • Conclusion: The conclusion states that linearized sparse techniques remain limited in data representation and fitting, especially for large-scale and complex cases.
Loading 2007.14006v1…