PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled-and-Coupled Networks: Self-Supervised Hyperspectral Image Super-Resolution with Subpixel Fusion

Danfeng Hong, Jing Yao, Deyu Meng, Naoto Yokoya, Jocelyn Chanussot

arXiv:2205.03742v1eess.IVcs.CV

TL;DR

HS super-resolution must fuse low-spatial-resolution hyperspectral data with high-spatial-resolution multispectral data despite sensor differences and mixed pixels. DC-Net addresses this through decoupled, coupled, and self-supervised fusion, outperforming existing state-of-the-art methods while exhibiting spectral degradation on real data.

  • Problem

    HS super-resolution fuses low-spatial-resolution HS and high-spatial-resolution MS images, but existing approaches rarely address sensor-specific differences and highly mixed pixels.

  • Method

    DC-Net progressively fuses HS-MS information by decoupling common and sensor-specific components, applying spectral-unmixing-inspired coupling, and enforcing abundance-map consistency.

  • Results

    DC-Net outperforms existing state-of-the-art fusion methods, while its D-Net improves C-Net by about 1.5 PSNR and just over 2 SAM.

  • Takeaways & Limitations

    Experiments on simulated and real data support DC-Net as a superior HS-super-resolution method over current state-of-the-art fusion methods.

  • Takeaways & Limitations

    The authors report spectral degradation in the HS super-resolution products generated from real data.

Abstract

from arXiv · show

Enormous efforts have been recently made to super-resolve hyperspectral (HS) images with the aid of high spatial resolution multispectral (MS) images. Most prior works usually perform the fusion task by means of multifarious pixel-level priors. Yet the intrinsic effects of a large distribution gap between HS-MS data due to differences in the spatial and spectral resolution are less investigated. The gap might be caused by unknown sensor-specific properties or highly-mixed spectral information within one pixel (due to low spatial resolution). To this end, we propose a subpixel-level HS super-resolution framework by devising a novel decoupled-and-coupled network, called DC-Net, to progressively fuse HS-MS information from the pixel- to subpixel-level, from the image- to feature-level. As the name suggests, DC-Net first decouples the input into common (or cross-sensor) and sensor-specific components to eliminate the gap between HS-MS images before further fusion, and then fully blends them by a model-guided coupled spectral unmixing (CSU) net. More significantly, we append a self-supervised learning module behind the CSU net by guaranteeing the material consistency to enhance the detailed appearances of the restored HS product. Extensive experimental results show the superiority of our method both visually and quantitatively and achieve a significant improvement in comparison with the state-of-the-arts. Furthermore, the codes and datasets will be available at https://sites.google.com/view/danfeng-hong for the sake of reproducibility.

I. INTRODUCTION · II. RELATED WORK

HS images provide rich spectral information but sacrifice spatial resolution, motivating fusion with spatially detailed MS images. DC-Net addresses HS-MS distribution gaps and mixed pixels through progressive decoupled-and-coupled fusion with self-supervised consistency.

  • I. INTRODUCTION: The coupled C-Net alleviates highly mixed spectral pixels and enables compact subpixel-level fusion beyond conventional pixel-level approaches.
  • I. INTRODUCTION: HS imaging captures subtle material differences through high spectral resolution, but its high spatial sampling distance limits practical applications.
  • I. INTRODUCTION: MS imaging complements HS data by providing finer spatial information, motivating HS-MS fusion for hyperspectral super-resolution.
  • I. INTRODUCTION: Prior HS-SR methods model HS-MS relationships using hand-crafted priors and sensor characteristics such as spectral response and point spread functions.
  • I. INTRODUCTION: DC-Net progressively fuses HS-MS information from pixel to subpixel and image to feature levels through a unified end-to-end architecture.
  • I. INTRODUCTION: Its decoupled D-Net separates common cross-sensor and sensor-specific components to reduce the HS-MS distribution gap before fusion.
  • I. INTRODUCTION: A plug-and-play self-supervised S-Net aligns HS- and MS-derived abundance maps to preserve semantic consistency and improve the restored high-resolution HS product.

A. Traditional Models … B. Problem Formulation

Traditional HS-MS fusion methods rely on pansharpening, subspace models, or handcrafted calibration, while the paper formulates HS super-resolution as latent spectral-spatial reconstruction and introduces DC-Net to address cross-sensor distribution gaps.

  • A. Traditional Models: Pansharpening methods inject multispectral spatial details into low-resolution hyperspectral images through component substitution or multiresolution analysis.These are the two main traditional pansharpening categories described for producing high-resolution hyperspectral products.
  • A. Traditional Models: Subspace-based methods enhance hyperspectral spatial resolution using matrix factorization or Bayesian estimation to estimate shareable spectral representations.A representative approach decomposes the low-resolution hyperspectral image into a spectral dictionary and sparse coefficients.
  • B. Deep Learning based Models: Traditional approaches can perform competitively but depend substantially on cross-sensor calibration and strong handcrafted priors.This limitation motivates the paper’s focus on deep-learning-based HS-MS fusion.
  • B. Deep Learning based Models: Deep-learning approaches use CNN priors, learned spectral response functions, or interpretable supervised models to improve HS-MS fusion and reconstruction.The cited methods include two-stage refinement, external or joint internal-external SRF learning, and interpretable deep supervision.
  • A. Overview: A self-supervised S-Net improves restored HS detail by correcting material correspondence between HS-MS images without requiring prior training.The learning objective combines adversarial, reconstruction, and self-supervised components in an end-to-end framework.
  • III. DECOUPLED-AND-COUPLED NETWORKS: DC-Net progressively fuses HS-MS information by decoupling common and sensor-specific components before coupling them through a model-guided spectral-unmixing network.The architecture is organized around D-Net and C-Net and targets the distribution gap caused by sensor-specific information and mixed spectral pixels.
  • B. Problem Formulation: The problem formulation models low-resolution HS and high-resolution MS observations as spectrally and spatially degraded versions of a high-resolution HS image.The degradation operators are the spectral response function H and point-spread function R, with observed noises N_x and N_y.
  • B. Problem Formulation: Under spectral mixing, reconstruction estimates nonnegative endmembers S and abundances A satisfying the sum-to-one constraint, while latent mappings separate coupled pixel information within each sensor.The high-resolution HS image is represented as Z = SA, and mappings f_x and f_y transform HS and MS inputs into a latent image space.

C. Decoupled Network (D-Net)

The decoupled network (D-Net) separates HS-MS inputs into common and sensor-specific representations, then recombines them to increase cross-source affinity before fusion.

  • C. Decoupled Network (D-Net): D-Net embeds HS-MS images into a shared content space and sensor-specific spaces using cross-sensor and sensor-specific encoders, generators, and a domain-common discriminator.The shared representations capture common information, while separate spaces retain domain-specific information.
  • C. Decoupled Network (D-Net): The recombined images provide higher affinity between HS-MS sources, creating greater potential for subsequent fusion.This recombination is intended to make the sources more sufficiently fusible after decoupling.
  • C. Decoupled Network (D-Net): D-Net interactively recombines common and sensor-specific information through generators to produce transformed HS-MS images.The transformations are defined as X̄ = G_X(X_C, Y_S) and Ȳ = G_Y(Y_C, X_S).

D. Coupled Network (C-Net)

C-Net jointly unmixed recombined HS-MS images into endmembers and abundances using coupled convolutional autoencoders. It estimates the high-resolution hyperspectral image while learning spectral and spatial degradation through cycle consistency.

  • D. Coupled Network (C-Net): C-Net uses two encoders and two decoders to jointly unmix recombined HS and MS images into endmembers and abundances.The coupled autoencoder reconstructs X and Y through separate encoder-decoder paths.
  • D. Coupled Network (C-Net): The network estimates the high-resolution hyperspectral image as Ẑ = SA using MS-derived abundances and HS-derived endmember decoding.The estimate is computed as fv(gu(Ȳ; Wg,u); Wf,v).
  • D. Coupled Network (C-Net): A Clamp function enforces nonnegative abundances, while regularization guarantees the abundance sum-to-one constraint.The constraint is expressed as 1^⊤A = 1^⊤.
  • Cycle consistency: C-Net learns the spectral response function and point spread function through cycle consistency rather than assuming either degradation function is known.This mechanism organizes spatial and spectral degradation into a closed cycle matching the observation models.

E. Self-Supervised Subnetwork (S-Net)

S-Net addresses chaotic HS–MS semantic correspondences by enforcing material consistency between abundance maps, using contrastive attraction and repulsion plus grouped representations to align materials one-to-one.

  • S-Net: S-Net learns high-level material consistency to align HS and MS abundance maps one-to-one, attracting same-material pairs and repelling different-material pairs.This corrects correspondence in both material order and category, improving the plausibility of abundance maps and endmembers and thereby HS reconstruction accuracy.
  • S-Net: To reduce computational cost, S-Net groups the abundance maps into m groups for group-to-group matching.The grouping parameter is experimentally set to m = 8.
  • S-Net: It encodes the grouped HS and MS abundance representations with two-stream CNN encoders whose pooling sizes and strides differ to produce equal final representation sizes.Each encoder contains two conv-relu-pool-fc blocks, with larger pooling size and stride in the HS stream.

F. Network Training

DC-Net is trained with a combined objective incorporating reconstruction, cycle-consistency, adversarial, ASC, and InfoNCE losses. Training uses PyTorch and Adam for 10,000 epochs with a linearly decayed learning rate and experimentally selected hyperparameters.

  • Loss functions: The training objective additionally includes adversarial loss in D-Net, ASC loss, and InfoNCE loss in S-Net for positive-negative sample distance measurement.
  • Loss functions: The reconstruction objective combines ℓ1 losses for image reconstruction, downsampled consistency, and cycle consistency, while ℓ1 measurement enhances detailed perception.
  • Loss functions: DC-Net is trained by minimizing an overall loss parameterized by {α, β, γ}.
  • Implementation details: The PyTorch networks use Adam for 10000 epochs with batch size 1, an initial learning rate of 0.005, linear decay, and Kaiming initialization.
  • Implementation details: Hyperparameters are experimentally determined through grid search, and an early stopping strategy is considered.

IV. EXPERIMENTAL RESULTS

The experimental section reviews datasets and settings, validates the networks through ablation studies, and evaluates DC-Net on simulated and real datasets against state-of-the-art methods.

  • The section first reviews the datasets and experimental settings.
  • It then verifies the proposed networks through extensive ablation studies.
  • Finally, it evaluates DC-Net on simulated and real datasets against state-of-the-art methods.

A. Dataset and Setup

DC-Net is evaluated on the indoor CAVE and remote-sensing Chikusei HS-MS datasets using controlled spectral and spatial degradation protocols. Performance is assessed with five standard image-fusion quality indices.

  • A. Dataset and Setup: Evaluation uses the indoor CAVE dataset and remote-sensing Chikusei dataset to assess DC-Net’s HS-MS fusion performance.CAVE contains 32 images, while Chikusei is acquired by the VNIR-C sensor.
  • A. Dataset and Setup: CAVE images are 512 × 512 × 31 across 400nm–700nm, while Chikusei provides 128 bands over a 2517 × 2335-pixel scene cropped into 12 subimages.The subimages have spatial size 576×448, and dataset splits follow proportions of 8:8:16 for CAVE and 4:4:4 for Chikusei.
  • A. Dataset and Setup: HrMS images are generated with sensor-specific spectral response functions, whereas LrHS images are spatially downsampled using Wald’s protocol with a 32 sampling ratio.The Nikon D700 and Landsat-8 MS sensors provide the CAVE and Chikusei spectral response functions, respectively.
  • A. Dataset and Setup: Fusion quality is quantified using PSNR, SAM, ERGAS, SSIM, and UIQI.These five metrics cover spectral, structural, and overall image-quality assessment.

B. Comparison with State-of-the-arts · 1) Experiments on CAVE Data:

On CAVE, the authors compare DC-Net and DC-Net-S with traditional and deep-learning SOTA HS-SR methods using quantitative and qualitative evaluations. DC-Net outperforms existing SOTA deep-learning approaches, while DC-Net-S produces stronger detailed appearances in visual comparisons.

  • 1) Experiments on CAVE Data:: NLSTF outperforms other traditional SOTA methods across all reported indices except the unspecified exception in the supplied passage.The passage attributes NLSTF’s performance to coupled tensor fusion and non-local spatial information preservation.
  • 1) Experiments on CAVE Data:: Table I reports average performance over 16 CAVE testing images using five quality indices for all compared methods, DC-Net, and DC-Net-S.The evaluation uses the same experimental configurations as the original literature as much as possible.
  • 1) Experiments on CAVE Data:: DC-Net outperforms existing SOTA deep-learning approaches on CAVE HS-SR comparisons.The comparison includes supervised and unsupervised deep-learning methods, including uSDN, MHF-Net, GDD, and CUCaNet.
  • 1) Experiments on CAVE Data:: Unsupervised GDD and CUCaNet achieve results broadly comparable to supervised MHF-Net and exceed it on certain indices.The comparison positions uSDN above non-deep-learning algorithms but below other deep-learning competitors.
  • 1) Experiments on CAVE Data:: The qualitative evaluation examines the 11th spectral band and selected regions of interest to compare reconstructed details.The supplied passages identify visual comparisons for chart and stuffed-toy images, with zoomed ROIs used for detailed observation.
  • 1) Experiments on CAVE Data:: Deep-learning models visibly outperform traditional methods, especially in coarser object structures, while the proposed methods surpass other unsupervised approaches.DC-Net-S performs better than DC-Net in detailed appearances in the supplied qualitative comparison.

2) Experiments on Chikusei Data: · 3) Visual Quality Evaluation:

On Chikusei, DC-Net-S outperforms competing methods across all reported PQIs, while both proposed methods produce finer spatial details and lower residual errors in visual evaluations.

  • 3) Visual Quality Evaluation:: Visual evaluation compares GT images, residual maps, and spectral RMSE curves for NSSR, uSDN, MHF-Net, CUCaNet, DC-Net, and DC-Net-S across six scenes.The evaluation covers four CAVE scenes and two Chikusei scenes.
  • 2) Experiments on Chikusei Data:: DC-Net-S is superior to competing methods on Chikusei across all five reported PQIs.The quantitative comparison is presented in Table II, with performance following the same general trend as on CAVE.
  • 2) Experiments on Chikusei Data:: Chikusei super-resolution results are generally higher quality than CAVE results because remote-sensing images are less spatially complex.This dataset-level difference is attributed to the lower complexity of remote-sensing imagery.
  • 3) Visual Quality Evaluation:: DC-Net and DC-Net-S produce better residual maps than competing methods, indicating finer spatial details in the reconstructed hyperspectral images.The visual comparison includes representative scenes from both CAVE and Chikusei datasets.
  • 3) Visual Quality Evaluation:: The visual study evaluates hyperspectral reconstruction quality using RMSE values measured across spectral bands for each selected scene.Bandwise RMSE values are plotted along the spectra for the six studied scenes.
  • 3) Visual Quality Evaluation:: Table II reports quantitative comparisons with the same state-of-the-art super-resolution methods using five different PQIs on Chikusei.The table provides the dataset-specific benchmark scope for the reported comparison.
  • 3) Visual Quality Evaluation:: Figure 9 visualizes real-data super-resolution using WV-3 high-resolution multispectral and Hyperion low-resolution hyperspectral imagery from three Cuprite sub-regions.The visualization uses the proposed method on imagery provided by Maxar.

C. Evaluation on Real Data … V. CONCLUSION

DC-Net and DC-Net-S restore high-resolution hyperspectral results on real WorldView-3/Hyperion scenes, while ablations show gains from physical constraints and network components. The framework uses subpixel fusion, spectral-unmixing properties, and self-supervision for HS-SR.

  • C. Evaluation on Real Data: DC-Net and DC-Net-S both restore high-resolution HrHS results on real registered WorldView-3 and Hyperion scenes, with DC-Net-S achieving higher visual verisimilitude.The evaluation uses 640 × 640 HrMS images with 8 bands and 160 × 160 LrHS images with 167 spectral channels; real HrHS ground truth is unavailable.
  • 1) Ablation Studies:: The ablation study evaluates stepwise additions of D-Net and S-Net alongside ANC and ASC using PSNR and SAM on 8 CAVE validation images.Configurations are selected by maximizing stepwise performance.
  • 1) Ablation Studies:: Jointly using ANC and ASC further improves C-Net’s PSNR and SAM over ANC alone, while ANC alone raises PSNR beyond 4 and approximately halves SAM.C-Net without ANC and ASC performs relatively poorly, demonstrating the importance of both physical constraints.
  • D. Model Analysis: Adding D-Net to C-Net improves performance by about 1.5 PSNR value, supporting the contribution of the decoupling subnetwork.The ablation evaluates average PSNR and SAM over 8 CAVE validation images.
  • 2) Computational Analysis:: DC-Net-S contains 3.43MB of parameters and has desirable, acceptable inference time per image on a PC equipped with one NVIDIA GeForce GTX 1080Ti GPU.The parameter size and running time are reported in Table IV.
  • V. CONCLUSION: DC-Net is a subpixel-level HS-SR framework that exploits affinity within data sources, exclusivity across HS-MS data, and mixed-pixel properties through spectral-unmixing-inspired fusion.The framework is intended to utilize intrinsic HS-MS properties for fusion in varied real cases.
  • V. CONCLUSION: A self-supervised module is introduced to further optimize DC-Net’s network performance.The conclusion identifies self-supervision as an additional optimization component.
Loading 2205.03742v1…