PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Spatial-Spectral Refinement and Calibrating Complementary Observations for Hyperspectral Image Super-Resolution

Liqian Yang, Xingchi Chen, Xinfeng Gui, Xiangyong Cao, Qianxin Yi

arXiv:2609.05303v1cs.CV

TL;DR

HMIF must fuse complementary spatial and spectral observations, but INR-based methods may underuse that complementarity. TSR-ITNR combines implicit Tucker refinement, a pretrained denoising prior, and parameter-free calibration, with theory and experiments supporting its reconstruction and downstream utility.

  • Problem

    INR-based HMIF methods may inadequately capture fine spatial structure and spectral dependencies while underexploiting complementary LR-HSI and HR-MSI information.

  • Method

    TSR-ITNR uses implicit Tucker refinement with spatial and spectral refiners, a fixed pretrained denoiser, and parameter-free complementary observation-guided calibration.

  • Results

    TSR-ITNR achieves superior performance across nearly all datasets, with theory establishing geometry preservation for spectral refinement and orthogonal complementarity for calibration.

  • Takeaways & Limitations

    The framework supports self-supervised HMIF without paired HR-HSI training data and is further assessed through downstream semantic segmentation.

  • Takeaways & Limitations

    The appendix states that some identities use no rank assumption, defining an explicit scope for that analysis.

Abstract

from arXiv · show

Hyperspectral and multispectral image fusion (HMIF) aims to reconstruct a high-resolution hyperspectral image (HR-HSI) by combining the fine spatial details of a high-resolution multispectral image (HR-MSI) with the rich spectral information of a low-resolution hyperspectral image (LR-HSI). Recent advances in implicit neural representations (INRs) have enabled flexible coordinate-based modeling for HMIF; however, existing INR-based approaches may not fully capture fine-grained spatial structures and rich spectral dependencies. Moreover, the LR-HSI and HR-MSI are primarily incorporated through degradation-consistency constraints, leaving their complementary information underexploited. To address these limitations, we propose Two-Stage Reconstruction with Implicit Tensor Neural Representation (TSR-ITNR), a unified self-supervised framework integrating representation refinement and observation-guided calibration. In Stage 1, TSR-ITNR learns an implicit Tucker representation and refines its low-rank spatial coefficient tensor and spectral basis to better capture fine spatial structures and interband correlations. A fixed pretrained denoiser further provides a deep prior for the preliminary reconstruction. In Stage 2, parameter-free calibration derives complementary and noninterfering corrections from both observations to recover information insufficiently captured in Stage 1. Theoretical analysis establishes the geometry-preserving property of spectral refinement and the orthogonal complementarity of calibration. Extensive experiments on multiple benchmark datasets demonstrate strong quantitative, visual, and spectral reconstruction performance without ground-truth HR-HSI supervision. Beyond conventional reconstruction metrics, we further assess the effectiveness of TSR-ITNR using downstream semantic segmentation accuracy.

I. INTRODUCTION

HMIF fuses LR-HSI spectral information with HR-MSI spatial detail to reconstruct HR-HSI, but INR-based methods still struggle to exploit complementary observations. TSR-ITNR addresses this with self-supervised representation refinement, observation-guided calibration, and theoretically supported components.

  • HMIF reconstructs HR-HSI by combining LR-HSI spectral information with HR-MSI spatial detail because neither modality alone provides both resolutions.
  • Existing HMIF methods use low-rank, deep-learning, or hybrid formulations, but predefined structural assumptions may limit adaptability across scenes.
  • INR-based hybrid methods provide compact continuous representations, yet exploiting complementary spatial and spectral information remains challenging.
  • TSR-ITNR first learns and refines an implicit Tucker representation using both observations and a fixed pretrained denoiser, then calibrates the preliminary reconstruction with complementary information.
  • The framework theoretically establishes basis independence and Gram preservation for AGPSR, plus orthogonal complementarity and minimum-change properties for COGC.
  • The HMIF formulation models LR-HSI and HR-MSI as spatially and spectrally degraded observations of the latent HR-HSI.

2) Tucker-Based Low-Rank Reconstruction Formulation:

The formulation represents HR-HSI with a compact Tucker model whose spatial coefficients and spectral basis are generated and refined through implicit neural functions. ITRR then combines these refined components into a preliminary reconstruction.

  • 2) Tucker-Based Low-Rank Reconstruction Formulation:: Direct HR-HSI recovery from two degraded observations is ill-posed, motivating a compact Tucker representation that exploits multilinear low-rank structure.
  • 2) Tucker-Based Low-Rank Reconstruction Formulation:: The Tucker model expresses X as G ×1 U ×2 V ×3 W = A ×3 W, separating spatial coefficients A from spectral basis W.
  • 2) Tucker-Based Low-Rank Reconstruction Formulation:: G, U, and V characterize spatial structure, while W maps the spatially low-rank coefficient tensor into the original spectral domain.
  • 2) Tucker-Based Low-Rank Reconstruction Formulation:: Recovery is reformulated as jointly estimating A and W from LR-HSI and HR-MSI, with fidelity terms enforcing consistency and R(·) providing regularization.
  • C. Stage 1: Implicit Tucker Representation and Refinement: ITRR constructs an implicit Tucker representation and refines its spatial coefficients and spectral basis to recover fine details and exploit spectral information.
  • C. Stage 1: Implicit Tucker Representation and Refinement: Four hash-encoder/GFINER branches generate the Tucker core and three factor matrices from their respective coordinate sets.
  • C. Stage 1: Implicit Tucker Representation and Refinement: The core and spatial factors form A, MSCR refines A, AGPSR refines W, and their product reconstructs the preliminary HR-HSI.
  • C. Stage 1: Implicit Tucker Representation and Refinement: MSCR enhances fine-grained, spatially varying detail while preserving A’s intrinsic low-rank structure.

1) Multi-scale Spatial Coefficient Refiner:

The refinement modules target complementary weaknesses of the Tucker factors: MSCR increases spatial flexibility, while AGPSR models spectral dependencies without changing the represented subspace geometry.

  • 1) Multi-scale Spatial Coefficient Refiner:: MSCR uses a four-level multiscale encoder-decoder with downsampling, upsampling, skip connections, and AFT blocks to refine spatial coefficients.
  • 1) Multi-scale Spatial Coefficient Refiner:: AFT combines convolutional feature transformation, channel attention, a shortcut, normalization, and nonlinear activation for spatial refinement.
  • 2) Attention-guided Geometry-preserving Spectral Refiner:: AGPSR forms band tokens from the basis-invariant projector ΠW, avoiding dependence on a particular orthonormal spectral basis.
  • 2) Attention-guided Geometry-preserving Spectral Refiner:: Its attention heads combine spectral-position and projector-row distances, enabling complementary modeling of inter-band dependencies.
  • 2) Attention-guided Geometry-preserving Spectral Refiner:: AGPSR projects the attention proposal onto the orthogonal complement of the current subspace, producing a valid Grassmann tangent direction.
  • 2) Attention-guided Geometry-preserving Spectral Refiner:: The Cayley-transform update preserves W⊤W, thereby preserving the spectral basis’s column inner products and singular values.

3) Optimization via PnP-HQS:

ITRR optimizes the preliminary HR-HSI with PnP-HQS, alternating representation fitting and denoising updates. A fixed pretrained DRUNet supplies a deep spatial prior while the algorithm iterates until convergence or a maximum iteration limit.

  • 3) Optimization via PnP-HQS:: After spatial and spectral refinement, ITRR substitutes Aref and Wref into the HMIF formulation to optimize the preliminary HR-HSI.
  • 3) Optimization via PnP-HQS:: PnP-HQS combines observation-adaptive representation learning with a deep prior from a fixed pretrained denoiser.
  • 3) Optimization via PnP-HQS:: Introducing auxiliary variable U decouples data fidelity from regularization, enabling alternating X0- and U-subproblem updates.
  • 3) Optimization via PnP-HQS:: The X0-subproblem is optimized with Adam using all three terms in the ITRR loss.
  • 3) Optimization via PnP-HQS:: The U-subproblem is interpreted as Gaussian denoising with noise level σ = λ/β.
  • 3) Optimization via PnP-HQS:: Pretrained DRUNet provides a more expressive deep spatial prior than handcrafted priors for complex and spatially varying structures.
  • 3) Optimization via PnP-HQS:: The implementation normalizes each spectral band, denoises with band-adjusted strength, restores original ranges, and updates U.
  • 3) Optimization via PnP-HQS:: Algorithm 1 updates X0 and U while increasing β and stopping when the convergence condition or maximum iteration limit is reached.

D. Stage 2: Complementary Observation-Guided Calibration

COGC calibrates the Stage-1 estimate by converting LR-HSI and HR-MSI residuals into complementary, noninterfering corrections. Its design preserves the preliminary reconstruction while enforcing observation consistency.

  • COGC retains the preliminary estimate X0 and converts LR-HSI and HR-MSI residuals into two targeted corrections using known degradation operators.
  • The HR-MSI branch lifts its residual into the hyperspectral domain, while the LR-HSI branch back-projects its residual and restricts it to spectral components unavailable to HR-MSI.
  • The final HR-HSI is obtained by adding both observation-derived corrections to the direct-path estimate.
  • COGC assigns corrections to complementary spectral components, reducing redundant or interfering calibration without learnable parameters or Stage-1 backpropagation.
  • AGPSR provides a basis-independent spectral refinement whose orthogonal update preserves WᵀW, singular values, rank, and associated norms and condition number.

B. Orthogonal Complementarity and Minimum-Change Property of COGC

The analysis shows that COGC recovers identifiable error components in orthogonal spectral subspaces and produces the closest reconstruction to the Stage-1 estimate that satisfies both observations.

  • COGC recovers sensor-identifiable components of the Stage-1 error through two orthogonal branches.
  • The result requires no rank assumption on the degradation operators.
  • The LR-HSI correction lies in the spectral-response null space and therefore cannot alter the HR-MSI calibration.
  • The remaining error is invisible to both sensors, while the recovered components form a Pythagorean error decomposition.
  • The calibrated estimate satisfies both observation equations exactly and is the unique minimum-change feasible reconstruction.

V. EXPERIMENTS AND ANALYSIS

Experiments evaluate TSR-ITNR on simulated Pavia, CAVE, and ICVL datasets using quantitative, visual, and spectral comparisons against representative methods. TSR-ITNR achieves superior performance across nearly all datasets and evaluation metrics, with darker residual maps and spectral curves closer to ground truth.

  • Experimental Settings: Experiments use simulated Pavia Centre, CAVE, and ICVL scenes, with scale factors of ×8 for Pavia and ×16 for CAVE and ICVL.Pavia Centre uses a 256 × 256 reference with 102 bands; CAVE and ICVL scenes provide additional benchmark settings.
  • Quantitative Comparisons: TSR-ITNR is compared with ten representative low-rank, deep-learning, and hybrid methods, including CTDF, GTNN, CAFE+, EDIP, CS2DIPs, DELTA, CNN-FUS, LRTFR, CLoRF, and SSLRDN.Several methods receive task-specific adaptations when they were not originally designed for HMIF.
  • Quantitative Comparisons: TSR-ITNR achieves superior performance across nearly all datasets and evaluation metrics, maintaining consistent accuracy across diverse spatial structures and spectral characteristics.Competing methods show more pronounced performance variations across datasets.
  • Visual and Spectral Analysis: Visual comparisons show that TSR-ITNR produces darker residual maps with fewer localized artifacts, indicating more accurate spatial reconstruction than competing methods.DELTA exhibits noticeable blurring and loss of fine edge details, while competing methods retain errors near structural boundaries and textured regions.
  • Visual and Spectral Analysis: TSR-ITNR follows the ground-truth spectral curves most closely across the spectral range in representative Pavia Centre, CAVE Lemons, and ICVL Hill scenes.Figure 4 compares spectral curves across the three scenes.

1) Ablation Study on Tucker-Based Representation:

Ablations show that Tucker-based structured modeling and joint refinement of spatial and spectral factors are important for reconstruction performance. Removing Tucker, MSCR, or AGPSR degrades performance, while jointly refining both components outperforms unrefined alternatives.

  • Tucker-Based Representation: Removing Tucker decomposition causes pronounced performance degradation, demonstrating the benefit of structured low-rank spatial–spectral modeling and refinement.The w/o Tucker variant reconstructs directly from hash-encoded spatial–spectral coordinates while retaining COGC.
  • MSCR: Removing MSCR causes clear performance degradation, indicating that multiscale spatial refinement captures dependencies and fine structural details beyond Tucker representation.The w/o MSCR variant combines an unrefined spatial coefficient tensor with the AGPSR-refined spectral basis.
  • AGPSR: Removing AGPSR produces a modest but consistent decline across all metrics, supporting the complementary benefit of geometry-aware inter-band refinement.The w/o AGPSR variant retains MSCR and COGC but uses an unrefined spectral basis.
  • Joint Refinement: Jointly removing MSCR and AGPSR performs better than removing Tucker but worse than complete TSR-ITNR, showing that both refiners improve representation of complex spatial–spectral details.The comparison separates the contribution of Tucker structure from the contribution of joint refinement.

5) Ablation Study on COGC:

The COGC ablation tests whether the second-stage calibration contributes beyond Stage 1. Removing COGC degrades performance, indicating that calibration exploits complementary information from the two observations.

  • COGC: Removing the second stage causes performance degradation, indicating that COGC calibrates the preliminary reconstruction by leveraging complementary LR-HSI and HR-MSI information.Without COGC, the Stage 1 reconstruction is used directly as the final output.

6) Ablation Study on the Effectiveness of Fusion:

The fusion ablation evaluates whether both observations are needed for reconstruction. Using either HR-MSI or LR-HSI alone causes substantial performance degradation because the observations provide complementary spatial and spectral information.

  • Observation Fusion: Using HR-MSI alone or LR-HSI alone causes substantial performance degradation, demonstrating the benefit of jointly exploiting both observations.HR-MSI provides fine spatial details, whereas LR-HSI preserves rich spectral information.

E. Semantic Segmentation Results

Semantic segmentation is used to assess the utility of reconstructed HSIs beyond conventional reconstruction metrics, with qualitative comparisons on the Pavia Centre test region.

  • Evaluation setup: The Pavia Centre evaluation reserves the upper-left 256 × 256 region for testing and uses spatially disjoint training and validation sets elsewhere.UNetFormer is adapted for semantic segmentation evaluation.
  • Ablation visualization: Figure 5 provides visual comparisons of different ablation settings on ICVL Hill at ×16.
  • Qualitative comparison: Figure 6 compares segmentation maps from Bicubic, reconstructed HSIs, and the reference HR-HSI against ground-truth labels.GT denotes the ground-truth labels.

VI. CONCLUSION

The conclusion presents TSR-ITNR as a self-supervised two-stage HMIF framework combining implicit Tucker modeling, refinement, denoising, and complementary observation-guided calibration. Theoretical analysis and experiments support its reconstruction, calibration, and downstream segmentation objectives, while future work targets broader restoration and fusion tasks.

  • Framework: TSR-ITNR uses implicit Tucker representation to preserve high-order structure and capture global low-rank spatial-spectral correlations in Stage 1.Spatial and spectral refinements enhance fine spatial details and interband dependencies.
  • Framework: PnP-HQS incorporates a pretrained denoiser as a deep prior, while parameter-free COGC derives complementary corrections from LR-HSI and HR-MSI in Stage 2.The calibration is described as using complementary corrections without mutual interference.
  • Theory: Theoretical analysis establishes spectral-refinement basis independence and Gram preservation, together with COGC orthogonal complementarity and a minimum-change property.The appendix identifies these as the principal claims of Theorems 1 and 2.
  • Empirical validation: Experiments on simulated datasets report strong quantitative, visual, and spectral reconstruction performance, with ablations and semantic segmentation providing further validation.
  • Calibration properties: The COGC branches are Frobenius-orthogonal, and the LR-HSI branch is invisible to the HR-MSI sensor and cannot disturb its calibration.These properties support noninterfering complementary corrections.
  • Calibration properties: The calibrated reconstruction is feasible for the spatial observation and is the unique minimum-change solution under the stated observation constraints.The minimum is attained by canceling the residual component in the joint null space.
Loading 2609.05303v1…