PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Interpretable RGB-Guided Hyperspectral Super-Resolution under Real Cross-resolution Misalignment

Mohamad Jouni, Aurélien Godet, Mauro Dalla Mura

arXiv:2609.01060v1eess.IVcs.CV

TL;DR

RGB-guided HSR must transfer spatial detail to low-resolution HSI despite unreliable close-range cross-camera correspondences. The paper combines cross-modal flow alignment with confidence-aware GSA fusion, improving Real-benchmark reconstruction over learned unaligned baselines while remaining substantially faster and supporting full 51-band VIS-NIR operation without retraining.

  • Problem

    Close-range RGB-HSI systems have locally unreliable correspondences, so RGB guidance can inject incorrect details into low-resolution HSI.

  • Method

    The framework warps RGB onto the HSI grid and uses an energy-based confidence map for weighted spectral regression and gated detail injection.

  • Results

    The method improves reconstruction over learned unaligned fusion baselines on Real while remaining substantially faster.

  • Takeaways & Limitations

    Native-device experiments validate genuine cross-sensor evaluation and demonstrate full 51-band VIS-NIR operation without retraining.

  • Takeaways & Limitations

    Confidence is an energy-based reliability score rather than a calibrated posterior probability, and unreliable correspondences can trigger local low-resolution fallbacks.

Abstract

from arXiv · show

Compact snapshot hyperspectral cameras provide rich instantaneous spectral measurements for ground-level machine vision, but at lower spatial resolution than standard RGB cameras. RGB-guided hyperspectral super-resolution (HSR) addresses this limitation by transferring spatial detail from a high-resolution RGB guide to a low-resolution hyperspectral image (HSI). These dual-camera systems are typically in a horizontal rig geometry, requiring cross-camera image alignment due to different fields of view. However, residual misregistration can inject spurious high-frequency details. Existing learned unaligned-fusion methods are usually trained for a fixed spectral support and spatial scale factors and can be computationally demanding, limiting their flexibility across sensors. We propose a lightweight and interpretable RGB-guided HSR framework combining cross-modal flow alignment with model-based Gram-Schmidt orthogonalization fusion. The method first warps the RGB guide onto the HSI grid, then estimates an energy-based confidence weight map by measuring local alignment reliability. This map is then used both in a weighted least-squares spectral regression and in a gated fusion between the super-resolved estimate and an HSI-preserving estimate. Unlike existing learned methods, the proposed framework has a low computational footprint and supports VIS-NIR spectral supports and scale factors without retraining. Experiments on the Real benchmark show that the proposed method improves reconstruction accuracy over learned fusion baselines while remaining substantially faster. On a 34-frame sequence acquired with our real RGB-HSI dual-camera setup, a reduced-resolution quantitative evaluation validates the method under genuine cross-sensor radiometric, noise, and geometric differences, while native-resolution qualitative results demonstrate deployment on the full 51-band VIS-NIR acquisition.

1 Introduction

RGB-guided HSR combines a low-resolution HSI with a high-resolution RGB image, but close-range dual-camera misalignment can corrupt transferred details. The paper couples cross-modal alignment with confidence-aware, interpretable fusion to address this problem.

  • Motivation: Compact snapshot cameras enable instantaneous spectral-cube acquisition for dynamic machine-vision applications but generally provide lower spatial resolution.Applications include robotic inspection, precision agriculture, food analysis, and non-destructive testing.
  • Motivation: RGB-guided HSR uses a high-resolution RGB image as a spatial guide for a low-resolution HSI.This configuration is standard in remote sensing, where parallax is typically negligible at scene scale.
  • Misalignment problem: Close-range RGB-HSI rigs suffer from viewpoint, optical, baseline, field-of-view, and spectral-response differences that create parallax and occlusions despite calibration.These effects make accurate pixel correspondence difficult near depth discontinuities.
  • Misalignment problem: Warped RGB details can produce ghosting, edge duplication, and spectral distortion when local registration is unreliable.The problem is especially pronounced in occluded and parallax regions.
  • Proposed direction: The proposed method couples cross-modal flow alignment with an explicit reliability map controlling both spectral regression and local detail injection.Its novelty lies in coupling CMF and GSA through confidence-aware fusion rather than using either component alone.
  • Proposed direction: The framework extends GSA with confidence-weighted least-squares regression and confidence-gated detail injection between HSI-preserving and RGB-guided estimates.The confidence map combines geometric validity with RGB-HSI consistency cues and bidirectional-flow consistency.

2 Related Work

Prior HSI super-resolution work spans single-image, fusion-based, learned unaligned, and model-based approaches. The paper focuses on real ground-level RGB-HSI pairs, where cross-sensor acquisition effects and local registration failures remain insufficiently represented by standard evaluation protocols.

  • HSI super-resolution: Single-image HSR reconstructs high-resolution HSI from a single low-resolution HSI but lacks observed high-frequency spatial information.ESSAformer is cited as an efficient transformer-based example exploiting spectral-spatial correlations.
  • HSI super-resolution: Fusion-based HSR supplements low-resolution HSI with an auxiliary high-resolution RGB, multispectral, or panchromatic image.Model-based optimization and deep networks represent complementary approaches to spatial-spectral fusion.
  • Ground-level unaligned fusion: Ground-level unaligned RGB-guided HSR requires registration because separate cameras observe different scene points under parallax and other geometric differences.HSIFN and SSC-HSR address this setting using learned alignment and feature aggregation.
  • Cross-modal alignment: Dense cross-modal flow is suited to pixel-wise correspondence under parallax and depth discontinuities that global affine transforms cannot capture.CMF adapts dense flow estimation to heterogeneous image pairs, although residual correspondences can remain locally unreliable.
  • Registration-aware fusion: Registration-aware multimodal fusion commonly combines alignment with fusion, whereas this paper explicitly uses registration confidence as a pixel-wise fusion weight.The paper targets real RGB-HSI VIS-NIR pairs rather than end-to-end learned reconstruction alone.
  • Evaluation gap: The Real protocol provides controlled quantitative evaluation but uses HSI-HSI pairs, projected RGB guides, 31 VIS bands, and manually cropped common support.The paper complements it with native RGB-HSI data from a Raspberry Pi HQ RGB camera and Ultris SR5 camera, including 34-frame reduced-resolution and full 51-band VIS-NIR evaluations.

3 Problem Formulation and Notation

The problem is to reconstruct a high-resolution HSI on the HSI grid from a low-resolution HSI and a high-resolution RGB image with unknown cross-camera correspondence. The formulation models degradation, spectral projection, dense flow warping, and spatially varying guide reliability.

  • Variables and objective: The target is a latent high-resolution HSI X on the HSI grid, reconstructed from low-resolution HSI Y and high-resolution RGB image R.The scale factor is σ = H/h = W/w.
  • Observation models: The low-resolution HSI is modeled as a spatially blurred and downsampled version of the latent high-resolution HSI.L_σ represents spatial blur and D_σ represents downsampling.
  • Observation models: The RGB observation is related to the latent HSI through an HSI-to-RGB spectral projection and an unknown coordinate map between HSI and RGB grids.The fixed projection Ψ produces an RGB proxy for the experimental pipeline.
  • Cross-camera correspondence: The unknown coordinate map is parameterized by a dense HSI-to-RGB flow field F_H→R.The flow is estimated between the upsampled HSI-derived RGB proxy and the observed RGB image.
  • Cross-camera correspondence: The RGB guide is warped onto the HSI grid as R_w(x) = R(x + F_H→R(x)).This back-warping produces the registered guide used by the reconstruction operator.
  • Reliability-aware reconstruction: The reconstruction should satisfy both the HSI degradation model and RGB-proxy agreement where correspondence is reliable.Because warped RGB reliability varies spatially, the operator uses a confidence map C as a reliability map.

4 Methodology

The method aligns RGB guidance to the HSI grid, estimates interpretable local reliability, and uses confidence-aware GSA fusion to inject spatial detail while preserving hyperspectral content.

  • Cross-modal flow alignment: Dense cross-modal flow aligns the RGB guide to the HSI grid through backward warping, while validity masks exclude samples outside the shared camera field of view.The flow can be instantiated with CMF/CrossRAFT, and backward warping uses bilinear interpolation in practice.
  • Confidence map and reliability cues: The confidence map assigns high trust only to geometrically valid warped RGB pixels that are locally consistent with the HSI-derived RGB proxy.It is an energy-based spatial reliability score rather than a learned uncertainty model.
  • Confidence map and reliability cues: RGB-proxy residuals compare bandwidth-matched images using an ℓ1 penalty, while structure residuals use smoothed local ZNCC to accommodate cross-sensor radiometric differences.Flat or low-texture regions are treated as uninformative, and bidirectional-flow disagreement can identify inconsistent correspondences.
  • Confidence-weighted Gram-Schmidt adaptive fusion: The confidence-aware GSA estimates spectral relations by penalized multivariate regression, then transfers RGB high-frequency residuals through HSI-derived band-wise gains.Tikhonov regularization stabilizes regression in the presence of snapshot-HSI high-frequency noise, while HSI bands remain anchored to the upsampled HSI.
  • Confidence-gated detail injection: A spatially varying convex gate blends the HSI-preserving estimate with the confidence-weighted GSA estimate, attenuating RGB detail where correspondence is unreliable.Confidence is broadcast across spectral bands; zero confidence falls back to the upsampled HSI, whereas one uses the RGB-guided estimate.

5 Experiments

Experiments evaluate the proposed confidence-gated CMF-GSA pipeline on the Real benchmark and a genuine Ultris-RPi dual-camera sequence. The method improves reconstruction metrics over learned baselines, remains memory-efficient, and supports full 51-band VIS-NIR evaluation without retraining.

  • Datasets and protocol: The Real benchmark uses 57 paired hyperspectral acquisitions with real non-rigid misalignment and evaluates all methods on common inputs and valid registered support.The reference HSI is spatially degraded after one HSI is spectrally projected to simulate the RGB guide.
  • Datasets and protocol: The Ultris-RPi experiment uses a genuine RGB-HSI dual-camera sequence with different fields of view, resolutions, sensor responses, and noise characteristics.Because simultaneous high-resolution HSI ground truth is unavailable, the sequence is evaluated quantitatively at reduced resolution and qualitatively at native resolution.
  • Real benchmark results: At ×4, CMF-GSA-wg achieves the best Real-test RMSE 0.0168, mPSNR 37.34 dB, mSSIM 0.94, and SAM 2.17° while outperforming learned baselines.At ×8, it again achieves the best four reconstruction metrics, including RMSE 0.0257 and SAM 2.94°.
  • Real benchmark results: The final two-expert gate provides the main improvement: at ×4, CMF-GSA-wg reduces RMSE by about 33% relative to CMF-GSA-w.Weighted regression alone produces only marginal RMSE and mPSNR changes, despite improving SAM and mSSIM.
  • Efficiency and ablations: CMF-GSA-wg has the lowest peak GPU memory at 0.69 GiB, although all 39.76M learned parameters belong to the frozen CMF registration backbone.The confidence estimation and SR/fusion stages add no learned parameters.
  • Efficiency and ablations: Removing the RGB residual cue degrades all four metrics, whereas removing cycle consistency leaves metrics nearly tied while roughly halving runtime.This identifies RGB-HSI proxy consistency as the dominant cue and cycle consistency as an auxiliary diagnostic.
  • Ultris-RPi results: On the 34-frame Ultris-RPi sequence, CMF-GSA-wg outperforms SSC-HSR across all four metrics on the common 31-band VIS support.Native-resolution results show direct operation on the full 51-band VIS-NIR cube without retraining, but no reference-based metrics are available there.

6 Discussion

The confidence gate provides the main reconstruction gain on Real by suppressing unreliable RGB detail, while the framework remains flexible on genuine RGB-HSI data and full 51-band VIS-NIR inputs. Its reliability score remains heuristic and can fall back locally to lower-resolution HSI content when correspondence is unavailable.

  • Real benchmark: The final confidence gate, rather than confidence-weighted regression alone, provides the main reconstruction gain on Real.The gate locally suppresses unreliable high-frequency RGB residuals, while weighted least squares modifies global RGB-HSI spectral calibration.
  • Qualitative comparison: At ×4 and ×8, CMF-GSA-wg reduces local RMSE in highlighted regions, while SSC-HSR shows more over-enhanced or spatially shifted structures at ×8.The qualitative comparison uses RGB previews, RMSE maps, and SAM maps on the Real test set.
  • Real-device evaluation: 34 frames from the Ultris-RPi setup validate the method under genuine cross-sensor differences, while native-resolution results operate on full 51-band VIS-NIR data without retraining.The learned baseline remains restricted to its 31-band VIS setting.
  • Limitations: The energy-based confidence is not a calibrated posterior probability and may confuse radiometric mismatch with geometric misregistration.A ZNCC-based structure cue reduces radiometric sensitivity on real-device data but remains heuristic.
  • Limitations: Confidence gating can miss RGB details without reliable correspondence, producing local low-resolution fallbacks.Stronger HSI-only priors or fallback super-resolution models may improve large unreliable regions.

7 Conclusion

The paper presents a lightweight, interpretable framework for RGB-guided hyperspectral super-resolution under real cross-resolution misalignment. It improves over learned unaligned fusion on Real while remaining substantially faster, and supports native full 51-band VIS-NIR deployment without retraining.

  • 7 Conclusion: The framework combines cross-modal flow, Gram-Schmidt fusion, an explicit reliability map, confidence-weighted regression, and gated detail injection.
  • 7 Conclusion: On Real, the method improves over learned unaligned fusion baselines while remaining substantially faster, with the final gate providing the main reconstruction gain.
  • 7 Conclusion: A 34-frame Ultris-RPi evaluation validates the method under real cross-sensor differences, while native-resolution results demonstrate full 51-band VIS-NIR deployment without retraining.
Loading 2609.01060v1…