PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperspectral Diffusion Equivariant Imaging (HyDiff-EI): A Self-supervised Framework for Hyperspectral Image Inpainting

Shuo Li, Mike Davies, Mehrdad Yaghoobi

arXiv:2608.26812v1cs.CVcs.LG

TL;DR

HSI inpainting must recover missing content from corrupted observations without relying on large-scale pretraining. HyDiff-EI performs self-supervised diffusion optimization with equivariant and noise-aware priors, achieving strong results across real-world datasets, especially under noiseless and low-to-moderate noise.

  • Problem

    Large-scale pretraining is costly for HSI inpainting, while the ill-posed reconstruction problem requires effective priors and must accommodate varied sensor configurations.

  • Method

    HyDiff-EI uses an untrained equivariant imaging network within a diffusion framework, with noise-aware scaling for noisy observations.

  • Results

    HyDiff-EI achieves superior overall performance in noiseless and low-to-moderate noise settings, remains competitive under severe noise, and achieves the best performance on the reported EMIT sample across four metrics.

  • Takeaways & Limitations

    Coupling diffusion with equivariant priors provides a flexible self-supervised approach for real-world HSI inpainting across Chikusei, Botswana, and EMIT.

Abstract

from arXiv · show

A novel Hyperspectral diffusion Equivariant Imaging (HyDiff-EI) framework for solving the hyperspectral image (HSI) inpainting problem has been presented here. Unlike conventional diffusion-based methods that rely on large-scale pretraining, HyDiff-EI is a test-time optimization framework that learns directly from a single corrupted HSI acquisition. This makes it flexible for different sensor configurations and particularly well-suited for practical remote sensing scenarios where large annotated hyperspectral datasets are limited. To address the ill-posed nature of unsupervised inpainting, we embed equivariant consistency constraints within the diffusion process. By leveraging the inherent geometric symmetries and intrinsic characteristics of HSIs, HyDiff-EI bridges the gap between generative diffusion modeling and self-consistent physical priors. We empirically show that coupling diffusion modeling with equivariant priors substantially enhances noise robustness and generalizability. Extensive experiments on real-world datasets including Chikusei, Botswana, and EMIT demonstrate that HyDiff-EI offers remarkable inpainting quality over existing self-supervised and diffusion-based algorithms in both noiseless and noisy cases.

I. INTRODUCTION

HSI inpainting addresses incomplete or corrupted spatial-spectral observations, while HyDiff-EI combines self-supervised diffusion with equivariant imaging priors to avoid pre-trained diffusion models and exploit geometric structure.

  • Motivation: HSI inpainting fills missing pixels or bands caused by sensor failures and data transmission errors.The task supports data completeness, analysis, and visualization.
  • Motivation: Self-supervised methods provide flexibility across datasets and generalize to unseen data without repeatedly training large models.This is especially useful because hyperspectral data collection and large-scale model training can be expensive.
  • Research gap: Existing diffusion inpainting methods depend heavily on pre-trained models and large-scale hyperspectral training datasets.This dependence can be computationally demanding and impractical for HSI inverse problems.
  • Proposed approach: HyDiff-EI is a self-supervised diffusion framework that learns directly from measurements, uses EI priors based on physical or geometric symmetries, and accommodates different spectral-band configurations.Unlike DHP-based methods, it does not rely solely on network architecture or pre-trained diffusion models.
  • Contributions: The paper introduces EI within a discrete-time diffusion framework for HSI inpainting and reports improved performance from integrating diffusion with EI.The authors describe this as the first such integration and report gains over conventional EI in noiseless and noisy settings.
  • Equivariant imaging: The EI constraint trains reconstructions so that transformed network outputs agree with re-estimated outputs obtained after masked re-injection.The workflow applies a group transformation to an estimated clean image and penalizes the discrepancy between the two outputs.

B. Diffusion Models

DDPM combines a forward process that progressively adds Gaussian noise with a backward process that iteratively denoises from random noise toward clean data. The EI workflow associates transformed and re-estimated outputs with the two sides of its consistency equation.

  • Forward process: DDPM's forward process gradually adds Gaussian noise over T time steps according to a predefined variance schedule.The process is represented as a Markov chain.
  • Backward process: The backward process starts from random noise xT and generates the preceding state xt−1 through iterative denoising.The model uses predicted posterior statistics in the reverse transitions.
  • EI workflow: In the EI workflow, x̃′ and x̃′′ correspond respectively to the transformed network output and the re-estimated output on the two sides of Eq. (5).The figure caption identifies these outputs as the right-hand and left-hand sides of the equation, respectively.
  • Sampling: Clean data x0 is reconstructed by repeatedly sampling reverse transitions from t = T down to t = 0.This iterative procedure uses the learned reverse conditional distribution.

C. Proposed Method

HyDiff-EI replaces the conventionally trained DDPM denoiser with a self-supervised equivariant imaging network optimized during diffusion. It combines data fidelity, equivariant constraints, and DDNM-style range/null-space refinement to reconstruct hyperspectral images.

  • Self-supervised diffusion design: HyDiff-EI trains an equivariant imaging network directly on the corrupted HSI instead of using a conventionally pretrained diffusion denoiser.The EI network directly estimates clean data at each diffusion time step.
  • Self-supervised diffusion design: The modified EI loss includes a data-fidelity term and equivariant consistency terms computed at each diffusion time step.The network output is substituted into the diffusion posterior mean after parameter updates.
  • Reverse diffusion reconstruction: The estimated clean image combines the observed range-space component with EI-generated null-space content during reverse diffusion sampling.This decomposition is the core DDNM mechanism, with HyDiff-EI refining the null-space contents using EI.
  • Reverse diffusion reconstruction: Algorithm 1 diffuses the corrupted HSI, updates EI parameters by minimizing the modified loss, re-estimates the clean image, and samples the preceding diffusion state.These operations repeat for t = T, ..., 1 while the corrupted input remains fixed.

D. Extension to Noisy HSI Inpainting

For noisy HSI inpainting, HyDiff-EI modifies the re-estimation and output steps to reduce reliance on unreliable observations while maintaining consistency with the diffusion noise schedule. The scaling matrix controls range-space corrections according to estimated signal and noise strengths.

  • Noise-aware re-estimation: The scaling matrix Σt is constructed from the singular value decomposition of the inpainting mask and scales the range-space correction at each time step.Its formulation uses singular values of M and adapts the correction to the estimated HSI.
  • Noise-aware re-estimation: In noisy observations, HyDiff-EI scales the range-space correction so that unreliable observation information receives less weight during later algorithm stages.When the scaling matrix is I, the noisy formulation reduces to the noiseless re-estimation step.
  • Noise-aware re-estimation: For inpainting, the mask is row-orthonormal, allowing its singular values to simplify to 1 or 0 while retaining the general operator formulation.The unsimplified notation supports extension beyond inpainting operators.
  • Noise-aware output sampling: An extra noise term ϵcon is introduced so the estimated image’s total noise variance at each update conforms to the forward diffusion noise level.The observation noise strength and current-estimate noise strength determine this noise-aware adjustment.

III. EXPERIMENTS

HyDiff-EI is evaluated on Chikusei, Botswana, and EMIT using pixel, structural, and spectral metrics, alongside visual and spectral-signature comparisons. Results show strong edge consistency, spectral fidelity, and performance across noiseless and noisy settings, with reduced advantage under severe noise.

  • Datasets and comparison: Experiments use Chikusei, Botswana, and real-time EMIT hyperspectral datasets, comparing HyDiff-EI with representative self-supervised and diffusion-based inpainting methods.DHP, R-DLRHyIn, DDS2M, HIR-Diff, and SHARE are included; several methods share a 2D Skip-Net backbone for fairness.
  • Evaluation metrics: MPSNR and MSSIM measure pixel and spatial accuracy, while SAM and SCC assess spectral fidelity against ground-truth spectra.Numerical results are reported for noiseless and noisy cases, with visual comparisons in Figures 3 and 4.
  • Visual comparison: HyDiff-EI better preserves consistency between generated content and surrounding edges than DDS2M and HIR-Diff, including when one-third of input pixels are missing.Under σy = 0.1 noise, DHP struggles or fails, while HyDiff-EI shows less edge smoothing than R-DLRHyIn and comparable competitors.
  • Noisy evaluation: HyDiff-EI achieves the best overall performance at σy = 0.1 and 0.2, while SHARE surpasses it at σy = 0.3 across MPSNR, MSSIM, SAM, and SCC.The narrowing performance gap indicates stronger SHARE robustness under severe noise and greater HyDiff-EI effectiveness under low-to-moderate noise.
  • Spectral fidelity: HyDiff-EI follows ground-truth spectra more closely at central and boundary missing pixels and preserves spectral shape more faithfully under noise.These observations align with strong SAM and SCC results and indicate preservation of material-dependent spectral signatures.
  • Real-time EMIT validation: 35.482 MPSNR, 0.933 MSSIM, 2.095° SAM, and 0.982 SCC are achieved by HyDiff-EI on the EMIT sample, leading all four reported metrics.It also preserves more fine-scale mountain structures within missing strips than DDS2M, HIR-Diff, and SHARE.

IV. ABLATION STUDIES

The ablations show that diffusion improves EI-based inpainting, equivariant consistency is most effective at αEI = 1, and spatial transformations such as shifting and rotation perform best. HyDiff-EI also offers a favorable restoration-quality and runtime trade-off, while physically meaningful spectral transformations remain open for future work.

  • Effect of diffusion: Diffusion steadily improves EI performance in noiseless inpainting and narrows the MPSNR gap with Robust-EI as noise increases while improving MSSIM.The authors attribute this to posterior sampling, which preserves sharper and more realistic structural details than MMSE-like point estimates.
  • Time efficiency: 38.54 s/image gives HyDiff-EI the best restoration performance with only a modest runtime overhead over DHP.It is approximately 26% faster than SHARE and more than four times faster than DDS2M.
  • Equivariance strength: αEI = 1 consistently provides the optimal balance between equivariance consistency and data fidelity, whereas αEI = 0 causes severe performance collapse under noise.The parameter controls the relative weight of the equivariance-consistency loss.
  • Transformation groups: Shifting and Rotation achieve the highest and most stable performance among the six evaluated transformations.Their effectiveness is consistent with reconstructing missing regions from local spatial context while preserving spectral consistency across bands.
  • Future directions: More effective noise-aware regularization and physically meaningful spectral-domain transformation groups remain future research directions.The conclusion identifies these extensions while reporting strong performance in noiseless and low-to-moderate noise settings and competitiveness under severe noise.

SUPPLEMENTARY MATERIAL: LOSS FUNCTION DETAILS

The supplementary derivation formulates the diffusion objective as a variational upper bound and connects its posterior-mean matching objective to the EI network output. It adds an equivariant consistency term weighted by αEI and explains which terms can be omitted or approximated.

  • Objective derivation: The EI network estimates clean data x0 from observation y through the posterior mean estimator.The derivation substitutes the network estimate into the diffusion posterior-mean formulation.
  • Variational bound: The diffusion objective is constructed as a variational upper bound whose components include KL divergences and boundary terms.Bayes’ theorem and the Markov structure factorize the bound, while DKL denotes Kullback–Leibler divergence.
  • Simplifications and loss choice: With fixed diffusion variance, LT and L0 are independent of training parameters and can be ignored, while learned variance would make them nonconstant.The derivation also notes that the simple loss produced better sample quality than the weighted loss in experiments.
  • Posterior-mean matching: The remaining KL term optimizes the diffusion posterior mean µq(xt, x0) to match the model mean µθ(xt, t).The clean-data term is replaced using the EI network’s estimate and the stated posterior-mean formulations.
  • Equivariant constraint: An equivariant constraint compares transformed and re-estimated network outputs, with its loss weighted by αEI.The transformed output and its re-estimation are introduced to enforce equivariance during training.

NOISY CONDITIONS

For noisy HSI inpainting, the supplementary derivation introduces observation noise into the diffusion updates and designs a consistency term to keep the total noise aligned with the diffusion schedule. The scaling matrix Σt controls how strongly noisy observations contribute at each step.

  • Noise model: The noisy-inpainting derivation assumes additive white Gaussian noise with known strength σy in the observation.The modified re-estimation and re-sampling steps are derived under this observation model.
  • Mask and re-estimation: The inpainting mask is decomposed by singular value decomposition to construct its pseudoinverse for re-estimation.The resulting decomposition is combined with the SVD of Σt in the update expression.
  • Noise injection: The observation contributes both a noiseless-recovery term and an introduced-noise term to the re-estimation update.The introduced noise is subsequently incorporated into the DDPM backward pass.
  • Consistency correction: ϵcon is constructed because the combined noise in the update may exceed the predefined diffusion noise level.Its design uses the observation-dependent introduced noise and the scaling matrix Σt.
  • Scaling design: Σt is chosen to keep each update’s noise variance bounded by the scheduled level σt while adapting the observation contribution to current noise.When current noise is large, Σt should remain close to I to use observation information; otherwise it is scaled using a thresholding rule.

SUPPLEMENTARY MATERIAL: OFFLINE TRAINING

HyDiff-EI can be extended from test-time single-image optimization to offline training on corrupted measurements. Training on prior measurements reduces inference optimization time while retaining the diffusion process at test time.

  • Offline-training extension: HyDiff-EI is not restricted to per-image test-time optimization and can be pretrained on corrupted HSI measurements.The supplementary experiment trains the EI network using 300 corrupted Chikusei samples.
  • Model variants: The pretrained EI network is integrated into the diffusion framework as HyDiff-EI (trained), distinct from the original HyDiff-EI (untrained).The untrained model performs test-time optimization, whereas the trained variant follows offline training on corrupted measurements.
  • Evaluation setup: Table VIII compares the untrained model with variants trained on 50 and 300 corrupted HSI samples on a 144 × 144 × 128 Chikusei sample.Figure 7 provides the corresponding visual comparison of trained and untrained models.
  • Inference efficiency: 15.15 seconds of remaining inference time is attributed to processing the full 1000-step diffusion procedure.Prior measurement training reduces optimization time but does not remove the diffusion-process cost.
  • Practical implication: Given sufficient corrupted training measurements, HyDiff-EI produces satisfactory test-time results and offers flexibility for real-world deployment.The supplementary results support adapting the framework beyond single-image optimization.
Loading 2608.26812v1…