PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Neural Networks for Spectral Snapshot Compressive Imaging

Ziyi Meng, Zhenming Yu, Kun Xu, Xin Yuan

arXiv:2108.12654v1eess.IVcs.CV

TL;DR

Spectral SCI reconstructs high-dimensional cubes from compressed measurements, while supervised networks depend on training data that may be limited or mismatched to real spectra. This paper integrates untrained neural-network priors into plug-and-play reconstruction, reporting competitive results without training data and state-of-the-art results with a pre-trained denoising prior.

  • Problem

    Supervised spectral SCI reconstruction relies on training data that can be limited and may not cover spectra encountered in real data.

    Source: §1.1 · p. 2

  • Method

    The method learns an untrained neural-network prior from compressed measurements and integrates it into plug-and-play reconstruction.

    Source: §1 · p. 1, §1.1 · p. 2, §5 · p. 8

  • Results

    The joint method with a pre-trained HSI denoising prior achieves state-of-the-art results on synthetic and real datasets from different spectral SCI systems.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The untrained method provides competitive spectral SCI reconstruction without training data, while adding a pre-trained denoising prior yields state-of-the-art results.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

We consider using {\bf\em untrained neural networks} to solve the reconstruction problem of snapshot compressive imaging (SCI), which uses a two-dimensional (2D) detector to capture a high-dimensional (usually 3D) data-cube in a compressed manner. Various SCI systems have been built in recent years to capture data such as high-speed videos, hyperspectral images, and the state-of-the-art reconstruction is obtained by the deep neural networks. However, most of these networks are trained in an end-to-end manner by a large amount of corpus with sometimes simulated ground truth, measurement pairs. In this paper, inspired by the untrained neural networks such as deep image priors (DIP) and deep decoders, we develop a framework by integrating DIP into the plug-and-play regime, leading to a self-supervised network for spectral SCI reconstruction. Extensive synthetic and real data results show that the proposed algorithm without training is capable of achieving competitive results to the training based networks. Furthermore, by integrating the proposed method with a pre-trained deep denoising prior, we have achieved state-of-the-art results. {Our code is available at \url{https://github.com/mengziyi64/CASSI-Self-Supervised}.}

1. Introduction

Spectral snapshot compressive imaging (SCI) reconstructs a high-dimensional data cube from compressed measurements, but supervised reconstruction is constrained by scarce training data. The paper integrates untrained neural-network priors into plug-and-play reconstruction and reports competitive results without training data, with state-of-the-art results when combined with a pre-trained denoising prior.

1. Introduction

  • SCI uses optical multiplexing to capture a 3D spectral or temporal data cube with a 2D detector, leaving a software decoder to reconstruct the cube.Source: §1 · p. 1

1.1. Motivation

  • Limited hyperspectral training data can leave supervised networks unable to reconstruct spectra absent from their training sets.The authors cite spectral errors on Bird data as an example of mismatch between training data and real data.Source: §1.1 · p. 2

1.2. Contributions

  • The proposed self-supervised framework learns a prior from measurements and combines it with plug-and-play reconstruction, without requiring training data.The network is optimized as plug-and-play iterations proceed.Source: §1.1 · p. 2, §1.2 · p. 2
  • The method is reported robust to Poisson noise in real measurements and competitive with supervised methods after per-dataset parameter tuning.The authors also report state-of-the-art results after integrating a pre-trained hyperspectral denoising prior.Source: §1.2 · p. 2

1.3. Related Work

  • Earlier SCI methods include handcrafted priors and supervised CNNs, whose speed, data needs, and system specificity pose different constraints.Iterative methods can be slow at large scale, while trained CNNs require substantial training and may need retraining for different systems.Source: §1.3 · p. 2
  • Unlike DeepRED, the method combines DIP with implicit conventional priors, uses two fidelity terms, and is evaluated on synthetic and real spectral SCI data.The authors report experimentally better results than DeepRED; a pre-trained HSI denoising prior is also identified as another PnP approach.Source: §1.3 · p. 3

2. Spectral SCI System

The spectral SCI system modulates a data cube with a coded aperture, disperses its spectral channels, and integrates them into a 2D measurement. Reconstruction then uses the captured measurement and calibrated sensing matrix to recover the cube.

2. Spectral SCI System

  • CASSI modulates the spectral cube with a coded aperture, disperses the channels, and integrates them into a 2D camera measurement.The coded aperture is a fixed mask, and the dispersing element shifts channels along the dispersion direction.Source: §2 · p. 3
  • Each spectral channel is shifted spatially according to its wavelength relative to the center wavelength.Source: §2 · p. 3
  • The sensor measurement is the sum of the tilted spectral channels plus measurement noise, yielding a compressed image.The cube is integrated across the designed wavelength range.Source: §2 · p. 3
  • Reconstruction seeks the cube from the captured measurement and calibrated sensing matrix, but the system's structured sensing matrix limits direct applicability of many CS theories.Source: §2 · p. 3

3. Methods

Section 3 formulates spectral SCI reconstruction with priors and develops a self-supervised framework that combines DIP with conventional priors. It solves the joint reconstruction and network-learning problem through alternating optimization.

3. Methods

  • The reconstruction objective balances measurement fit against R(x), while existing spectral SCI priors include hand-crafted TV, sparsity, and low-rank penalties.Source: §3 · p. 3

3.1. Deep Image Prior (DIP)

  • DIP represents x as TΘ(e), learning measurement-specific Θ from y without ground truth; implicit network regularization and early stopping avoid overfitting.Source: §3.1 · p. 4

3.2. Proposed Joint Framework

  • DIP is paired with a conventional prior because DIP alone may miss spectral information; the joint framework couples x and TΘ(e) while enforcing measurement fidelity.Source: §3.1 · p. 4, §3.2 · p. 4
  • ADMM splits the coupled optimization into subproblems, and the x update has a closed-form solution computed efficiently using H's structure and Woodbury.Source: §3.2 · pp. 4–5
  • The DIP network uses measurement loss plus proximity to x−b, serving both denoising and measurement fit; the alternating loop also updates the denoiser and auxiliary variables.Source: §3.2 · pp. 4–5

4. Results

The section evaluates PnP-DIP and its HSI-denoiser variant on synthetic and real spectral SCI data against optimization and supervised methods. PnP-DIP-HSI reaches state-of-the-art PSNR and SSIM on synthetic data, while real-data reconstructions show spatial and spectral gains.

4.1. Results on Synthetic Data

  • PnP-DIP uses a U-net with a fixed random input per measurement, resets network parameters each ADMM iteration, and increases DIP iterations later.The implementation uses early stopping, with longer DIP runs in later ADMM iterations.Source: §4.1 · pp. 5–6
  • PnP-DIP improves PSNR by 0.33dB over TSA-Net but has lower SSIM; PnP-DIP-HSI reaches state-of-the-art PSNR and SSIM.Table 1 reports PSNR in dB and SSIM for 10 synthetic scenes.Source: §4.1 · pp. 6–7, Table 1 · p. 6
  • On Scene 9, PnP-DIP shows sharper edges and higher spectral-curve correlation; on Scenes 1 and 7, PnP-DIP-HSI mitigates detail loss and noise.Figure 3 compares eight algorithms; Figure 4 compares three algorithms on two scenes.Source: Figure 3 · p. 6, Figure 4 · p. 6, §4.1 · p. 7

4.2. Ablation Study

  • 30.48dB and 0.854: PnP-DIP’s average PSNR and SSIM exceed the deep decoder’s 28.44dB and 0.819.The authors suggest the gap may reflect the deep decoder’s smaller parameter count and the 3D spectral-cube reconstruction task.Source: §4.2 · p. 7
  • 26.99dB: sole DIP averages this PSNR, 3.5dB below proposed PnP-DIP; single-fidelity PnP-DIP is 1.6dB lower.Table 2 compares sole DIP, single-fidelity PnP-DIP, and the proposed double-fidelity PnP-DIP.Source: §4.2 · p. 7, Table 2 · p. 7
  • 2.75dB and 2.61dB: PnP-DIP and PnP-DIP-HSI degradation, respectively, versus 6.00dB for TSA-Net at SNR=25dB.The authors report less performance degradation for their methods under Poisson noise.Source: §4.2 · p. 7, Table 3 · p. 7

4.3. Real Data Results

  • On real CASSI and endomicroscopy data, the proposed method shows fewer artifacts, higher spectral accuracy, higher spatial resolution, and cleaner details than comparison methods.The reported real datasets include Lego plant, Strawberry, Object, Resolution target, and Red blood cell.Source: §4.3 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

5. Conclusions

The paper integrates an untrained neural-network prior into a plug-and-play framework for spectral SCI reconstruction. It reports competitive results without training data and state-of-the-art results when combined with a pre-trained HSI denoiser.

5. Conclusions

  • The PnP-DIP framework learns a prior from compressed measurements without training data and achieves results competitive with supervised deep-learning algorithms.Combining it with a pre-trained HSI denoiser yields state-of-the-art results on synthetic and real datasets from different spectral SCI systems.Source: §5 · pp. 8–9
  • The joint reconstruction using a pre-trained HSI denoising prior achieves state-of-the-art results on synthetic and real datasets from different spectral SCI systems.The untrained-network framework is solved using ADMM within the plug-and-play regime.Source: §5 · p. 9
  • The authors identify video SCI reconstruction as a possible extension of the self-supervised framework.This is stated as future work.Source: §5 · p. 9

6. Derivation of Single Fidelity Formulation

For the single-fidelity formulation, ADMM splits the problem into iterative subproblems for reconstruction, network parameters, and the auxiliary variable. The network update combines measurement fitting with proximity to the current reconstruction estimate.

6. Derivation of Single Fidelity Formulation

  • ADMM frames the formulation as separate subproblems, including denoising for x and optimization of Θ against measurement and proximity terms.The x subproblem is expressed as a denoising problem with prior R(x).Source: §6 · p. 11
  • The x update is a denoising step, applying the selected operator to the network output plus the auxiliary variable.The denoiser's estimated noise level depends on λ/μ.Source: §6 · p. 11
  • The Θ update minimizes measurement error and proximity to x−b, so the learned network output both denoises x−b and fits the measurement.For the implemented U-net, both terms in the objective are used as the loss.Source: §6 · p. 11
  • The auxiliary variable is updated by subtracting the difference between x^k and the current network output from b^k.The three ADMM steps are performed iteratively, and some can include inner loops such as Θ optimization.Source: §6 · p. 11

7. Ablation Study Results

The ablations compare network priors, fidelity terms, an added TV prior, and U-net sampling choices. The reported findings favor DIP over the deep decoder for detailed regions, double fidelity over single fidelity, and ConvTranspose with average-pooling.

7.1. DIP vs. Deep Decoder

  • PnP-DD reconstructs some smooth regions well but produces significant artifacts and oversmoothing in regions with many spatial details.The authors suggest this may be due to the deep decoder's limited number of network parameters; the comparison uses four synthetic examples with three channels.Source: §7.1 · p. 11, Figure M1 · p. 11

7.2. Single Fidelity vs. Dual Fidelity in DIP

  • Double-fidelity PnP-DIP produces clearer details with less noise and fewer artifacts than sole DIP or single-fidelity PnP-DIP.The comparison is shown on two synthetic examples with five spectral channels.Source: §7.2 · p. 11, Figure M2 · p. 12

7.3. Incorporating DIP with TV Prior

  • PnP-DIP-TV averages 30.44 dB and 0.852 SSIM, close to PnP-DIP at 30.48 dB and 0.854 SSIM.The authors report that TV helps real-data reconstruction, possibly because of measurement noise, while final results rely on DIP.Source: §7.3 · p. 12

7.4. Choice of the Up/downsampling

  • The DIP network with ConvTranspose performs best among the tested upsampling choices, while average-pooling outperforms max-pooling.These choices are used in the experiments; Table M1 reports average PSNR and SSIM for the tested configurations.Source: §7.4 · p. 12, Table M1 · p. 12

8. Supplementary Results

The supplementary results examine comparisons with PnP-HSI and Autoencoder, synthetic-data benchmarks, and reconstructions across several real-data sets. In the reported large-shift settings, PnP-DIP outperforms PnP-HSI, while the other supplied comparisons describe visual or dataset scope without reporting detailed outcomes.

8.1. PnP-DIP vs. PnP-HSI

  • With small mask shifts, PnP-DIP and PnP-HSI give similar results; with large shifts, PnP-DIP leads by 4.36 dB and 3.82 dB on the two datasets.The supplementary table and figure compare the methods under two simulation settings, and the large-shift pattern is also reported for real data.Source: §8.1 · pp. 12–13, Table M2 · p. 13, Figure M3 · p. 13

8.2. PnP-DIP vs. Autoencoder

  • On the datasets with a binary mask and 30-pixel shift, PnP-DIP achieves much better visual results than ADMM-TV and Autoencoder.The authors attribute Autoencoder's spatial blur to the single-disperser CASSI model differing from the dual-disperser model mainly used in.Source: §8.2 · p. 13, Figure M5 · p. 13

8.3. Results on the Synthetic Data

  • The synthetic-data comparison covers ten scenes, reconstructing 10 of 28 spectral channels with PnP-DIP and PnP-DIP-HSI against TSA-Net, with PSNR and SSIM reported.The accompanying figures show reconstructed results for the scenes.Source: §8.3 · p. 13, Figure M6 · p. 15, Figure M7 · p. 15

8.4. Results on the Real Data

  • Real-data comparisons span CASSI and endomicroscopy datasets, contrasting PnP-DIP and PnP-DIP-HSI with methods including ADMM-TV, TSA-Net, TwIST, PnP-HSI, and supervised networks.The supplied figures include reconstructions from both CASSI datasets and endomicroscopy scenes, generally displaying 10 spectral channels.Source: §8.4 · pp. 13–14, Figure M16 · p. 18, Figure M20 · p. 19
Loading 2108.12654v1…