Source-linked AI summary
Plug-and-Play Algorithms for Large-scale Snapshot Compressive Imaging
Xin Yuan, Yang Liu, Jinli Suo, Qionghai Dai
TL;DR
Large-scale SCI reconstruction is limited by methods that are too slow or insufficiently flexible to encoding changes. The paper develops PnP-ADMM and lower-workload PnP-GAP algorithms with deep denoising priors and convergence analysis under SCI hardware constraints. It reports recovery of FHD and UHD color videos from snapshot measurements, including a 3840 × 1644 × 48 video with PSNR above 30dB.
Problem
Existing SCI reconstruction methods are challenged by the speed required for large-scale HD, FHD, and UHD videos, while end-to-end networks are not flexible to encoding changes.
Method
The paper extends PnP-ADMM to SCI and proposes lower-workload PnP-GAP, integrating bounded and deep denoisers under SCI hardware constraints.
Results
A UHD color video of size 3840 × 1644 × 48 is recovered from a snapshot measurement with PSNR above 30dB, alongside results on simulation and real datasets.
Takeaways & Limitations
PnP reconstruction provides efficient, flexible SCI algorithms that achieve results on-par with DeSCI while substantially reducing computational time.
Takeaways & Limitations
The previously reported global-convergence proof of PnP-GAP contained an error, so this version supplies another convergence proof.
Abstract
from arXiv · showhide
Snapshot compressive imaging (SCI) aims to capture the high-dimensional (usually 3D) images using a 2D sensor (detector) in a single snapshot. Though enjoying the advantages of low-bandwidth, low-power and low-cost, applying SCI to large-scale problems (HD or UHD videos) in our daily life is still challenging. The bottleneck lies in the reconstruction algorithms; they are either too slow (iterative optimization algorithms) or not flexible to the encoding process (deep learning based end-to-end networks). In this paper, we develop fast and flexible algorithms for SCI based on the plug-and-play (PnP) framework. In addition to the widely used PnP-ADMM method, we further propose the PnP-GAP (generalized alternating projection) algorithm with a lower computational workload and prove the convergence of PnP-GAP under the SCI hardware constraints. By employing deep denoising priors, we first time show that PnP can recover a UHD color video ($3840\times 1644\times 48$ with PNSR above 30dB) from a snapshot 2D measurement. Extensive results on both simulation and real datasets verify the superiority of our proposed algorithm. The code is available at https://github.com/liuyang12/PnP-SCI.
1. Introduction
The paper targets SCI reconstruction at large video scales, where existing methods face speed or encoding-flexibility limitations. It develops convergent, plug-and-play algorithms that combine computational efficiency, reconstruction quality, and flexibility across sensing processes.
- Motivation: Existing SCI methods struggle with large-scale video because DeSCI is slow, while end-to-end networks are inflexible to encoding changes.DeSCI requires more than one hour even for a 256 × 256 × 8 video, motivating alternatives for HD, FHD, and UHD applications.
- Contributions: PnP-ADMM is extended to SCI, with fixed-point convergence established under the hardware constraints and sensing-matrix structure.The analysis exploits the special SCI sensing matrix rather than treating the problem as generic compressive sensing.
- Contributions: PnP-GAP uses bounded denoisers within generalized alternating projection, reducing computational workload relative to PnP-ADMM while retaining a convergence guarantee under proper assumptions.The method is designed as a faster alternative within the same plug-and-play framework.
- Large-scale reconstruction: A FHD video of size 1920 × 1080 × 24 is recovered within 2 minutes with PSNR close to 30dB using one GPU and a normal computer.Compared with an end-to-end network, no retraining is required; the paper also reports UHD recovery at PSNR above 30dB.
- Large-scale reconstruction: A UHD color video of size 3840 × 1644 × 48 is reconstructed from a snapshot measurement with PSNR above 30dB, while PnP-FFDNet takes 55 minutes in the illustrated comparison.GAP-TV takes 180 minutes, and the other listed methods require more than 12 hours in Figure 1.
- Evaluation: Experiments on simulation and real SCI-camera datasets report results on-par with DeSCI while substantially reducing computational time and maintaining robustness to different coding processes.The framework is presented as suitable for adaptive sensing because denoisers can be integrated without tying the reconstruction to one encoding process.
2. Mathematical Model of SCI
Video SCI modulates multiple video frames with sensing masks and compresses them into one 2D measurement. Its sparse sensing structure yields a sampling rate of 1/B, while color reconstruction independently handles Bayer channels before demosaicing.
- Sensing process: A B-frame video X is modulated by B sensing matrices and compressed into a 2D measurement frame Y.The measurement includes additive noise Z and combines masked video frames through element-wise multiplication.
- Sensing structure: The SCI sensing matrix H is sparse and formed by concatenating diagonal matrices derived from the sensing masks.For each frame, Db is the diagonal matrix of the vectorized mask Cb.
- Sensing structure: The SCI sampling rate is 1/B, and reconstruction error remains bounded even when B > 1.This follows from the structured sensing matrix used for SCI.
- Color reconstruction: For color video, Bayer RGGB channels are reconstructed independently before demosaicing produces the final RGB video.Demosaicing the mask-modulated grayscale measurement directly cannot generate the correct color.
3. Plug-and-Play ADMM for SCI
PnP-ADMM incorporates an off-the-shelf denoiser into ADMM for SCI reconstruction while exploiting SCI’s structured sensing model. Under bounded-denoiser and hardware-related assumptions, its iterates converge to a fixed point.
- Model: SCI reconstruction minimizes a quadratic forward-model loss ∥y − Hx∥2^2 together with a prior g(x), typically acting as a regularizer.The loss is f(x) = 1/2∥y − Hx∥2^2.
- PnP-ADMM formulation: PnP-ADMM replaces one ADMM subproblem with an off-the-shelf denoiser Dσ, where σ is the assumed additive white Gaussian noise standard deviation.The denoiser update is v^(k+1) = Dσ(x^(k) + 1/ρ u^(k)).
- PnP-ADMM formulation: The denoiser parameter is coupled to the ADMM penalty through σk = √(λ/ρk), with ρ updated multiplicatively as ρk+1 = γkρk and γk ≥ 1.This parameterization supports the bounded-denoiser convergence analysis.
- SCI hardware constraints: SCI’s block-diagonal sensing structure makes HH⊤ diagonal and bounds the forward-model gradient under normalized masks.For sensing values in [0,1], the text gives ∥H⊤Hx∥2 ≤ B∥x∥2.
- Convergence: With nonzero mask energy at every spatial location and a bounded denoiser, PnP-ADMM’s x^(k) and θ^(k) converge to a fixed point.The hardware assumption requires Rj > 0 for every spatial location and Rmax > Rmin.
4. Plug-and-Play GAP for SCI
PnP-GAP combines projection onto the SCI measurement manifold with denoising, reducing each iteration to two subproblems. Under a bounded denoiser whose strength decreases to zero, the method converges, while its noisy-case accuracy can differ from ADMM’s.
- 4. Plug-and-Play GAP for SCI: PnP-GAP has lower computational workload than PnP-ADMM because each iteration uses two subproblems rather than three.The paper presents this reduction as making PnP-GAP faster.
- 4.1. Algorithm: The x-update projects the current denoised variable onto the linear SCI manifold M: y = Hx, while the v-update is a denoising problem.The SCI structure makes (HH⊤)^−1 diagonal, simplifying the projection.
- 4.1. Algorithm: PnP-GAP adaptively decreases λ when the relative residue fails to decrease sufficiently, otherwise retaining its current value.The rule is λk+1 = ξλk when ∆k+1 ≥ η∆k, and λk+1 = λk otherwise.
- 4.2. Convergence: A non-increasing denoiser schedule requires σk+1 ≤ σk and σk → 0 as k → +∞.The paper motivates this by expecting increasingly accurate estimates to require denoising of less noisy signals.
- 4.2. Convergence: Under the SCI sensing model and a bounded denoiser used in non-increasing order, PnP-GAP’s iterates x^(k) converge.The proof uses σk → 0 and ∥x^(k+1) − x^(k)∥2^2 → 0.
- 4.3. PnP-ADMM vs. PnP-GAP: In noisy settings, GAP enforces y = Hx, whereas ADMM minimizes measurement residual and may achieve a smaller distance to the true signal depending on initialization.The paper does not guarantee ADMM is more accurate than GAP in every noisy case.
5. Integrate Various Denoisers into PnP for SCI Reconstruction
The PnP framework supports interchangeable denoisers with different speed–quality trade-offs, and can combine them across iterations. DeSCI is a special case of PnP-GAP when WNNM is treated as bounded.
- Denoiser choice controls reconstruction quality and runtime through the bounded-denoiser error term.Smaller denoiser bound C yields a reconstruction closer to the true signal; TV is faster, while WNNM generally provides better results at higher cost.
- Integrating different denoisers into PnP-GAP or PnP-ADMM produces algorithms with distinct results.
- DeSCI is equivalent to PnP-WNNM and exploits correlations across video frames, whereas FFDNet-based priors are primarily image-based.
6. Results
Across simulated and real SCI data, PnP-FFDNet offers a strong quality–speed trade-off and remains effective as video scale and compression ratio increase. The color reconstruction pipeline handles Bayer channels separately before recombination and demosaicing.
- Benchmark data: GPU-accelerated PnP-FFDNet is faster than GAP-TV and provides about 3dB higher PSNR, with average PSNR close to 30dB.
- Benchmark data: DeSCI remains best on average, but PnP-WNNM-FFDNet achieves comparable results using one-sixth the computational time.
- Benchmark data: PnP-FFDNet is comparable to DeSCI while reducing simulated reconstruction time from 103 minutes to 3 seconds.
- Color video SCI: Color SCI measurements use shifted binary masks and an RGGB Bayer sensor, then split into four sub-measurements for independent reconstruction and later recombination.
- Large-scale data: The large-scale simulations include 1920 × 1080 videos and a 3840 × 1644 × 48 Football video reconstructed from one snapshot.
- Varying compression ratios: Across compression ratios from 8 to 48, PnP-FFDNet has the highest quality and fastest speed among the three fast PnP-SCI algorithms.
- Real data: Real-data reconstructions are comparable or better than DeSCI, while a 512 × 512 × 22 video takes around 12 seconds.
- Real data: PnP-FFDNet is more than 5× faster than GAP-TV on regular-size real datasets and more than 7× faster on large-scale datasets.
7. Conclusions
The paper presents PnP algorithms for SCI reconstruction that combine deep denoisers with efficient, flexible optimization. Experiments report strong reconstruction quality and short runtimes, including large-scale color video recovery.
- PnP algorithms combine deep denoisers with the reconstruction framework to achieve strong results on simulated and real datasets in short time.
- The running-time comparison identifies PnP-FFDNet as the fastest algorithm among the reported methods.