Source-linked AI summary
Plug-and-Play Algorithms for Video Snapshot Compressive Imaging
Xin Yuan, Yang Liu, Jinli Suo, Frédo Durand, Qionghai Dai
TL;DR
Video SCI compresses high-speed video into measurements captured by a low-speed sensor, but large-scale reconstruction faces speed and flexibility challenges. The paper develops PnP algorithms that integrate image and video denoisers, including a joint reconstruction-and-demosaicing approach for color SCI. The authors report reconstruction results on simulation and real data, including large-scale HD, FHD, and UHD videos.
Problem
Large-scale video SCI reconstruction is challenging because optimization algorithms can be slow, while end-to-end networks are inflexible to changes in the encoding process.
Source: §1 · p. 1Method
The paper integrates deep image and video denoisers into PnP-ADMM and PnP-GAP, and develops joint reconstruction and demosaicing for color SCI.
Source: §1.2 · p. 3, §2 · p. 4, §3 · p. 5Results
The authors report strong results on simulation and real datasets, including large-scale HD, FHD, and UHD video reconstruction, with PnP-FastDVDnet improving on image-denoiser results.
Source: §8 · p. 16Takeaways & Limitations
The authors conclude that their PnP algorithms provide fast, flexible reconstruction and demonstrate SCI on large-scale daily-life videos.
Source: §8 · p. 16Deep learning denoisers are usually not robust to noise levels, and their results are poor in high-noise cases.
Source: §5.2 · p. 8
Abstract
from arXiv · showhide
We consider the reconstruction problem of video snapshot compressive imaging (SCI), which captures high-speed videos using a low-speed 2D sensor (detector). The underlying principle of SCI is to modulate sequential high-speed frames with different masks and then these encoded frames are integrated into a snapshot on the sensor and thus the sensor can be of low-speed. On one hand, video SCI enjoys the advantages of low-bandwidth, low-power and low-cost. On the other hand, applying SCI to large-scale problems (HD or UHD videos) in our daily life is still challenging and one of the bottlenecks lies in the reconstruction algorithm. Exiting algorithms are either too slow (iterative optimization algorithms) or not flexible to the encoding process (deep learning based end-to-end networks). In this paper, we develop fast and flexible algorithms for SCI based on the plug-and-play (PnP) framework. In addition to the PnP-ADMM method, we further propose the PnP-GAP (generalized alternating projection) algorithm with a lower computational workload. We first employ the image deep denoising priors to show that PnP can recover a UHD color video with 30 frames from a snapshot measurement. Since videos have strong temporal correlation, by employing the video deep denoising priors, we achieve a significant improvement in the results. Furthermore, we extend the proposed PnP algorithms to the color SCI system using mosaic sensors, where each pixel only captures the red, green or blue channels. A joint reconstruction and demosaicing paradigm is developed for flexible and high quality reconstruction of color video SCI systems. Extensive results on both simulation and real datasets verify the superiority of our proposed algorithm.
1 INTRODUCTION
Video snapshot compressive imaging (SCI) offers low-bandwidth, low-power capture, but large-scale reconstruction must balance speed, accuracy, and flexibility. This paper develops plug-and-play (PnP) algorithms that combine denoisers with SCI reconstruction, including video denoising and joint color reconstruction and demosaicing.
1 INTRODUCTION
- SCI encodes sequential video frames into compressed measurements, while reconstruction remains a bottleneck for large-scale video because optimization can be slow and end-to-end networks are inflexible to changes in encoding.The paper frames the desired decoder as efficient and flexible, particularly for large-scale problems.Source: §1 · pp. 1–2
1.1 Related Work
- PnP integrates denoisers into an inverse-problem framework to combine optimization methods’ flexibility with deep learning’s speed.The paper positions this approach as a response to slow optimization algorithms and end-to-end networks that cannot readily adapt to different modulation patterns or compression rates.Source: §1.1 · p. 2
1.2 Contributions of This Work
- PnP-ADMM is shown to converge to a fixed point, and PnP-GAP is proposed with lower computational workload and a fixed-point convergence result under proper assumptions.The contributions also report FHD and UHD video reconstruction and results on par with DeSCI with reduced computational time.Source: §1.2 · pp. 2–3
- The paper adds FastDVDnet video denoising to PnP-GAP and proposes joint reconstruction and demosaicing for color SCI; it also evaluates the method on real camera measurements.The reported camera measurements span compressive sampling rates from 10 to 50.Source: §1.2 · p. 3
1.3 Organization of This Paper
- The paper proceeds from grayscale and color SCI models to PnP-ADMM, PnP-GAP, denoiser integration and color demosaicing, then evaluates simulation and real data.The final section concludes the paper.Source: §1.3 · p. 3
2 MATHEMATICAL MODEL OF SCI
The section formulates grayscale and color video SCI as forward measurement models. For Bayer color SCI, it describes both channel-wise reconstruction followed by demosaicing and a joint reconstruction-and-demosaicing framework.
2 MATHEMATICAL MODEL OF SCI
- A grayscale SCI measurement is formed by modulating each video frame with its sensing mask and integrating the modulated frames into one measurement, with additive noise.The masks and frames are combined element-wise before summation.Source: §2 · p. 3
- In Bayer color SCI, each pixel captures one color channel, and conventional processing reconstructs four RGGB channels separately before demosaicing.The sensor mosaic pattern determines the interleaved channel samples.Source: §2 · p. 4
- The color-video model represents red, green, and blue as interpolated full-resolution videos, with red and blue interpolated from one pixel in four and green from two pixels in four.The corresponding channel forward models include the sensor’s interleaved mosaic downsampling.Source: §2 · p. 5
- The proposed unified PnP framework jointly reconstructs and demosaics color SCI from the model, allowing different demosaicing and color-denoising algorithms.Unlike separate channel reconstruction followed by demosaicing, the joint framework processes the color video directly.Source: §2 · p. 5
3 PLUG-AND-PLAY ADMM FOR SCI
The section formulates SCI reconstruction as data fitting plus a prior and develops PnP-ADMM by replacing an optimization subproblem with denoising. For SCI, bounded-denoiser conditions yield fixed-point convergence.
3 PLUG-AND-PLAY ADMM FOR SCI
- SCI reconstruction is formulated as measurement fidelity plus a prior, with this work focusing on deep denoising priors, especially video denoising that can use temporal correlation.Source: §3 · p. 5
3.1 Review the Plug-and-Play ADMM
- PnP-ADMM introduces an auxiliary variable and solves iterative subproblems, replacing the denoising-related optimization step with an off-the-shelf denoiser.The denoiser parameter is tied to the assumed Gaussian noise level, while the penalty parameter ρ is updated across iterations.Source: §3.1 · p. 5
3.2 PnP-ADMM for SCI
- For the SCI sensing model, PnP-ADMM with a bounded denoiser makes x and θ converge to a fixed point.The section establishes bounded gradients for the SCI data-fidelity function under its stated pixel and sensing-matrix conditions.Source: §3.2 · p. 6
4 PLUG-AND-PLAY GAP FOR SCI
PnP-GAP adapts generalized alternating projection to SCI and has lower computational workload than PnP-ADMM. Its convergence is established under bounded-denoiser and non-increasing-denoiser conditions; its noisy-case behavior differs from ADMM.
4 PLUG-AND-PLAY GAP FOR SCI
- PnP-GAP has lower computational workload than PnP-ADMM because it uses two subproblems rather than three.Source: §4 · p. 6, §4.3 · p. 7
4.1 Algorithm
- Each PnP-GAP iteration projects onto the measurement manifold, denoises the result, and conditionally decreases λ when the relative residue fails to decrease sufficiently.The algorithm keeps λ unchanged when the residue decreases by the specified threshold; otherwise it multiplies λ by ξ < 1.Source: §4.1 · pp. 6–7
4.2 Convergence
- With a bounded denoiser in non-increasing order, PnP-GAP's SCI iterates x(k) converge.The stated denoiser condition requires σk+1 ≤ σk and σk to approach zero.Source: §4.2 · p. 7
4.3 PnP-ADMM vs. PnP-GAP
- In the noise-free case, ADMM and GAP can perform similarly with appropriate parameters, while in noise GAP may be farther from the truth and ADMM is not guaranteed to be more accurate.The ADMM outcome depends on initialization; the comparison is illustrated with a two-dimensional sparse signal.Source: §4.3 · p. 7, Figure 4 · p. 7
FOR SCI RECONSTRUCTION
The section surveys denoiser choices for PnP-SCI and develops video, hybrid, color, and sequential-measurement variants. It emphasizes trade-offs among reconstruction quality, flexibility, and running time.
5.1 Non-deep Denoiser
- Non-deep denoisers trade speed against quality: TV is fast but lower quality, while stronger methods such as WNNM are slower; DeSCI uses WNNM and achieves state-of-the-art SCI results at longer running time.Source: §5.1 · p. 8
5.2 Deep Denoiser
- FastDVDnet's video denoising prior accounts for temporal correlation and produces better grayscale and color SCI results than FFDNet.Image-based FFDNet is applied frame-wise; using color denoising rather than grayscale channel-wise denoising can improve color results.Source: §5.2 · p. 8
5.3 Hybrid Denoiser
- Image-denoiser PnP-GAP/ADMM results are not as good as DeSCI, while video denoising priors can improve them; combining denoisers sequentially or within an iteration can balance performance and running time.Source: §5.3 · p. 8
5.4 Joint Demosaicing and SCI Reconstruction
- For color SCI, the proposed PnP denoising step pairs demosaicing with color denoising; Malvar04 is used because Menon07 takes about 4× longer for limited gain, and existing deep demosaicing is reported as unstable in this setting.Source: §5.4 · p. 9
- A lightweight color-SCI option denoises a half-size RGB video each iteration and demosaics only once at the end, saving time at some cost in result quality.Source: §5.4 · p. 9
5.5 Online PnP for Sequential Measurements
- For sequential measurements, initializing each reconstruction from the previous measurement's result can reduce reconstruction time after a warm start on the first measurement.Source: §5.5 · p. 9
6 SIMULATION RESULTS
The simulations evaluate PnP reconstruction on grayscale, mid-scale color Bayer, and large-scale video SCI data. Across these settings, video denoising, joint color reconstruction and demosaicing, and high-resolution tests show strong results, with runtime varying by processing mode.
6.1 Benchmark Data: Grayscale Videos
- On grayscale benchmarks, DeSCI leads average PSNR, while PnP-FastDVDnet alone exceeds its SSIM; using temporal correlation gains more than 3 dB PSNR and 0.05 SSIM over PnP-FFDNet.Figure 5 compares reconstructed frames across six datasets; PnP-FastDVDnet is close to DeSCI in most cases and can show finer detail.Source: Table 1 · p. 10, §6.1 · p. 10, Figure 5 · p. 10
6.2 Benchmark Data: Color RGB-Bayer Videos
- The mid-scale color setup uses six 512 × 512 × 3 scenes, four compressed measurements per scene, and B = 8; color PnP methods take about 1.6 minutes per measurement versus 0.3 minutes for grayscale counterparts.The joint color procedure reconstructs and demosaics together; most of the color runtime is consumed by demosaicing.Source: §6.2 · pp. 10–11
- On six mid-scale color Bayer datasets, both joint color-denoising methods outperform DeSCI, and color denoising improves PSNR over grayscale denoising by 3.45 dB for FFDNet and 4.1 dB for FastDVDnet.Figure 6 shows smooth motion and fine detail for the color methods, with FastDVDnet-color providing the finest details in complicated backgrounds.Source: Table 2 · p. 11, §6.2 · pp. 11–12, Figure 6 · p. 11
6.3 Large-scale Data
- On four large-scale datasets, PnP-FFDNet and PnP-FastDVDnet improve PSNR over GAP-TV by at least 3.69 dB; FastDVDnet leads on three datasets but trails FFDNet by 0.39 dB on football40.Figure 7 shows the reconstructed frames; the tested compression rates range from 15 to 40, with results up to or close to 30 dB.Source: Figure 7 · p. 12, §6.3 · p. 13
- For HD videos with B up to 30, color PnP reconstruction finishes within an hour, whereas UHD videos at B = 40 take hours and DeSCI takes days.Grayscale-denoising PnP finishes all tested large-scale reconstructions within an hour; color denoising takes 10× longer.Source: Table 3 · p. 12, §6.3 · p. 13
- At B = 48, PnP-FastDVDnet reaches 0.85 SSIM, more than 0.15 above PnP-FFDNet and more than 0.25 above GAP-TV.Both PnP-FFDNet and PnP-FastDVDnet reconstruct the video at PSNR close to 27 dB at this compression rate.Source: §6.3 · p. 13
7 REAL DATA
Real-camera tests span grayscale and color systems with different spatial sizes, compression rates, and modulation patterns, covering fixed-rate grayscale, variable-rate grayscale, and color video.
7.1 Grayscale Videos with Fixed Compression Rate
- PnP-FFDNet is at least 4× faster than GAP-TV and better at reducing motion blur and preserving detail; PnP-FastDVDnet trades about 10× runtime for higher quality yet remains 20× shorter than DeSCI.The real-data comparisons also report PnP-FFDNet taking seconds while DeSCI takes hours.Source: Table 4 · p. 14, §7.1 · p. 14
7.2 Grayscale Videos with Various Compression Rates
- At B=50, Hand reconstructions resolve several pixels of frame-to-frame motion; PnP-FFDNet takes 2.4 minutes and PnP-FastDVDnet 3.4, versus DeSCI’s 210 minutes.Trained networks in [19] and [29] produced poor-quality results above B=20, while PnP-FastDVDnet reconstructed clear details.Source: §7.2 · pp. 14, 16, Figure 11 · p. 15
7.3 Color Videos
- For Bayer color data, PnP-FFDNet reconstructs a 22-frame 512 × 512 RGB video from one measurement in about 12 seconds, with sharper edges than DeSCI.PnP-FastDVDnet gives sharper boundaries and finer detail in 136 seconds, but produces a greenish background that may result from brightness-smoothing artifacts across frames.Source: Figure 12 · p. 16, §7.3 · p. 16
8 CONCLUSIONS
The paper concludes that plug-and-play reconstruction with deep denoisers gives short-time, flexible SCI reconstruction, with results on simulation and real data. It also identifies demosaicing speed and real large-scale systems as future-work directions.
8 CONCLUSIONS
- The proposed PnP algorithms integrate deep denoisers for SCI reconstruction and provide results on simulation and real data in short time with sufficient flexibility.The paper also proves convergence results for PnP-GAP.Source: §8 · p. 16
- The authors report SCI reconstruction on large-scale HD, FHD, and UHD daily-life videos as a step toward practical applications.The paper describes this as paving the way for practical SCI applications.Source: §8 · p. 16
- Future work includes an efficient demosaicing network to speed reconstruction and improve video quality, and building a real large-scale video SCI system.Source: §8 · p. 16