Source-linked AI summary
Quanta Video Restoration
Prateek Chennuri, Yiheng Chi, Enze Jiang, G. M. Dilshan Godaliyadda, Abhiram Gnanasambandam, Hamid R. Sheikh, Istvan Gyongy, Stanley H. Chan
TL;DR
Recovering grayscale video from noisy, few-bit photon counts is difficult when scenes move and only a few frames are available. QUIVER is an end-to-end quanta video restoration network, accompanied by the I2-2000FPS dataset. On simulated and real data, it significantly outperforms the methods compared in the paper.
Problem
Reconstructing grayscale images or video from noisy 1-bit or few-bit photon counts in moving scenes without motion blur remains challenging, particularly when only a few frames are available.
Source: §1 · p. 2Method
QUIVER is an end-to-end trainable quanta video restoration network that adopts core ideas from classical quanta restoration methods.
Source: §1 · p. 2, §3.1 · p. 5Results
QUIVER significantly outperforms all methods compared on an evaluation dataset containing simulated and real data.
Source: §1 · p. 2Takeaways & Limitations
Experiments indicate that QUIVER generalizes across single-photon sensors, and I2-2000FPS provides fast-moving-scene video captured at 2000 fps.
Source: §7 · p. 14The experiments assume a monochromatic sensor, matching the monochromatic real data used.
Source: §5.1 · p. 10
Abstract
from arXiv · showhide
The proliferation of single-photon image sensors has opened the door to a plethora of high-speed and low-light imaging applications. However, data collected by these sensors are often 1-bit or few-bit, and corrupted by noise and strong motion. Conventional video restoration methods are not designed to handle this situation, while specialized quanta burst algorithms have limited performance when the number of input frames is low. In this paper, we introduce Quanta Video Restoration (QUIVER), an end-to-end trainable network built on the core ideas of classical quanta restoration methods, i.e., pre-filtering, flow estimation, fusion, and refinement. We also collect and publish I2-2000FPS, a high-speed video dataset with the highest temporal resolution of 2000 frames-per-second, for training and testing. On simulated and real data, QUIVER outperforms existing quanta restoration methods by a significant margin. Code and dataset available at https://github.com/chennuriprateek/Quanta_Video_Restoration-QUIVER-
1 Introduction
The paper targets grayscale video reconstruction from noisy 1-bit or few-bit photon counts without motion blur, especially when only a few frames are available. It introduces QUIVER and I2-2000FPS to address this problem.
1 Introduction
- QUIVER addresses reconstruction of grayscale images or video from noisy 1-bit or few-bit measurements of moving scenes without incurring motion blur.The paper highlights the difficulty of existing quanta burst reconstruction when the temporal window narrows and noise remains.Source: §1 · p. 2
- QUIVER is an end-to-end trainable quanta video restoration method built on the core ideas of traditional quanta restoration algorithms.The paper specializes it for 3-bit data at 2000 fps with motion of 1 to 7 pixels per frame.Source: §1 · p. 2
- QUIVER significantly outperforms all methods compared on an evaluation dataset containing both simulated and real data.The comparison is reported as a significant margin.Source: §1 · p. 2
- I2-2000FPS provides high-speed video at 2000 fps for training and testing image and video reconstruction networks.The collection contains 280 videos across 114 scenes, with ground truth and simulated 3-bit videos.Source: §1 · p. 2
- For the targeted motion range, 3-bit single-photon detectors provide the best trade-off between motion blur and SNR.The figure uses real 1-bit SPAD captures; higher-bit outputs are generated through temporal frame averaging.Source: Figure 2 · p. 3
2 Background
Single-photon detectors can provide low-bit-depth photon counts, but bit depth trades off integration time, motion blur, and noise. Existing denoising and reconstruction approaches have limitations for low-light, moving scenes.
2.1 Few-bit Single-Photon Detectors
- QIS and SPAD offer extraordinary photon-counting capability: QIS uses a two-stage pump-gate and correlated double sampling to suppress read noise, while SPAD uses avalanche multiplication to amplify photocharge; both resolve photons at single-photon sensitivity.Source: §2.1 · p. 3
- Higher bit depth requires longer integration and can blur motion, while high-frame-rate 1-bit sensing is blur-free but extremely noisy.The paper describes an optimal bit depth as dependent on motion and the minimum per-frame SNR needed for reconstruction.Source: §2.1 · pp. 3–4
- At equal exposure, 1-bit video at 10k fps requires 96 Mb/sec versus 1.73 Mb/sec for 9-bit video at 20 fps, and more frames accumulate read noise.The paper therefore notes that recording 1-bit data is not always the best option.Source: §2.1 · p. 4
2.2 Related Work
- Image and video denoisers make simplistic noise assumptions, while burst methods depend on robust alignment and often use higher-photon CMOS data.Learnable alignment and practical camera-noise models are among the approaches discussed.Source: §2.2 · p. 4
- Event and spike cameras record brightness variation above a threshold, unlike QIS and SPADs, and are not designed for single-photon counting in extreme low light.The cited prior work also describes SPAD capture at very high temporal resolutions, which can hinder consumer deployment when bandwidth is a bottleneck.Source: §2.2 · p. 4
- Prior QIS reconstruction methods struggle with moving scenes; the cited dynamic-scene method targets only slow motion at 1 pixel per frame.The reconstruction problem is also characterized as challenging because of Poisson-Gaussian statistics.Source: §2.2 · pp. 4–5
3 QUanta VIdeo REstoration (QUIVER)
QUIVER reworks classical quanta-restoration stages into an end-to-end trainable pipeline designed to handle noise and motion jointly. Its modules progressively denoise, align and fuse features, then reconstruct and refine at multiple scales.
3.1 Design Philosophy
- Classical methods use sequential, independent stages that struggle with extreme noise and strong motion; QUIVER makes its stages interdependent and trains them end to end.The classical pipeline sums frames to boost SNR, estimates alignment, warps and combines frames, then refines the result.Source: §3.1 · p. 5
3.2 Design of QUIVER
- A lightweight single-image denoiser provides preliminary noise reduction, avoiding motion-vulnerable averaging and the detail loss that can follow removing pre-denoising.The design uses Residual Dense Blocks and avoids a computationally demanding multi-frame denoiser.Source: §3.2 · pp. 6–7
- QUIVER estimates learnable, multiscale optical flow and uses deformable convolution with residual offsets to align features from noisy and denoised frames.A GLU-based MLP fuses aligned features along the channel dimension; recurrence is used for alignment but not for fusion.Source: §3.2 · p. 7
- RMDF recurrently fuses each frame’s multiscale features, combining spatial correlations within frames with temporal correlations across frames.It progressively processes frame features using residual dense blocks and a hidden state.Source: §3.2 · pp. 7–8
- TCA attends across features from all input frames, while RFRM refines residual frames and produces reconstructed images at multiple scales.Reconstruction proceeds from high-level features at the lowest scale toward intricate details at scales 2 and 1.Source: §3.2 · pp. 8–9
- QUIVER is trained with a multiscale loss combining image and horizontal- and vertical-gradient errors against downsampled ground-truth frames.The loss supervises reconstructions at scales 1, 2, and 4.Source: §3.2 · p. 9
4 Proposed I2-2000FPS dataset
I2-2000FPS was created to address gaps in existing high-frame-rate datasets, including motion blur and limited coverage of high-speed scenes. It provides 2000-FPS videos across 114 scenes for training and testing reconstruction networks.
4 Proposed I2-2000FPS dataset
- Existing high-frame-rate datasets often contain severe motion blur and do not always provide high motion speed or enough videos for this setting.I2-2000FPS was introduced to capture high-speed motion with precision.Source: §4 · p. 9
- I2-2000FPS comprises 280 unique videos spanning 114 diverse scenes, with temporal resolution of 2000 FPS and spatial resolution of 512 × 1024 pixels, captured using a Chronos 1.4 high-speed CMOS camera.Source: §4 · p. 9
- The dataset’s outdoor capture maintained analog and digital gain at 0 dB and used dark-current calibration to mitigate dark-current effects.Outdoor ambient lighting was used to minimize noise.Source: §4 · p. 9
5 Experiments
The experiments use a Poisson-Gaussian sensor model and evaluate QUIVER against existing methods on simulated and real quanta data. QUIVER and its scaled-down variant outperform the baselines across simulated light levels, while real-SPAD results show recovery of high-frequency information.
5.1 Image Formation Model
- The simulated sensor model represents photon arrivals with a Poisson process, adds Gaussian read noise, and converts the signal to integer values through an ADC.The experiments assume a monochromatic sensor with QE 0.80, dark current 1.6 e−/pix/sec, and read noise 0.2 e−/pix.Source: §5.1 · p. 10
5.2 Experimental Settings
- Training uses 249 I2-2000FPS videos with simulated 3-bit frames at 3.25 PPP; testing includes 31 I2-2000FPS videos, 15 X4K1000FPS videos, and real SPAD data.The real-data models are trained at 4.9 PPP, and the real frames are generated from SPAD captures at 10000 FPS.Source: §5.2 · pp. 10–11, Figure 10 · p. 12, §5.3 · p. 12
5.3 Results
- For visual comparisons, methods process 11 simulated 3-bit frames at 3.25 PPP; on real SPAD data, QUIVER recovers high-frequency information while smoothing low-frequency regions.The authors report that the real-data results indicate generalization across QIS and SPAD imaging models.Source: Figure 9 · p. 11, §5.3 · p. 13
5.2 Experimental Settings
- The comparisons include eight dynamic-scene reconstruction methods and QBP with BM3D, with evaluation using PSNR and SSIM on simulated data.The synthetic results predict 6017 I2-2000FPS frames and 345 X4K1000FPS frames.Source: §5.2 · p. 11, §5.3 · p. 12
5.3 Results
- On synthetic 3-bit quanta frames at 3.25, 9.75, 19.5, and 26 PPP, QUIVER and QUIVER-s substantially outperform all baselines on PSNR and SSIM across I2-2000FPS and X4K1000FPS.For SPAD visual results at 4.9 PPP after summation, with networks trained at 4.9 PPP, QUIVER recovers high-frequency information.Source: §5.3 · pp. 12–13
6 Ablation Study
The ablations assess the denoiser, optical flow, and multi-scale reconstruction modules, as well as pretrained optical-flow weights. Removing modules worsens performance, while pretrained SPyNet weights yield sub-optimal results.
6 Ablation Study
- Fig. 11 presents visual comparisons of the predenoiser, optical flow, and multi-scale reconstruction modules.The caption identifies the modules being compared but does not report specific visual outcomes.Source: Figure 11 · p. 14
- Removing one or more modules substantially worsens performance, even when feature dimensions and RDB blocks are adjusted to maintain similar model capacity.The tested combinations evaluate pre-denoising, learnable optical flow, and multi-scale reconstruction.Source: §6 · pp. 13–14
- Initializing SPyNet with pretrained weights results in sub-optimal performance because the estimator is not robust to photon shot and read noise, especially in low light.The pretrained weights were finetuned during QUIVER training.Source: §6 · p. 14, Table 4 · p. 14
7 Conclusion
The conclusion presents QUIVER for reconstructing blur-free grayscale images and videos from 1-bit or few-bit quanta data. It reports strong results across synthetic and real data, generalization across single-photon sensors, and the introduction of the I2-2000FPS dataset.
7 Conclusion
- QUIVER reconstructs blur-free grayscale images and videos captured using 1-bit or few-bit quanta data.The framework uses pre-filtering, learnable optical flow, and multi-scale reconstruction.Source: §7 · p. 14
- Experiments on synthetic and real data indicate that QUIVER beats state-of-the-art methods and generalizes across single-photon sensors.Source: §7 · p. 14
- The paper introduces I2-2000FPS, a high-speed video dataset capturing fast-moving scenes at 2000 fps.The authors describe it as the world’s first high-speed video dataset.Source: §7 · p. 14