Source-linked AI summary
FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation
Matias Tassano, Julie Delon, Thomas Veit
TL;DR
Video denoising needs temporal coherence and efficient processing, yet neural approaches had been comparatively underexplored. FastDVDnet uses a cascaded convolutional architecture with noise-map input and implicit motion handling, achieving coherent, detailed denoising without flow estimation. The method runs one to three orders of magnitude faster than other state-of-the-art competitors, although the reported results focus on Gaussian noise and patch-based methods can perform better on highly repetitive structures.
Problem
Video denoising must preserve temporal coherence while processing challenging, high-definition footage efficiently, but neural video denoising remained comparatively underexplored.
Method
FastDVDnet is a cascaded convolutional denoiser that uses temporal neighbors and a noise map while handling motion implicitly without explicit flow estimation.
Results
One to three orders of magnitude faster than other state-of-the-art competitors, FastDVDnet produces temporally coherent videos with very low flickering and excellent detail preservation.
Takeaways & Limitations
FastDVDnet offers a practical route toward high-quality real-time deep video noise reduction with a single model handling different noise levels.
Takeaways & Limitations
Reported results focus on Gaussian noise, and patch-based methods can surpass FastDVDnet on sequences containing many repetitive structures.
Abstract
from arXiv · showhide
In this paper, we propose a state-of-the-art video denoising algorithm based on a convolutional neural network architecture. Until recently, video denoising with neural networks had been a largely under explored domain, and existing methods could not compete with the performance of the best patch-based methods. The approach we introduce in this paper, called FastDVDnet, shows similar or better performance than other state-of-the-art competitors with significantly lower computing times. In contrast to other existing neural network denoisers, our algorithm exhibits several desirable properties such as fast runtimes, and the ability to handle a wide range of noise levels with a single network model. The characteristics of its architecture make it possible to avoid using a costly motion compensation stage while achieving excellent performance. The combination between its denoising performance and lower computational load makes this algorithm attractive for practical denoising applications. We compare our method with different state-of-art algorithms, both visually and with respect to objective quality metrics.
1. Introduction
Video denoising remains less explored and more demanding than image denoising because it must preserve temporal coherence while processing high-definition data efficiently. FastDVDnet addresses this gap with a deep network designed for strong quality and very fast runtimes.
- Video restoration must exploit temporal information while maintaining temporal coherence, making it more demanding than still-image denoising.Video contains additional information that can aid restoration, but temporal consistency increases the challenge.
- High-definition video processing requires very fast and efficient denoising algorithms.
- FastDVDnet implicitly handles motion without explicit motion estimation, while delivering high-quality denoised videos at very fast running times.The paper reports runtimes up to thousands of times faster than other relevant methods.
- CNN-based image denoisers can handle several noise levels with a single trained model, a property relevant to FastDVDnet's design context.
- Video denoising has received less research attention than image denoising, with patch-based methods forming the majority of recent approaches.Neural-network methods for video denoising have been rarer still.
2. FastDVDnet
FastDVDnet uses neighboring frames and noise information in a cascaded, multi-scale CNN architecture. Its modified U-Net blocks learn motion handling without explicit flow estimation while supporting efficient processing and temporal consistency.
- Modified multi-scale U-Net blocks implicitly handle motion instead of using an explicit motion-estimation or compensation stage.Multi-scale U-Net-like architectures are described as able to learn misalignment.
- FastDVDnet uses four neighboring frames with the noisy center frame, processing five consecutive frames to denoise the middle frame.
- The architecture uses cascaded spatio-temporal denoising blocks, with the first-step blocks sharing weights to reduce memory requirements and facilitate training.
- A separate noise map supplies the expected per-pixel noise standard deviation and can control the trade-off between noise removal and detail preservation.The noise map is constant for Gaussian noise and can vary with image intensity for Poisson noise.
- The denoising blocks use residual learning, PixelShuffle upsampling, and pixel-wise feature addition to support training and reduce memory or gridding artifacts.
3. Discussion
The paper evaluates architectural choices for motion handling and temporal consistency. Cascaded two-step denoising, multi-scale blocks, and end-to-end training improve performance or reduce artifacts, while avoiding explicit flow also limits flow-related failures.
- 3.1. Two-step denoising: The two-step cascade outperforms a one-step five-input denoiser by up to 0.9dB in PSNR and produces fewer temporal artifacts.The one-step alternative cannot handle object motion as well as FastDVDnet.
- 3.2. Multi-scale architecture and end-to-end training: Replacing FastDVDnet's multi-scale blocks with single-scale DVDnet blocks considerably worsens denoising results.
- 3.2. Multi-scale architecture and end-to-end training: End-to-end training reduces temporal artifacts considerably compared with separately training the multi-scale blocks in each cascade step.Separate training improved over single-scale blocks but left noticeable flickering.
- 3.3. Handling of motion: Avoiding explicit flow estimation helps prevent artifacts caused by erroneous flow in challenging cases such as occlusions or strong noise.The multi-scale blocks, cascaded design, and end-to-end training jointly support motion handling and temporal consistency.
4. Training details
FastDVDnet is trained on spatio-temporal clean/noisy patch pairs generated with additive white Gaussian noise across a range of standard deviations. Training uses DAVIS samples, ADAM optimization, learning-rate decay, augmentation, and early orthogonal-kernel regularization.
- Training pairs are formed from five-frame spatial patches and clean central patches corrupted with AWGN at σ ∈[5, 50].The noise map is constant with all elements equal to σ in this setting.
- The training set contains 384000 DAVIS samples with 96 × 96 spatial patches and temporal size 2T + 1 = 5.
- The network output is optimized using ADAM for 80 epochs with batch size 96 and step-decayed learning rates.The learning rate changes from 1e−3 to 1e−4 and then 1e−6 across training.
- Training includes rescaling and random-flip augmentation, with orthogonalization of convolutional kernels applied during the first 60 epochs.
5. Results
FastDVDnet delivers strong denoising quality across noise levels and datasets, with low flickering, good detail preservation, and substantially lower runtime than competing methods.
- FastDVDnet and DVDnet produce temporally coherent sequences with especially little flickering in flat areas affected by residual low-frequency noise.Patch-based methods often leave temporally decorrelated residual noise in such regions.
- Patch-based methods can outperform DVDnet and FastDVDnet on sequences containing many repetitive structures, where their non-local similarity prior is advantageous.The proposed methods handle nonrepetitive textures well, including denoised text and vegetation.
- ST-RRED evaluates both image quality and temporal distortions, complementing PSNR in the Set8 and DAVIS comparison.Table 1 reports PSNR and ST-RRED, with larger PSNR and smaller ST-RRED being preferable.
- FastDVDnet avoids explicit motion compensation and can implicitly handle motion, reducing occlusion-related artifacts seen in V-BM4D, VNLB, and DVDnet.The hypersmooth sequence uses σ = 50 and shows artifacts in methods dependent on erroneous flow.
- FastDVDnet performs consistently well across noise levels, while FastDVDnet and VNLnet lead DAVIS at small noise levels on PSNR and ST-RRED, respectively.For larger noise, DVDnet surpasses VNLB; FastDVDnet remains consistent across cases.
- FastDVDnet clipped outperforms ViDeNN by a wide margin on clipped AWGN.The clipped-AWGN model was trained specifically for that noise case.
6. Running times
FastDVDnet achieves very fast GPU inference while retaining the denoising quality described in the paper.
- 100ms is sufficient for FastDVDnet to denoise a 960×540 color frame on a Titan Xp GPU.This is more than 3 orders of magnitude faster than V-BM4D and VNLB, and more than an order faster than DVDnet and VNLnet.
7. Conclusion
The paper concludes that FastDVDnet combines high-quality, temporally coherent denoising with substantially faster execution, while the reported results concern Gaussian noise.
- FastDVDnet provides remarkable temporal coherence, very low flickering, and excellent detail preservation without a flow-estimation step.The algorithm runs between one and three orders of magnitude faster than other state-of-the-art competitors.
- Figure 6 compares the time required to denoise a 960×540 color frame across algorithms.VNLB's displayed time excludes motion-estimation time.
Supplemental Materials
The supplemental materials include an architecture diagram for the Den Block 5inputs denoiser.
- Figure S1 presents the architecture of the Den Block 5inputs denoiser.
1. Two-step denoising
FastDVDnet’s two-step cascaded design uses temporal neighbors to improve denoising and temporal consistency, outperforming a one-step five-input variant.
- The two-step cascade uses temporal-neighbor information and enforces temporal correlation in the remaining output noise.
- The experiment compares FastDVDnet with a modified Denoising Block that accepts five frames instead of three.
- 0.95dB average PSNR advantage favors FastDVDnet over the one-step Den Block 5inputs variant.The comparison covers four 854 × 480 color sequences.
- Den Block 5inputs produces sharply increased flickering and handles object motion less effectively than FastDVDnet.
3. Ablation studies
Ablations show that multi-scale blocks and batch normalization materially improve FastDVDnet, while activation, loss, and additional input-frame changes have little or no benefit.
- Replacing ReLU with Leaky ReLU or ELU changes average PSNR by less than 0.05dB across the tested sequences and noise levels.
- Using the Huber loss instead of L2 produces no significant performance change, with a 0.04dB average PSNR difference favoring L2.
- Removing batch normalization reduces performance by 0.18dB on average.
- Using 7 or 9 input frames yields no performance improvement and makes training convergence more difficult than with 5 frames.
4. Upscaling layers
FastDVDnet’s decoder uses PixelShuffle for multi-scale upsampling, while the architecture is designed for Gaussian denoising, ordered video frames, and five-frame processing.
- Upscaling layers: PixelShuffle repacks 4n_ch × h/2 × w/2 inputs into n_ch × h × w outputs during decoder upsampling.It constructs non-overlapping 2 × 2 output patches from pixels in different input channels.
- Gaussian noise model: The method targets Gaussian denoising because this noise model facilitates fair comparison across architectures and datasets.
- Permutation invariance: Input-frame order is used to enforce temporal coherence, so permutation invariance is not necessarily desirable for video denoising.
- Recursive processing: In practice, FastDVDnet processes five input frames, whereas recursive methods can suffer from artifacts that propagate through subsequent outputs.