Source-linked AI summary
DVDnet: A Fast Network for Deep Video Denoising
Matias Tassano, Julie Delon, Thomas Veit
TL;DR
Video denoising methods based on neural networks had not matched patch-based methods, despite strong results in image denoising. The paper introduces DVDnet, a CNN that combines spatial and temporal denoising with motion compensation and a noise map. DVDnet compares favorably with state-of-the-art methods while providing temporal coherence, low flickering, detail preservation, and substantially faster inference.
Problem
Neural video denoising methods had not matched patch-based methods, even though neural networks were state-of-the-art for image denoising.
Method
DVDnet uses spatial and temporal CNN denoisers, motion compensation of neighboring frames, and a noise map as input.
Results
DVDnet produces temporally coherent sequences with very low flickering and detail preservation, while comparing favorably with state-of-the-art methods.
Takeaways & Limitations
DVDnet combines competitive video-denoising quality with fast inference and a small, efficient network design suitable for practical denoising applications.
Takeaways & Limitations
The experiments focus on additive white Gaussian noise, although the authors state that the algorithm can be extended to other noise types.
Abstract
from arXiv · showhide
In this paper, we propose a state-of-the-art video denoising algorithm based on a convolutional neural network architecture. Previous neural network based approaches to video denoising have been unsuccessful as their performance cannot compete with the performance of patch-based methods. However, our approach outperforms other patch-based competitors with significantly lower computing times. In contrast to other existing neural network denoisers, our algorithm exhibits several desirable properties such as a small memory footprint, and the ability to handle a wide range of noise levels with a single network model. The combination between its denoising performance and lower computational load makes this algorithm attractive for practical denoising applications. We compare our method with different state-of-art algorithms, both visually and with respect to objective quality metrics. The experiments show that our algorithm compares favorably to other state-of-art methods. Video examples, code and models are publicly available at \url{https://github.com/m-tassano/dvdnet}.
1. INTRODUCTION
Video denoising remains less developed than image denoising: neural approaches have struggled against patch-based methods, while DVDnet targets comparable quality with faster processing.
- DVDnet is introduced as a deep video denoising network that compares favorably with state-of-the-art methods while running quickly.The authors report temporally coherent outputs, low flickering, strong noise reduction, and detail preservation.
- Video denoising is described as largely underexplored relative to image denoising, where deep learning methods have achieved outstanding performance.
- CNN image denoisers can handle multiple noise levels with one trained model and achieve favorable quantitative and visual performance.DnCNN, for example, handles noise standard deviations σ ∈[0, 55] with one model through residual learning.
- Earlier neural video denoisers were limited by grayscale operation or unsatisfactory results, whereas patch-based methods define the video-denoising state of the art.
- VNLB achieves the best reported video-denoising quality but can require several minutes per frame, motivating faster alternatives such as DVDnet.The authors state that DVDnet compares favorably with VNLB for moderate to large noise while offering significantly faster inference.
2. OUR METHOD
DVDnet combines spatial denoising, motion alignment, and temporal denoising to use neighboring frames while reducing flicker. Its design also accepts a noise map and lowers computational demands through compact feed-forward blocks.
- DVDnet denoises a target frame together with 2T neighboring frames in two stages: individual spatial denoising followed by temporal denoising.
- Optical-flow motion compensation aligns pre-denoised neighboring frames before concatenation and temporal processing.
- Using temporal neighbors helps reduce flickering because residual errors across frames are correlated.
- A noise map is supplied to both denoisers, enabling processing of spatially varying noise without additional parameters beyond the sequence and noise estimate.
- The method is evaluated primarily on additive white Gaussian noise, while the authors state it can be extended to other noise types such as Poissonian noise.
- Spatial and temporal blocks use 12 and 6 convolutional layers, respectively, and process inputs at quarter resolution to reduce runtime and memory requirements.Both blocks use residual connections and upscale outputs back to full resolution.
3. TRAINING DETAILS
DVDnet trains its spatial and temporal denoisers separately on image or video patches generated with additive white Gaussian noise. Temporal training additionally uses motion-compensated neighboring patches and a noise map.
- The spatial and temporal denoisers are trained separately, with the spatial denoiser trained first.
- Both denoisers use patch-based training with residual learning for the spatial model and ADAM optimization under a shared 80-epoch schedule.The learning rate changes from 1e−3 to 1e−4 and then 1e−6 during training.
- 1,024,000 spatial patches from the Waterloo Exploration Database use 50×50 crops with AWGN σ ∈[0, 55].
- 450,000 temporal samples from DAVIS use 44×44 patches across five contiguous frames, with neighboring frames motion-compensated using DeepFlow.
4. RESULTS
DVDnet is evaluated on DAVIS and Set8 against established video denoisers using visual quality, PSNR, and running-time comparisons. It shows strong temporal quality and substantially faster inference, while its PSNR advantage over VNLB depends on noise level.
- Visual quality: 26.62dB PSNRseq is reported for DVDnet in the illustrated comparison, exceeding VNLB at 26.34dB and V-BM4D at 24.91dB.Neat Video is reported at 23.11dB, while the noisy frame is 14.15dB.
- Visual quality: DVDnet produces temporally coherent sequences with notably low flickering, strong noise reduction, and preserved detail.
- PSNR comparisons: For smaller noise values, VNLB performs better on the reported PSNR comparisons, while DVDnet tends to over-denoise in these cases.
- Running times: Less than 8s are required to denoise a 960 × 540 color frame, making DVDnet about 20 times faster than V-BM4D and about 50 times faster than VNLB.Of this time, 6s are spent compensating motion of temporal neighboring frames; VNLB values exclude motion-estimation time.
5. CONCLUSIONS
The paper presents DVDnet as a video denoising algorithm that improves the state of the art through temporally coherent, low-flicker outputs with preserved detail.
- DVDnet is presented as a video denoising algorithm that improves the state of the art.
- Its denoising results feature remarkable temporal coherence, very low flickering, and excellent detail preservation.