PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervised Raw Video Denoising with a Benchmark Dataset on Dynamic Scenes

Huanjing Yue, Cong Cao, Lei Liao, Ronghe Chu, Jingyu Yang

arXiv:2003.14013v1eess.IVcs.CVcs.LG

TL;DR

Realistic raw video denoising lacks noisy-clean pairs for dynamic scenes, limiting supervised learning. The paper constructs such a dataset and proposes RViDeNet, which processes Bayer-packed sequences using spatial, channel, and temporal correlations; experiments show superiority over state-of-the-art methods on indoor and outdoor videos.

  • Problem

    Realistic raw video denoising is limited by the lack of noisy-clean pairs for dynamic scenes, which cannot be captured cleanly with long exposure or repeated shots.

  • Method

    The paper creates dynamic noisy-clean videos by repeatedly capturing controllable object motions and proposes RViDeNet to denoise four Bayer-derived RGBG subsequences using spatial, channel, and temporal correlations.

  • Results

    The method outperforms state-of-the-art video and raw image denoising algorithms in raw and sRGB domains on captured indoor and outdoor videos.

  • Takeaways & Limitations

    The dataset facilitates research on realistic raw video denoising, while dual raw and sRGB outputs provide flexibility for generating results with preferred ISPs.

Abstract

from arXiv · show

In recent years, the supervised learning strategy for real noisy image denoising has been emerging and has achieved promising results. In contrast, realistic noise removal for raw noisy videos is rarely studied due to the lack of noisy-clean pairs for dynamic scenes. Clean video frames for dynamic scenes cannot be captured with a long-exposure shutter or averaging multi-shots as was done for static images. In this paper, we solve this problem by creating motions for controllable objects, such as toys, and capturing each static moment for multiple times to generate clean video frames. In this way, we construct a dataset with 55 groups of noisy-clean videos with ISO values ranging from 1600 to 25600. To our knowledge, this is the first dynamic video dataset with noisy-clean pairs. Correspondingly, we propose a raw video denoising network (RViDeNet) by exploring the temporal, spatial, and channel correlations of video frames. Since the raw video has Bayer patterns, we pack it into four sub-sequences, i.e RGBG sequences, which are denoised by the proposed RViDeNet separately and finally fused into a clean video. In addition, our network not only outputs a raw denoising result, but also the sRGB result by going through an image signal processing (ISP) module, which enables users to generate the sRGB result with their favourite ISPs. Experimental results demonstrate that our method outperforms state-of-the-art video and raw image denoising algorithms on both indoor and outdoor videos.

1. Introduction

The paper addresses realistic raw video denoising by introducing dynamic noisy-clean video data and RViDeNet, which models spatial, channel, and temporal correlations. It produces both raw and sRGB denoising outputs and outperforms state-of-the-art video and raw-image methods on indoor and outdoor videos.

  • Motivation: Realistic sRGB noise is difficult to model because nonlinear ISP operations make it more complex than Gaussian noise.Prior realistic denoising methods commonly construct noisy-clean image pairs from high-ISO short exposures and averaged clean images.
  • Motivation: Raw-domain denoising is appealing because raw sensor data retains original information and has simpler noise statistics than sRGB data.Existing raw-image datasets do not provide noisy-clean video pairs for dynamic scenes.
  • Contributions: The dataset contains 55 groups of dynamic noisy-clean videos captured at ISO values from 1600 to 25600, enabling temporal modeling and quantitative evaluation.Objects are manually moved, and repeated captures of each moment are averaged to obtain clean frames.
  • Contributions: RViDeNet explores non-local spatial, channel, and temporal correlations by processing four Bayer-derived RGBG subsequences separately before spatial fusion.The design includes pre-denoising, alignment, non-local attention, temporal fusion, and spatial fusion modules.
  • Contributions: The method outperforms state-of-the-art video denoising and raw image denoising algorithms in both raw and sRGB domains on indoor and outdoor videos.The network also provides raw and RGB outputs through an ISP module.

2. Related Work

Prior video denoising work largely targets Gaussian or synthesized noise, while raw-domain video processing and dynamic realistic noisy-clean datasets remain limited. This paper responds with a dynamic dataset and a network that maps raw data to both raw and flexible sRGB outputs.

  • 2.1. Video Denoising: Existing video denoising methods generally target Gaussian or synthesized noise, leaving complex real noise from low-light capture insufficiently addressed.Only limited prior work considered realistic video noise, and its training data contained static sequences.
  • 2.2. Image and Video Processing with Raw Data: Raw-domain processing is attractive because raw images retain original sensor information and have simpler noise than images after ISP processing.Most prior video methods rarely process video directly in the raw domain.
  • 2.2. Image and Video Processing with Raw Data: Unlike prior raw-to-sRGB video denoising with static training frames, the proposed method maps raw data to both raw and sRGB outputs for user-selected ISP processing.This provides flexible result generation across different users.
  • 2.3. Noisy Image and Video Datasets: Prior noisy-clean image datasets commonly use averaged frames from static scenes or low-ISO images as clean references.These strategies do not directly solve dynamic video pair construction.
  • 2.3. Noisy Image and Video Datasets: The paper constructs a dynamic noisy-clean video dataset because long-exposure or repeated-shot capture of moving scenes introduces blur.Manually created object motions allow each motion to be captured repeatedly and averaged into a clean frame.

3. Raw Video Dataset

The dataset combines captured indoor raw videos with synthesized raw videos to support realistic video denoising research. Captured clean targets are formed by averaging repeated high-ISO frames, while raw noise synthesis models shot and read noise.

  • 3.1. Captured Raw Video Dataset: Captured dynamic videos use controllable indoor object motions, repeated noisy frames, and averaged frames as clean ground truth.The camera records consecutive frames while each object motion is held still, then sequences are ordered temporally into paired noisy and clean videos.
  • 3.1. Captured Raw Video Dataset: The captured dataset supports supervised training and quantitative evaluation, with indoor scenes for training, validation, and testing plus 50 outdoor videos for further testing.The indoor collection contains 11 scenes, while the outdoor videos are captured under different ISO levels.
  • 3.2. Synthesized Raw Video Dataset: The synthesized raw-video pipeline converts sRGB videos to clean raw videos through image unprocessing and adds sensor noise as supplementary training data.The selected motion videos include scene motion, camera motion, or both.
  • 3.2. Synthesized Raw Video Dataset: The raw noise model combines Poisson shot noise and Gaussian read noise, with parameters varying as sensor gain or ISO changes.The noisy observation is modeled from true pixel intensity using shot- and read-noise parameters.
  • 3.2. Synthesized Raw Video Dataset: Noise parameters are calibrated for specific cameras using flat-field and bias frames rather than relying only on many repeated captures.Flat-field frames estimate shot-noise behavior, while bias frames are used for read-noise calibration.

4. The Proposed Method

RViDeNet denoises raw video by packing Bayer frames into four RGBG sub-sequences, aligning neighboring features with pre-denoising-guided deformable convolutions, and fusing spatial, channel, and temporal information. It reconstructs raw outputs and can convert them to sRGB through a selectable ISP.

  • 4.1. Overview: RViDeNet splits Bayer frames into four RGBG sub-sequences, aligns neighboring features, fuses them temporally and spatially, and reconstructs the raw video.The pipeline uses deformable convolutions for alignment and separate fusion modules for the four packed sequences.
  • 4.2. Alignment: A pre-denoising U-Net estimates cleaner features before deformable alignment, enabling correspondence estimation despite noise in the raw input.The U-Net is trained on synthesized raw noisy-clean pairs and produces the pre-denoising result used by the alignment module.
  • 4.2. Alignment: Pyramidal deformable alignment estimates offsets across multiple feature scales to align neighboring frames with the central frame under large movements.The method uses three pyramid levels, with offsets refined from coarser to finer levels.
  • 4.3. Non-local Attention and 4.4. Temporal Fusion: Non-local attention aggregates long-range spatial, channel, and temporal correlations after alignment, while temporal fusion weights neighboring features by their similarity to the central frame.The separated attention modules avoid the high cost of full 3D non-local attention; temporal fusion accounts for occlusions and alignment errors.
  • 4.5. Spatial Fusion and 4.6. Image Signal Processing (ISP): The spatial fusion network combines the four temporally fused sub-sequences into a full-resolution raw result, after which an ISP module produces sRGB output.The spatial fusion network uses residual blocks and CBAM, while the ISP can be trained for different cameras or replaced by traditional pipelines.

5. Experiments

Experiments compare the proposed method with state-of-the-art denoising approaches on indoor and outdoor videos, including ablations of its processing modules.

  • 5.3. Comparison with State-of-the-art Methods: Table 2 evaluates raw-domain processing, RGBG packing, pre-denoising, and non-local attention through PSNR and SSIM ablations across ISO 1600–25600.Removing raw-domain processing decreases sRGB PSNR by more than 1 dB; RGBG packing is then evaluated as a separate component.
  • 5.3. Comparison with State-of-the-art Methods: The comparison includes VBM4D, TOFlow, ViDeNN, SMD, EDVR, DIDN, and a variant trained only on synthetic data.TOFlow and EDVR are retrained on the authors’ sRGB noisy-clean video pairs, while SMD is reported with both pretrained and retrained results.
  • 5.3. Comparison with State-of-the-art Methods: 0.72 dB and 1.12 dB gains over DIDN are achieved in the raw and sRGB domains, respectively, on 25 indoor videos.The method also achieves nearly 1 dB gain over EDVR.
  • 5.3. Comparison with State-of-the-art Methods: The method removes noise clearly while preserving fine-grained details and temporally consistent textures in indoor and outdoor visual comparisons.EDVR and DIDN produce over-smooth outdoor results, while DIDN lacks consistency between neighboring frames.
  • 5.3. Comparison with State-of-the-art Methods: Outdoor evaluation lacks ground truth, so the authors additionally conduct a user study to assess denoising performance.The user-study results and video demonstrations are provided in supplementary material.

6. Conclusion

The paper introduces RViDeNet and a noisy-clean dynamic-video dataset for realistic raw video denoising. Its method uses RGBG decomposition and spatiotemporal processing, produces raw and sRGB outputs, and yields temporally consistent denoised videos.

  • 6. Conclusion: RViDeNet processes RGBG raw sub-sequences through alignment, non-local attention, temporal fusion, and spatial fusion modules.The design exploits spatial, channel, and temporal correlations in raw sequences.
  • 6. Conclusion: The first noisy-clean dynamic video dataset facilitates research on realistic raw video denoising.The conclusion identifies the dataset as a contribution for future work on this topic.
  • 6. Conclusion: Raw and sRGB outputs give users flexibility to generate results with their preferred image signal processing pipeline.The network produces both output domains through its processing design.
  • 6. Conclusion: Experimental results demonstrate realistic noise removal and temporally consistent video production.The conclusion states this outcome without restricting it to a single scene or domain.
Loading 2003.14013v1…