PaperlayerView exact revision on arXiv

Source-linked AI summary

Unpaired Learning of Deep Image Denoising

Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, Wangmeng Zuo

arXiv:2008.13711v1eess.IVcs.CV

TL;DR

The paper asks how blind denoising can be learned from unpaired clean and noisy images despite complex real noise. It combines self-supervised blind-spot denoising and noise modeling with knowledge distillation, and reports favorable results on synthetic noise and real photographs. Its scope is constrained by the pixel-independent heteroscedastic Gaussian assumption and estimation errors in self-supervision.

  • Problem

    Paired noisy-clean data are difficult to obtain and real photographic noise is complex, while existing unpaired or self-supervised methods have practical and performance limitations.

  • Method

    A two-stage scheme learns D-BSN and image-specific CNNest from noisy images, then distills a state-of-the-art denoiser using synthetic noisy-clean pairs and real noisy-denoising pairs.

  • Results

    The method performs better than N2V and Laine19 on synthetic noise, on par with a fully supervised counterpart, and favorably on real-world photographs, surpassing GCBD on DND.

  • Takeaways & Limitations

    Unpaired clean and noisy images can support training CNN denoisers through complementary self-supervised and distilled supervision.

  • Takeaways & Limitations

    The method assumes pixel-independent heteroscedastic Gaussian noise, although real noise can be more complex and spatially correlated; estimation errors may also remain.

Abstract

from arXiv · show

We investigate the task of learning blind image denoising networks from an unpaired set of clean and noisy images. Such problem setting generally is practical and valuable considering that it is feasible to collect unpaired noisy and clean images in most real-world applications. And we further assume that the noise can be signal dependent but is spatially uncorrelated. In order to facilitate unpaired learning of denoising network, this paper presents a two-stage scheme by incorporating self-supervised learning and knowledge distillation. For self-supervised learning, we suggest a dilated blind-spot network (D-BSN) to learn denoising solely from real noisy images. Due to the spatial independence of noise, we adopt a network by stacking 1x1 convolution layers to estimate the noise level map for each image. Both the D-BSN and image-specific noise model (CNN\_est) can be jointly trained via maximizing the constrained log-likelihood. Given the output of D-BSN and estimated noise level map, improved denoising performance can be further obtained based on the Bayes' rule. As for knowledge distillation, we first apply the learned noise models to clean images to synthesize a paired set of training images, and use the real noisy images and the corresponding denoising results in the first stage to form another paired set. Then, the ultimate denoising model can be distilled by training an existing denoising network using these two paired sets. Experiments show that our unpaired learning method performs favorably on both synthetic noisy images and real-world noisy photographs in terms of quantitative and qualitative evaluation.

1 Introduction

Deep denoising commonly relies on paired noisy-clean images, whereas unpaired clean and noisy collections are practical but technically challenging. The paper addresses this gap with a two-stage self-supervised and distillation scheme that performs favorably on synthetic and real photographs.

  • Motivation: Most CNN denoisers depend on large paired noisy-clean datasets, but real noise and ISP processing are difficult to characterize with simple models.This motivates learning from separately collected clean and noisy images.
  • Motivation: Noise2Noise requires identical underlying images and independently sampled noise, while Noise2Void and related methods use only noisy images and have training or utilization limitations.These constraints reduce practicality or prevent exploiting available clean images.
  • Proposed scheme: The proposed method combines self-supervised learning on noisy images with knowledge distillation from two complementary paired sets built using clean and noisy data.One set uses learned noise models to synthesize noisy images; the other pairs real noisy images with first-stage denoising results.
  • Proposed scheme: D-BSN and CNNest jointly model denoising and image-specific signal-dependent noise under a pixel-wise independent assumption using constrained log-likelihood.D-BSN uses dilated blind-spot processing, while CNNest estimates noise levels with a fully convolutional 1 × 1 network.
  • Results: Experiments report that the unpaired method outperforms N2V and Laine19 on synthetic noise, matches a fully supervised counterpart, and surpasses GCBD on DND.The method also performs well on real-world noisy photographs.

2 Related Work

Prior work progressed from supervised CNN denoisers and noise modeling toward self-supervised and unpaired learning. This paper develops a non-GAN approach that combines efficient blind-spot learning, image-specific noise modeling, and knowledge distillation.

  • Supervised denoising: Deep CNN denoisers improved substantially over traditional methods, while later work addressed complex noise and real-world photographs.Examples include DnCNN, RED30, MWCNN, N3Net, and NLRN.
  • Noise modeling: Complex noise models remain insufficient for real sensor noise, whose ISP processing can make noise signal-dependent and spatially correlated.Paired noisy-clean images or specialized acquisition procedures are commonly used to address this setting.
  • Self-supervised denoising: SURE-based methods avoid paired clean targets in some settings but handle AWGN and require known noise levels.These constraints limit their scope relative to the paper’s setting.
  • Self-supervised denoising: Noise2Noise avoids clean-image acquisition but requires matched underlying images and identically distributed independent noise; Noise2Void uses blind spots but is inefficient and omits the center pixel.Follow-up self-supervised methods improve aspects of this design.
  • Unpaired denoising: GCBD introduced GAN-based unpaired denoising but remained weaker on real photographs, motivating the paper’s non-GAN two-stage alternative.The proposed method uses D-BSN, 1 × 1 convolution for signal-dependent noise, constrained log-likelihood, and distillation.

3 Proposed Method

The proposed method learns denoising from unpaired clean and noisy images through self-supervised blind-spot learning followed by knowledge distillation. It combines D-BSN and CNNest with complementary synthetic and real paired sets, while pixel-shuffle down-sampling addresses spatially correlated noise in photographs.

  • Self-Supervised Learning: The scheme first jointly learns D-BSN and image-specific CNNest from noisy images using a constrained negative log-likelihood.D-BSN produces denoising-related outputs, while CNNest models pixel-independent, signal-dependent noise through 1×1 convolutions.
  • Knowledge Distillation: The second stage distills an existing convolutional denoiser using synthetic noisy-clean pairs from X and real noisy-denoised pairs from Y.The two sets complement one another: real clean images preserve details, while real noisy images compensate for noise-model estimation errors.
  • Self-Supervised Learning: D-BSN preserves the blind-spot requirement with centrally masked and dilated convolutions, avoiding four branches or rotated inputs.Stacked centrally masked convolutions alone would break the blind spot, whereas suitable dilated convolutions maintain it.
  • Self-Supervised Learning: CNNest uses an all-1×1 fully convolutional architecture so each pixel’s noise level depends only on the input value at that position.The model estimates an image-specific noise-level function from noisy images to approximate the function defined on the unavailable clean image.
  • Real-World Photographs: Pixel-shuffle down-sampling enables application to real photographs whose demosaicking noise violates the pixel-independent assumption.The method exploits the short-range nature of this spatial correlation by training on images downsampled with factor 4.

4 Experimental Results

Experiments evaluate the method on synthetic noise and real-world photographs across multiple datasets and supervision settings. The unpaired model performs competitively with supervised and self-supervised alternatives, while remaining limited by real-image noise correlations and missing ISP or paired-data information.

  • Experimental setup: The experiments use synthetic AWGN, heteroscedastic Gaussian, and multivariate Gaussian noise across gray- and color-image datasets.Evaluations include BSD68, CBSD68, KODAK24, and McMaster, with real-world tests on CC15 and DND.
  • Synthetic noisy images: D-BSN outperforms N2V on BSD68 in most reported settings and is generally on par with MWCNN trained using paired noisy-clean data.The comparison covers AWGN noise levels 15, 25, and 50, plus heteroscedastic Gaussian noise with α = 40 and δ = 10.
  • Efficiency: 0.020s: MWCNN(unpaired) processes a 320 × 480 image faster than N2V at 0.034s and Laine19 at 0.044s.Training also takes about 10 hours on two 2080Ti GPUs for D-BSN, compared with about 14 hours for Laine19 on four Tesla V100 GPUs.
  • Synthetic noisy images: MWCNN(unpaired) outperforms CBM3D, CDnCNN, and FFDNet by a non-trivial margin for all tested noise levels across three color-image datasets.For AWGN, it performs on par with NLRN and better than BM3D, DnCNN, and N3Net; for multivariate Gaussian noise, it performs on par with FFDNet on CBSD68.
  • Synthetic noisy images: 0.8dB: MWCNN(unpaired) gains PSNR against CBDNet on heteroscedastic Gaussian noise.The evaluation models heteroscedastic Gaussian noise as a proxy for RAW-image noise and reports that CNN denoisers outperform CBM3D on CBSD68.
  • Real-world noisy photographs: 0.5dB on CC15 and 2.3dB on DND: MWCNN(unpaired) exceeds specified unsupervised or GAN-based alternatives in PSNR.On CC15, the comparison is against N2S and N2V; on DND, it is against GCBD. Figure 4 reports comparable or better qualitative results versus competing methods.

5 Concluding Remarks

The paper combines self-supervised learning and knowledge distillation to train CNN denoisers from unpaired clean and noisy images. Its main limitation is the assumed noise model and possible estimation errors during distillation.

  • Method: The method combines a dilated blind-spot network and a 1 × 1-convolution FCN, trained through constrained log-likelihood maximization on unorganized noisy images.The learned denoising image and noise model are then used to distill existing CNN denoisers such as DnCNN and MWCNN.
  • Method: Knowledge distillation uses estimated denoising images and noise models to train state-of-the-art CNN denoisers.
  • Results: The proposed method is effective for synthetic noise, including AWGN, heteroscedastic Gaussian, and multivariate Gaussian, as well as real-world noisy photographs.
  • Limitations: The method assumes pixel-independent heteroscedastic Gaussian noise, whereas real noise may be more complex and spatially correlated.
  • Limitations: Estimation errors for clean images and noise models may remain unavoidable during self-supervised learning, motivating more robust and accurate CNN-denoiser distillation.

A Description for the blind-spot mechanisms.

The D-BSN blind-spot mechanism prevents the feature at a target position from depending directly on the input pixel at that same position. Dilated convolutions then expand the receptive field while preserving this requirement.

  • Masked convolution: After the 3 × 3 masked convolution, the target feature depends on neighboring pixels but excludes the target input pixel.For position (4, 4), the affected inputs are y33, y34, y35, y43, y45, y53, y54, and y55.
  • Dilated convolution: Applying a dilated convolution with s = 2 expands the receptive field of the blind-spot feature.

B Additional Visualization Results

Additional visualizations compare MWCNN(unpaired) with traditional and discriminative denoisers on real noisy images from RNI6 and RNI15. The reported comparisons are favorable across grayscale and color images.

  • RNI6: On grayscale RNI6 images, MWCNN(unpaired) outperforms DnCNN-B and performs favorably against BM3D.
  • RNI15: On color RNI15 images, MWCNN(unpaired) achieves comparable or better visual denoising results than CBM3D.
  • RNI15: Compared with CBDNet, MWCNN(unpaired) shows comparable visualization results without using ISP details or paired noisy-clean images.
Loading 2008.13711v1…