PaperlayerView exact revision on arXiv

Source-linked AI summary

C2N: Practical Generative Noise Modeling for Real-World Denoising

Geonwoon Jang, Wooseok Lee, Sanghyun Son, Kyoung Mu Lee

arXiv:2202.09533v1cs.CVeess.IV

TL;DR

Real-world denoising lacks a practical way to model complex noise without paired clean/noisy examples, while aligned pairs are difficult to collect. C2N learns a clean-to-noisy generator without paired data, and the paper reports that denoisers trained with its generated pairs outperform existing unsupervised methods on real-world benchmarks.

  • Problem

    Real-world noise differs from signal-independent AWGN, aligned noisy/clean pairs are difficult to acquire at scale, and self-supervised noise statistics can be insufficient for real-world cases.

    Source: §1 · p. 1

  • Method

    C2N learns a clean-to-noisy noise generator without paired data, using designed components to represent real-world noise characteristics.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    C2N-generated training pairs let denoisers outperform existing unsupervised methods on real-world benchmarks, with denoised SIDD examples showing noise removal and detail preservation comparable in quality to a supervised model.

    Source: §5 · p. 8, §4.4 · p. 8

  • Takeaways & Limitations

    C2N can be used with different denoising backbones, including DnCNN and DIDN, rather than a fixed backbone.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Learning-based image denoising methods have been bounded to situations where well-aligned noisy and clean images are given, or samples are synthesized from predetermined noise models, e.g., Gaussian. While recent generative noise modeling methods aim to simulate the unknown distribution of real-world noise, several limitations still exist. In a practical scenario, a noise generator should learn to simulate the general and complex noise distribution without using paired noisy and clean images. However, since existing methods are constructed on the unrealistic assumption of real-world noise, they tend to generate implausible patterns and cannot express complicated noise maps. Therefore, we introduce a Clean-to-Noisy image generation framework, namely C2N, to imitate complex real-world noise without using any paired examples. We construct the noise generator in C2N accordingly with each component of real-world noise characteristics to express a wide range of noise accurately. Combined with our C2N, conventional denoising CNNs can be trained to outperform existing unsupervised methods on challenging real-world benchmarks by a large margin.

1. Introduction

Real-world noise differs from idealized assumptions, while collecting aligned clean/noisy training pairs is challenging. The paper introduces C2N to model complex noise without paired data, against a backdrop of supervised, self-supervised, and generation-based denoising approaches.

1. Introduction

  • Real-world noise is not signal-independent like AWGN, and collecting large-scale aligned noisy/clean pairs is difficult; self-supervised methods also rely on noise statistics insufficient for real-world cases.Training on synthetic examples therefore may not generalize to real-world images.Source: §1 · p. 1
  • Earlier generation-based methods learn noise from target noisy images to pair pseudo-noisy samples with clean images, but a successful approach without paired clean examples had not been established.Source: §1 · p. 2
  • C2N is a generative noise-modeling framework trained without paired data, designed to simulate complex noise and train denoisers using generated pairs.The paper says its generator uses explicit modules for noise characteristics and avoids handcrafted formulations or unrealistic assumptions.Source: §1 · p. 2

2. Related Works

  • CNN denoisers can perform well on AWGN, but require many training pairs containing the target noise distribution, which is unknown for real-world noise.Source: §2 · p. 2
  • Real-world denoising approaches include paired datasets, noise estimation, self-supervision, adaptive methods, and noise generation; prior generation methods still faced paired-data or modeling constraints.DND and SIDD provide real-world datasets, while Noise Flow requires true noisy/clean pairs and UIDNet addresses unpaired generative modeling.Source: §2 · pp. 2–3

3. Method

C2N learns complex real-world noise from unpaired clean and noisy images, then trains a denoiser on generated pairs. Its generator uses explicit modules for distinct noise characteristics and does not depend on the choice of denoising architecture.

3.1. Complexity of Real-World Noise

  • Real-world noise can be signal-dependent and spatially correlated, properties that make precise modeling challenging beyond simple AWGN assumptions.The heteroscedastic Gaussian model approximates realistic noise only to some extent.Source: §3.1 · p. 3

3.2. Learning to Generate Pseudo-Noisy Images

  • C2N learns a noise generator from unpaired clean and real noisy images, then uses generated noisy-clean pairs to train a denoiser.The generator maps a clean image x and random vector r to pseudo-noisy image ŷ = x + G(x, r).Source: §3.2 · p. 3, Figure 2 · p. 3
  • Adversarial training matches generated and real noisy-image distributions, while a stabilizing loss pushes channel-wise average generated noise toward zero to prevent color shifts.The stabilizer does not average over a single sample or local area, allowing varying nonzero local means.Source: §3.2 · pp. 3–4

3.3. C2N Architecture

  • GI 1×1 simulates spatially i.i.d. noise, while GI 3×3 and GD 3×3 produce features for spatially correlated signal-independent and signal-dependent noise, respectively.Source: §3.3 · pp. 4–5

3.4. Learning to Denoise with the Generated Pairs

  • A denoiser is trained with an L1 reconstruction loss on generated pairs, and C2N remains independent of the denoising architecture because its generator receives no reconstruction-loss supervision.The generated images are therefore not specialized for a particular denoising model.Source: §3.4 · p. 5

4. Experiment

The experiments test C2N on synthetic and real-world noise, examining its components and its denoising performance. The full framework models complex noise and outperforms existing unsupervised methods on real-world benchmarks.

4.1. Experimental Setup

  • The experimental setup trains C2N and denoisers using separate clean and noisy image sets, with 36,000 training patches and DND and SIDD benchmark evaluation.For DND, the clean images come from SIDD because DND provides no ground-truth images for learning.Source: §4.1 · p. 5

4.2. Model Analysis of C2N on Synthetic Noise

  • On synthetic noise, signal-dependent GD transforms model Poisson noise, while adding 3 × 3 transforms enables spatially correlated noise without the artifacts of 1 × 1-only models.The experiments use BSD images corrupted with signal-dependent Poisson noise and spatially correlated Gaussian noise.Source: §4.2 · pp. 6–7

4.3. Model Analysis of C2N on Real-World Noise

  • Ablations show that C2N’s stabilizing loss, signal-independent and signal-dependent modules, and 3 × 3 convolutional transforms are essential for simulating challenging real-world noise; using both modules also yields higher PSNR than others.Among compared methods, C2N achieves the lowest KL-divergence between generated and ground-truth noise maps.Source: §4.3 · pp. 7–8

4.4. Results of Real-World Denoising

  • C2N with DnCNN outperforms UIDNet on SIDD, while C2N with DIDN further improves performance on the DND and SIDD benchmarks.The method uses fewer training patches than UIDNet and can train denoisers with different backbones.Source: Table 4 · p. 7, §4.4 · p. 8
  • Fine-tuning a C2N model trained on SIDD for DND improves performance over training directly on DND.The paper also reports denoised SIDD examples with noise removal and detail preservation comparable in quality to the supervised model.Source: §4.4 · p. 8

5. Conclusion

The conclusion presents C2N as a framework for real-world denoising that learns noise distributions from unpaired data. Its components model signal dependence and spatial correlation, and generated pairs train denoisers that outperform existing methods without actual data pairs.

5. Conclusion

  • C2N explicitly models signal dependence and spatial correlation to simulate noise distributions from unpaired noisy and clean images.The conclusion reports that experiments on synthetic and real-world noise demonstrate the corresponding characteristics of each module.Source: §5 · p. 8
  • Denoisers trained on C2N-generated pairs outperform existing methods without using actual data pairs.The paper frames C2N as a framework for practical real-world denoising.Source: §5 · p. 8
  • The authors propose C2N as a potential key to addressing challenges in practical real-world denoising.Source: §5 · p. 8

Supplementary Material for

The supplied section contains author and affiliation information and an arXiv identifier.

S1. Measurement of KL-Divergence

The section defines KL-divergence evaluation by comparing histograms of ground-truth and generated noise maps. It specifies how the ground-truth histogram is calculated.

S1. Measurement of KL-Divergence

  • KL-divergence is evaluated between the distributions of ground-truth and generated noise maps.Their probability histograms represent those distributions.Source: S1. Measurement of KL-Divergence · p. 11
  • The ground-truth noise histogram counts pixel intensities across the evaluated noise maps and color channels.The generated-noise histogram is calculated in the same manner.Source: S1. Measurement of KL-Divergence · p. 11
  • Histogram calculation pools pixels across color channels, height, width, and the evaluated noise maps.The indicator function counts pixels with intensity i.Source: S1. Measurement of KL-Divergence · p. 11

S2. Implementation Details

Implementation details specify the discriminator architecture, an eight-pass self-ensemble for final denoising, and generator fine-tuning on DND.

S2. Implementation Details

  • The discriminator uses six 3 × 3 ResBlocks followed by a 1 × 1 convolution, with spatially averaged outputs indicating real versus generated images.All ResBlocks have 64 channels.Source: S2. Implementation Details · p. 11
  • Self-ensemble averages eight denoiser outputs after input flips and 90° rotations, mapping each output back to the original geometry.The eight evaluations include the untransformed input.Source: S2. Implementation Details · p. 11
  • For DND fine-tuning, the SIDD-trained C2N generator is trained for 16 additional epochs with an initial learning rate of 10−5.The learning rate is multiplied by 0.8 every three epochs.Source: S2. Implementation Details · p. 11

S3. Architectures in the Model Analysis

The model analysis contrasts C2N variants with different signal-dependence and spatial-correlation capabilities. The accompanying synthetic-noise figures and table describe the denoising comparisons and their evaluation scope.

S3. Architectures in the Model Analysis

  • GI 1×1 uses only a signal-independent 1×1 module, while GI 1×1 + GD 1×1 adds a signal-dependent module but retains only 1×1 convolutions.The variants are shown in the synthetic-experiment architecture figure.Source: S3. Architectures in the Model Analysis · p. 11, Figure S2 · p. 12
  • Figure S3 presents denoising examples for the model ablation on Poisson and spatially correlated Gaussian noise.It shows noisy images and results from the corresponding C2N variants and denoisers.Source: Figure S3 · p. 12
  • The full C2N combines signal-independent and signal-dependent modules with 1 × 1 and 3 × 3 convolutions to generate signal-dependent, spatially correlated noise.The architecture diagram identifies this as the complete generator.Source: S3. Architectures in the Model Analysis · p. 12, Figure S2 · p. 12
  • Table S1 reports denoising performance as PSNR on CBSD68 for the various synthetic noise settings.Source: Table S1 · p. 12

S4. Denoising on Synthetic Noise

Synthetic denoising comparisons show that variants lacking the needed noise characteristics leave mismatches: the 1×1-only design cannot handle signal-dependent noise or spatial correlation.

S4. Denoising on Synthetic Noise

S5. Visualizing Generated Noise

C2N-generated noisy samples visually resemble ground-truth noise without significant artifacts. Interpolating the input random vector changes synthesized noise properties, including noise level and conditions associated with different cameras.

S5. Visualizing Generated Noise

S6. Practical Data Constraints in C2N

The section considers limited clean images for generator training and different clean-image scene distributions for denoiser training. C2N fairly preserves performance under both constraints, while the authors identify resolving them entirely as future work.

S6. Practical Data Constraints in C2N

  • Practical use may provide few clean images from the noisy images’ scene distribution and require denoising training on clean images from a different distribution.Real-world scenes can differ in content types, scale, and illumination, making it hard to learn noise distinct from scene content.Source: S6. Practical Data Constraints in C2N · p. 14
  • Earlier generative noise methods used many clean samples and assumed the denoiser’s clean-image distribution matched the generator’s training distribution.The section characterizes this setting as impractical without a sufficiently large noisy and clean image dataset.Source: S6. Practical Data Constraints in C2N · p. 14
  • C2N fairly preserves performance with too few generator-training clean images and with denoiser-training images from a different scene distribution.The evaluation uses SIDD validation data; the authors say fully resolving these constraints remains future work.Source: S6. Practical Data Constraints in C2N · p. 14, Table S2 · p. 15

Related papers

Loading 2202.09533v1…