PaperlayerView exact revision on arXiv

Source-linked AI summary

Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images

Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, Jianzhuang Liu

arXiv:2101.02824v3eess.IVcs.CV

TL;DR

Large noisy-clean datasets are difficult to obtain, while existing single-image self-supervised denoisers impose practical, architectural, or noise-modeling constraints. Neighbor2Neighbor forms neighboring sub-sampled pairs from one noisy image and trains with regularization; experiments show it outperforms traditional and existing single-image self-supervised denoisers.

  • Problem

    CNN denoisers require difficult-to-collect noisy-clean pairs, while existing self-supervised methods rely on multiple observations, blind spots, synthetic noise, or noise assumptions.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    Neighbor2Neighbor randomly sub-samples neighboring pixels from one noisy image to form training pairs, then trains a denoiser with a regularization term.

    Source: §1 · p. 2

  • Results

    Neighbor2Neighbor outperforms traditional denoisers and existing self-supervised methods trained from only single noisy images on synthetic and real-world experiments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework trains existing denoising networks without clean targets, network modifications, or explicit noise model assumptions.

    Source: §1 · p. 2

    The theoretical construction assumes pixel-wise conditional independence of noise, and the method's future work includes spatially correlated noise and extremely dark images.

    Source: §4.1 · p. 5, §6 · p. 8

Abstract

from arXiv · show

In the last few years, image denoising has benefited a lot from the fast development of neural networks. However, the requirement of large amounts of noisy-clean image pairs for supervision limits the wide use of these models. Although there have been a few attempts in training an image denoising model with only single noisy images, existing self-supervised denoising approaches suffer from inefficient network training, loss of useful information, or dependence on noise modeling. In this paper, we present a very simple yet effective method named Neighbor2Neighbor to train an effective image denoising model with only noisy images. Firstly, a random neighbor sub-sampler is proposed for the generation of training image pairs. In detail, input and target used to train a network are images sub-sampled from the same noisy image, satisfying the requirement that paired pixels of paired images are neighbors and have very similar appearance with each other. Secondly, a denoising network is trained on sub-sampled training pairs generated in the first stage, with a proposed regularizer as additional loss for better performance. The proposed Neighbor2Neighbor framework is able to enjoy the progress of state-of-the-art supervised denoising networks in network architecture design. Moreover, it avoids heavy dependence on the assumption of the noise distribution. We explain our approach from a theoretical perspective and further validate it through extensive experiments, including synthetic experiments with different noise distributions in sRGB space and real-world experiments on a denoising benchmark dataset in raw-RGB space.

1. Introduction

Neighbor2Neighbor addresses the difficulty of training neural denoisers without large noisy-clean datasets by creating paired samples from a single noisy image. Its random neighbor sub-sampling and regularized training scheme are evaluated on synthetic and real-world data.

  • Large aligned noisy-clean image datasets are difficult and expensive to collect for real-world photography, limiting CNN-based denoising.Source: §1 · p. 1
  • Existing self-supervised approaches rely on multiple observations, blind-spot architectures, noise models, or synthetic noisier-noisy pairs, limiting practicality in real-world scenarios.Source: §1 · p. 1
  • Neighbor2Neighbor generates training input-target pairs by randomly sub-sampling neighboring pixels from one noisy image.Source: §1 · pp. 1–2
  • Experiments on synthetic and real-world noisy images show that Neighbor2Neighbor outperforms traditional denoisers and existing single-image self-supervised methods.Source: §1 · p. 2
  • The framework trains denoising networks on these pairs with an additional regularization term and does not require clean targets, network modifications, or noise model assumptions.Source: §1 · p. 2

2. Related Work

Prior denoising research spans traditional methods, supervised CNNs, and self-supervised approaches using single or multiple noisy observations. These methods differ in their data requirements, architectural constraints, and dependence on noise assumptions.

  • Traditional denoisers such as BM3D, NLM, and WNNM use noisy-image structures without learning from ground-truth images.Source: §2.1 · p. 2, §2.2 · p. 2
  • Supervised CNN denoisers achieve strong performance but require large amounts of aligned noisy-clean image pairs that are challenging and expensive to collect.Source: §2.1 · p. 2
  • Noise2Noise trains denoisers from multiple noisy observations of the same scenes, whereas Noise2Void and Noise2Self use blind-spot networks with one noisy observation.Source: §2.2 · p. 2
  • Later blind-spot methods introduce explicit noise modeling and probabilistic inference, while Noisier2Noise creates training pairs by adding synthetic noise.Source: §2.2 · p. 2
  • SURE-based approaches target Gaussian additive noise and require the noise level to be known as a prior.Source: §2.2 · p. 2

3. Motivation

The section extends Noise2Noise reasoning from paired noisy observations to single noisy images by generating neighboring sub-sampled pairs and correcting their non-zero ground-truth gap with regularization.

Noise2Noise Revisit

  • Noise2Noise trains denoising models from independent noisy images of the same scene without clean ground-truth images.Source: §3.1 · p. 3

Motivation

  • Multiple noisy observations are difficult to capture, and their underlying clean images can differ because of occlusion, motion, and lighting variation.Source: §3.2 · p. 3

Extension to Single Noisy Images

  • Neighbor2Neighbor generates paired training images from one noisy image using an image pair sampler whose corresponding elements are neighboring pixels.Source: §3.3 · p. 3
  • The sampled pair has a non-zero ground-truth gap, so directly applying Noise2Noise training is inappropriate and can lead to over-smoothing.Source: §3.3 · p. 3, §3.2 · p. 3
  • A correction term accounts for the ground-truth gap, while the resulting constraint is satisfied by the ideal denoiser.Source: §3.3 · p. 3

4. Proposed Method

Neighbor2Neighbor trains denoising networks from single noisy images by generating paired sub-samples with a random neighbor sub-sampler and optimizing a regularized self-supervised loss.

4.2. Self-Supervised Training with a Regularizer

  • The regularized loss combines reconstruction against the noisy target with a regularization term accounting for the non-zero ground-truth gap between paired sub-sampled images.The regularization strength is controlled by γ, whose value is gradually increased during training; gradients through denoised versions of the original image are stopped for stabilization.Source: §4 · p. 4, Figure 1 · p. 4, §4.2 · p. 5

4.1. Generation of Training Image Pairs

  • Neighbor2Neighbor generates two noisy training images from one observation by randomly selecting neighboring pixels within local cells.The sub-sampler divides the image into k × k cells, empirically uses k = 2, and selects two neighboring locations per cell.Source: §4.1 · p. 4, Figure 2 · p. 4
  • Paired pixels are close in the original image, making their underlying ground-truth values similar while conditionally independent under pixel-wise noise.The independence condition holds when the noisy image is conditionally pixel-wise independent given the ground truth.Source: §4.1 · pp. 4–5

4.2. Self-Supervised Training with a Regularizer

  • The denoising network uses one sub-sampled image as input and the other as target in a self-supervised training scheme.Different random neighbor sub-samplers can generate noisy training pairs from single noisy images.Source: §4.2 · p. 5
  • During each training iteration, the method samples a noisy image, creates a random neighbor pair, computes the reconstruction loss, and applies the regularized update.The algorithm derives fθ(g1(y)) for the input and also computes fθ(y) without gradients for the regularization procedure.Source: §4.2 · p. 5

5. Experiments

Experiments evaluate Neighbor2Neighbor on synthetic Gaussian and Poisson noise in sRGB space, real-world SIDD data in raw-RGB space, and ablations of its regularizer and sampling strategy. Across these settings, the method compares favorably with traditional and self-supervised denoisers, while ablations show that moderate regularization and random sampling are important.

Synthetic Experiments

  • Experiments compare Neighbor2Neighbor with supervised, traditional, and six self-supervised denoisers on synthetic Gaussian and Poisson noise.The comparisons use PSNR and SSIM, with methods trained from single noisy images included among the baselines.Source: §5 · p. 5, §5.2 · p. 5, Table 1 · p. 6
  • Neighbor2Neighbor outperforms traditional and six self-supervised denoisers in synthetic experiments without explicit noise modeling.It achieves comparable performance to Laine19-pme, which uses noise-distribution modeling, especially when noise levels vary.Source: §5.2 · p. 6

Real-world Experiments

  • On SIDD raw-RGB data, Neighbor2Neighbor consistently outperforms traditional and self-supervised denoisers under the same network architecture.It also performs better than a network trained only on CycleISP-generated synthetic image pairs.Source: §5.2 · p. 6, Table 2 · p. 7
  • Explicit probabilistic post-processing performs worse in real-world experiments because Gaussian, heteroscedastic Gaussian, or Poisson noise models do not adequately represent real-world photography.Replacing U-Net with a more advanced denoising architecture can further improve Neighbor2Neighbor performance.Source: §5.2 · p. 7

Ablation Study

  • Removing the regularizer causes severe oversmoothing and detail loss, whereas excessive regularization leaves substantial noise; moderate γ balances smoothness and noisiness.The paper uses γ = 2 for synthetic experiments and γ = 1 for real-world experiments.Source: §5.3 · pp. 7–8, Table 3 · p. 8
  • The random neighbor sub-sampler performs better than the less-random fix-location sub-sampler, demonstrating the importance of randomness in sampling.Both strategies generate sub-sampled images from one noisy image, but fix-location selects pixels from the same location within each k × k cell.Source: §5.3 · p. 8, Table 4 · p. 8

6. Conclusion

Neighbor2Neighbor enables single-image denoising by generating sub-sampled paired images with random neighbor sub-samplers and adding a regularization term. Experiments demonstrate effectiveness and superiority over existing methods, while future work targets spatially correlated noise and extremely dark images.

  • Neighbor2Neighbor generates training pairs from sub-sampled versions of a single noisy image using random neighbor sub-samplers.Source: §6 · p. 8
  • The framework uses a self-supervised training scheme with an additional regularization term.Source: §6 · p. 8
  • Extensive experiments demonstrate the method’s effectiveness and superiority over existing denoising methods.Source: §6 · p. 8
  • Future work will extend the method to spatially correlated noise and extremely dark images.Source: §6 · p. 8

A. Proof of Theorem 1

The theorem analyzes the squared-error objective when two noisy images are conditionally independent given the clean image but have different conditional means. The resulting decomposition includes the target variance and a bias-dependent cross term.

  • The theorem assumes y and z are conditionally independent noisy images given x, with conditional means x and x + ε, respectively.Source: A. Proof of Theorem 1 · p. 11
  • The final decomposition subtracts σ_z^2 and adds a term proportional to ε and the expected prediction error relative to x.Source: A. Proof of Theorem 1 · p. 11
  • The squared-error expression is expanded by inserting z − x into fθ(y) − x.Source: A. Proof of Theorem 1 · p. 11
  • Conditional independence factors the cross term into a product involving fθ(y) − x and the conditional mean deviation of z from x.Source: A. Proof of Theorem 1 · p. 11

B. Details of Fix-Location Sampling Strategy

The fix-location sampler creates paired sub-sampled images by selecting locations consistently within each k × k cell. For k = 2, four sub-sampled images use the same location pattern across all cells.

Related papers

Loading 2101.02824v3…