Source-linked AI summary
Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images
Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, Jianzhuang Liu
TL;DR
Large noisy-clean datasets are difficult to obtain, while existing single-image self-supervised denoisers impose practical, architectural, or noise-modeling constraints. Neighbor2Neighbor forms neighboring sub-sampled pairs from one noisy image and trains with regularization; experiments show it outperforms traditional and existing single-image self-supervised denoisers.
Problem
CNN denoisers require difficult-to-collect noisy-clean pairs, while existing self-supervised methods rely on multiple observations, blind spots, synthetic noise, or noise assumptions.
Method
Neighbor2Neighbor randomly sub-samples neighboring pixels from one noisy image to form training pairs, then trains a denoiser with a regularization term.
Results
Neighbor2Neighbor outperforms traditional denoisers and existing self-supervised methods trained from only single noisy images on synthetic and real-world experiments.
Takeaways & Limitations
The framework trains existing denoising networks without clean targets, network modifications, or explicit noise model assumptions.
Takeaways & Limitations
The theoretical construction assumes pixel-wise conditional independence of noise, and the method's future work includes spatially correlated noise and extremely dark images.
Abstract
from arXiv · showhide
In the last few years, image denoising has benefited a lot from the fast development of neural networks. However, the requirement of large amounts of noisy-clean image pairs for supervision limits the wide use of these models. Although there have been a few attempts in training an image denoising model with only single noisy images, existing self-supervised denoising approaches suffer from inefficient network training, loss of useful information, or dependence on noise modeling. In this paper, we present a very simple yet effective method named Neighbor2Neighbor to train an effective image denoising model with only noisy images. Firstly, a random neighbor sub-sampler is proposed for the generation of training image pairs. In detail, input and target used to train a network are images sub-sampled from the same noisy image, satisfying the requirement that paired pixels of paired images are neighbors and have very similar appearance with each other. Secondly, a denoising network is trained on sub-sampled training pairs generated in the first stage, with a proposed regularizer as additional loss for better performance. The proposed Neighbor2Neighbor framework is able to enjoy the progress of state-of-the-art supervised denoising networks in network architecture design. Moreover, it avoids heavy dependence on the assumption of the noise distribution. We explain our approach from a theoretical perspective and further validate it through extensive experiments, including synthetic experiments with different noise distributions in sRGB space and real-world experiments on a denoising benchmark dataset in raw-RGB space.
1. Introduction
Neighbor2Neighbor addresses the difficulty of training neural denoisers without large noisy-clean datasets by creating paired samples from a single noisy image. Its random neighbor sub-sampling and regularized training scheme are evaluated on synthetic and real-world data.
- Large aligned noisy-clean image datasets are difficult and expensive to collect for real-world photography, limiting CNN-based denoising.
- Existing self-supervised approaches rely on multiple observations, blind-spot architectures, noise models, or synthetic noisier-noisy pairs, limiting practicality in real-world scenarios.
- Neighbor2Neighbor generates training input-target pairs by randomly sub-sampling neighboring pixels from one noisy image.
- Experiments on synthetic and real-world noisy images show that Neighbor2Neighbor outperforms traditional denoisers and existing single-image self-supervised methods.
- The framework trains denoising networks on these pairs with an additional regularization term and does not require clean targets, network modifications, or noise model assumptions.
2. Related Work
Prior denoising research spans traditional methods, supervised CNNs, and self-supervised approaches using single or multiple noisy observations. These methods differ in their data requirements, architectural constraints, and dependence on noise assumptions.
- Traditional denoisers such as BM3D, NLM, and WNNM use noisy-image structures without learning from ground-truth images.
- Supervised CNN denoisers achieve strong performance but require large amounts of aligned noisy-clean image pairs that are challenging and expensive to collect.
- Noise2Noise trains denoisers from multiple noisy observations of the same scenes, whereas Noise2Void and Noise2Self use blind-spot networks with one noisy observation.
- Later blind-spot methods introduce explicit noise modeling and probabilistic inference, while Noisier2Noise creates training pairs by adding synthetic noise.
- SURE-based approaches target Gaussian additive noise and require the noise level to be known as a prior.
3. Motivation
The section extends Noise2Noise reasoning from paired noisy observations to single noisy images by generating neighboring sub-sampled pairs and correcting their non-zero ground-truth gap with regularization.
- Noise2Noise Revisit: Noise2Noise trains denoising models from independent noisy images of the same scene without clean ground-truth images.
- Motivation: Multiple noisy observations are difficult to capture, and their underlying clean images can differ because of occlusion, motion, and lighting variation.
- Extension to Single Noisy Images: Neighbor2Neighbor generates paired training images from one noisy image using an image pair sampler whose corresponding elements are neighboring pixels.
- Extension to Single Noisy Images: The sampled pair has a non-zero ground-truth gap, so directly applying Noise2Noise training is inappropriate and can lead to over-smoothing.
- Extension to Single Noisy Images: A correction term accounts for the ground-truth gap, while the resulting constraint is satisfied by the ideal denoiser.
4. Proposed Method
Neighbor2Neighbor trains denoising networks from single noisy images by generating paired sub-samples with a random neighbor sub-sampler and optimizing a regularized self-supervised loss.
- 4.2. Self-Supervised Training with a Regularizer: The regularized loss combines reconstruction against the noisy target with a regularization term accounting for the non-zero ground-truth gap between paired sub-sampled images.The regularization strength is controlled by γ, whose value is gradually increased during training; gradients through denoised versions of the original image are stopped for stabilization.
- 4.1. Generation of Training Image Pairs: Neighbor2Neighbor generates two noisy training images from one observation by randomly selecting neighboring pixels within local cells.The sub-sampler divides the image into k × k cells, empirically uses k = 2, and selects two neighboring locations per cell.
- 4.1. Generation of Training Image Pairs: Paired pixels are close in the original image, making their underlying ground-truth values similar while conditionally independent under pixel-wise noise.The independence condition holds when the noisy image is conditionally pixel-wise independent given the ground truth.
- 4.2. Self-Supervised Training with a Regularizer: The denoising network uses one sub-sampled image as input and the other as target in a self-supervised training scheme.Different random neighbor sub-samplers can generate noisy training pairs from single noisy images.
- 4.2. Self-Supervised Training with a Regularizer: During each training iteration, the method samples a noisy image, creates a random neighbor pair, computes the reconstruction loss, and applies the regularized update.The algorithm derives fθ(g1(y)) for the input and also computes fθ(y) without gradients for the regularization procedure.
5. Experiments
Experiments evaluate Neighbor2Neighbor on synthetic Gaussian and Poisson noise in sRGB space, real-world SIDD data in raw-RGB space, and ablations of its regularizer and sampling strategy. Across these settings, the method compares favorably with traditional and self-supervised denoisers, while ablations show that moderate regularization and random sampling are important.
- Synthetic Experiments: Experiments compare Neighbor2Neighbor with supervised, traditional, and six self-supervised denoisers on synthetic Gaussian and Poisson noise.The comparisons use PSNR and SSIM, with methods trained from single noisy images included among the baselines.
- Synthetic Experiments: Neighbor2Neighbor outperforms traditional and six self-supervised denoisers in synthetic experiments without explicit noise modeling.It achieves comparable performance to Laine19-pme, which uses noise-distribution modeling, especially when noise levels vary.
- Real-world Experiments: On SIDD raw-RGB data, Neighbor2Neighbor consistently outperforms traditional and self-supervised denoisers under the same network architecture.It also performs better than a network trained only on CycleISP-generated synthetic image pairs.
- Real-world Experiments: Explicit probabilistic post-processing performs worse in real-world experiments because Gaussian, heteroscedastic Gaussian, or Poisson noise models do not adequately represent real-world photography.Replacing U-Net with a more advanced denoising architecture can further improve Neighbor2Neighbor performance.
- Ablation Study: Removing the regularizer causes severe oversmoothing and detail loss, whereas excessive regularization leaves substantial noise; moderate γ balances smoothness and noisiness.The paper uses γ = 2 for synthetic experiments and γ = 1 for real-world experiments.
- Ablation Study: The random neighbor sub-sampler performs better than the less-random fix-location sub-sampler, demonstrating the importance of randomness in sampling.Both strategies generate sub-sampled images from one noisy image, but fix-location selects pixels from the same location within each k × k cell.
6. Conclusion
Neighbor2Neighbor enables single-image denoising by generating sub-sampled paired images with random neighbor sub-samplers and adding a regularization term. Experiments demonstrate effectiveness and superiority over existing methods, while future work targets spatially correlated noise and extremely dark images.
- Neighbor2Neighbor generates training pairs from sub-sampled versions of a single noisy image using random neighbor sub-samplers.
- The framework uses a self-supervised training scheme with an additional regularization term.
- Extensive experiments demonstrate the method’s effectiveness and superiority over existing denoising methods.
- Future work will extend the method to spatially correlated noise and extremely dark images.
A. Proof of Theorem 1
The theorem analyzes the squared-error objective when two noisy images are conditionally independent given the clean image but have different conditional means. The resulting decomposition includes the target variance and a bias-dependent cross term.
- The theorem assumes y and z are conditionally independent noisy images given x, with conditional means x and x + ε, respectively.
- The final decomposition subtracts σ_z^2 and adds a term proportional to ε and the expected prediction error relative to x.
- The squared-error expression is expanded by inserting z − x into fθ(y) − x.
- Conditional independence factors the cross term into a product involving fθ(y) − x and the conditional mean deviation of z from x.
B. Details of Fix-Location Sampling Strategy
The fix-location sampler creates paired sub-sampled images by selecting locations consistently within each k × k cell. For k = 2, four sub-sampled images use the same location pattern across all cells.
- The fix-location sampler randomly selects a pair from k^2 possible sub-sampled images.
- Each sub-sampled image draws pixels from one consistent location within every k × k cell.
- For k = 2, four sub-sampled images correspond to red, blue, yellow, and green locations repeated across all 2 × 2 cells.