Source-linked AI summary
Noisier2Noise: Learning to Denoise from Unpaired Noisy Data
Nick Moran, Dan Schmidt, Yu Zhong, Patrick Coady
TL;DR
The paper addresses denoising when clean targets and paired noisy realizations are unavailable. It trains from one noisy realization using sampled synthetic noise and reports quality close to paired-data methods, while exceeding BM3D and Noise2Void in the reported comparisons. The approach applies to independent and spatially correlated noise but requires a noise-distribution model and can produce mean-like outputs under clean-image ambiguity.
Problem
Image denoising often lacks clean training examples or paired noisy realizations, limiting standard supervised and Noise2Noise training.
Method
The method samples synthetic noise, adds it to a single noisy training image, and trains the network to predict the original noisy image from the doubly noisy input.
Results
The method produces reconstructions within 0.5dB of Noise2Noise, with higher quality than Noise2Void and BM3D, and reaches average PSNR 31.73 on KODAK under one reported setting.
Takeaways & Limitations
The approach supports denoising with single noisy realizations and spatially correlated noise when a statistical noise model can be sampled.
Takeaways & Limitations
Inference requires adding extra noise because the network is trained on doubly noisy images, and L2 training can yield means over plausible clean images.
Abstract
from arXiv · showhide
We present a method for training a neural network to perform image denoising without access to clean training examples or access to paired noisy training examples. Our method requires only a single noisy realization of each training example and a statistical model of the noise distribution, and is applicable to a wide variety of noise models, including spatially structured noise. Our model produces results which are competitive with other learned methods which require richer training data, and outperforms traditional non-learned denoising methods. We present derivations of our method for arbitrary additive noise, an improvement specific to Gaussian additive noise, and an extension to multiplicative Bernoulli noise.
1. Introduction
The paper introduces a denoising training algorithm that uses one noisy realization per image plus a noise-distribution model, avoiding clean and paired noisy targets. It adds synthetic noise during training and predicts the original noisy image from the doubly noisy input.
- Unlike standard supervised denoising, the approach avoids comparing predictions with a true clean image as the training target.Standard denoising networks typically learn from noisy-input and clean-target pairs.
- The method trains a denoiser using only a single noisy realization of each image, no clean images, and a statistical noise model.
- Figure 1 evaluates Gaussian-noise removal at σ = 0.05, 0.1, and 0.2 on a [0, 1] pixel-intensity scale.
- Training adds synthetic noise to an already noisy image and asks the network to predict the original noisy image from the doubly noisy input.
- The network can learn a clean-image estimate by treating the observed noise as a mixture of natural and synthetically added noise, followed by a simple correction.
2. Related Work
Prior methods either require clean or paired noisy targets, or rely on pixel-wise independent noise for self-supervision. This paper removes paired noisy-data requirements while supporting both independent and spatially correlated noise, using a statistical noise model.
- Earlier neural denoising methods commonly required known clean targets, while Noise2Noise replaced them with a second independent noisy realization.
- Noise2Void and Noise2Self use a single noisy realization by predicting pixels from neighboring values, relying on spatially uncorrelated noise.
- The proposed method removes paired noisy-data requirements like Noise2Void and Noise2Self while also allowing spatially correlated noise models.
- The method requires the ability to sample from the noise distribution, unlike Noise2Noise, Noise2Void, and Noise2Self.
- Its reconstructions are reported as close in quality to methods requiring clean or paired noisy training images.
3. Method
Noisier2Noise trains a denoising network from one noisy realization by adding synthetic noise, predicting the original noisy image, and correcting the output to estimate the clean image. The method extends this framework to Gaussian noise with adjustable augmentation and multiplicative Bernoulli noise, while recognizing uncertainty and inference-time trade-offs.
- Core training procedure: Because the original and synthetic noise samples are identically distributed, the network learns an estimate halfway between the doubly-noisy input and the clean image.The conditional expectation of one noise component is half the observed sum, yielding the midpoint interpretation.
- Core training procedure: The method starts from a single noisy image, adds synthetic noise sampled from the known noise distribution, and trains a network to predict the original noisy image.For Y = X + N, it constructs Z = Y + M = X + N + M and uses a pixel-wise L2 loss with stochastic gradient descent.
- Correction step: Figure 2 depicts the pipeline from singly noisy training target to doubly noisy input, raw network output, implicit remaining-noise estimate, and final reconstruction.The same augmentation and correction steps are used at inference time, so extra noise must be added to the input.
- Correction step: The clean-image estimate is obtained by doubling the network output and subtracting its input: E[X|Z] = 2E[Y|Z] − Z.This correction removes the network’s implicit estimate of the remaining noise from its raw output.
- Gaussian improvement: For Gaussian noise, reducing synthetic-noise intensity gives the correction E[X|Z] = (1 + α^2)E[Y|Z] − Z over α^2, with α = 1 recovering the standard formula.Smaller α gives the network a clearer view but increases sensitivity to prediction errors; α = 0.5 works well across several noise levels.
- Non-additive noise: The framework also analyzes multiplicative Bernoulli noise, where adjusting the masking parameter reduces synthetic corruption but magnifies errors in the network estimate.This extends the method beyond additive noise while preserving an explicit correction trade-off.
4. Experiments
Experiments evaluate the method on synthetic Gaussian, structured, and multiplicative Bernoulli noise, comparing it with learned and traditional denoisers. Results show competitive reconstruction quality, with performance depending on inference strategy and noise characteristics.
- Experimental setup: Experiments use ImageNet-derived training images and KODAK for testing, with synthetic Gaussian noise and comparisons against Noise2Noise, Noise2Void, and BM3D.The evaluation also includes structured noise and multiplicative Bernoulli noise.
- Gaussian noise: Our method achieves higher PSNR than BM3D and Noise2Void for suitable α values, while Noise2Noise remains a few tenths of a dB better.The comparison reflects different assumptions about training data and noise-model knowledge.
- Gaussian noise: α = 0.5 is successful across noise levels, whereas α = 0.25 is slightly worse in many settings because correction errors become more pronounced.Lower α provides a clearer input but increases the correction multiplier.
- Inference strategies: Singly-noisy inference can improve performance at smaller α but harm it at larger α because it trades cleaner inputs against train/test distribution shift.It can also produce overly smooth detail even when PSNR is higher.
- Structured noise: Structured-noise training removes noise without disrupting fine sails and water detail, but struggles with cloud texture at frequencies similar to the noise.The structured noise is generated by convolving white noise with a 21×21 Gaussian filter.
5. Future Work
The method’s L2 objective averages over uncertainty about both noise and the underlying clean image. The authors therefore identify output regularization as a future direction for reducing blur across plausible reconstructions.
- 5. Future Work: L2 loss handles uncertainty about noise allocation by inducing mean-seeking predictions, but also averages over plausible clean images.The model is therefore expected to output the mean of the plausible clean-image distribution given the noisy input.
- 5. Future Work: Additional output regularization may discourage undesirable blurring across plausible clean images.The proposed direction is to penalize residuals that do not resemble samples from the modeled noise distribution.
- 5. Future Work: Future experiments will test residual regularization based on simple statistics and GAN-style discrimination against true noise samples.The discriminator would distinguish residuals from the noise distribution rather than learning the manifold of natural images.
6. Conclusion
The method enables denoising-network training from single noisy images in settings where paired noisy images are unavailable, while supporting spatially structured noise. Its reconstructions approach Noise2Noise quality and exceed Noise2Void and BM3D in the reported comparison.
- 6. Conclusion: The method requires only single noisy images, unlike Noise2Noise, which requires paired noisy images.This addresses settings where multiple views of the same scene are unavailable.
- 6. Conclusion: The method applies to spatially structured noise, whereas Noise2Void requires pixel-wise i.i.d. noise.This broadens the noise settings covered by the training approach relative to Noise2Void.
- 6. Conclusion: Within 0.5dB of Noise2Noise and higher in quality than Noise2Void and BM3D, the method produces competitive reconstructions.The comparison is reported in the paper’s summary of results.
A. Saturation and Clipping
The paper treats saturation and clipping as important departures from purely additive, unbounded noise. For fair comparison, training and testing use unclipped values, while displayed and inferred outputs are converted to legal image values.
- A. Saturation and Clipping: Physical, hardware, and data-format constraints restrict pixel values, so noisy images are clipped to an allowed range.This makes a purely additive noise model unrealistic in such settings.
- A. Saturation and Clipping: Unclipped noise models remain common in prior work, motivating the same assumption here for fair comparison.The paper explicitly adopts the unbounded-value assumption used by many other techniques.
- A. Saturation and Clipping: During training and testing, noisy input images remain unclipped.This convention preserves the noise magnitude presented to the model.
- A. Saturation and Clipping: PSNR is computed using unclipped noisy inputs, which more accurately represents the noise magnitude seen by the model.The metric calculation differs from the legal-value representation used for outputs.
- A. Saturation and Clipping: Inference outputs are clipped and converted to uint8 RGB, while training loss uses unclipped, unquantized outputs; display clipping preserves original colors.Display values are clipped rather than rescaled across the image.