Source-linked AI summary
Noise2Noise: Learning Image Restoration without Clean Data
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, Timo Aila
TL;DR
The paper asks whether image restoration can be learned without clean training targets, whose acquisition is often difficult. It trains neural networks using corrupted observations and relies on statistical properties of the corruption rather than explicit priors or likelihood models. Across photographic denoising, Monte Carlo rendering, and undersampled MRI, restoration reaches performance equal or close to clean-target training.
Problem
Clean training targets are often difficult or tedious to obtain for photographic, MRI, and rendering restoration tasks.
Method
The method trains a convolutional neural network on corrupted input-target pairs whose conditional expectation matches the clean target, without explicit corruption likelihood or image prior models.
Results
Across photographic denoising, synthetic Monte Carlo rendering, and undersampled MRI reconstruction, the approach achieves restoration performance equal or close to training with clean targets.
Takeaways & Limitations
Clean data is not necessary for denoising, and deep neural networks can achieve high restoration performance entirely from corrupted data.
Takeaways & Limitations
The expectation-based condition fails under saturation because gamut clipping removes part of the corruption distribution.
Abstract
from arXiv · showhide
We apply basic statistical reasoning to signal reconstruction by machine learning -- learning to map corrupted observations to clean signals -- with a simple and powerful conclusion: it is possible to learn to restore images by only looking at corrupted examples, at performance at and sometimes exceeding training using clean data, without explicit image priors or likelihood models of the corruption. In practice, we show that a single model learns photographic noise removal, denoising synthetic Monte Carlo images, and reconstruction of undersampled MRI scans -- all corrupted by different processes -- based on noisy data only.
1. Introduction
Signal reconstruction can learn mappings from corrupted observations to clean signals, but obtaining clean targets is often difficult or tedious. Noise2Noise shows that corrupted examples alone can support restoration without explicit corruption likelihoods or image priors.
- 1. Introduction: Signal reconstruction uses regression models such as CNNs to map corrupted observations to unobserved clean signals.Training traditionally minimizes empirical risk over pairs of corrupted inputs and clean targets.
- 1. Introduction: Training examples can pair short and long exposure photographs, incomplete and complete MRI k-space samplings, or noisy and converged ray-traced renderings.
- 1. Introduction: Clean targets are often difficult or tedious to obtain because noise-free photographs require long exposures and full MRI sampling precludes dynamic subjects.
- 1. Introduction: Noise2Noise learns image restoration from corrupted examples alone, often matching or exceeding training with clean examples.The approach requires neither an explicit statistical likelihood model of the corruption nor an image prior.
- 1. Introduction: The reference TensorFlow implementation for Noise2Noise training is available on GitHub.
2. Theoretical Background
The theoretical basis is that common loss functions estimate distributional summaries, allowing corrupted targets to replace clean ones when their conditional expectation matches the clean target. This extends regression training to restoration without explicit corruption or image-manifold models.
- 2. Theoretical Background: L2 minimization estimates the arithmetic mean, whereas L1 minimization estimates the median of the observations.
- 2. Theoretical Background: Neural-network regression generalizes point estimation, so its training inherits the properties of the underlying loss function.
- 2. Theoretical Background: The mapping from inputs to targets can be multiple-valued because a single low-resolution image may correspond to many high-resolution images.This reflects the complex conditional distribution p(y|x) in superresolution.
- 2. Theoretical Background: With L2 loss, zero-mean noise can corrupt training targets without changing the learned optimum in principle.The result follows because the expected corrupted target remains unchanged.
- 2. Theoretical Background: Corrupted inputs and targets can be used when E{ˆyi|ˆxi} = yi; with infinite data, the solution matches training with clean targets.For finite data, target-corruption variance is averaged over the number of training samples.
- 2. Theoretical Background: The method needs neither an explicit p(noisy|clean) likelihood nor a p(clean) prior, provided training data are distributed according to them.
- 2. Theoretical Background: In many restoration tasks, the expected corrupted input equals the clean target, enabling photon-noise removal from pairs of noisy images.The same reasoning extends to L1 loss, which can recover medians with up to 50% outlier content.
- 2. Theoretical Background: The paper presents subsequent examples to demonstrate that these theoretical capabilities are efficiently realizable in practice.
3. Practical Experiments
The experiments show that noisy-target training can match or exceed clean-target training across several restoration settings, although convergence and quality depend on corruption structure, loss choice, and data allocation.
- Additive Gaussian Noise: Correlated brown noise slowed convergence as inter-pixel correlation increased, but eventual performance remained within 0.1 dB of the less-correlated setting.The slower convergence is attributed to noisier weight updates caused by reduced effective averaging.
- Additive Gaussian Noise: Using corrupted targets improved Gaussian-noise performance through both more noise realizations per latent image and more latent clean images under a fixed capture budget.With 2000 training pairs, using 1000 latents and two noisy realizations each outperformed using 100 latents and 20 realizations each by several tenths of a dB.
- Other Synthetic Noises: 30.57 ± 0.02 dB with noisy targets matched 30.59 ± 0.02 dB with clean targets for Poisson noise, while outperforming the Anscombe-plus-BM3D comparison by 2 dB.Both target types converged at similar speed.
- Other Synthetic Noises: For random impulse noise, L0 showed little bias even when 90% of pixels were corrupted, whereas L1 became biased beyond 50% corruption and L2 was biased toward gray.The L0 behavior follows from the correct pixel value remaining the most common value among possible corruptions.
- Monte Carlo Rendering: For HDR Monte Carlo denoising, the custom relative loss LHDR outperformed L2, input tone mapping helped, and target tone mapping produced incorrect dark predictions.The proposed loss addresses the nonlinearity problem associated with comparing or predicting tone-mapped values.
- Other Synthetic Noises: 31.83 dB was reached by noisy-target text removal after 4000 epochs, approximately twice the convergence time, while the quality gap to clean targets remained appreciable.The authors attribute the remaining gap partly to the limited number of training pairs available when clean targets are costly.
4. Discussion
The paper shows that deep neural networks can recover signals under complex corruptions using only corrupted observations, without explicit corruption models or clean targets. This approach can match or closely approach clean-target performance and may reduce the burden of collecting clean data.
- Deep neural networks recover signals under complex corruptions without observing clean signals or explicitly characterizing the corruption.
- Performance is equal or close to training with clean target data across learned signal-recovery tasks.
- The approach uses a general-purpose deep convolutional model rather than requiring a separate explicit prior or likelihood model.
- AmbientGAN differs by requiring an explicit forward model of the corruption.
A.1. Network architecture
The experiments use a U-network architecture, with one exception for the first test in Section 3.1, and vary channel counts for different restoration tasks.
- The U-network is used in all tests except the first Section 3.1 test, which uses the RED30 network.
- RGB noise and text-removal experiments use three input and three output channels.
- Monte Carlo denoising uses nine input channels containing RGB color, RGB albedo, and a 3D normal vector, with three output channels.
- The MRI reconstruction setup uses monochrome data.
A.2. Training parameters
Training uses He initialization and ADAM, with fixed learning rates except for a brief rampdown and a lower rate for Monte Carlo denoising.
- Network weights are initialized following He et al. (2015).
- ADAM training uses β1 = 0.9, β2 = 0.99, and ϵ = 10^-8.
- The learning rate remains constant during training except for a brief smooth rampdown to zero.
- The learning rate is 0.001 for all experiments except Monte Carlo denoising, where 0.0003 improves stability.
- All experiments use minibatches of 4.
A.3. Finite corrupted data in L2 minimization
For L2 minimization with finite corrupted data, the estimate remains correct on expectation, with variance determined by corruption covariance and reduced by more samples.
- The expected error is analyzed when corrupted targets replace clean targets in an L2 minimization task.The derivation considers arbitrary random variables whose corrupted-target expectation equals the clean target.
- Uncorrelated corruptions simplify the covariance expression for the expected squared difference between estimates.
- The estimate’s variance equals the average corruption covariance divided by the number of samples N.
- The error approaches zero as the number of samples grows, while the estimate is unbiased even with finite data.Unbiased here means correct on expectation.
- For images, N counts all scalar values across images, pixels, and color channels rather than only the number of images.
A.4. Mode seeking and the “L0” norm
The paper characterizes norm minimization as expected distance minimization and shows that the L0 solution is a zero of the data density’s Hilbert transform, typically near its mode.
- In the L0 case, minimization recovers a zero-crossing of the density’s Hilbert transform rather than an exact probability-density mode.
- The L0 estimate is typically close to the true mode because the Hilbert transform approximates differentiation with a sign flip.
- For continuous density q(x), Lp minimization finds the point x∗ minimizing expected p-norm distance from samples drawn from q.
- The minimizer is obtained by finding a root of the derivative of the expected-distance expression.