Source-linked AI summary
High-Quality Self-Supervised Deep Image Denoising
Samuli Laine, Tero Karras, Jaakko Lehtinen, Timo Aila
TL;DR
The paper addresses denoising from unorganized corrupted-image collections without clean references or paired corruptions. It uses an efficient blind-spot architecture and posterior-mean estimation, achieving near-baseline quality across several pixelwise-independent noise models, with some losses for impulse noise and estimated variable Poisson noise.
Problem
Clean references or multiple independent corruptions are often unavailable, motivating self-supervised denoising from collections of single corrupted images.
Method
The method combines an efficient convolutional blind-spot network with posterior mean estimation and incorporates the blind-spot data into denoising.
Results
The method is virtually as good as baseline methods for additive Gaussian noise, loses approximately 0.4 dB for fixed impulse noise and 0.3 dB for variable impulse noise, and has difficulty estimating variable Poisson noise.
Takeaways & Limitations
Self-supervised deep denoising can reach similar quality to models trained with clean references when self-supervision's drawbacks are addressed.
Takeaways & Limitations
The approach assumes pixelwise-independent noise with a known analytic likelihood model, and its self-supervised optimum is not guaranteed to match supervised training.
Abstract
from arXiv · showhide
We describe a novel method for training high-quality image denoising models based on unorganized collections of corrupted images. The training does not need access to clean reference images, or explicit pairs of corrupted images, and can thus be applied in situations where such data is unacceptably expensive or impossible to acquire. We build on a recent technique that removes the need for reference data by employing networks with a "blind spot" in the receptive field, and significantly improve two key aspects: image quality and training efficiency. Our result quality is on par with state-of-the-art neural network denoisers in the case of i.i.d. additive Gaussian noise, and not far behind with Poisson and impulse noise. We also successfully handle cases where parameters of the noise model are variable and/or unknown in both training and evaluation data.
1 Introduction
Self-supervised denoising is motivated by the scarcity of clean references and paired corruptions. The paper uses blind-spot networks to learn from individual noisy images while improving efficiency and recovering denoising quality.
- Supervised denoising traditionally requires corrupted images paired with clean targets.
- NOISE2NOISE removes clean targets but still requires two independent corrupted realizations for each training image.
- Self-supervised training instead learns from individual corrupted images, motivated by the widespread availability of single poor-quality images.
- Blind-spot training predicts each output from surrounding context while excluding the center pixel, allowing the same noisy image to serve as input and target.
- The paper improves prior blind-spot methods with an architecture that trains efficiently without masking and combines outputs with blind-spot data to improve quality.
2 Convolutional blind-spot network architectures
The proposed convolutional blind-spot architecture combines directional branches whose receptive fields exclude the center pixel. Rotating inputs through one shared branch implements the same function with shared kernels and fewer trainable weights.
- Four directional branches restrict their receptive fields to half-planes that exclude the center pixel, then combine through 1×1 convolutions.
- A one-pixel feature-map offset separates each branch’s receptive field from the center pixel before branch combination.
- Kernel offsets implement directional convolution restrictions by shifting input data and cropping the corresponding output rows.
- The architecture distinguishes its intentional blind spot from PixelCNN’s term for pixels omitted because of poor receptive-field design.
3 Self-supervised Bayesian denoising with blind-spot networks
The method learns a context-conditioned Gaussian prior from noisy images using an explicit noise likelihood, then incorporates the observed pixel through Bayesian posterior-mean estimation. This retains measurement information without requiring clean or paired references, but relies on a Gaussian prior approximation.
- Denoising estimates the clean pixel from its noisy measurement and neighboring context, represented by p(x|y, Ωy).
- Unlike blind-spot regression that ignores the noisy measurement at test time, the method explicitly models the corruption likelihood p(y|x).
- The network maps context Ωy to the mean µx and covariance Σx of a multivariate Gaussian approximation to p(x|Ωy).
- At test time, Bayesian inference combines the learned context prior with the noisy observation, using the posterior mean as the estimate.
- Training consists of learning the context-conditioned Gaussian parameters, followed by a two-step testing procedure that incorporates the observation.
- The approach is not guaranteed to match supervised training because it approximates the prior as Gaussian rather than approximating the posterior.
4 Practical experiments
Experiments show that the proposed blind-spot method achieves near-baseline Gaussian denoising quality, remains competitive with Poisson and impulse noise, and improves substantially over center-pixel-ignoring ablations. Architectural blind spots also make training far more efficient than masking-based alternatives.
- Additive Gaussian noise: The method with convolutional blind-spot networks and posterior mean estimation is virtually as good as supervised and paired-corruption baselines, even when Gaussian noise is unknown.For fixed noise, N2C is best for the given architecture and N2N matches it; the proposed method remains virtually as good.
- Additive Gaussian noise: Variable Gaussian noise preserves the relative method ordering, with the proposed method conceding 0.1 dB relative to the baseline.Knowing the noise level in advance does not change the results.
- Additive Gaussian noise: For smaller Gaussian training sets, the method is roughly on par with the baseline without augmentation and surpasses it for datasets smaller than 1000 images.Rotation augmentation benefits the baseline more than the proposed method in very small-data settings.
- Training efficiency: RANDOM masking is better than COPY masking in the Bayesian setting, while COPY performs better when the center pixel is ignored.COPY may leak center-pixel information, which conflicts with the Bayesian approach's assumptions.
- Training efficiency: 20–100× more computation is estimated for masking-based training to reach the quality obtained after 0.5M minibatches with the convolutional blind-spot architecture.A 10× longer masking-based run still did not reach comparable output quality.
- Poisson and impulse noise: For fixed Poisson noise, the method is within 0.1–0.2 dB of N2C, while variable noise causes a 1.2 dB SET14 drop when parameters must be estimated.The SET14 drop is concentrated in one POWERPOINT clip art image, which incurs a 13 dB penalty; for impulse noise, the method trails N2C by approximately 0.4 dB with fixed noise and 0.3 dB with variable noise.
5 Discussion and future work
The paper situates its self-supervised denoising method among approaches that use only noisy images and concludes that it can match clean-reference training when self-supervision’s drawbacks are addressed. It assumes pixelwise-independent noise with a known analytic likelihood, while identifying broader noise-model learning as future work.
- Related work: Self-supervised denoising methods, including Bayesian approaches and deep image prior, train or restore images using noisy data rather than clean references.Related methods include non-local means, BM3D, WNNM, Stein’s unbiased risk estimator, and FC-AIDE.
- Conclusion: The paper reports that self-supervised deep denoisers can reach similar quality to comparable models trained with clean reference data.The result depends on remedying the drawbacks imposed by self-supervision.
- Limitations and future work: The method assumes pixelwise-independent noise and a known analytic likelihood model.Individual corruption-model parameters can nevertheless be deduced from noisy data.
- Limitations and future work: Real corrupted images may not follow theoretical noise models exactly, motivating future work that learns more of the noise model directly from data.The authors also identify experiments on real-life data as a valuable next step.
A Network architecture, training and evaluation details
The experiments use modified five-level U-Nets, with blind-spot variants adding rotation, shifting, and unrotation operations to exclude the center pixel from receptive fields. Training uses Adam on random 256×256 crops, while evaluation includes padding and repeated noise realizations.
- Network architecture: The networks use a modified five-level U-Net with three final 1×1 convolution layers added to equalize blind-spot and baseline depth.All convolutions use leaky ReLU with α = 0.1 except the final linear 1×1 convolution.
- Network architecture: Blind-spot variants add ROTATE, SHIFT, and UNROTATE layers to combine receptive fields that exclude each pixel’s center.ROTATE creates four rotated inputs, while SHIFT moves feature maps downward by one pixel.
- Training and evaluation: Training uses Adam with learning rate λ = 0.0003, minibatches of 4, and random 256×256 crops.Most networks train for 0.5M minibatches, while impulse-noise blind-spot networks train twice as long.
- Training and evaluation: The training set contains 44,328 suitably sized images from the ILSVRC2012 validation set, with image data represented in the range [0, 1].Test images are mirror-padded to squares, denoised, and cropped back to their original sizes.
- Training and evaluation: A typical run takes approximately 4 hours for the baseline architecture and 14 hours for the blind-spot architecture.The blind-spot cost reflects a fourfold increase in internal minibatch size, with further optimization possible.
- Training and evaluation: Masking-based training evaluates an exponentially smoothed network, whose results were verified to align with fixed-length training using learning-rate rampdown.This removes the need to decide training length in advance for convergence measurement.
B Additional result images
Additional images evaluate Gaussian, Poisson, and impulse noise with fixed but unknown parameters. The full method is usually visually indistinguishable from the baseline, whereas ablations exhibit color leakage or checkerboard artifacts.
- Additional results: The additional results cover Gaussian, Poisson, and impulse noise with fixed but unknown parameters, and PSNRs are reported per image.The corresponding examples appear in Figures 5, 6, and 7.
- Ablation behavior: Diagonal signal covariance produces color artifacts because it models each color channel as independent and cannot represent hue–luminance uncertainty.Color leakage is visible in examples such as rows 6 and 9 of Figure 5.
- Ablation behavior: Discarding the center-pixel information entirely produces strong pixel-scale diamond or checkerboard artifacts.Similar artifacts also appear in some results from Krull et al.’s method.
- Full method: The full method shows slight checkerboarding in some high-frequency regions, especially with impulse noise, but is usually visually indistinguishable from the baseline.The comparison is intended to be inspected by zooming into the displayed images.
- Additional results: The displayed image sets correspond to Gaussian noise at σ = 25, Poisson noise at λ = 30, and impulse noise at α = 0.5.These parameter values label Figures 5, 6, and 7 respectively.