Source-linked AI summary
Noise2Void - Learning Denoising from Single Noisy Images
Alexander Krull, Tim-Oliver Buchholz, Florian Jug
TL;DR
Image denoising traditionally relies on clean targets or paired noisy images, which are often unavailable, especially for biomedical data. Noise2Void introduces self-supervised training from single noisy images and finds moderate performance loss relative to better-informed methods while comparing favorably with training-free denoising.
Problem
The central gap is denoising when neither clean target images nor noisy image pairs are available, particularly in biomedical imaging.
Method
Noise2Void trains denoising CNNs self-supervised from single noisy acquisitions using predictable signal and conditionally pixel-wise independent noise.
Results
N2V denoising performance drops only moderately compared with methods having additional training information and still outperforms BM3D.
Takeaways & Limitations
N2V can train networks directly on the same data body that will be processed, including biomedical datasets lacking ground truth.
Takeaways & Limitations
N2V cannot distinguish signal from structured noise that violates pixel-wise independence, potentially revealing systematic imaging patterns.
Abstract
from arXiv · showhide
The field of image denoising is currently dominated by discriminative deep learning methods that are trained on pairs of noisy input and clean target images. Recently it has been shown that such methods can also be trained without clean targets. Instead, independent pairs of noisy images can be used, in an approach known as Noise2Noise (N2N). Here, we introduce Noise2Void (N2V), a training scheme that takes this idea one step further. It does not require noisy image pairs, nor clean target images. Consequently, N2V allows us to train directly on the body of data to be denoised and can therefore be applied when other methods cannot. Especially interesting is the application to biomedical image data, where the acquisition of training targets, clean or noisy, is frequently not possible. We compare the performance of N2V to approaches that have either clean target images and/or noisy image pairs available. Intuitively, N2V cannot be expected to outperform methods that have more information available during training. Still, we observe that the denoising performance of Noise2Void drops in moderation and compares favorably to training-free denoising methods.
1. Introduction
N2V addresses the lack of clean targets and noisy image pairs by training denoising CNNs directly from single noisy images. Its performance remains competitive despite using less training information than supervised and N2N approaches.
- Problem: Image denoising separates an observed image into signal s and degrading noise n, using contextual dependencies to predict unobserved pixel values.The formulation is x = s + n, and useful prediction depends on signal pixels not being statistically independent.
- Existing training schemes: Traditional CNN denoising requires noisy-clean pairs, while N2N instead learns from independently degraded versions of the same underlying image.N2N still requires paired noisy acquisitions with approximately constant signal, limiting it to quasi-static scenes.
- Noise2Void: N2V trains denoising CNNs from a body of single noisy images without clean targets or noisy image pairs.Its assumptions are predictable signal structure and noise that is conditionally pixel-wise independent given the signal.
- Evaluation: On BSD68 and simulated microscopy data, N2V is compared with traditionally trained, N2N-trained, and self-supervised denoising methods.The comparisons include BM3D, non-local means, mean and median filters, and supervised CNN baselines.
- Results and practical scope: N2V performance drops only moderately relative to methods with additional training information and still outperforms BM3D.The method is also applied to cryo-TEM and Cell Tracking Challenge datasets where traditional training lacks ground truth and N2N is unavailable for most examples.
- Contributions: The paper contributes N2V, comparisons with existing trained and non-trained methods, theoretical motivation, and an efficient implementation.These contributions support training directly on data that will itself be denoised.
2. Related Work
Related work spans supervised CNN denoising, internal-statistics methods, and self-supervised restoration. N2V is distinguished by training directly on noisy test data while remaining limited to denoising under its noise assumptions.
- Scope: N2V addresses denoising rather than general image restoration because its rationale depends on multiple noisy observations revealing the true signal.The method therefore does not directly extend to perturbations such as blur.
- CNN denoising: Supervised CNN denoising treats restoration as regression against clean ground truth and includes deep residual and encoder-decoder architectures.Examples include networks that predict noise instead of the clean signal and architectures with symmetric skip connections.
- Paired-data methods: CARE acquires low- and high-exposure image pairs, but keeping biological samples fixed between exposures can be difficult.N2V avoids this paired-acquisition requirement by masking inputs during gradient calculation.
- Internal statistics: Internal-statistics methods extract information directly from the test image rather than relying on pre-collected ground truth training data.N2V belongs to this category because it trains directly on the image body to be processed.
- Other approaches: Other related approaches include non-local means, deep-image-prior restoration, GAN-based restoration requiring clean images, and autoregressive pixel-distribution modeling.These methods differ in training requirements, outputs, or whether they address denoising specifically.
- Internal statistics: BM3D groups similar repeated patterns and jointly filters them, whereas N2V concentrates computation during training and applies the trained CNN efficiently afterward.BM3D incurs substantial computation at test time, while a trained N2V network can process additional images of the same data type efficiently.
3. Methods
Noise2Void trains a denoising CNN from single noisy images by preventing identity learning through a blind spot, then approximating this architecture with random masking. The method relies on spatially dependent signal and conditionally independent, zero-mean noise, enabling predictions from surrounding pixels.
- Image Formation: Denoising separates a noisy image into signal and noise, assuming neighboring signal pixels are statistically dependent.This dependence allows surrounding context to predict an unobserved pixel's signal intensity.
- Training Baselines: Traditional CNN training maps noisy inputs to clean ground-truth targets, whereas Noise2Noise uses independently degraded versions of the same signal.Both procedures optimize a pixel-wise loss using corresponding target values.
- Blind-Spot Network: A blind-spot network excludes the center pixel from each receptive field, preventing a same-image input-target network from learning the identity.Under conditionally independent noise, neighboring pixels cannot predict the center noise, while signal dependencies still permit signal estimation.
- Noise2Void Training: Noise2Void extracts both input patches and target pixels from one noisy image, using the masked pixel's original value as the target.The target is treated as equivalent to the Noise2Noise target under the stated signal and noise assumptions.
- Implementation Details: Randomly replacing center pixels with values from surrounding areas implements blind spots with standard CNNs and prevents direct identity mapping.The practical masking scheme preserves efficient CNN operation while erasing the selected pixel's information.
4. Experiments
N2V is evaluated on natural images, simulated biological data, and acquired microscopy images against supervised, Noise2Noise, and training-free denoising methods.
- N2V is tested on natural images, simulated biological image data, and acquired microscopy images.
4.1. Denoising of BSD68 Data
BSD68 experiments use augmented noisy natural-image data and compare N2V with BM3D, traditionally trained networks, and N2N. N2V performs moderately below BM3D in this setting.
- 400 grayscale 180 × 180 images are augmented and corrupted with zero mean Gaussian noise at σ = 25 for training and BSD68 testing.Training uses random 64 × 64 pixel patches.
- All methods are applicable to BSD68 data because the experiment provides the needed training conditions.
- The BSD68 comparison includes BM3D, traditionally trained networks, N2N-trained networks, and N2V-trained networks.Average PSNR values are reported for the methods.
- N2V denoising performance drops moderately below BM3D on this dataset.
4.2. Denoising of Simulated Microscopy Data
Simulated fluorescence microscopy data provides high-SNR ground truth and two corresponding low-SNR inputs, enabling traditional, N2N, and N2V training. N2V reaches virtually the same denoising quality as the trained baselines and all trained networks outperform BM3D.
- The simulation generates high-SNR ground truth images and two corresponding low-SNR inputs by applying Poisson and zero mean Gaussian noise.This supports traditional, N2N, and N2V training.
- The simulated-data networks use U-Net architectures with depth 2 and N2V training manipulates N = 64 pixels per input patch.
- N2V reaches virtually the same denoising quality as traditional and N2N training.
- All trained networks clearly outperform BM3D on the simulated microscopy data.
4.3. Denoising of Real Microscopy Data
Real microscopy experiments evaluate N2V where ground truth is unavailable, preventing PSNR computation. N2V preserves structures comparably to N2N on cryo-TEM data and produces visually smooth results on fluorescence data while using less runtime than BM3D.
- 4.3. Denoising of Real Microscopy Data: Ground-truth-quality microscopy data is typically unavailable, so PSNR values cannot be computed for real microscopy experiments.
- 4.3.1 Cryo-TEM Data: For cryo-TEM data, N2V preserves marked structures as well as the N2N baseline and both trained methods run roughly 25 times faster than BM3D.High-SNR acquisition is impossible because of beam-induced damage.
- 4.3.2 Fluorescence Microscopy Data: For Cell Tracking Challenge fluorescence datasets, only BM3D and N2V training are applicable because neither ground-truth images nor second noisy images are available.
- 4.3.2 Fluorescence Microscopy Data: N2V produces subjectively smooth and appealing fluorescence-microscopy results while requiring only a fraction of BM3D’s runtime.
- 4.3.2 Fluorescence Microscopy Data: Failure cases show that N2V misses a bright isolated pixel and loses more high-frequency detail when preserving grainy structure.
4.4. Errors and Limitations
N2V errors increase when signal structure is difficult to predict or when noise is structured rather than pixel-wise independent. In the latter case, N2V removes unpredictable noise components but can preserve or reveal structured patterns.
- N2V predictions are most error-prone on images containing highly irregular signals that are difficult to predict from surrounding pixels.N2V blocks the center pixel, unlike traditional and N2N training, which can use it.
- N2V cannot distinguish structured noise from signal when the noise violates pixel-wise independence.The method removes unpredictable components but leaves structured components, which can reveal hidden patterns.
- In real microscopy data, N2V denoising revealed a systematic imaging-system error as a striped pattern.
4.5. Performance over Various Noise Levels
The BSD68 evaluation compares N2V with classical filters, non-local means, and DnCNN across multiple Gaussian-noise levels using average PSNR and qualitative results.
- Figure 7 compares N2V with mean filters, median filters, non-local means, and DnCNN.DnCNN results are taken from previously reported numbers.
- BSD68 performance is evaluated across various Gaussian-noise levels using average PSNR values.The comparison includes mean and median filters with widths of 3, 5, and 7 pixels.
- The figure distinguishes methods trained with ground truth, noisy image pairs, or only single noisy images.
5. Conclusion
The paper introduces N2V, which trains denoising CNNs using only single noisy acquisitions. It reports applicability across imaging modalities and competitive performance when its assumptions hold.
- N2V trains denoising CNNs using only single noisy acquisitions.
- N2V is demonstrated on photography, fluorescence microscopy, and cryo-Transmission Electron Microscopy.
- When predictable signal and pixel-wise independent noise assumptions hold, N2V networks can compete with traditionally and N2N trained networks.
- The paper presents training on the same body of data that will subsequently be processed, including potential biomedical applications.