Source-linked AI summary
Zero-Shot Noise2Noise: Efficient Image Denoising without any Data
Youssef Mansour, Reinhard Heckel
TL;DR
Existing denoising methods can require costly datasets, explicit noise knowledge, or substantial computation, motivating a dataset- and noise-model-free alternative. ZS-N2N uses a simple 2-layer network trained from one noisy image, and experiments report good performance across simulated and real-world noise with a favorable resource trade-off. Its scope relies on pixel-wise independent, unstructured, zero-mean noise.
Problem
Existing denoising methods can require expensive datasets, explicit noise models, or substantial computation, while dataset-trained models may lose performance under distribution shifts.
Method
ZS-N2N uses a simple 2-layer network, fixed-filter downsampling, and regularization to denoise without training examples or knowledge of the noise model or level.
Results
ZS-N2N performs well on simulated, real-world camera, and microscope noise and achieves a good trade-off among generalization, denoising quality, and computational resources.
Takeaways & Limitations
ZS-N2N is suitable for denoising when data are scarce and computational resources are limited, including CPU-only execution.
Takeaways & Limitations
The method assumes noise is pixel-wise independent, unstructured, and zero mean; model-based methods can degrade when noise distributions are unknown or must be estimated.
Abstract
from arXiv · showhide
Recently, self-supervised neural networks have shown excellent image denoising performance. However, current dataset free methods are either computationally expensive, require a noise model, or have inadequate image quality. In this work we show that a simple 2-layer network, without any training data or knowledge of the noise distribution, can enable high-quality image denoising at low computational cost. Our approach is motivated by Noise2Noise and Neighbor2Neighbor and works well for denoising pixel-wise independent noise. Our experiments on artificial, real-world camera, and microscope noise show that our method termed ZS-N2N (Zero Shot Noise2Noise) often outperforms existing dataset-free methods at a reduced cost, making it suitable for use cases with scarce data availability and limited computational resources. A demo of our implementation including our code and hyperparameters can be found in the following colab notebook: https://colab.research.google.com/drive/1i82nyizTdszyHkaHBuKPbWnTzao8HF9b
1 Introduction
Image denoising removes distortions and reconstructs details, but existing dataset-based and zero-shot methods face data, generalization, quality, or computational limitations. ZS-N2N addresses these constraints with a small dataset-free network and aims for a better trade-off across denoising quality, generalization, and resources.
- Medical image denoising is especially important because diagnostic accuracy depends on reconstructing fine details.
- Dataset-based denoising requires costly data collection and can lose performance when test images differ from the training distribution.
- Existing zero-shot methods variously require known noise, substantial compute, suffer poor quality, or generalize poorly across noise conditions.
- ZS-N2N trains a lightweight regularized network to map between two downsampled images produced from fixed-filter convolutions of one noisy test image.
- Compute: 20k parameters enable the 2-layer method to often outperform million-parameter networks while reducing computational cost and supporting CPU execution.
- Generalisation: ZS-N2N generalizes better across noise distributions and levels, while S2S sometimes performs better but is orders of magnitude slower and can fail at low noise levels.
2 Related Work
Prior denoising methods span dataset-free classical, blind-spot, deep-prior, and Noise2Noise-derived approaches, each with limitations in noise assumptions, compute, quality, or training requirements. ZS-N2N builds on Noise2Noise and Neighbour2Neighbour while using a smaller network and a consistency loss to reduce computational demands.
- Zero-Shot/ Dataset free Methods: Noise2Fast is conceptually similar to this approach, but uses a relatively large network and requires early stopping.
- Zero-Shot/ Dataset free Methods: ZS-N2N uses a network twelve times smaller and a forward pass seven times faster than Noise2Fast, while retaining information that Noise2Fast drops during downsampling.
- BM3D and Anscombe work well for Gaussian and Poisson noise respectively, but require the noise level as input.
- DIP depends on difficult-to-select early stopping, while Self2Self uses ensembling and can take 1.2 hours to denoise one 256 × 256 image on a GPU.
- Noise2Noise learns from two noisy images of the same scene when noise is zero mean, but obtaining such pairs is difficult.
- Blind-spot variants can perform well for artificial noise models but show significant performance drops on real-world camera noise.
- Neighbour2Neighbour creates noisy training pairs by sub-sampling a single noisy image, extending Noise2Noise without requiring paired images.
3 Method
ZS-N2N extends Noise2Noise and Neighbour2Neighbour to train on a single noisy image by creating a downsampled noisy pair and fitting a small regularized network. The method uses residual and consistency losses with a lightweight two-layer architecture to produce a denoised estimate efficiently.
- Image Pair Downsampler: ZS-N2N generates two half-resolution noisy images from one input by averaging diagonal and anti-diagonal pixels within non-overlapping 2 × 2 patches.The resulting images D1(y) and D2(y) each have size H/2 × W/2 × C.
- Zero-Shot Noise2Noise: The method fits a small image-to-image network to map the first downsampled image D1(y) to the second, D2(y), using a pairwise loss.The trained network is then applied to the original noisy observation to estimate its denoised image.
- Loss Function: Residual learning trains the network to fit noise rather than the image, while a symmetric loss yields the residual objective.The consistency term additionally compares denoising-before-downsampling with downsampling-before-denoising.
- Loss Function: The consistency loss regularizes training by enforcing agreement between the two processing orders and helps avoid overfitting.Only this term exposes the network to the full-resolution image; the residual loss uses downsampled inputs.
- Optimization: The total loss combines residual and consistency terms, and gradient descent typically converges in 1k to 2k iterations.Using the lightweight network, optimization takes less than half a minute on a GPU and around one minute on a CPU.
- Network: The architecture is a very simple two-layer image-to-image network with about 20k parameters, using convolutional operators without normalization or pooling layers.The paper motivates this shallow design as a way to avoid overfitting when training on one noisy image and reduce computational cost.
4 Experiments
Experiments compare ZS-N2N with dataset-based and zero-shot baselines across synthetic and real-world noise, emphasizing generalization, denoising quality, and computational cost. ZS-N2N generally matches or exceeds competing dataset-free methods while using substantially fewer resources.
- Results: ZS-N2N is on par with or better than dataset-free baselines on Gaussian, Poisson, and real-world camera and microscope noise.It only falls short of Self2Self at high noise levels, while Self2Self performs insufficiently at low noise levels.
- Results: Dataset-based methods underperform with small training sets and lose performance when test noise levels differ from training conditions.Noise2Clean surpasses other baselines when trained on 4000 images, showing sensitivity to dataset size and composition.
- Results: ZS-N2N produces sharper microscope denoising results than Noise2Fast despite similar scores, preserving more detail in fine-structure images.Noise2Fast images are described as relatively smooth, especially for MRI images.
- Computational Efficiency: On CPU, ZS-N2N is 200 times faster than S2S and 35 times faster than DIP, requiring 2% and 1% of their memory respectively.Only BM3D is computationally more efficient, while S2S also requires ensembling and long denoising times.
- Results: Noise-model-based methods perform well for their target distributions but generalize poorly to unknown or real-world noise.BM3D is competitive on Gaussian noise but drops on Poisson and real-world noise, while ZS-N2N and S2S remain consistent across Gaussian and Poisson noise.
- Discussion: ZS-N2N uses a small two-layer network and offers a trade-off among denoising quality, generalization, and computational resources.The method is reported to work across different noise distributions and levels with relatively fast denoising and few parameters.
5 Conclusion
The proposed zero-shot denoising algorithm requires neither training examples nor knowledge of the noise model or level. Its simple two-layer network performs well across simulated and real-world noise while balancing generalization, quality, and computational resources.
- ZS-N2N requires no training examples or knowledge of the noise model or level.
- The method uses a simple two-layer network for relatively short denoising, including without a GPU.
- ZS-N2N performs well on simulated, real-world camera, and microscope noise.
- The method achieves a good trade-off between generalization, denoising quality, and computational resources compared with existing dataset-free methods.
A Ablation Studies
The ablation studies examine loss components, network size, training-data scaling, and optimization behavior. They show that the residual loss is essential, very small networks can perform well, supervised methods benefit from sufficient data, and ZS-N2N avoids manual early stopping.
- Loss function: The residual loss is necessary because removing it makes the network learn the identity mapping.
- Network size: A 500-parameter ZS-N2N network outperforms DIP with 2 million parameters, while a UNet fails, likely because single-image training causes overfitting.
- Data scaling: 4000 training images are sufficient for N2C to significantly outperform all other dataset-free methods.
- Performance vs optimization iterations: ZS-N2N performance improves with optimization steps, unlike DIP, whose optimal early stopping point varies with noise type and level.
- Data scaling: Dataset-free methods maintain constant performance as the Noise2Clean training set grows.
- Performance vs optimization iterations: Figure 5 plots MSE between the clean image and denoised output against gradient-descent iterations for one Kodak24 image.
B Appendix
The appendix states that ZS-N2N is strongest at low and moderate noise levels but loses performance at high noise. It also notes that dataset-based methods can be substantially better when abundant similar training data are available, and outlines the Noise2Noise assumptions used in the proof.
- Weaknesses and Limitations: ZS-N2N performs strongly at low and moderate noise levels but drops at high Gaussian or Poisson noise levels.
- Weaknesses and Limitations: At high noise, ZS-N2N remains better than DIP and BM3D but worse than S2S.
- Weaknesses and Limitations: Dataset-free methods do not use training data, so dataset-based methods significantly outperform them when abundant similar data are available.
- Proof of equation 1: Noise2Noise considers two noisy observations y1 and y2 of the same clean image x, with independent noise e1 and e2.
- Proof of equation 1: The proof’s penultimate equality follows from noise independence and zero mean.
C Sample Reconstructions
The sample reconstructions compare Gaussian and Poisson denoising across multiple noise levels. They highlight failures or inconsistency in baseline methods under particular noise conditions.
- Gaussian denoising: For Gaussian denoising, Self2Self fails at σ = 10 and can produce an output noisier than the input.
- Poisson denoising: For Poisson denoising, BM3D performance is inconsistent because its noise-level estimation varies across images.