PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images

Kwanyoung Kim, Jong Chul Ye

arXiv:2106.07009v2eess.IVcs.CVcs.LG

TL;DR

Self-supervised denoising methods lack a coherent mathematical structure, while supervised methods require paired clean images and SURE has noise-dependent training limitations. Noise2Score uses Tweedie’s formula with score-function estimation, achieving strong results across exponential-family noises and enabling identical training across noise models. Its scope is limited when the noise distribution is unknown or cannot be modeled by an exponential-family distribution.

  • Problem

    Existing self-supervised denoising approaches appear mathematically disparate, while supervised learning requires paired clean images that are difficult or impossible to acquire.

  • Method

    Noise2Score uses Tweedie’s formula to express denoising through a score function estimated with AR-DAE from noisy images.

  • Results

    Noise2Score surpasses SURE, PURE, Noise2X, and related methods, with experiments showing significant gains for Gaussian and Poisson noise and comparable performance to supervised learning and Noise2Noise.

  • Takeaways & Limitations

    Identical neural-network training can be used across exponential-family noise models and supports noise-parameter estimation with minimal complexity.

  • Takeaways & Limitations

    The method is limited when the noise distribution is unavailable or cannot be modeled by an exponential-family distribution.

Abstract

from arXiv · show

Recently, there has been extensive research interest in training deep networks to denoise images without clean reference. However, the representative approaches such as Noise2Noise, Noise2Void, Stein's unbiased risk estimator (SURE), etc. seem to differ from one another and it is difficult to find the coherent mathematical structure. To address this, here we present a novel approach, called Noise2Score, which reveals a missing link in order to unite these seemingly different approaches. Specifically, we show that image denoising problems without clean images can be addressed by finding the mode of the posterior distribution and that the Tweedie's formula offers an explicit solution through the score function (i.e. the gradient of log likelihood). Our method then uses the recent finding that the score function can be stably estimated from the noisy images using the amortized residual denoising autoencoder, the method of which is closely related to Noise2Noise or Nose2Void. Our Noise2Score approach is so universal that the same network training can be used to remove noises from images that are corrupted by any exponential family distributions and noise parameters. Using extensive experiments with Gaussian, Poisson, and Gamma noises, we show that Noise2Score significantly outperforms the state-of-the-art self-supervised denoising methods in the benchmark data set such as (C)BSD68, Set12, and Kodak, etc.

1 Introduction

The paper addresses the lack of a coherent mathematical theory connecting self-supervised denoising methods by using Bayesian statistics and Tweedie’s formula to unify denoising through score-function estimation. Noise2Score uses identical neural-network training across exponential-family noise models and reports strong empirical performance against existing self-supervised approaches.

  • The paper identifies the lack of a principled mathematical framework relating SURE and Noise2X despite their practical promise.
  • Tweedie’s formula provides an explicit denoising representation through the score function for exponential-family noise models.This reduces self-supervised denoising without clean images to estimating the score function.
  • Noise2Score combines score-function estimation with Tweedie’s formula, building on DAE and AR-DAE methods closely related to Noise2X training.AR-DAE is described as improving numerical stability and accuracy.
  • The same neural-network training step applies across exponential-family noise models, unlike SURE’s noise-model- and parameter-dependent retraining.The paper also connects this universality to blind settings with unknown noise parameters.
  • Experiments with additive Gaussian, Poisson, and Gamma noise show that Noise2Score significantly outperforms SURE and Noise2X under similar conditions.

2 Related Works

Existing denoising approaches differ mainly in their training targets, distance functions, or risk estimators. Their practical limitations include the need for paired clean images, unavailable multiple noisy realizations, and SURE losses that depend on noise models and parameters.

  • 2.1 Image denoising using supervised learning: Supervised denoising minimizes distance between a noisy input’s network output and paired clean ground truth.The approach requires large numbers of paired clean images, which are difficult or impossible to acquire in real applications.
  • 2.2 Noise2X for image denoising without clean images: Noise2X methods train without clean images by defining a target y′ related to the input y through algorithm-specific transformations.Noise2Noise uses another noisy realization, while Noise2Void and Noise2Self use blind spots or complementary masks.
  • 2.2 Noise2X for image denoising without clean images: These Noise2X target modifications are designed to prevent the network from learning a trivial identity mapping.
  • 2.3 Image denoising using Stein’s Unbiased Risk Estimate (SURE): SURE adds a divergence-based regularization term to the data-fidelity loss to prevent trivial identity mapping.Because calculating the divergence is difficult, Monte Carlo SURE introduces additional hyperparameters.
  • 2.3 Image denoising using Stein’s Unbiased Risk Estimate (SURE): SURE requires different loss functions for different noise models and retraining when Gaussian noise variance changes.This creates additional computational burden compared with Noise2X’s noise-model-independent loss.

3 Main Contribution: Noise2Score

Noise2Score reframes self-supervised denoising as posterior estimation via Tweedie’s formula and estimates the required score function with AR-DAE. The same training step applies across exponential-family noise models and supports blind parameter estimation without retraining.

  • Tweedie’s formulation: For Gaussian noise, the posterior mean is obtained as E[x|y] = y + σ2l′(y), where l′(y) is the score function of the noisy measurement.This posterior mean is the MMSE estimator when the score function is known.
  • Tweedie’s formulation: Noise2Score uses Tweedie’s formula to derive denoising estimates from the posterior and the score function for exponential-family noise distributions.The framework covers distributions including Gaussian, Poisson, and Gamma noise.
  • Score function estimation: The method estimates the score function with an amortized residual denoising autoencoder, whose training provides a direct and numerically stable estimator.AR-DAE addresses the numerical instability of the ordinary DAE score estimate as the auxiliary noise scale approaches zero.
  • Connections to prior methods: Noise2Score connects its score-estimation step to SURE and its noisy-image training to Noise2X while retaining Tweedie-based post-processing for the noise model.Unlike SURE, the approach uses the same training loss across noise models rather than changing the loss for each model.
  • Universal training: Noise2Score uses identical neural-network training across exponential-family noise models because model and parameter dependence enters through sampling noisy inputs.The sampling procedure can use random patch cropping regardless of whether the noise is Gaussian, Poisson, or Gamma.
  • Blind denoising: The same precomputed score estimate enables on-the-fly noise-parameter estimation, avoiding network retraining for different Gaussian, Gamma, or Poisson parameters.Unknown parameters can be selected by optimizing an image-quality penalty such as total variation.

4 Experimental Results

Noise2Score was evaluated on grayscale and color benchmark datasets under Gaussian, Poisson, and Gamma noise, including known- and unknown-parameter settings. It performed best among self-supervised methods and produced results comparable to supervised learning and Noise2Noise in the reported experiments.

  • Datasets and setup: Experiments covered BSD68, Set12, CBSD68, and Kodak using synthetic Gaussian, Poisson, and Gamma noise.The evaluation included grayscale and color images, with shared implementation settings for comparison methods.
  • Gaussian noise: Noise2Score achieved the best performance among self-supervised Gaussian denoising methods across all datasets.Its performance was only marginally below supervised learning and Noise2Noise, while blind estimation remained comparable to supervised learning.
  • Poisson noise: Noise2Score delivered significant performance gains for Poisson noise and remained comparable to supervised learning and Noise2Noise.The method also handled experiments with randomly sampled unknown noise levels.
  • Qualitative comparison: Figure 2 compares qualitative CBSD68 reconstructions for Gaussian σ = 50, Poisson ζ = 0.05, and Gamma k = 100 using PSNR values in dB.The figure labels noisy inputs and ground-truth images for visual comparison.
  • Gamma noise: Noise2Score produced the best results among self-supervised Gamma denoisers and remained comparable to Noise2Noise and supervised learning.This pattern also held for blind Gamma noise with k randomly selected from [40, 120].

5 Conclusion

The paper concludes that Noise2Score provides a Bayesian framework for self-supervised denoising without clean data. Tweedie’s formula and AR-DAE enable denoising across exponential-family noise models with identical training, including noise-parameter estimation.

  • Framework: Noise2Score introduces a Bayesian framework for self-supervised image denoising without clean data.The conclusion presents the framework as surpassing SURE, PURE, and Noise2X.
  • Core mechanism: Tweedie’s formula represents denoised images explicitly through the score function.The score function is estimated using AR-DAE.
  • Universality: Noise2Score applies to exponential-family noises using identical neural-network training regardless of the noise model.This shared training enables noise-parameter estimation with minimal complexity.

Limitation and negative societal impacts

The paper identifies risks from denoising failures and limits Noise2Score’s scope when noise priors are unavailable or the noise is not modeled by an exponential-family distribution.

  • Negative societal impacts: Removing both noise and texture from medical images could lead to misdiagnosis.The paper presents this as a possible negative societal impact of denoising failures.
  • Scope limitation: Noise2Score is limited when real environments lack prior knowledge of the noise distribution.The stated boundary concerns blind deployment settings where the noise prior is unavailable.
  • Scope limitation: The method’s stated noise-model scope excludes distributions that cannot be modeled by exponential-family noises.The paper identifies extension to such scenarios as future work.

B Quality metric for blind noise removal

Blind noise removal uses quality metrics tailored to each noise distribution, combining total variation with data fidelity terms for Poisson and Gamma noise. The reported estimates closely matched the target parameters for Gaussian, Poisson, and Gamma cases.

  • Quality metrics: Blind parameter estimation uses a quality penalty based on total variation, with distribution-specific data fidelity for Poisson and Gamma noise.Gaussian noise uses only the total variation norm, while Poisson and Gamma use additional fidelity terms.
  • Gaussian noise: 24.8 versus 25: the estimated Gaussian parameter σ* closely matched the target σ = 25.The estimate was obtained from the Gaussian quality metric Q(x̂(σ)).
  • Poisson noise: 0.00985 versus 0.01: the estimated Poisson parameter ζ* closely matched the target ζ = 0.01.The Poisson metric combines total variation with a data fidelity term and uses α = 0.1 for ζ = 0.01.
  • Gamma noise: 98.5 versus 100: the estimated Gamma parameter k* closely matched the target k = 100.The Gamma metric uses α = β = k and hyperparameters α = 0.5, β = 0.5, and γ = 1 under the stated stability condition.

C Detail of Dataset

The evaluation uses Kodak, CBSD68, Set12, and BSD68 as test datasets, with DIV2K and CBSD400 used for training.

  • The test datasets comprise Kodak, CBSD68, Set12, and BSD68, with BSD68 formed by converting CBSD68 images to grayscale.
  • DIV2K and CBSD400 serve as the training datasets for evaluating Noise2Score.
  • Training uses cropped 128×128 patches with random horizontal, vertical flip, and flop augmentation.
  • The dataset URLs are provided, including a Kodak dataset link.

D Implementation Detail

Implementation trains Noise2Score separately for each noise distribution, sampling either fixed or random noise parameters depending on the known-parameter setting. Additional comparisons use common training data and inference datasets.

  • Each Noise2Score model is trained using one noise distribution, with one fixed noise level for known-parameter experiments and randomly sampled levels for unknown parameters.
  • Annealed perturbation levels for learning the score function are specified separately for Gaussian, Gamma, and Poisson noise.The passage gives ranges for the Gaussian and Gamma cases and a separate range for Poisson noise.
  • Additional comparisons train all methods on combined DIV2K and BSD400 grayscale data and test them on BSD68 under Gaussian and Poisson noise.
  • Noise2Score outperformed Laine et al., Noisier2Noise, and Noise2Same in these additional comparisons.

F Application on Real Data

Real-data and mismatch experiments assess Noise2Score beyond synthetic matched-noise settings. The paper applies it to mixed Poisson-Gaussian fluorescence microscopy data and examines mismatched noise statistics and levels.

  • Real fluorescence microscopy data: The fluorescence microscopy dataset contains mixed Poisson-Gaussian noise, addressed by sequentially applying Gaussian and Poisson Tweedie denoising.The dataset contains 1,000 images, with 900 used for training and 100 for testing.
  • Noise-statistics mismatch: When training and inference noise statistics differ, all denoisers deteriorate, while the proposed method surpasses other methods in almost all cases.
  • Noise-level mismatch: With Poisson noise trained at ζ = 0.01, increasing the inference noise level reduces performance for all methods, but Noise2Score is most robust among self-supervised methods.

H Qualitative Results

Qualitative experiments compare denoising across Gaussian, Poisson, and Gamma corruption on grayscale and color benchmarks, including unknown-parameter settings. Noise2Score shows a reported 0.5∼3dB PSNR gain over state-of-the-art self-supervised approaches and comparable blind results to known-parameter results.

  • Known parameters: On Kodak and CBSD68 with known noise parameters, Noise2Score provides a 0.5∼3dB PSNR gain over state-of-the-art self-supervised approaches.The comparison covers color images and reports superb subjective image qualities.
  • Unknown parameters: For unknown noise parameters, the blind approach provides results comparable to the known-parameter method on grayscale and color datasets.
  • Known parameters: Figures compare Gaussian σ=50, Poisson ζ=0.05, and Gamma k=50 corruption using PSNR values shown on the images.The methods include SURE, PURE, Noise2Void, and Noise2Self.
  • Unknown parameters: The unknown-parameter qualitative comparisons use Set12 and BSD68 for grayscale images and Kodak and CBSD68 for color images.
Loading 2106.07009v2…