PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise2Same: Optimizing A Self-Supervised Bound for Image Denoising

Yaochen Xie, Zhengyang Wang, Shuiwang Ji

arXiv:2010.11971v1cs.CVcs.LG

TL;DR

Clean targets are often hard to obtain, while existing self-supervised denoisers require J-invariance that may reduce performance. Noise2Same derives a self-supervised upper-bound loss without that requirement and improves over previous self-supervised baselines on ImageNet, HànZì, and CARE.

  • Problem

    Clean images are hard to obtain in many denoising applications, and J-invariance may reduce self-supervised denoising performance.

    Source: §1 · pp. 1–2

  • Method

    Noise2Same derives a self-supervised upper-bound loss that does not require the denoising function to be J-invariant.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Noise2Same improves over previous self-supervised baselines on ImageNet, HànZì and CARE.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    Noise2Same can use information from the entire noisy input without post-processing or extra assumptions about the noise.

    Source: §3.2 · p. 5

    The invariance-term analysis uses additive Gaussian noise N(0, σ), though applying Noise2Same does not require knowing the noise model.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Self-supervised frameworks that learn denoising models with merely individual noisy images have shown strong capability and promising performance in various image denoising tasks. Existing self-supervised denoising frameworks are mostly built upon the same theoretical foundation, where the denoising models are required to be J-invariant. However, our analyses indicate that the current theory and the J-invariance may lead to denoising models with reduced performance. In this work, we introduce Noise2Same, a novel self-supervised denoising framework. In Noise2Same, a new self-supervised loss is proposed by deriving a self-supervised upper bound of the typical supervised loss. In particular, Noise2Same requires neither J-invariance nor extra information about the noise model and can be used in a wider range of denoising applications. We analyze our proposed Noise2Same both theoretically and experimentally. The experimental results show that our Noise2Same remarkably outperforms previous self-supervised denoising methods in terms of denoising performance and training efficiency. Our code is available at https://github.com/divelab/Noise2Same.

1 Introduction

Deep denoising has benefited from stronger neural architectures, while the need for training data has become more critical. Noise2Same targets self-supervised denoising from individual noisy images without requiring J-invariance or a known noise model.

1 Introduction

  • Supervised denoising requires matched noisy-clean image pairs, but clean images can be difficult to obtain because of instrument or cost limitations.Noise2Noise instead uses paired noisy images corresponding to the same unknown clean image, so it remains supervised with noisy targets.Source: §1 · p. 1
  • Noise2Noise is not always applicable because paired noisy images may be unavailable or have registration problems.Other individual-image methods either require a known noise model or operate without extra information.Source: §1 · pp. 1–2
  • Noise2Same introduces a self-supervised loss based on an upper bound of supervised loss, without requiring J-invariance or extra noise-model information.The authors report that it consistently outperforms previous self-supervised denoising methods.Source: §1 · p. 2
  • The authors argue that the J-invariance requirement underlying existing self-supervised frameworks may reduce denoising performance.Noise2Same is evaluated theoretically and experimentally, with reported improvements over previous self-supervised methods.Source: §1 · p. 2

2 Background and Related Studies

The section contrasts supervised denoising with individual-image self-supervision, whose standard theory relies on J-invariant denoisers. It introduces Noise2Same as a way to avoid that restriction and contrasts it with noise-model-dependent Bayesian post-processing.

2 Background and Related Studies

  • Supervised denoising learns a function from noisy-clean image pairs by minimizing expected squared error between its output and the clean image.The clean image is denoted y, and the denoiser maps noisy images x to reconstructed images.Source: §2 · p. 2
  • Individual-image self-supervised methods assume zero-mean noise independent across dimensions and use J-invariance to prevent learning the identity function.For each partition J, the output on J must not depend on the input values on J.Source: §2 · p. 2
  • Blind-spot networks implement J-invariance by predicting masked image values from their context, using masking or shifted receptive fields.The self-supervised objective compares the denoiser output with the noisy image.Source: §2 · p. 2
  • Noise2Same proposes a new self-supervised loss without requiring the denoising function to be J-invariant.The section argues that the existing J-invariant self-supervised objective is not optimal for denoising.Source: §2 · pp. 2–3
  • Bayesian post-processing uses the noise model to incorporate information from the target region, but requires a known noise model and assumes one noise type across dimensions.Unknown, inconsistent, or combined noise types therefore fall outside the stated applicability of this post-processing; Noise2Same uses the entire input without post-processing or a known noise model.Source: §2 · p. 3

3 Analysis of the J -Invariance Property

The section questions whether J-invariance is both achieved by mask-based methods and necessary for strong self-supervised denoising. Its analyses motivate Noise2Same, which uses a loss that does not require J-invariance.

3.1 Mask-Based Blind-Spot Denoising: Is the Optimal Function Strictly J -Invariant?

  • Masking blocks access to J during training, but mask-based models retain a weak dependency on J at inference and therefore are not guaranteed to be strictly J-invariant.The authors test this dependency using D(f), where smaller values indicate greater J-invariance.Source: §3.1 · p. 3
  • On BSD68, D(f) is close to zero for the strictly J-invariant shifted-receptive-field model but large for mask-based methods, varying across replacement strategies.Table 1 reports D(f) and PSNR for the replacement strategies; the prose attributes the near-zero value to mutual influences within J and numerical precision.Source: §3.1 · p. 4, Table 1 · p. 4
  • With a fixed replacement strategy, dataset choice also strongly affects J-invariance strictness, and similar large D(f) trends appear during training.The authors note that dataset effects are not under the denoising approach’s control.Source: §3.1 · p. 4, Table 2 · p. 4

3.2 Shifting Receptive Field: How do the Strictly J -Invariant Models Perform?

  • Strict J-invariance does not guarantee the best performance: shifted receptive fields beat Gaussian-replacement mask methods, while UPS performs best despite lower strictness.The authors suggest UPS can sometimes replace a masked location with its original value, yielding a linear combination of noisy input and a strictly J-invariant network output.Source: §3.2 · p. 4
  • Noise2Same replaces the J-invariant-loss requirement with a new self-supervised loss and can use information from the entire noisy input without noise-model assumptions or post-processing.Figure 1 depicts reconstruction loss against the full noisy image and an invariance loss between outputs for full and masked inputs.Source: §3.2 · p. 5, Figure 1 · p. 5

4 The Proposed Noise2Same Method

Noise2Same replaces the J-invariance requirement with a self-supervised upper-bound loss. Its invariance term controls how strictly the model behaves like a J-invariant function, with the appropriate strictness depending on noise strength.

4.1 Noise2Same: A Self-Supervised Upper Bound without the J -Invariance Requirement

  • Noise2Same minimizes a self-supervised upper bound on supervised loss without requiring the denoising function to be J-invariant, under the theorem’s stated noise assumptions.The bound is for normalized images with zero-mean i.i.d. noise.Source: §4.1 · pp. 5–6
  • The loss combines full-image reconstruction MSE with a weighted invariance MSE that discourages identity mapping and controls the degree of J-invariance.The default invariance weight is λinv = 2; other values may help in some cases.Source: §4.1 · p. 6

4.2 Analysis of the Invariance Term

  • Under additive zero-mean Gaussian noise, stronger noise calls for a more strictly J-invariant function that relies more on context and less on the noisy input.The analysis links the noise standard deviation σ to the weight of the invariance term.Source: §4.2 · p. 6
  • Unlike prior post-processing approaches [1] [12], Noise2Same can tune invariance without a known noise model, extending its scope to unknown, inconsistent, or combined noise.The cited post-processing methods cannot be applied when the noise model is unknown.Source: §4.2 · p. 6

5 Experiments

Experiments compare Noise2Same with traditional, supervised, and self-supervised methods across four datasets with different noise types and levels. Noise2Same improves on previous self-supervised baselines on three datasets, while experiments also examine training efficiency and the invariance term.

5.1 Comparisons with Baselines

  • Across ImageNet, HànZì, Planaria, and BSD68, the experiments include different noise types and levels; fair self-supervised comparisons use images alone when noise information is unavailable.ImageNet and HànZì have combined noise, while Planaria has an unknown noise model.Source: §5 · p. 7, §5.1 · pp. 7–8
  • Noise2Same remarkably improves over previous self-supervised baselines on ImageNet, HànZì, and CARE in PSNR comparisons.Table 3 summarizes PSNR across datasets and denoising methods.Source: §5.1 · p. 8, Table 3 · p. 8
  • Supervised methods still outperform self-supervised models, especially on Planaria and BSD68, and Noise2Same moves toward closing that gap.The authors suggest supervised training may carry extra information implicitly, such as noise-model information.Source: §5.1 · p. 8
  • Noise2Same has a similar convergence speed to the convolutional blind-spot network, while Noise2Self is expected to train more slowly because each step uses only a subset of output pixels.Figure 3 compares training cost using batch sizes of 64 for Noise2Same, 128 for Noise2Self, and 32 for Laine et al.; one cost unit is 50 minibatch steps.Source: §5.1 · pp. 8–9, Figure 3 · p. 8

5.2 Effect of the Invariance Term

  • On HànZì datasets with additive Gaussian noise, the best performance is achieved when σloss is close to σnoise, consistent with Theorem 2.The experiments vary σloss from 0.1 to 1.0 across four noise levels.Source: §5.2 · p. 9, Figure 4 · p. 9

6 Conclusion and Future Work

The conclusion presents Noise2Same as a self-supervised method that removes the J-invariance restriction. It reports denoising capability and identifies noise-model-augmented Noise2Same as future work.

6 Conclusion and Future Work

  • Noise2Same removes the assumption and restriction that the denoising network be J-invariant.The conclusion describes it as a self-supervised denoising method.Source: §6 · p. 9
  • The paper reports that its analyses and experiments demonstrate Noise2Same’s denoising capability.Source: §6 · p. 9
  • Future work will explore Noise2Same augmented with a noise model.The conclusion describes this direction as orthogonal work.Source: §6 · p. 9

Broader Impact

The broader-impact discussion describes potential uses of Noise2Same in image denoising, computer vision, and research imaging. It also cautions that failures could affect downstream tasks built on denoising algorithms.

Broader Impact

  • Noise2Same could denoise digital images captured under poor conditions and serve as preprocessing for object detection or segmentation.The authors suggest this could make downstream algorithms more robust to noisy images.Source: Broader Impact · p. 9
  • In microscopy, denoising low-quality data could reduce the need to capture high-quality data directly, which the authors note is costly in budget and time.Source: Broader Impact · p. 9
  • The self-supervised framework could extend beyond image denoising to audio noise reduction and single-cell applications.Source: Broader Impact · p. 9
  • If Noise2Same fails, imaging and computer-vision tasks built on denoising algorithms could potentially inherit biases or failures.Source: Broader Impact · p. 10

A Proof of Theorem 1

The proof bounds the cross-term in the supervised-loss decomposition rather than setting it to zero under J-invariance. It uses the condition E[x|y] = y and controls the bound for normalized inputs.

A Proof of Theorem 1

  • The proof controls the cross-term in Equation (1) with an upper bound, assuming E[x|y] = y rather than J-invariance.The bound starts from a conditional covariance term and uses Cauchy-Schwarz for normalized inputs.Source: A Proof of Theorem 1 · pp. 12–13
  • Uniformly sampling a subset J lets the proof express the covariance bound over the selected image dimensions.The derivation uses the independence of noise and the fact that j is outside Jc.Source: A Proof of Theorem 1 · p. 12
  • Equation (20) decomposes the supervised loss into the noisy-image reconstruction term and a cross-term involving f(x) − y and x − y.Source: A Proof of Theorem 1 · p. 13

B Proof of Theorem 2

Theorem 2 repeats the cross-term control with the stronger assumption of known additive, zero-mean noise. Its standard deviation determines the conditional noise variance used in the proof.

B Proof of Theorem 2

  • Known additive noise with standard deviation σ and zero mean gives Var(xj − yj|y) = σ².Source: B Proof of Theorem 2 · p. 13
  • The proof applies the Theorem 1 control strategy to bound Equation (1) under the known-noise assumption.Equation (28) gives the resulting decomposition into reconstruction and cross-term losses.Source: B Proof of Theorem 2 · p. 13

C Dataset Constructions

The experiments use constructed natural-image and character datasets alongside physically acquired 3D microscopy data. The appendix also specifies default training choices and baseline configuration exceptions.

C Dataset Constructions

  • The RGB, HànZì, and BSD68 datasets provide natural-image and character benchmarks with specified noise and train/test construction procedures.RGB uses combined Poisson, Gaussian, and Bernoulli noise; HànZì uses Gaussian and Bernoulli noise; BSD68 uses Gaussian noise with σ = 25.Source: C Dataset Constructions · pp. 13–14
  • Planaria microscopy tests use 20 large 3D images at three noise levels, with noise increasing from C1 to C3.Training uses 17,005 patches of size 16 × 64 × 64 from the three conditions.Source: C Dataset Constructions · p. 14

D Implementation Details

  • The default model is a depth-3 U-Net, and 3D augmentation rotates only width and height because of depth anisotropy.Training also uses masking, normalization, and a decaying learning rate.Source: D Implementation Details · p. 14

E Model Configurations

  • For BSD68, Noise2Same uses λinv = 0.95; some baselines are omitted on Planaria because they are inapplicable or lack a public 3D version.BM3D does not apply to 3D images, and Noise2Noise requires unavailable paired noisy images.Source: E Model Configurations · p. 14
  • Figure 5 compares NLM, CARE, Noise2Self, and Noise2Same, with rows for C1–C3 and columns ordered by denoising quality.Source: Figure 5 · p. 15

F Denoising Results on the Planaria Dataset

The Planaria denoising results are shown as 2D projections of the 3D images.

F Denoising Results on the Planaria Dataset

Related papers

Loading 2010.11971v1…