PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise2Self: Blind Denoising by Self-Supervision

Joshua Batson, Loic Royer

arXiv:1901.11365v2cs.CVcs.LGstat.ML

TL;DR

The paper addresses denoising high-dimensional measurements without signal priors, noise estimates, clean targets, or repeated measurements. It introduces J-invariant self-supervision using conditionally independent noisy dimensions, and demonstrates blind denoising across analytical, image, microscopy, and single-cell settings with neural networks performing comparably to clean-target and independently noisy-target training.

  • Problem

    Traditional denoising requires assumptions about signal or noise structure, while supervised and Noise2Noise approaches require clean or repeated measurements that can be difficult to obtain.

  • Method

    The framework trains or calibrates J-invariant denoisers by predicting each group of conditionally independent noisy features from the remaining features.

  • Results

    A masked J-invariant deep convolutional network achieves state-of-the-art blind denoising performance on three diverse datasets, while self-supervised neural nets are comparable to clean-target and independently noisy-target training.

  • Takeaways & Limitations

    The framework supports blind denoising and calibration of both traditional filters and large neural networks from single noisy measurements, including under-sampled single-cell gene-expression data.

  • Takeaways & Limitations

    The theoretical guarantee assumes unbiased measurements and conditional independence between each noise subset and its complement; image clipping can violate conditional mean-zero noise and induce bias.

Abstract

from arXiv · show

We propose a general framework for denoising high-dimensional measurements which requires no prior on the signal, no estimate of the noise, and no clean training data. The only assumption is that the noise exhibits statistical independence across different dimensions of the measurement, while the true signal exhibits some correlation. For a broad class of functions ("$\mathcal{J}$-invariant"), it is then possible to estimate the performance of a denoiser from noisy data alone. This allows us to calibrate $\mathcal{J}$-invariant versions of any parameterised denoising algorithm, from the single hyperparameter of a median filter to the millions of weights of a deep neural network. We demonstrate this on natural image and microscopy data, where we exploit noise independence between pixels, and on single-cell gene expression data, where we exploit independence between detections of individual molecules. This framework generalizes recent work on training neural nets from noisy images and on cross-validation for matrix factorization.

1. Introduction

The paper introduces blind denoising from single noisy measurements by exploiting conditional independence in the noise and correlation in the signal. J-invariant self-supervision estimates denoising performance from noisy data alone and applies from simple filters to deep networks and single-cell data.

  • Motivation: High-dimensional measurements can be extremely noisy even when the underlying objects are structured and feature values are correlated.The framework targets measurements from cameras, electron microscopes, and DNA sequencers without requiring prior signal knowledge.
  • Motivation: Traditional denoisers depend on assumptions about noise or signal structure, and their hyperparameters require calibration that can strongly affect performance.Examples include Gaussian noise, smoothness, self-similarity, low rank, filter scale, and matrix rank.
  • Framework: The proposed framework uses groups of conditionally independent noisy features to predict one another, enabling denoising from a single noisy measurement of each object.It also calibrates traditional methods such as median filters and non-local means.
  • Framework: A J-invariant function predicts values on each dimension subset without using the measurements in that subset, preventing the identity function from minimizing the self-supervised objective.The function must use information outside each subset to predict the values inside it.
  • Framework: Under unbiasedness and conditional independence between each subset and its complement, self-supervised loss equals supervised denoising loss plus noise variance.Thus minimizing the self-supervised loss over J-invariant functions can identify the optimal denoiser for the dataset.
  • Applications: For independent pixel noise, donut median filters allow the self-supervised loss to select the optimal filter radius for the image at hand.The same framework extends to analytical optimality, deep convolutional networks, and independently detected RNA molecules.

2. Related Work

Related denoising methods exploit assumptions such as smoothness, self-similarity, generative structure, Gaussian noise, sparsity, or compressibility. Noise2Noise and masking-based approaches instead exploit statistical independence, but differ in data requirements and theoretical guarantees.

  • Signal and noise assumptions: Smoothness-based methods locally average noisy measurements, with the degree of smoothing commonly tuned by a filter-width hyperparameter.Gaussian and median filters are simple examples.
  • Signal and noise assumptions: Self-similarity methods denoise by combining similar image patches, as in non-local means and BM3D, whose hyperparameters strongly affect performance.Deep image prior methods exploit self-similarity through a generative convolutional network and early stopping.
  • Signal and noise assumptions: Other approaches use generative models, Gaussian-noise-specific Stein losses, sparsity, or lossy compression to constrain denoising.These methods depend on the applicability of their model or on choices such as basis, threshold, and compression degree.
  • Neural architectures: Autoencoders compress noisy data through a bottleneck, while UNets can reproduce fine detail and may learn the identity function when trained directly on noisy data.UNets trained with clean targets can instead learn accurate denoising functions.
  • Statistical independence: Noise2Noise trains on independent noisy measurements of the same signal and can be expressed using J-invariant functions, whereas video extensions use optical-flow alignment.These approaches require repeated measurements of each target, which can be difficult in practice.
  • Statistical independence: A concurrent masking approach predicts held-out pixels but is not fully J-invariant and lacks a theoretical guarantee against fitting the noise.It nevertheless performs well on natural and microscopy images.

3. Calibrating Traditional Models

The paper calibrates denoising parameters without ground truth by minimizing a self-supervised loss for J-invariant functions. This approach selects effective settings for classical image denoisers and principal component regression on undersampled single-cell data.

  • Calibrating Traditional Models: Self-supervised loss matches ground-truth denoising loss up to the noise variance for J-invariant denoisers, preserving the same minimizer.For general denoisers, the noisy-data loss is unrelated to ground-truth performance.
  • Calibrating Traditional Models: The donut median selects the optimal radius r = 3, while the ordinary median’s self-supervised loss is strictly increasing and uninformative.The donut loss minimum aligns with the ground-truth minimum; its vertical offset equals the noise variance.
  • Calibrating Traditional Models: Masking pixels before applying a classical denoiser creates a J-invariant version whose output cannot depend on the masked subset.This construction enables self-supervised calibration of filters such as wavelets and non-local means.
  • Calibrating Traditional Models: Among three J-invariant denoisers, the model with the best self-supervised loss also achieves the best PSNR.Performance can exceed default classical-model settings and may improve further by optimally mixing the noisy input with the J-invariant output.
  • 3.1. Single-Cell: Single-cell measurements can be split into conditionally independent molecule groups, allowing self-supervised principal component regression to select 17 components.The method is demonstrated on 2730 bone marrow cells, comparing too few, too many, and optimal components.

4. Theory

The theory identifies the optimal J-invariant predictor as a conditional expectation and studies how its recovery quality depends on feature correlation and signal structure. It also shows that combining a J-invariant estimate with the noisy input can improve performance.

  • Correlation: As feature correlation increases, the optimal J-invariant denoiser approaches the optimal general denoiser.The framework studies this convergence analytically and illustrates it using Gaussian processes.
  • Signal structure: Gaussian processes provide a worst-case recovery error for a given covariance structure, while nonlinear structure can yield better predictions.For noisy handwritten digits, the alphabet predictor visibly outperforms a Gaussian process with the same covariance matrix.
  • Signal structure: The optimal J-invariant predictor can be substantially better than a Gaussian process because it exploits nonlinear structure in finite template datasets.For alphabet data with Gaussian noise, predictions weight templates by their posterior probabilities and approach the closest template at low noise.
  • Improving the estimate: An optimal linear combination of a J-invariant output and the noisy input produces an estimate with lower variance than either component alone.The optimal mixing weight equals the noise variance divided by the self-supervised loss; a 10 dB PSNR improvement can yield another 0.4 dB.

5. Deep Learning Denoisers

The paper trains J-invariant deep convolutional networks using self-supervised loss from one noisy sample per image. Across three diverse datasets, Noise2Self outperforms default classical denoisers and matches neural networks trained with clean or independently noisy targets.

  • Training: A self-supervised loss trains a deep convolutional neural network using only one noisy sample of each image.The network is made J-invariant through a masking procedure and trained on data with independently applied pixel noise.
  • Datasets and noise: The experiments cover three domains with varied heteroscedastic synthetic noise, including Poisson, Gaussian, Bernoulli, and realistic sCMOS camera noise.Hànzì and ImageNet use mixtures of synthetic noise, while CellNet uses simulated sCMOS noise.
  • Results: Noise2Self outperforms NLM and BM3D at default parameters and performs comparably to Noise2Truth and Noise2Noise.The comparison is reported across the three datasets in Table 2.
  • Results: Applying the trained network directly to noisy input performs 0.5 dB better than using its J-invariant version.The reported images use the directly applied network rather than the J-invariant output.
  • Single-image training: A DnCNN with 560,000 parameters trained self-supervised on one noisy camera image achieves a PSNR of 31.2.The training image contains 260,000 pixels.

6. Discussion

The framework calibrates or trains any J-invariant denoising function using a self-supervised loss, while leaving the choice of partition J as an open problem. The paper discusses applications across imaging and single-cell gene-expression data, with broader domains suggested for future work.

  • Figure 6 compares classic, supervised, and self-supervised denoising methods across natural images, Chinese characters, and fluorescence microscopy images.
  • A self-supervised loss can calibrate or train any J-invariant class of denoising functions.
  • The optimal partition J remains problem-dependent because it must reflect signal dependence and noise independence.Partition size creates a bias-variance tradeoff by exchanging predictive information for information about prediction quality.
  • The authors suggest possible applications in agricultural or geological sensor networks, whole-brain neuronal time series, and telescope observations.

1. Notation

The notation defines coordinate restrictions, partitions, and singleton-complement notation for vectors in R^m. These operators specify how subsets of measurement dimensions are represented.

  • For x ∈ R^m and J ⊂ {1, . . . , m}, x_J denotes the restriction of x to coordinates in J.
  • For a function f: R^m → R^m, f(x)_J denotes the restriction of its output to coordinates in J.
  • A partition J is a collection of disjoint subsets whose union is the full set.
  • When J = {j} is a singleton, x_-j denotes the restriction to all coordinates except j.

2. Gaussian Processes

This section develops the conditional-mean foundation for denoising and analyzes optimal J-invariant predictors, including a Gaussian-process comparison based on covariance structure.

  • The minimum-MSE estimator of y from x is the conditional expectation E[y|x].Its expected MSE equals the conditional variance of y given x.
  • The Gaussian-process argument uses covariance factorization and projection onto the column span of a covariance factor.
  • Gaussian random variables with the same covariance matrix support a comparison of their optimal J-invariant predictors.
  • The proof invokes conditional-variance identities, the law of total variance, and a covariance calculation for E[y|x].
  • For J-invariant prediction, each coordinate is predicted from the complementary coordinates, with f*_J(x)_j = E[y_j|x_Jc].

3. Masking

The masking section constructs J-invariant functions by replacing selected inputs with values that exclude the target coordinate, then examines how masking and estimator mixing affect denoising.

  • Masking construction: A masking transformation combines an interpolation function s(x) with an indicator for J to produce a J-invariant function.
  • Masking construction: For calibration, s is a local average that excludes the center pixel; a random uniform replacement is another possible construction.
  • Uniform Pixel Selection: UPS masking can leak information because the target pixel helps estimate the local replacement distribution, so the resulting function is not genuinely J-invariant.
  • Uniform Pixel Selection: Among J-invariant functions, the self-supervised optimum for coordinate j is E[x_j|x_-j].
  • Uniform Pixel Selection: UPS training can produce a linear combination of the noisy input and the Noise2Self optimum, with smaller selection regions increasing the noise contribution.
  • Uniform Pixel Selection: The proposed modification excludes x_j when estimating the local distribution, for example by replacing it with a random neighbor.
  • Linear combinations: The variance of a mixed estimator is UV/(U + V), and when component variances match, mixing halves the variance.
  • Linear combinations: If a denoiser reduces noise variance by a factor of 10, the optimal mixing weight is λ* = 1/11 and PSNR is 0.43 higher than for the denoiser alone.

4. Calibrating Traditional Denoising Methods

The authors evaluate traditional denoisers on a standard camera image and calibrate their parameters without ground truth. Calibration plots are shown for median, wavelet, and non-local means methods.

  • The experiments use the full scikit-image camera image, with inset regions displayed in the figures.
  • Median, wavelet, and non-local means implementations are evaluated with noise standard deviation 0.1 on a [0, 1] scale.
  • Calibration plots compare parameter choices for the median filter in the text and for wavelet and non-local means denoisers in Supplementary Figure 1.

5. Neural Net Examples

The neural-network experiments test blind denoising across synthetic character, microscopy, and natural-image datasets. A masked J-invariant UNet is trained and evaluated alongside direct application to noisy inputs, with PSNR used for reconstruction quality.

  • Datasets: The experiments use Hànzì, CellNet, and ImageNet datasets spanning synthetic characters, fluorescence microscopy, and natural photographs.Hànzì contains 13029 characters; CellNet contains 34630 image tiles; ImageNet contributes 60000 cropped images.
  • Datasets: ImageNet tiles receive combined Poisson, Gaussian, and Bernoulli noise, while Hànzì receives Gaussian and Bernoulli noise.
  • Model and training: The model is a UNet with skip connections, convolutional blocks, InstanceNorm, and channel widths [32, 64, 128, 256].
  • Model and training: A neural network is converted into a random J-invariant function using masking, and training restricts the loss to randomly selected coordinate subsets.
  • Inference: Inference compares a genuinely J-invariant reconstruction requiring 25 network applications with direct application of the trained network.
  • Inference: Direct application performs about 0.5 dB better than the J-invariant reconstruction across the three datasets.
  • Evaluation: Reconstruction quality is evaluated with PSNR on fully reconstructed hold-out test images.PSNR is a log-transformation of mean-squared error for images with range [0, 1].
  • Evaluation: Clipping makes image noise conditionally non-zero-mean, causing blind methods to exhibit shrinkage that is corrected by matching output mean and variance to ground truth.

6. Single-Cell Gene Expression

The single-cell analysis models molecule detections with Poisson statistics and applies a normalized representation to accommodate the large dynamic range of gene expression. The resulting optimal denoiser is directionally informative but not an unbiased estimate after inversion.

  • Observation model: Molecule counts are modeled as xi ∼ Poisson(Nλi), approximating multinomial sampling from a cell’s gene-expression density.
  • Normalization: Because gene expression spans about five orders of magnitude, the analysis uses a normalized variable with a nonlinearity such as square root or logarithm.
  • Normalization: For the Paul et al. dataset, normalization uses the median total molecule count per cell and a square-root nonlinearity.
  • Denoising target: The optimal denoiser is defined on the normalized variable rather than directly on molecular counts.
  • Denoising target: The resulting function of λi is positive, monotonic, and maps zero to zero, providing a directionally informative estimate of gene expression.
  • Caveat: Inverting the nonlinear expectation does not produce an unbiased estimate of λi, although the transformed quantity accommodates gene-expression dynamic range.
  • Measurement context: Unique molecular identifiers can deduplicate reads originating from the same molecule after PCR amplification.
Loading 1901.11365v2…