PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Learning with Stein's Unbiased Risk Estimator

Christopher A. Metzler, Ali Mousavi, Reinhard Heckel, Richard G. Baraniuk

arXiv:1805.10531v3stat.MLcs.CVcs.LG

TL;DR

Image recovery from noisy linear measurements is difficult to train when clean target images are unavailable. The paper uses SURE and its generalizations to train CNNs for denoising and compressed-sensing recovery without ground truth, and reports effective recovery across these settings. It also identifies limitations in the stability and speed of the single-image approach.

  • Problem

    CNN recovery training typically needs representative images and clean targets, which may be unavailable even when noisy measurements can be collected.

    Source: §1 · pp. 1–2

  • Method

    The paper uses SURE and generalized SURE to estimate reconstruction MSE and train CNNs from noisy observations or measurements without clean images.

    Source: §2 · pp. 2–3, §8 · p. 10

  • Results

    SURE supports CNN denoising without training data, denoising from noisy training data, and compressive-sensing recovery from noisy measurements.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work suggests SURE as a hands-off route to image reconstruction and divergence as a possible proxy for estimator variance beyond SURE’s specific setting.

    Source: §8 · p. 10

    The Deep Image Prior training dynamics can favor fitting noisy data or driving divergence downward, although recovered solutions remain stable for many hundreds of iterations.

    Source: §7 · p. 10

Abstract

from arXiv · show

Learning from unlabeled and noisy data is one of the grand challenges of machine learning. As such, it has seen a flurry of research with new ideas proposed continuously. In this work, we revisit a classical idea: Stein's Unbiased Risk Estimator (SURE). We show that, in the context of image recovery, SURE and its generalizations can be used to train convolutional neural networks (CNNs) for a range of image denoising and recovery problems without any ground truth data. Specifically, our goal is to reconstruct an image $x$ from a noisy linear transformation (measurement) of the image. We consider two scenarios: one where no additional data is available and one where we have measurements of other images that are drawn from the same noisy distribution as $x$, but have no access to the clean images. Such is the case, for instance, in the context of medical imaging, microscopy, and astronomy, where noise-less ground truth data is rarely available. We show that in this situation, SURE can be used to estimate the mean-squared-error loss associated with an estimate of $x$. Using this estimate of the loss, we train networks to perform denoising and compressed sensing recovery. In addition, we also use the SURE framework to partially explain and improve upon an intriguing results presented by Ulyanov et al. in "Deep Image Prior": that a network initialized with random weights and fit to a single noisy image can effectively denoise that image. Public implementations of the networks and methods described in this paper can be found at https://github.com/ricedsp/D-AMP_Toolbox.

1 Introduction

Image recovery reconstructs an image from noisy measurements by combining the measurement operator with prior information about image distributions. CNN priors often perform well, but standard training depends on representative images and clean targets, motivating recovery from noisy data alone.

1 Introduction

  • Image recovery reconstructs x from noisy linear measurements y = Hx + w using image-distribution priors and knowledge of H.The setup covers denoising, inpainting, superresolution, deblurring, and compressive sensing.Source: §1 · p. 1
  • CNN priors often offer strong image-recovery performance, while the role of training data is complicated by evidence that CNN structure itself encodes prior information.The paper places CNNs among priors ranging from sparse models to non-local self-similarity priors.Source: §1 · pp. 1–2
  • Standard CNN training learns a mapping from measurements to images by minimizing MSE against clean images, requiring representative examples that may be unavailable.Even when example images exist, additional measurements may be available to refine the reconstruction algorithm.Source: §1 · p. 2
  • SURE can train neural networks to denoise without training data, learn from noisy training data, and solve compressive sensing using only noisy measurements.The paper focuses on image recovery, where learning from noisy data had been studied less than noisy-label learning in classification.Source: §1 · p. 2

2 SURE and its Generalizations

SURE estimates an estimator’s mean-squared error under Gaussian noise, allowing parameter optimization without clean targets. Its practical use depends on estimating divergence, while generalized SURE extends the approach to noisy linear measurements.

2 SURE and its Generalizations

  • SURE’s generalized form addresses reconstruction from noisy linear measurements when H is not the identity.Source: §2 · pp. 2–3
  • SURE provides an unbiased estimate of an estimator’s MSE for Gaussian observations, and its loss balances fidelity to observations against a divergence penalty on input sensitivity.The fidelity term is described as bias-related and the divergence term as variance-related.Source: §2 · p. 2
  • Advanced denoisers make analytic divergence computation difficult or impossible, creating the main practical challenge in using SURE.Source: §2 · p. 2
  • Monte Carlo SURE estimates divergence using random Gaussian perturbations, with the expectation approximated by sampling.The method is introduced for bounded functions; the text says a single sample approximates the expectation well for high-dimensional images and a small epsilon approximates the limit.Source: §2 · p. 3
  • Combining SURE with a Monte Carlo divergence estimate enables denoising-algorithm MSE minimization without ground-truth data.Source: §2 · p. 3
  • For linear measurements with known-covariance exponential-family noise, GSURE estimates reconstruction MSE; with i.i.d. Gaussian noise, its expression can be optimized without knowing x.The expression uses P, the orthonormal projection onto the range of H, and H†, the pseudoinverse.Source: §2 · p. 3

3 Denoising Without Training Data

A randomly initialized CNN can recover a single noisy image, but fitting it too long overfits the noise. Training with SURE avoids overfitting and achieves a better final NMSE than optimal stopping with fidelity loss.

3.1 Deep Image Prior

  • A randomly initialized CNN fit to one corrupted image can recover it, but training to convergence overfits noise, so early stopping is needed.For a 512 × 512 Mandrill image with AWGN standard deviation 25, NMSE begins to rise after a few iterations while training loss continues to decay.Source: §3 · p. 3, §3.1 · p. 4

3.2 SURE Deep Image Prior

  • The fidelity loss alone gives little indication of when to stop, whereas the SURE loss tracks NMSE and can outperform optimal stopping.The SURE-trained network is competitive with DnCNN and CBM3D, but is very slow; training across additional images is beneficial.Source: §3.1 · p. 4, §3.2 · pp. 4–5
  • SURE training avoids overfitting and achieves a better final NMSE than optimal stopping with fidelity loss.The SURE estimate of loss lies directly on top of observed NMSE.Source: §3.2 · p. 4

4 Denoising with Noisy Training Data

With additional noisy training images, the authors train DnCNN using SURE instead of requiring clean targets. Reconstructions are almost equivalent to clean-image MSE training, and both approaches outperform BM3D, though SURE training takes roughly twice as long.

4.1 DnCNN and Experimental Setup

  • The denoising experiment trains DnCNN on grayscale BSD-500 images and tests it on six standard images.Training images were transformed into 204,800 overlapping 40 × 40 patches.Source: §4.1 · p. 5
  • DnCNN trained with SURE on noisy data produces reconstructions almost equivalent to clean-image MSE training, and both outperform BM3D.Calculating SURE requires two calls to the denoiser, making training roughly twice as long as MSE training.Source: §4.1 · p. 6, Figure 5 · p. 6

5 Compressive Sensing Recovery with Noisy Training Measurements

This section trains an image-recovery network from noisy, undersampled measurements rather than ground-truth labels. LDAMP turns recovery into layerwise denoising problems, and SURE-based training performs roughly on par with BM3D-AMP while GSURE performs worse.

5 Compressive Sensing Recovery with Noisy Training Measurements

  • The section trains recovery from noisy, undersampled linear measurements without ground-truth training labels.This differs from conventional compressive-sensing learning approaches that use training labels.Source: §5 · p. 6

5.1 LDAMP

  • LDAMP unrolls DAMP into layers that denoise effective noisy versions of the image, with an Onsager correction that removes bias from intermediate solutions.The correction makes the effective noise variance accurately predicted at each layer.Source: §5.1 · p. 7

5.2 Training LDAMP and Experimental Setup

  • LDAMP SURE trains using only noisy measurements by applying SURE to layerwise denoising problems with known noise variance.The comparison also includes LDAMP MSE, which minimizes MSE on training data, and LDAMP GSURE, which uses generalized SURE.Source: §5.2 · p. 7
  • LDAMP trained with SURE and MSE performs roughly on par with BM3D-AMP and runs significantly faster, while GSURE offers significantly reduced performance.For the reported setting, SURE-trained LDAMP is on par with BM3D-AMP using modified fast JL transform matrices at n = 0.2 and Gaussian noise with standard deviation 1.Source: Table 2 · p. 8, Figure 6 · p. 8, §5.2 · p. 8
  • GSURE performs poorly in compressive sensing because it minimizes error between projected images, and the small measurement range makes that distance nonproportional to full-image error.The SURE and GSURE methods estimate divergence using Monte Carlo.Source: §5.2 · pp. 7–8

6 Related Work

Related work traces SURE from wavelet denoising to noisy-data learning and connects compressive-sensing recovery to risk-based algorithm tuning and universal compressed sensing.

Denoising with SURE

  • SureShrink uses SURE to set parameters in wavelet-domain thresholding, followed by other wavelet-thresholding denoisers that improve on the basic idea.Source: §6 · p. 8
  • SURE and related techniques have also been generalized to Poisson, multiplicative, and other noise distributions and used in least-squares estimation, kernel regression, and channel estimation.Source: §6 · p. 9

Denoising with Noisy Training Data

  • Related noisy-training approaches include risk-estimator training for discrete signals and images, GAN training from corrupted images, and restoration from pairs of corrupted images without knowing the noise distribution.Source: §6 · p. 9

Compressive Sensing

  • Related compressive-sensing methods use (G)SURE to tune coordinate-descent or AMP algorithms, or use SURE to adapt an AMP denoiser at each iteration.Source: §6 · p. 9
  • The proposed method is described as a practical form of universal compressed sensing, estimating a signal distribution from noisy linear measurements for reconstruction.The passage also relates it to blind compressed sensing, where signals sparse in an unknown dictionary are reconstructed from compressive measurements.Source: §6 · p. 9

7 Interpreting the Deep Image Prior

The paper interprets Deep Image Prior through a Taylor-expanded loss and contrasts its Jacobian-norm regularization with SURE’s divergence penalty. Although the SURE loss can permit pathological solutions, the reported learning dynamics remain stable for many hundreds of iterations.

7 Interpreting the Deep Image Prior

  • Noise regularization dramatically improves Deep Image Prior performance, and adding noise at each optimization iteration makes the optimization a stochastic problem.The paper uses this observation as the starting point for its interpretation of the method.Source: §7 · p. 9
  • A Taylor expansion of fθ around z expresses loss function (8), leading to the interpretation that deep image prior minimizes loss function (9) rather than loss function (7).Source: §7 · pp. 9–10
  • When z = y, the Jacobian’s Frobenius norm penalizes loss curvature around y and encourages the vector field fθ to be small there.This is the geometric interpretation of the Taylor-expanded loss in the setting used in Section 3.Source: §7 · p. 10
  • SURE instead penalizes the Jacobian trace, which geometrically encourages fθ to be a sink at y.Unlike the Deep Image Prior interpretation, the SURE loss does not require jittering the input.Source: §7 · p. 10
  • With enough parameters, the network can fit the noisy signal exactly while minimizing regularization, and the SURE loss could drive divergence toward negative infinity.In practice, the recovered solutions remain stable for many hundreds of iterations.Source: §7 · p. 10

8 Conclusions

The paper identifies three SURE-based image-recovery contributions and points to possible implications for imaging practice and machine learning. It suggests both sensor-driven prior selection in novel imaging environments and divergence as a possible proxy for estimator variance.

8 Conclusions

  • SURE supports CNN denoising without training data, CNN denoiser training from noisy data, and neural-network compressive sensing from noisy measurements.The three contributions cover a single image without training data, noisy training examples, and noisy measurements for compressive sensing.Source: §8 · p. 10
  • In imaging, SURE suggests a hands-off approach in which a sensor could determine and apply an appropriate image prior in a novel environment.This is presented as a possible application of the proposed framework.Source: §8 · p. 10
  • In machine learning, divergence may be an overlooked proxy for estimator variance, and penalizing it could help address overfitting beyond SURE’s specific settings.The paper presents this as a suggestion, not a demonstrated general result.Source: §8 · p. 10

Related papers

Loading 1805.10531v3…