Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser

Zahra Kadkhodaie; Eero P. Simoncelli

Source-linked AI summary

Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser

Zahra Kadkhodaie, Eero P. Simoncelli

arXiv:2007.13640v3Jul 27, 2020cs.CVeess.IVstat.ML

TL;DR

High-dimensional natural-image priors are difficult to model explicitly, although CNN denoisers implicitly encode useful image structure. The paper converts denoiser outputs into noisy-density gradients for stochastic prior sampling and constrained linear inverse problems. The same pretrained-denoiser procedure achieves state-of-the-art unsupervised performance across deblurring, super-resolution, inpainting, and compressive sensing.

Problem
Explicitly modeling high-dimensional natural-image densities is difficult, while CNN denoisers contain sophisticated implicit priors that are intertwined with their training task.
Method
The method uses Miyasawa’s denoising result to perform stochastic coarse-to-fine gradient ascent from a denoiser residual, with constrained sampling for linear measurements.
Results
The same algorithm and parameters are applied to deblurring, super-resolution, inpainting, and compressive sensing, using only changed measurement matrices and values.
Takeaways & Limitations
A denoiser trained for blind least-squares Gaussian denoising can provide a transferable implicit prior for sampling and multiple linear inverse problems without further training.
Takeaways & Limitations
Performance is not well captured by comparisons with the original image using PSNR or SSIM, and human-observer or no-reference perceptual evaluation is suggested instead.

Abstract

from arXiv · show

Prior probability models are a fundamental component of many image processing problems, but density estimation is notoriously difficult for high-dimensional signals such as photographic images. Deep neural networks have provided state-of-the-art solutions for problems such as denoising, which implicitly rely on a prior probability model of natural images. Here, we develop a robust and general methodology for making use of this implicit prior. We rely on a statistical result due to Miyasawa (1961), who showed that the least-squares solution for removing additive Gaussian noise can be written directly in terms of the gradient of the log of the noisy signal density. We use this fact to develop a stochastic coarse-to-fine gradient ascent procedure for drawing high-probability samples from the implicit prior embedded within a CNN trained to perform blind (i.e., with unknown noise level) least-squares denoising. A generalization of this algorithm to constrained sampling provides a method for using the implicit prior to solve any linear inverse problem, with no additional training. We demonstrate this general form of transfer learning in multiple applications, using the same algorithm to produce state-of-the-art levels of unsupervised performance for deblurring, super-resolution, inpainting, and compressive sensing.

1 Introduction

The paper addresses the difficulty of modeling high-dimensional natural-image densities by using the implicit prior embedded in CNN denoisers. It derives a stochastic sampling and constrained-inference framework from the connection between least-squares denoising and noisy-density gradients.

Motivation: Traditional image priors use simple parametric or transformed-coordinate models that cannot represent complex visual features or demanding inference problems.These models include Gaussian, exponential, Gaussian-mixture, Fourier, and wavelet formulations.
Motivation: CNNs achieve strong denoising performance because their implicit priors reflect training data, architecture, regularization, and optimization, but remain intertwined with the trained task.The paper seeks access to this implicit prior without replacing it with an explicit density model.
Contribution: The proposed stochastic ascent procedure samples high-probability images from a denoiser’s implicit prior and extends to sampling conditioned on arbitrary linear measurements.The approach uses a denoiser-estimated gradient and adaptively controlled step sizes and injected noise, without requiring an explicit density model.
Image priors, manifolds, and noisy observations: For Gaussian-corrupted observations y = x + z, the observation density p(y) is obtained by marginalizing the image prior and forms a Gaussian scale-space representation.The noise distribution is Gaussian, so p(y) is a Gaussian-blurred version of p(x).
Exposing the implicit prior through Empirical Bayes estimation: Miyasawa’s result rewrites the least-squares denoising estimate as a function of the gradient of the log noisy-image density, directly linking denoiser outputs to an implicit prior.The relevant density is p(y), not p(x), and the optimal denoising adjustment is a single step for any noise level σ.

2 Drawing high-probability samples from the implicit prior

The sampling algorithm repeatedly follows the denoiser residual toward the image manifold while injecting controlled noise and reducing effective noise across scales. Adaptive control improves convergence, and experiments produce natural-looking samples from denoisers trained on photographic and digit data.

Sampling procedure: The algorithm combines denoiser-residual gradient steps with injected Gaussian noise to sample from a sequence of progressively finer implicit densities.The denoiser residual f(y) = ˆx(y) − y supplies the gradient direction, while injected noise helps avoid local maxima.
Sampling procedure: The denoiser moves samples toward the image manifold by suppressing noise orthogonal to it while preserving variation parallel to the manifold.This mechanism reduces effective noise while synthesizing image content.
Convergence control: The method assumes the denoiser reduces noise standard deviation by a factor of (1 − h_t), an assumption justified empirically for recent DNN denoisers [38].The effective noise combines residual noise after correction with variance from injected noise.
Convergence control: Adaptive residual-based control reduces gradient steps as samples approach the manifold and adjusts injected noise to maintain convergence.The residual magnitude estimates effective noise, producing efficient and reliable convergence empirically.
Convergence control: Increasing step sizes over iterations accelerates convergence relative to a small constant fractional step, while adjusted noise amplitude preserves convergence.The schedule begins with h_0 ∈ [0, 1] and uses h_t = h'_0 t/(1 + h'_0(t−1)).
Image synthesis examples: With β = 1, samples typically converge in fewer than 40 iterations; injecting more noise yields smoother, higher-probability images but lengthens convergence.Samples include natural-looking contours, junctions, shading, and texture, and the method also generates handwritten digits from MNIST-trained denoisers.

3 Solving linear inverse problems using the implicit prior

The method samples high-probability images consistent with arbitrary linear measurements by combining denoiser-based prior gradients with explicit measurement constraints. Using the same algorithm across inverse problems produces plausible inpainting, missing-pixel recovery, super-resolution, deblurring, and compressive-sensing reconstructions.

Constrained sampling: The conditional-density gradient splits into a denoiser-prior component orthogonal to the measurements and a constraint-correction component in the measurement subspace.The update substitutes this combined gradient for the unconstrained denoiser residual in stochastic coarse-to-fine ascent.
Experimental setup: The same stochastic algorithm and parameters are applied across linear inverse problems, changing only the measurement matrix and measured values.Evaluation emphasizes perceptual quality when measurements are very low rank because the prior must supply missing information.
Inpainting: Different initializations produce distinct plausible inpainting samples consistent with the measured pixels, reflecting multiple points where the image manifold intersects the constraint hyperplane.The missing block is 30 × 30 pixels, smaller than the BF-CNN receptive field of 40 × 40.
Random missing pixels: With only 10% of pixels retained, the method recovers images described as remarkably similar to the originals.The measurements preserve a randomly selected subset of pixel coordinates.
Spatial super-resolution: For 4 × 4 block-averaged spatial super-resolution, reconstructions are compared with DIP and DeepRED [14], while averaging 10 samples improves PSNR but produces blurrier images.The downsampled measurements retain 6.25% of the original dimensionality; Tables 1 and 2 report Set5 and Set14 performance.
Deblurring and compressive sensing: The method also addresses deblurring from retained low-frequency Fourier coefficients and compressive sensing from random linear projections.For compressive sensing, it outperforms almost all compared methods on Set68 using PSNR and SSIM.

4 Related work

The method is related to plug-and-play and score-matching approaches, but derives an explicit denoiser–prior connection and samples constrained high-probability images rather than computing only MAP estimates.

The method derives an exact, direct relationship between denoising and the implicit prior, unlike plug-and-play methods whose prior connection is less explicit.
It samples high-probability images consistent with linear constraints, favoring perceptual quality over the sharper PSNR typically obtained by MAP-based alternatives.
Unlike RED and related ADMM methods, the algorithm automatically adjusts step size and uses only two primary hyper-parameters, with robust performance across their choices.
Score-matching research similarly connects denoising autoencoders with gradients of log density and develops implicit-prior sampling methods.

5 Discussion

The discussion presents denoiser-based stochastic sampling as a broadly reusable inverse-problem framework that requires no further training, while noting limitations of conventional image-fidelity metrics.

The constrained stochastic sampler solves linear inverse problems using a denoiser’s implicit prior, extending the same framework across multiple problem types without additional training.
Discriminative denoiser training can use virtually unlimited unlabeled data, broadening the reach of supervised learning to problems beyond the original denoising task.
PSNR and SSIM do not adequately capture the method’s performance, motivating human-observer experiments or no-reference perceptual-quality metrics.

A Description of BF-CNN denoiser

The BF-CNN is a bias-free convolutional denoiser trained with Gaussian noise over varying amplitudes on image patches from the Berkeley Segmentation Dataset.

Architecture: The BF-CNN contains 20 bias-free convolutional layers with 3 × 3 filters, 64 channels, batch normalization, and ReLU nonlinearities.
Training Scheme: Training corrupts images with i.i.d. Gaussian noise whose standard deviation ranges from 0 to 0.4 relative to the [0, 1] intensity range.
Training Scheme: The model is trained on overlapping 40 × 40 patches from the Berkeley Segmentation Dataset for 70 epochs with batches of 128.

B Block diagram of Universal Linear Inverse Sampler

The Universal Linear Inverse Sampler embeds a trained blind denoiser in an iterative sampler, with measurement constraints and parameters controlling step size and stochasticity.

The block diagram trains a blind denoiser on Gaussian-corrupted images, then embeds its learned parameters in an iterative computation that draws samples from the image distribution.
For constrained sampling, xc = M^T x specifies the measurements, while h0 controls step size and β controls stochasticity; an empty M yields unconstrained samples.

C Visualization of Universal Inverse Sampler on a 2D manifold prior

The Universal Inverse Sampler is visualized in two dimensions using signals sampled from a uniform prior on a manifold. Increasing noise spreads measurements and changes the denoising solutions shown along the associated density.

Fifty signals sampled from a uniform prior on a manifold are used to visualize the Universal Inverse Sampler.The prior is represented by a green curve, with sampled signals shown as example points.
For three noise levels, the visualization compares corrupted measurements, their noisy signal density, and least-squares denoising outputs.Measurements are red points, density is shown with grayscale intensity, and denoising solutions terminate red line segments.

D Convergence

The iterative sampler converges across stochasticity settings, while injected noise slows effective-noise reduction relative to deterministic updates but remains compatible with convergence.

As β decreases, injected noise makes convergence slower, although effective noise still decreases faster than the expected schedule.The behavior is observed across three stochasticity settings and is described as well-behaved and efficient.
With larger injected-noise amounts, convergence is faster than expected because some additive noise is parallel to the manifold and does not contribute to calculated variance.This explanation is offered for the observed convergence behavior.
For β = 1, effective noise falls at approximately the rate predicted by the sampling schedule.This setting uses no injected noise and provides the fastest convergence.
The sampler tracks both removed and injected noise, with their balance determining the effective-noise reduction at each iteration.The effective noise follows σ_t = (1 − βh_t)σ_t−1, while the added noise is represented by γ_t.
Run times are reported in seconds for the evaluated applications, averaged over Set12 images on an NVIDIA RTX 8000 GPU.

E Sampling from the implicit prior of a denoiser trained on MNIST

Training a bias-free CNN on MNIST produces an implicit prior distinct from the photographic-image prior, and the resulting samples vary with injected noise.

A BF-CNN trained on MNIST handwritten digits produces a different implicit prior than the photographic-image model.The figure compares samples from the MNIST-trained network with those associated with the earlier prior.
The sampled digits change across injected-noise levels, with β ∈ {1.0, 0.3, 0.01} increasing from left to right.Each panel contains 16 samples drawn from the implicit prior.