Source-linked AI summary
A Bayesian Perspective on the Deep Image Prior
Zezhou Cheng, Matheus Gadelha, Subhransu Maji, Daniel Sheldon
TL;DR
The paper addresses overfitting and uncertainty in deep image prior inference, where gradient-based estimation relies on early stopping. It derives a stationary Gaussian-process limit for infinitely wide convolutional networks and uses SGLD for posterior inference, improving denoising and inpainting while avoiding early stopping. The paper also identifies computational and finite-width boundaries on the GP interpretation.
Problem
Deep image prior estimation from a single image risks overfitting, relies on early stopping, and does not directly capture parameter uncertainty.
Method
The paper derives the stationary Gaussian-process limit and kernel of infinitely wide convolutional networks, then performs posterior sampling with stochastic gradient Langevin dynamics.
Results
SGLD avoids early stopping, improves denoising and inpainting over gradient-descent baselines, and improves a reported inpainting result from 33.48 to 34.51 PSNR.
Takeaways & Limitations
The Bayesian formulation provides posterior uncertainty estimates while making the deep image prior more robust and effective for reconstruction.
Takeaways & Limitations
Exact GP inference is expensive for high-resolution images, and whether the posterior matches the GP posterior beyond small networks remains open.
Abstract
from arXiv · showhide
The deep image prior was recently introduced as a prior for natural images. It represents images as the output of a convolutional network with random inputs. For "inference", gradient descent is performed to adjust network parameters to make the output match observations. This approach yields good performance on a range of image reconstruction tasks. We show that the deep image prior is asymptotically equivalent to a stationary Gaussian process prior in the limit as the number of channels in each layer of the network goes to infinity, and derive the corresponding kernel. This informs a Bayesian approach to inference. We show that by conducting posterior inference using stochastic gradient Langevin we avoid the need for early stopping, which is a drawback of the current approach, and improve results for denoising and impainting tasks. We illustrate these intuitions on a number of 1D and 2D signal reconstruction tasks.
1. Introduction
The paper recasts the deep image prior as a Bayesian model whose infinitely wide convolutional networks induce stationary Gaussian-process priors. It uses posterior sampling to avoid early stopping and improve reconstruction while estimating uncertainty.
- Randomly initialized convolutional networks can encode non-trivial image properties without prior training.The deep image prior operates by initializing parameters randomly.
- Infinitely wide convolutional networks with stationary inputs converge to stationary Gaussian processes with analytically derived kernels.The kernel accounts for architectural components including convolutions, nonlinearities, sampling, and skip connections.
- SGLD posterior sampling avoids early stopping and outperforms vanilla gradient descent for image denoising and inpainting.It also provides variances of estimates as an uncertainty measure.
- Figure 1 contrasts SGD overfitting with SGLD behavior near the image noise level and shows inpainting estimates with posterior variance heat maps.The heat map assigns lower variance to uniform missing regions near the top left.
2. Related work
Related work connects random and trained convolutional networks to image representations, neural-network Gaussian processes, and Bayesian inference. This paper instead studies spatial covariance within convolutional networks viewed as random processes over image coordinates.
- The deep image prior represents images with a convolutional network and random initialization, optimizing reconstruction by gradient descent with early stopping to limit overfitting.
- Prior work established Gaussian-process limits and covariance functions for wide, deep, and convolutional neural networks.
- This paper studies spatial covariance within convolutional-network layers for a single image, rather than covariance between scalar network outputs for different inputs.
- Bayesian neural-network inference is motivated by preventing overfitting and quantifying uncertainty, but full posterior inference remains computationally challenging.
3. Limiting GP for Convolutional Networks
The paper shows that wide convolutional networks with stationary random inputs converge to stationary Gaussian processes, with covariances determined by the input covariance and preserved through suitable network layers.
- Limiting distribution for fixed X: A convolutional building block converges to a zero-mean Gaussian distribution as the number of filters grows, with covariance derived from the convolutional construction.The result applies a central-limit argument under appropriate weight scaling.
- Limiting distribution for stationary X: For stationary inputs, a two-layer convolutional network with sigmoid nonlinearity converges to a zero-mean stationary Gaussian process as input channels and filters grow.The limiting covariance is given analytically from the input covariance.
- Limiting distribution for stationary X: The same stationary Gaussian-process convergence holds with ReLU nonlinearity, using the corresponding recursive covariance formula.The result extends the theorem beyond the sigmoid case.
- Limiting distribution for stationary X: For both nonlinearities, output covariance at offset r depends only on input covariance at the same offset and at zero offset.This gives a local covariance transformation across the network layer.
- Beyond two layers: Stationarity is preserved through additional convolutional layers, downsampling, average pooling, and skip connections, while upsampling requires suitable filters such as band-limited filters.The covariance can be propagated recursively through deep convolutional networks.
4. Bayesian Inference for Deep Image Prior
The paper recasts deep image prior reconstruction as Bayesian posterior inference over network inputs and parameters. Stochastic gradient Langevin dynamics produces posterior samples that avoid the early-stopping dependence of standard optimization, while MAP and MLE can still overfit.
- Current inference: Deep image prior denoising represents the image as a convolutional-network output and optimizes its random input and parameters using SGD with early stopping.The reconstruction is f(x*, θ*) after optimization, with x and θ initialized randomly.
- Bayesian formulation: MLE and MAP inference do not capture parameter uncertainty and can overfit the observed image.MAP adds a prior but remains a point estimate rather than a posterior average.
- Posterior inference: SGLD adds Gaussian noise to SGD gradient updates to obtain samples whose averages estimate the posterior reconstruction.The update treats w=(x, θ), and under suitable step-size conditions converges to the posterior distribution.
- Empirical behavior: SGLD samples remain stable in PSNR after burn-in, avoiding early stopping, while the posterior mean exceeds the highest PSNR of the other approaches in the peppers experiment.The comparison is reported for Figure 2's PSNR curves.
5. Experiments
Experiments characterize how convolutional architectures induce stationary covariance and show that SGLD improves reconstruction while avoiding the early-stopping dependence of SGD-based inference. Across toy signals, denoising, and inpainting, SGLD produces competitive or superior results and uncertainty estimates.
- 5.1. Toy examples: Increasing input covariance and depth produces longer-range spatial covariances, while autoencoder upsampling also induces long-range interactions.These effects are observed in the one-dimensional convolutional experiments.
- 5.1. Toy examples: Larger input covariance or depth yields smoother posterior interpolations, analogous to selecting a larger stationary-kernel scale.Changing the covariance affects both posterior means and variances.
- 5.2.1 Image denoising: SGLD samples remain stable in PSNR over optimization, whereas SGD variants peak and then degrade through overfitting.The posterior mean after burn-in consistently performs better, with the 20K-iteration posterior mean achieving the best result among the compared inference methods.
- 5.2.1 Image denoising: 30.81 average PSNR is achieved by SGLD for denoising, compared with 30.33±0.03 for SGD + Input + Early + Avg and 31.68 for BM3D.The reported values are averages over the denoising comparisons.
- 5.2.2 Image inpainting: SGLD outperforms baseline methods in denoising and inpainting, improving the reported inpainting result from 33.48 to 34.51 PSNR.The SGLD posterior mean also has fewer artifacts than the best SGD-variant result in qualitative comparisons.
- 5.2.2 Image inpainting: SGLD provides uncertainty estimates that are lower in uniform missing regions and higher near boundaries between different objects.This spatial variation is illustrated for image inpainting.
- 5.3. Equivalence between GP and DIP: DIP prior samples are nearly identical to samples from a Gaussian process with the equivalent kernel, while DIP PSNR approaches GP performance as channel count increases.The DIP kernel is more effective than an RBF kernel for the reported inpainting comparison.
- 5.3. Equivalence between GP and DIP: Exact GP inference requires O(n^2) memory and O(n^3) time, whereas DIP inference scales linearly in pixels; equivalence of posteriors for deeper networks remains open.A 500×500 image is given as requiring 233 GB memory for exact GP inference.
6. Conclusion
The paper presents a Bayesian view of the deep image prior by characterizing random convolutional networks as stationary Gaussian processes and using SGLD for posterior inference. This approach improves reconstruction performance and avoids early stopping.
- 6. Conclusion: Random convolutional networks converge to stationary zero-mean Gaussian processes as the number of channels per layer tends to infinity.The paper also derives the realized covariance and analyzes spatial covariance on a single input image.
- 6. Conclusion: SGLD performs fully Bayesian posterior inference for the deep image prior, improving performance and preventing the need for early stopping.The conclusion identifies further study of convolutional-network kernels and their inductive bias as future work.
Appendix
The appendix provides the proof of Theorem 1 and additional visualizations for denoising and inpainting. The proof uses stationarity, independent channels, and the strong law of large numbers to establish convergence.
- The appendix proves Theorem 1 and includes additional visualizations of denoising and inpainting results.
- Stationarity makes the terms x(k, t1 −i)x(k, t2 −i) identically distributed with expected value Kx(r), despite possible dependence.
- Independent channels make the summed expressions iid with mean dKx(r), enabling the convergence argument.
- The strong law of large numbers gives almost-sure convergence of an, bn, and cn to dKx(r), dKx(0), and dKx(0).
- The continuous mapping theorem is then applied to the convergent quantities Yn.
- Figures 7 and 8 visualize results for various inputs and inference schemes in image denoising and image inpainting.