Source-linked AI summary
Deep Image Prior
Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
TL;DR
Image restoration priors are often attributed to learning from large datasets, but the role of generator architecture remains unclear. The paper fits randomly initialized ConvNets to individual degraded images and finds that their handcrafted structural prior performs competitively across standard restoration tasks.
Problem
The role of generator-network structure in capturing useful image priors, independently of learned data priors, remains unclear for image generation and restoration.
Method
The method fits a randomly initialized generator ConvNet to a single degraded image, using its weights as a task-specific parametrization of the reconstruction.
Results
The untrained ConvNet prior performs competitively on denoising, inpainting, super-resolution, and detail enhancement without learning network parameters from data.
Takeaways & Limitations
The results suggest that suitably designed generator architectures provide handcrafted image priors on which learned ConvNet restoration can build.
Takeaways & Limitations
The approach is probably too slow for most practical applications and does not match or exceed problem-specific methods.
Abstract
from arXiv · showhide
Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, super-resolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs. Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity. Code and supplementary material are available at https://dmitryulyanov.github.io/deep_image_prior .
1 Introduction
The paper argues that generator ConvNet architecture itself captures substantial image statistics without learning, enabling restoration and analysis from a single degraded image. It demonstrates this untrained prior across standard restoration problems and deep-network activation inversion.
- Contribution: Generator ConvNet structure captures substantial image statistics independently of learning, especially those needed to restore images after degradation.The authors argue that image priors need not all be learned from data.
- Method: The method fits a randomly initialized generator to one degraded image, using its weights as the restored image’s parametrization under a task-dependent observation model.Unlike standard approaches, it does not train the ConvNet on a large dataset of example images.
- Results: The untrained-network formulation is competitive for denoising, inpainting, super-resolution, and detail enhancement without learning any network component from data.The result illustrates the image prior implicitly captured by network structure.
- Novelty: The study directly investigates the prior captured by deep convolutional generative networks independently of learning their parameters from images.The authors present this as the first study to do so.
- Applications: The technique also uses an untrained deep convolutional generator to replace the surrogate natural prior when inverting classification-network activations.This application targets understanding information contained in activations and characterizing learned invariants.
2 Method
The deep image prior treats an untrained generator network as an implicit, handcrafted prior: a fixed random code is mapped to an image through a standard convolutional architecture, whose parameters are optimized only against the observed corrupted image. Its architecture and optimization dynamics favor natural-image structure, enabling restoration without a learned explicit regularizer.
- Network parametrization: A generator maps a code vector z to an image x through x = fθ(z), with convolution, upsampling, and nonlinear activation layers.The parameters θ are the network’s filter weights and biases, while the code is a fixed random tensor.
- Optimization: Starting from random parameters, gradient-based optimization minimizes the task-dependent data term, and the restored image is x∗ = fθ∗(z).The restoration process uses no empirical information beyond the observed corrupted image x0.
- Applications: The prior is effectively handcrafted because no network component is learned beforehand, yet it performs well across denoising, super-resolution, and inpainting.The paper positions this prior as exceeding standard handcrafted priors and approaching learning-based methods in many cases.
- Implicit prior: The method replaces the explicit regularizer R(x) in energy minimization with the implicit prior imposed by the neural-network parametrization.The data term compares the reconstruction with the noisy, low-resolution, or occluded observation x0.
- Architecture: A U-Net-like hourglass architecture with skip connections is used in most experiments, while z remains a fixed randomly initialized 3D tensor.The architecture choice affects results; the networks have several millions of parameters, and the code is not optimized unless noted otherwise.
- Optimization bias: Although the network can eventually fit nearly any image, its architecture makes gradient descent reach naturally-looking images faster and resist random-noise solutions.Natural images and noisy natural images converge faster than randomly scrambled images and white noise, which exhibit significant optimization inertia.
3 Applications
The deep image prior is applied to diverse reconstruction problems, including denoising, super-resolution, inpainting, and inversion of deep representations. Across these tasks, it works without pretrained image data, while its architecture and optimization behavior determine both strengths and limitations.
- Denoising: The method performs blind denoising without requiring a known noise model, recovering a clean image by optimizing the network parameters for the observed noisy image.The same formulation is used under the blindness assumption, with the recovered image defined as x∗ = fθ∗(z).
- Denoising: 31.00 PSNR is achieved on nine color images with noise strength σ = 25 after averaging over two optimization runs, compared with 31.42 for CMB3D and 30.26 for non-local means.Without these additional averaging steps, the method reaches 29.22 PSNR after 1800 steps and 30.43 PSNR with last-iteration averaging.
- Denoising: 41.95 PSNR is obtained for real-world non-Gaussian noise, compared with 30.13 for CBM3D; changing architectures yields 35.05 PSNR for UNet and 31.95 for another architecture.The differing results across architectures emphasize that they impose different priors.
- Super-resolution: For super-resolution, the deep prior outperforms non-trained methods such as bicubic while remaining behind learning-based approaches, visually narrowing the gap to trained ConvNets.The task is ill-posed because infinitely many high-resolution images can downsample to the same low-resolution observation, so regularization is required.
- Inpainting: Inpainting removes overlaid text with almost perfect results and virtually no artifacts, while large-hole inpainting works well in non-semantic cases by interpolating textures from known image regions.The method is not expected to work correctly for highly semantic large holes such as face inpainting.
- Deep representation inversion: The handcrafted prior produces interpretable inversions at least as good as a learned prior and clearer images than the TV norm, especially for deeper VGG layers such as fc6 and fc7.The method is not biased toward a particular training set, and similar improvements are observed when inverting VGG-19 layers.
4 Technical details
The method primarily uses fully convolutional hourglass networks, optionally with skip connections, whose structure provides a robust prior across a broad range of architectures and hyper-parameters. Input design and optimization procedures further shape the prior and fitting process, including meshgrid inputs for large-hole inpainting and noise-based regularization.
- Network architecture: The experiments primarily use fully convolutional encoder-decoder (“hourglass”) networks, optionally augmented with skip connections.The input and output share the same spatial resolution, with the input z ∈ R^C′×W×H and output fθ(z) ∈ R^3×W×H.
- Network architecture: A wide range of architectures and hyper-parameters gives acceptable results, although task- or image-specific tuning can improve performance.
- Input initialization: Meshgrid initialization adds a smoothness prior and benefits large-hole inpainting, but not other tasks.The alternative random initialization fills z with uniform noise between zero and 0.1.
- Optimization: Noise-based regularization can impede optimization, but the network eventually reaches zero objective for any reasonable noise variance given sufficiently many iterations.The input is perturbed at each iteration with additive zero-mean normal noise of standard deviation σp.
- Optimization: Optimization destabilization is mitigated by reverting to the previous parameters when consecutive-iteration loss increases exceed a threshold.Destabilization appears as a significant loss increase and blur in the generated image.
5 Related work
The approach is positioned among learning-free self-similarity methods, dataset-based convolutional models, and non-deep inverse-scale-space denoising. It is also contrasted with zero-shot super-resolution and related extensions using the deep image prior framework.
- Self-similarity methods: Self-similarity-based restoration methods avoid hold-out-set training and model groups of similar patches within corrupted images, especially under complex, variable corruption.Examples include spatially varying blur.
- Self-similarity methods: Zero-shot super-resolution trains a feed-forward ConvNet on synthetic patches from one image, exploiting cross-scale self-similarity rather than the proposed within-scale self-similarity at multiple scales.The two approaches are described as complementary.
- Learned convolutional models: Related dataset-based methods learn image priors through filter responses, patch dictionaries, convolutional sparse coding, or deep convolutional representations trained on natural-image datasets.A ConvNet combined with self-similarity denoising is presented as bridging dataset learning and within-image self-similarities.
- Non-deep methods: Inverse scale-space denoising gradually recovers finer details from a uniform image toward the noisy image, with early stopping producing a denoised result.These methods remain driven by a total-variation prior that does not model image self-similarity.
- Subsequent extensions: The manuscript expands its conference version with additional intuition, visualizations, technical details, experiments, activation maximization, and high-frequency enhancement, while later work applies the framework to ego-motion, adversarial defense, and phase retrieval.The cited applications include event-camera reconstruction, adversarial-attack defense, and Fourier ptychography.
6 Discussion
The discussion attributes restoration performance substantially to architectural priors: fitting a randomly initialized ConvNet works across restoration problems, though it is likely too slow for most practical applications. The authors argue that handcrafted architectures provide strong priors underlying learned ConvNet performance and motivate further architecture development.
- Architectural priors: Fitting a randomly initialized ConvNet to corrupted images works as a versatile “Swiss knife” for restoration problems.The discussion separates the architectural prior from information transferred through learning.
- Limitations: The approach is probably too slow for most practical applications.
- Why the prior emerges: Convolutional generation may impose self-similarity because filters operate across the visual field, encouraging stationarity in convolutional-layer outputs.
- Why the prior emerges: Hourglass architectures with skip connections may impose self-similarity across multiple scales.
- Implications: The results challenge the narrative that image-restoration success primarily reflects learning, instead emphasizing properly handcrafted architectures as better handcrafted priors.The authors suggest that learning ConvNets builds on this architectural basis.
- Implications: This observation validates the importance of developing new deep learning architectures.