PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Image Deconvolution using Deep Generative Priors

Muhammad Asim, Fahad Shamshad, Ali Ahmed

arXiv:1802.04073v4cs.CV

TL;DR

Blind image deblurring is ill-posed, and end-to-end methods may be sensitive to changes in blur, images, or noise. This paper searches the latent spaces of pretrained image and blur generators using alternating gradient descent. Experiments report promising deblurring results, while showing that performance depends on the generator’s range.

  • Problem

    Blind deblurring is ill-posed because many image–kernel pairs can explain an observation, and end-to-end methods may be sensitive to test conditions unlike their training data.

    Source: §I · p. 1

  • Method

    The method alternates gradient descent over latent representations of pretrained image and blur generators to find a pair that explains the blurry observation.

    Source: §I · p. 2

  • Results

    Experiments report promising results on face images and house numbers, with performance depending on how well the image generator’s range spans the image class.

    Source: §I · p. 2, 1) Impact of Generator Range on Image… · p. 8

  • Takeaways & Limitations

    Untrained convolutional generators can act as image priors, and the approach reports better results than conventional image priors in general.

    Source: §I · p. 2

    Performance is limited by the generator’s expressive range, which can be affected by issues such as mode collapse.

    Source: 1) Impact of Generator Range on Image… · p. 8

Abstract

from arXiv · show

This paper proposes a novel approach to regularize the \textit{ill-posed} and \textit{non-linear} blind image deconvolution (blind deblurring) using deep generative networks as priors. We employ two separate generative models --- one trained to produce sharp images while the other trained to generate blur kernels from lower-dimensional parameters. To deblur, we propose an alternating gradient descent scheme operating in the latent lower-dimensional space of each of the pretrained generative models. Our experiments show promising deblurring results on images even under large blurs, and heavy noise. To address the shortcomings of generative models such as mode collapse, we augment our generative priors with classical image priors and report improved performance on complex image datasets. The deblurring performance depends on how well the range of the generator spans the image class. Interestingly, our experiments show that even an untrained structured (convolutional) generative networks acts as an image prior in the image deblurring context allowing us to extend our results to more diverse natural image datasets.

I. INTRODUCTION

The paper uses generative priors in an alternating deblurring scheme and reports promising results, while identifying generator limitations and untrained convolutional networks as important considerations. Its approach is motivated by blind deblurring’s ambiguity and the sensitivity of end-to-end methods to changes in test conditions.

I. INTRODUCTION

  • Blind deblurring is severely ill-posed because many image–kernel pairs can fit the same observation, while end-to-end methods may be sensitive to test-set shifts.Conventional engineered priors can also admit unrealistic images.Source: §I · p. 1
  • Experiments report promising results on face images and house numbers, and deep generative priors generally outperform conventional image priors in deblurring.The paper also reports that its explicit use of the convolution forward map yields robust results compared with end-to-end frameworks.Source: §I · p. 2
  • Alternating gradient descent searches the ranges of pretrained image and blur-kernel generators for a pair that explains the blurry observation.The lower-dimensional latent representations reduce the number of unknowns and allow gradients to be computed through the generators.Source: §I · p. 2
  • Strictly constraining recovery to a generator’s range can be counterproductive when the model fails to faithfully learn the image distribution.The paper modifies the loss to allow recovered images some slack beyond the generator range.Source: §I · p. 2
  • Untrained convolutional generative networks can act as image priors, with network structure alone capturing some image statistics and helping on rich datasets.The untrained network weights are initialized using a single blurred image.Source: §I · p. 2

II. RELATED WORK

The paper positions generative-network priors as a way to combine iterative deblurring with learned image and blur structure. It formulates recovery over pretrained generator ranges and optimizes the resulting latent variables with alternating gradient descent.

II. RELATED WORK

  • Conventional priors favor sparse or otherwise structured images, while this approach constrains full-size images to a learned nonlinear generator range.The paper also contrasts full-image generators with learned dictionaries for sparse image-patch representations.Source: §II · p. 2
  • The iterative framework retains adaptability to blur-model and noise-statistics changes, while limiting blur-model retraining to a shallow network.The paper contrasts this with end-to-end methods that require retraining when test conditions differ from training data.Source: §II · p. 2
  • The authors identify their work, to their knowledge, as the first use of pretrained generative models as priors for blind image deconvolution.Pretrained generative priors had already been used for compressed sensing, phase retrieval, Fourier ptychography, and image inpainting.Source: §II · p. 3

III. PROBLEM FORMULATION

  • The formulation assumes images and blur kernels belong to structured classes represented by fixed pretrained generators, then constrains recovery to those generator ranges.The range constraint is described as reducing ambiguities in the ill-posed problem and keeping solutions within the modeled classes.Source: §III · p. 3
  • The objective can be optimized in lower-dimensional latent space, where alternating gradient descent updates image and kernel representations through the fixed generators.The estimated image and kernel are obtained by passing the optimized latent vectors through their respective generators.Source: §III · p. 3

IV. IMAGE DEBLURRING ALGORITHM

  • The approach uses pretrained GANs or VAEs for clean images and blur kernels; the latent variables are also regularized with ℓ2 penalties.The penalty follows the assumption that latent representations come from standard Gaussian distributions.Source: §IV · p. 3

A. Training the Generative Models

The paper describes GAN- and VAE-based generative models and their use in naive deblurring. Simply choosing the generated image closest to the blurred observation fails because this approach ignores the forward blur model.

A. Training the Generative Models

  • GANs train a generator to produce images that fool a discriminator, while VAEs learn the image distribution by maximizing a lower bound on log likelihood.The VAE’s generative network produces image samples from latent representations.Source: A. Training the Generative Models · pp. 3–4
  • The face and shoe image generators use adversarial learning, while other image generators and the blur-kernel generator use variational inference.Source: A. Training the Generative Models · p. 4

B. Naive Deblurring

  • Naive deblurring searches for the generated clean image closest to the observation, but this back-projection fails to produce reasonable recovery results.The generator weights remain fixed, and the approach fails because it ignores the forward blur model.Source: B. Naive Deblurring · p. 4, Figure 2 · p. 4

C. Deconvolution using Deep Generative Priors

The method searches for an image and blur kernel in their respective generator ranges whose convolution matches the observation. It minimizes a regularized measurement loss by alternating updates to the two latent vectors.

  • The method seeks a clean image and blur kernel from their respective generator ranges whose convolution comes as close as possible to the blurred observation.Source: C. Deconvolution using Deep Generative… · p. 4
  • The objective adds ℓ2 penalties on both latent vectors to the measurement loss, and alternating gradient steps update one vector while holding the other fixed.Random restarts may be used when measurement loss does not decrease sufficiently after a reasonable number of iterations.Source: C. Deconvolution using Deep Generative… · p. 4
  • For computational efficiency, the loss and gradients are implemented in the Fourier domain, where the DFT diagonalizes circular convolution.The objective uses the Fourier-transformed observation and generated image and kernel, together with latent-vector penalties.Source: C. Deconvolution using Deep Generative… · p. 4
  • Each alternating latent update requires backpropagation through the corresponding image or blur-kernel generator.The block diagram depicts zi and zk updates minimizing measurement loss and producing image and blur estimates.Source: C. Deconvolution using Deep Generative… · p. 5, Figure 3 · p. 5

D. Beyond the Range of Generator

Strictly constraining the recovered image to the generator range can cause artifacts when that range does not span complex image classes. The proposed alternative permits out-of-range images while tying them to generated images and adding a classical image prior.

  • When the generator range does not span the image class, strict range constraints may cause artifacts, especially for complex natural images.Source: D. Beyond the Range of Generator · p. 5
  • The modified objective combines in-range and out-of-range measurement losses, a range-error penalty, and total variation to guide images beyond the generator range.Weights on the loss terms control how much the recovered image may deviate from the generator output.Source: D. Beyond the Range of Generator · p. 5
  • The modified algorithm alternates gradient steps over the unknowns and returns the recovered image together with a generated blur kernel.Its iterative scheme is identified as Algorithm 2.Source: D. Beyond the Range of Generator · p. 5

E. Untrained Generative Priors

The paper uses an untrained convolutional image generator as a prior, updating its weights from the blurred observation during deblurring. The authors report that network structure alone captures some image statistics and can provide a prior for complex images that pretrained generators learn poorly.

  • An untrained structured generator can act as a prior for natural-image deblurring, though it is less effective than a pretrained prior.The authors attribute this ability to image statistics captured by the network structure, even without pretraining.Source: E. Untrained Generative Priors · p. 6
  • The image-generator weights are learned from the single blurred observation during deblurring, while a total-variation penalty is added to its output.Source: E. Untrained Generative Priors · p. 6
  • The method initializes generator weights by fitting the generator output to the blurred observation for a fixed random input, then obtains the deblurred image and kernel from the optimized generators.Source: E. Untrained Generative Priors · p. 6

V. EXPERIMENTAL RESULTS

The experiments evaluate deblurring across SVHN, Shoes, and CelebA using pretrained image generators, alongside a separately generated motion-blur dataset. The setup also specifies dataset splits and the models used for images and blur kernels.

V. EXPERIMENTAL RESULTS

A. Datasets and Generative Models

B. Baseline Methods

The evaluation compares the proposed approach with engineered-prior and learned deblurring baselines, and examines pretrained-generator performance under varying blur and noise. The analysis identifies generator coverage as a constraint and relates recovery error to distance from the generator range.

B. Baseline Methods

  • Baselines include four engineered-prior methods and two learned methods: a CNN and DeblurGAN, trained on SVHN and CelebA.The engineered-prior baselines are dark prior, extreme channel prior, outlier handling, and learned data fitting.Source: B. Baseline Methods · p. 7

C. Deblurring Results under Pretrained Generative Priors

  • Algorithm 1 is evaluated under small-to-heavy blurs and varying measurement noise; its performance depends on how well the image generator spans the image class.The passage contrasts complex face images with compact number images, noting that generators may represent the latter more adequately.Source: C. Deblurring Results under Pretrained… · p. 7
  • More expressive generators can reduce mismatch between a test image and its closest generator-range image.Figure 5 illustrates range-image correspondence and notes that recovered shoes may miss texture, while recovered faces can differ semantically.Source: C. Deblurring Results under Pretrained… · p. 7, Figure 5 · p. 8
  • Overall recovery error is bounded by the deblurred image’s distance to the range image plus the range image’s distance to the test image.The second term is the range error; the passages provide this as an upper bound, not an equality.Source: C. Deblurring Results under Pretrained… · pp. 7–8

1) Impact of Generator Range on Image Deblurring:

The experiments show that deblurring quality depends on how well the image generator’s range represents the test image. Across CelebA, Shoes, and SVHN, the best-performing proposed algorithm varies with the dataset, and allowing images to move beyond the generator range can improve recovery.

1) Impact of Generator Range on Image Deblurring:

  • Algorithm 1’s recovered image approximates the closest image in the generator range, so its accuracy depends on how closely that image matches the test image.The generator’s expressive power, influenced by its training scheme and architecture, constrains this range match; Algorithm 2 mitigates the issue by allowing deviation from the range.Source: 1) Impact of Generator Range on Image… · p. 8

2) Qualitative Results on CelebA:

3) Qualitative Results on SVHN:

4) Quantitative Results:

5) Robustness against Noise:

The experiments examine robustness to noise, initialization, and latent dimension. Results vary with image and noise conditions, while multiple random starts and larger latent dimensions can improve PSNR within the tested settings.

5) Robustness against Noise:

  • Generative priors outperform other methods on deblurred range images across noise levels, while arbitrary-image results vary by dataset and generator range.For arbitrary images, performance is second best on SVHN but below competing approaches on CelebA; fixed-1% noise end-to-end models deteriorate rapidly as noise changes.Source: 5) Robustness against Noise · pp. 10–11

6) Random Restarts:

7) Latent Dimension:

  • On CelebA, increasing the image latent length above 10 sharply raises PSNR, with gains tapering after length 200.The paper estimates that setting the latent length to 200 would improve average PSNR by roughly 1 dB over its evaluation setting of 100.Source: 7) Latent Dimension · p. 12, Figure 12 · p. 12

8) Robustness against Large Blurs:

  • With large blurs, Algorithm 2 outperforms the other compared methods on CelebA and Shoes, while end-to-end methods lag as blur size increases.The paper also reports recovery of a face from an image described as completely unrecognizable due to blur.Source: 7) Latent Dimension · p. 12

8) Robustness against Large Blurs:

The paper reports promising deblurring under large blurs and explores untrained generators for complex natural images. It concludes that generative priors may also apply to other nonlinear inverse problems.

8) Robustness against Large Blurs:

  • Figure 13 presents excellent deblurring results from Algorithm 2 under large blurs, and Figure 14 compares proposed methods with baselines as blur length increases.The comparison covers CelebA in the first row and Shoes in the second.Source: Figure 13 · p. 12, Figure 14 · p. 12

D. Extension to Natural Images using Untrained Generators

  • An untrained generator performs competitively against classical-prior baselines on complex blurry images, with PSNR and SSIM reported for the examples.The authors present these initial results as evidence of potential, and attribute the prior behavior to the network structure alone.Source: D. Extension to Natural Images using… · pp. 12–13, Figure 15 · p. 13

VI. CONCLUSION

  • The authors argue that generative priors can serve as effective methods for challenging nonlinear inverse problems beyond image deblurring.The conclusion presents this broader use as a general strategy, alongside discussion of limitations and remedies for complex images.Source: §VI · p. 14

APPENDIX

The appendix provides additional visual deblurring results across datasets, blur types, and generator settings.

APPENDIX

  • The appendix also shows Algorithm 1 results for Gaussian blur on SVHN and CelebA, plus PG-GAN results under heavy noise.Figures 19–21 provide these additional settings.Source: APPENDIX · p. 16, Figure 19 · p. 19, Figure 20 · p. 19
  • On SVHN, Algorithm 1 produces results reported as superior to the baseline methods, especially under large blurs.Figure 16 compares Algorithm 1 with baseline methods.Source: Figure 16 · p. 16
  • On CelebA and Shoes, Algorithm 2’s results are reported as superior to the baselines, especially under large blurs.Figures 17 and 18 also note that DeblurGAN and iDeGAN outputs, though sharp, deviate from the original images.Source: Figure 17 · p. 17, Figure 18 · p. 18
Loading 1802.04073v4…