PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Blind Deconvolution Using Deep Priors

Dongwei Ren, Kai Zhang, Qilong Wang, Qinghua Hu, Wangmeng Zuo

arXiv:1908.02197v2cs.CV

TL;DR

Blind deconvolution must recover both clean images and blur kernels, while handcrafted priors and existing deep models are limited for complex, large kernels. SelfDeblur combines separate generative networks with unconstrained neural optimization and reports favorable quantitative and visual results on benchmarks and real-world images.

  • Problem

    Blind deconvolution remains challenging because it must estimate both the latent clean image and blur kernel, while fixed handcrafted priors are insufficient and existing deep models struggle with complex, large kernels.

  • Method

    SelfDeblur uses an asymmetric Autoencoder for the clean-image prior and an FCN with SoftMax-constrained output for the blur-kernel prior, optimized jointly from the blurry input.

  • Results

    SelfDeblur achieves notable performance gains over state-of-the-art methods and produces visually favorable clean images while estimating blur kernels on benchmark and real-world blurry images.

  • Takeaways & Limitations

    The method can estimate the blur kernel and generate a latent clean image with satisfying visual quality, so non-blind deconvolution is not compulsory.

  • Takeaways & Limitations

    Neural blind deconvolution remains difficult to optimize under blur-kernel constraints, and DIP denoising relies on costly averaging across iterations and runs for blurry noisy images.

Abstract

from arXiv · show

Blind deconvolution is a classical yet challenging low-level vision problem with many real-world applications. Traditional maximum a posterior (MAP) based methods rely heavily on fixed and handcrafted priors that certainly are insufficient in characterizing clean images and blur kernels, and usually adopt specially designed alternating minimization to avoid trivial solution. In contrast, existing deep motion deblurring networks learn from massive training images the mapping to clean image or blur kernel, but are limited in handling various complex and large size blur kernels. To connect MAP and deep models, we in this paper present two generative networks for respectively modeling the deep priors of clean image and blur kernel, and propose an unconstrained neural optimization solution to blind deconvolution. In particular, we adopt an asymmetric Autoencoder with skip connections for generating latent clean image, and a fully-connected network (FCN) for generating blur kernel. Moreover, the SoftMax nonlinearity is applied to the output layer of FCN to meet the non-negative and equality constraints. The process of neural optimization can be explained as a kind of "zero-shot" self-supervised learning of the generative networks, and thus our proposed method is dubbed SelfDeblur. Experimental results show that our SelfDeblur can achieve notable quantitative gains as well as more visually plausible deblurring results in comparison to state-of-the-art blind deconvolution methods on benchmark datasets and real-world blurry images. The source code is available at https://github.com/csdwren/SelfDeblur

1. Introduction

Blind deconvolution must recover both a latent clean image and blur kernel from a blurry observation, but handcrafted MAP priors and existing deep models remain limited. SelfDeblur uses separate generative networks and neural optimization to address this problem without requiring non-blind deconvolution.

  • Problem: Blind deconvolution estimates the latent clean image x and blur kernel k from blurry image y under y = k ⊗ x + n.The noise term n is additive white Gaussian noise with level σ.
  • Limitations of prior methods: Handcrafted MAP priors can insufficiently characterize clean images and blur kernels, while non-convex optimization may converge to a trivial delta kernel.Existing methods also depend on optimization details such as projected alternating minimization and delayed kernel normalization.
  • SelfDeblur: SelfDeblur models clean-image and blur-kernel priors with an asymmetric Autoencoder and a fully-connected network, respectively.SoftMax at the FCN output enforces the blur-kernel non-negative and equality constraints.
  • Optimization: The method performs zero-shot self-supervised learning of its generative networks using only the input blurry image.A joint optimization algorithm is suggested for estimating the kernel and generating the latent clean image.
  • Results: SelfDeblur achieves notable quantitative gains and visually plausible deblurring results on benchmark datasets and real-world blurry images.It can estimate the blur kernel and generate a clean image with satisfying visual quality, making non-blind deconvolution unnecessary as a compulsory step.

2. Related Work

Prior blind-deconvolution research spans handcrafted optimization methods and deep networks for kernel estimation or direct image restoration. SelfDeblur combines deep image statistics with a dedicated blur-kernel generator to target limitations of both approaches.

  • Optimization-based methods: Optimization-based blind deconvolution includes Variational Bayes and MAP-based methods, with the former theoretically promising but computationally expensive.MAP methods use priors such as total variation, gradient norms, patch statistics, low rank, and dark channels.
  • Optimization-based methods: After blur-kernel estimation, non-blind deconvolution is usually required to recover fine texture details in the latent clean image.The priors useful for kernel estimation can differ from those favoring natural-image restoration.
  • Deep learning methods: Deep CNNs have been used to estimate blur kernels or learn direct mappings to clean images for motion and video deblurring.Approaches include architectures imitating alternating minimization, Fourier-coefficient prediction, parametric-kernel prediction, multi-scale networks, and adversarial losses.
  • Deep learning methods: These supervised deep methods do not succeed in handling various complex and large-size blur kernels in blind deconvolution.Their use typically involves learning from training sets or mappings tailored to particular deblurring settings.
  • Deep priors: DIP captures image statistics for several low-level vision tasks but is limited for blur-kernel priors, while Double-DIP performs poorly for blind deconvolution.SelfDeblur addresses this gap by pairing DIP for clean images with an FCN for blur kernels.

3. Proposed Method

SelfDeblur replaces handcrafted image and kernel regularizers with generative networks optimized directly from the blurry image. Its unconstrained formulation uses tailored architectures, constraint-enforcing nonlinearities, and joint or alternating optimization.

  • Proposed Model: SelfDeblur models latent clean images with an asymmetric Autoencoder and blur kernels with a fully-connected network, replacing explicit regularization terms.The image generator uses skip connections, while the kernel generator uses SoftMax at its output to satisfy kernel constraints.
  • Neural Blind Deconvolution: The neural formulation removes the MAP regularizers by substituting the latent image and kernel with generator outputs driven by sampled uniform noise.The kernel generator output is reshaped into a two-dimensional blur kernel.
  • Constraint Handling: SoftMax and Sigmoid output nonlinearities enforce the blur-kernel and clean-image range constraints, respectively.The kernel is constrained to be non-negative and sum to one, while clean-image pixels remain within [0, 1].
  • Noise Handling: A noise-level-controlled TV regularizer is added to improve robustness across blurry images with different noise levels.The regularization parameter λ is controlled by the noise level σ, and the authors report that combining the image generator with another image prior is beneficial.
  • Optimization: The optimization is zero-shot self-supervised learning because both generators are trained using only the test blurry image, without a ground-truth clean image.SelfDeblur supports alternating optimization and joint optimization; empirically, joint optimization performs better in most cases.
  • Output Generation: Unlike MAP pipelines that generally require a subsequent non-blind deconvolution, SelfDeblur can directly generate a visually favorable latent clean image.The generative image model also presents high impedance to image noise, although DIP denoising heuristics add computational cost and are not directly suitable for blurry noisy images.

4. Experimental Results

The paper evaluates SelfDeblur through ablations, benchmark comparisons, and real-world blurry images using alternating and joint optimization procedures. The supplied experimental passages specify the algorithms and implementation settings but do not report numerical outcomes.

  • Evaluation Scope: The experiments include ablation studies of the optimization algorithm and network architecture, followed by benchmark and real-world evaluations.The evaluation covers two benchmark datasets and several real-world blurry images.
  • Alternating Optimization: Algorithm 1 alternates updates of the blur-kernel and clean-image generators using ADAM while fixing the other generator during each update.Gradients for either generator are obtained through automatic differentiation.
  • Joint Optimization: Algorithm 2 jointly computes gradients for both generators and updates them during each iteration.The supplied algorithm passages identify joint gradient computation and generator updates but provide no quantitative result.
  • Implementation Settings: The default implementation runs for T = 5,000 iterations with uniform-noise inputs, random perturbation of z_x, and scheduled learning-rate decay.The learning rate starts at 0.01 and is multiplied by 0.5 at iterations 2,000, 3,000, and 4,000.

4.1. Ablation Study

The ablation studies compare optimization strategies and blur-kernel generator architectures, showing that joint optimization and the default FCN design are preferred. Intermediate results also show simultaneous refinement of the latent image and blur kernel.

  • Alternating Optimization vs. Joint Optimization: SelfDeblur-J significantly outperforms SelfDeblur-A on average PSNR and SSIM, demonstrating the superiority of joint optimization.
  • Alternating Optimization vs. Joint Optimization: SelfDeblur-A may converge to a delta kernel and a worse solution, whereas SelfDeblur-J performs favorably on these cases.
  • Network Architecture of Gk: SelfDeblur significantly outperforms Double-DIP, indicating that an asymmetric Autoencoder is less suitable than a simple FCN for modeling the blur-kernel prior.
  • Network Architecture of Gk: The FCN without a hidden layer succeeds but performs much worse than SelfDeblur, while the three-layer FCN improves over it yet remains inferior to SelfDeblur.
  • Alternating Optimization vs. Joint Optimization: As iterations increase, Gx and Gk generate finer details while simultaneously estimating the latent clean image and blur kernel.

4.2. Comparison with State-of-the-arts

On benchmark datasets, SelfDeblur generally outperforms competing blind deconvolution methods quantitatively and qualitatively, including for large blur kernels. Non-blind deconvolution can further improve metrics on Levin et al., but is not essential to the method.

  • Results on dataset of Levin et al. [19]: SelfDeblur significantly outperforms competing methods in PSNR and Error Ratio on Levin et al., while its average SSIM is slightly inferior to Sun et al. and Zuo et al.
  • Results on dataset of Levin et al. [19]: SelfDeblurΔ further boosts quantitative performance and outperforms all other methods after incorporating non-blind deconvolution from.
  • Results on dataset of Levin et al. [19]: SelfDeblur is time-consuming because it optimizes two generative networks, but its runtime is comparable with Sun et al. and Pan-DCP.
  • Results on dataset of Levin et al. [19]: The performance and visual-quality gaps between SelfDeblur and SelfDeblurΔ are not significant, so non-blind deconvolution is not compulsory for SelfDeblur.
  • Results on dataset of Lai et al. [18]: On Lai et al., SelfDeblur and SelfDeblurΔ achieve better quantitative metrics than competing methods across five image categories.
  • Results on dataset of Lai et al. [18]: SelfDeblur estimates more accurate blur kernels and produces more visually plausible textures than competing methods on Lai et al.

4.3. Evaluation on Real-world Blurry Images

On real-world blurry images, SelfDeblur estimates less noisy blur kernels and produces clean images with more visually plausible structures and textures than the compared methods.

  • SelfDeblur produces less noisy kernel estimates and more visually plausible structures and textures than Xu&Jia and Pan-DCP on real-world blurry images.

5. Conclusion

SelfDeblur combines separate deep priors for the latent clean image and blur kernel in a joint unconstrained optimization framework. Experiments report performance gains and visually favorable reconstructions.

  • SelfDeblur uses an asymmetric Autoencoder and an FCN to capture deep priors for the latent clean image and blur kernel, respectively.
  • SoftMax applied to the FCN output enforces the blur kernel’s non-negative and equality constraints.
  • A joint optimization algorithm solves the resulting unconstrained neural blind deconvolution model.
  • Experiments show notable performance gains over state-of-the-art methods, with effective blur-kernel estimation and visually favorable clean-image textures.
Loading 1908.02197v2…