Source-linked AI summary
Learning to Deblur
Christian J. Schuler, Michael Hirsch, Stefan Harmeling, Bernhard Schölkopf
TL;DR
Blind image deconvolution must recover a sharp image while the blur is also unknown. The paper learns this task as a nonlinear regression procedure using a layered architecture with deconvolution-specific computations and artificial training examples. The resulting system is comparable to hand-crafted state-of-the-art methods and can exceed them when specialized to an image category.
Problem
Blind image deconvolution seeks to recover a sharp image from a blurred, noisy recording when both the image and blur are unknown.
Method
The method unrolls reconstruction as nonlinear regression and trains a layered architecture end-to-end on artificially generated data, including deconvolution-specific computations.
Results
The results are comparable to state-of-the-art hand-crafted approaches and go beyond them when the model specializes to a particular image category.
Takeaways & Limitations
A trainable model can learn blind deconvolution and approximate the underlying image from only a blurry measurement.
Takeaways & Limitations
Performance drops for larger blur kernels, which the authors associate with a suboptimal multi-scale architecture at higher resolution scales.
Abstract
from arXiv · showhide
We describe a learning-based approach to blind image deconvolution. It uses a deep layered architecture, parts of which are borrowed from recent work on neural network learning, and parts of which incorporate computations that are specific to image deconvolution. The system is trained end-to-end on a set of artificially generated training examples, enabling competitive performance in blind deconvolution, both with respect to quality and runtime.
1. Introduction
Blind image deconvolution recovers a sharp image when both the image and blur are unknown, making it a difficult inverse problem. The proposed system learns this reconstruction procedure end-to-end using a layered nonlinear model trained on artificial data.
- Motivation: Blind image deconvolution recovers a sharp image from a blurred, noisy recording while the blur itself is unknown.The setting arises in photography, astronomy, and remote sensing.
- Motivation: Because both the latent image and blur are unknown, blind deconvolution is an especially difficult inverse problem.
- Approach: The method unrolls the conventional reconstruction procedure as a nonlinear regression problem whose parameters are learned from artificially generated data.
- Approach: Its layered architecture combines convolutional layers with non-standard computations tailored to blind deconvolution.
- Training: Extensive training on images with simulated camera shakes teaches the network an efficient procedure for approximating the underlying image from only the blurry measurement.
- Contributions: The authors report state-of-the-art-comparable results, with further gains when the model specializes to a particular image category.
2. Related work
Earlier neural-network approaches to image deconvolution addressed restricted blur identification, blur modeling, inverse filtering, optimization, or noise removal. This work instead learns a general blind-deconvolution procedure applicable across images and blurs, distinguishing it from both single-image learning and known-kernel deblurring.
- Neural-network approaches: These prior approaches differed from the proposed work in their tasks and deconvolution settings.
- Neural-network approaches: Earlier neural-network methods identified blur types, modeled blurry images, learned inverse filters, optimized regularized objectives, or removed colored noise.
- Learning-based blind deconvolution: Some learning-based methods learn a deconvolution solution or sparse representation for one fixed image.
- Learning-based blind deconvolution: The proposed strategy learns a general procedure applicable to other images and different blurs rather than fitting a single image.
- Learning-based blind deconvolution: Regression tree fields are closest in strategy but address non-blind deblurring, where the blur kernel is known.
- Problem distinction: Blind deconvolution is harder than non-blind deconvolution because the unknown image and kernel form an underdetermined bilinear problem.
3. Blind deconvolution as a layered network
The network unrolls the three alternating steps of blind deconvolution into a trainable, stacked architecture. Learned feature extraction feeds fixed-form quotient-based kernel and image estimation, repeated across stages and scales.
- Layered network: The network alternates feature extraction, kernel estimation, and sharp-image estimation to represent the standard blind-deconvolution procedure.Feature extraction is learned, while kernel and image estimation use fixed computations with regularization parameters.
- Feature extraction module: Learned convolutional features replace hand-designed gradient representations and are shared across image locations.Tanh nonlinearities and pixel-wise linear recombination produce multiple gradient-like representations for kernel estimation.
- Kernel estimation module: The quotient layer estimates the blur kernel in one Fourier-space step from paired learned feature representations.The solution uses element-wise Fourier-domain operations, a Gaussian kernel prior, cropping, and thresholding of negative values.
- Image estimation module: The estimated kernel updates the latent image through another quotient layer, after which later feature extraction receives both the blurry and latent images.The latent-image regularization parameter βx is learned during training.
- Iterations as stacked networks: Stacking multiple three-operation stages improves performance while increasing runtime.A multi-scale procedure applies networks to progressively finer resolutions for larger target kernels.
- Training: Training uses artificially generated sharp–blurred pairs from ImageNet, with Gaussian-process blur trajectories and added Gaussian noise.The sharp images are sampled from about 1.6 million images, and each setting generates 1 million noise-free examples before noise is added.
4. Implementation
The method requires substantial one-time training for each blur class, but trained networks can be applied quickly using Matlab alone. Runtime is dominated by constructing multiple hidden convolutional representations.
- Training and deployment: Training a network for a blur class takes about two days per stage, after which inference runs very fast in Matlab without additional dependencies.The authors provide code for both training and testing.
- Runtime: The most expensive runtime operation is creating the 32 hidden representations in the convolutional layer.The reported runtimes were measured on an Intel i5 using only Matlab.
5. Experiments
Experiments evaluate the learned deblurring system across content, noise, spatially varying blur, benchmark datasets, and real-world images. Performance is competitive with state-of-the-art methods, while content- and noise-specific training improves adaptation in the corresponding settings.
- Experimental setup: The default experiments use a multi-scale, triple-stage architecture supporting kernel sizes of 17×17, 25×25, and 33×33 pixels.Each scale uses a 32-filter convolutional feature extraction module followed by nonlinear recombination and gradient-like outputs.
- Noise-specific training: PSNR 23.41 is obtained with noise-specific training, compared with PSNR 23.29 for noise-agnostic training and PSNR 23.17 for.The noise-specific network was trained with 5% Gaussian noise, matching the noisy input image.
- Image content specific training: Content-specific training lets the network focus on image features informative for the target category and yields improved performance.Experiments train separate networks on valley and blackboard ImageNet categories.
- Spatially-varying blur: The learned prediction step is combined with a fast forward model to estimate spatially varying blur using local kernels, motion-basis projection, and thresholding.The additional projection constrains estimates to translations and in-plane rotations, while thresholding removes small kernel entries.
- Evaluation: On standard benchmark and real-world images, the method achieves results comparable in quality with state-of-the-art deblurring methods.The benchmark evaluation compares against Levin et al., Cho and Lee, and Xu et al.; real-world tests include invariant and spatially varying blur.
6. Discussion
The discussion examines what the network learns, how its behavior changes with image content and input size, and where performance degrades. Learned features support kernel estimation, while larger blur kernels remain a limitation.
- Limitations: The method falls short for blur kernels larger than 25×25 pixels.The authors attribute this performance drop to a suboptimal multi-scale architecture at higher-resolution scales.
- Learned filters: The feature extraction module uses both the restored latent image and the blurry image to estimate the blur kernel.This differs from approaches that use only a linearly filtered blurry input for kernel estimation.
- Learned filters: The first-stage filters resemble gradient, Gabor, and Laplace-like filters, whereas later-stage filters are more intricate and harder to interpret.The filters change across the three stages of the learned network.
- Content dependence: Training image content changes the learned filters; valley filters are often mirror- or rotationally symmetric unlike many filters of the generic network.These content-specific features are described as informative about the unknown blur kernel.
- Input-size dependence: Larger inputs lead to better kernel estimates, with the learned algorithm showing monotonic MSE decreases as observed-image size increases.The comparison reports no clear trend for the non-learned state-of-the-art method.
7. Conclusion
The method learns blind deconvolution as a nonlinear regression problem while incorporating the forward model’s shared blur kernel. It adapts to varied imaging settings, but large kernel sizes remain a scalability issue.
- The approach maps blurry inputs to predicted kernels through a single large nonlinear regression problem.It incorporates the fact that the same blur kernel convolves the image everywhere, combines local features globally for kernel estimation, and propagates image information across the whole image.
- Image estimation propagates information throughout the image, reducing the difficulty of the problem for the following iteration.
- The approach can adapt to strong noise, specific image classes, and extensions combining deblurring with other imaging-pipeline steps.The proposed extensions include over-saturation, Bayer filtering, HDR, and super-resolution.
- Recorded spatially varying camera shakes and other blur sources, including lens aberrations and atmospheric turbulence, are identified as future research directions.The authors also suggest that trained-system insights could inform hand-crafted methods through higher-order gradient representations and extended gradient filters.
- Scalability to large kernel sizes remains an issue that may require improved neural-network architectures and training.
Appendix A Quotient Layer
The quotient layer estimates the blur kernel from predicted sharp and blurry images using a Fourier-domain operation with learned regularization. Its parameters receive gradients through back-propagation during end-to-end training.
- The quotient layer estimates the kernel ˜k from images ˜xi and blurry counterparts ˜yi predicted by earlier network layers.
- The quotient layer includes a learned regularization parameter βk.
- Training back-propagates the current example’s loss through later layers to determine gradient steps for ˜xj, ˜yk, and βk.The kernel gradient ∆˜k is propagated to obtain ∆˜xj, ∆˜yk, and ∆βk.
A.1 Derivative with respect to sharp images
The appendix derives the gradient with respect to sharp images for the quotient layer using differential forms and matrix calculus. The resulting gradient step is expressed through the transpose of the differential operator.
- The derivative calculation begins by expressing the differential in numerator layout for vectors u, v, and matrix M.
- The derivation expands vector and matrix differentials componentwise before applying matrix-calculus rules.
- With only ˜xj variable, the differential of ˜k is written as M d˜xj, where M combines A, B, and C.
- The resulting gradient step is ∆˜xj = M^T∆˜k.The derivation also uses the Hadamard product and the Diag operation that forms a diagonal matrix from a vector.
- Hermitian symmetry ensures the relevant inverse-Fourier-transformed product is real, and the real part operator is denoted ℜ(v).
A.2 Derivative with respect to blurry images
The derivative with respect to blurry images is derived analogously to the sharp-image derivative. In this case, the derivation treats the sharp-image variable and βk as fixed.
- The derivation for the gradient with respect to blurry images starts from the corresponding differential form.
- The calculation assumes d˜xj and dβk are zero while deriving the gradient for the blurry-image variable.It again uses that a real vector is unchanged by conjugation.
A.3 Derivative with respect to the regularization parameter
The procedure differentiates with respect to the regularization parameter βk while treating βk as the variable, then multiplies ∆˜k by the transpose of d˜k.
- The derivation follows the previous procedure for the regularization parameter βk.
- The calculation assumes only βk is variable.
- The next algebraic step multiplies ∆˜k by the transpose of d˜k.