Source-linked AI summary
Learning a Convolutional Neural Network for Non-uniform Motion Blur Removal
Jian Sun, Wenfei Cao, Zongben Xu, Jean Ponce
TL;DR
The paper addresses estimating and removing spatially varying motion blur from a single blurry image, especially in complex scenes. It uses CNN-based local kernel prediction, candidate-set extension, and MRF field inference before deblurring, and reports effective estimation and removal of challenging non-uniform blur.
Problem
Strongly non-uniform motion blur in complex scenes remains difficult to remove with existing approaches.
Method
A CNN predicts motion-kernel distributions from local patches, rotations extend its candidate set, and an MRF fuses estimates into a dense motion-kernel field for deblurring.
Results
The approach effectively predicts spatially varying motion kernels in strongly blurred real and synthetic examples and achieves state-of-the-art motion deblurring results.
Takeaways & Limitations
The method can handle complex, strongly non-uniform motion blur that previous approaches estimate poorly.
Takeaways & Limitations
Motion-kernel estimation uses only local image regions and makes no global parametric motion assumptions, making the estimation challenging.
Abstract
from arXiv · showhide
In this paper, we address the problem of estimating and removing non-uniform motion blur from a single blurry image. We propose a deep learning approach to predicting the probabilistic distribution of motion blur at the patch level using a convolutional neural network (CNN). We further extend the candidate set of motion kernels predicted by the CNN using carefully designed image rotations. A Markov random field model is then used to infer a dense non-uniform motion blur field enforcing motion smoothness. Finally, motion blur is removed by a non-uniform deblurring model using patch-level image prior. Experimental evaluations show that our approach can effectively estimate and remove complex non-uniform motion blur that is not handled well by previous approaches.
1. Introduction
The paper targets strongly non-uniform motion blur, which remains difficult for existing methods, and proposes a CNN-based estimation and deblurring pipeline. It predicts local motion kernels, extends their candidates, fuses them into a dense field, and deconvolves the image.
- Strongly non-uniform motion blur in complex scenes remains challenging for existing approaches.
- The method predicts patch-level motion-kernel probabilities with a CNN and extends its candidate kernels using image rotations.The extension is designed to improve motion-kernel estimation.
- An MRF fuses patch estimates into a dense, spatially varying field of motion kernels.
- The estimated motion field is used to deconvolve the blurry image with a non-uniform deblurring model.Figure 1 presents the sequence from blurry input to kernel-field estimation and deconvolution.
- Unlike methods based on global camera motion or latent sharp-image estimation, this approach estimates kernels directly from local patches.The local formulation does not require estimating camera motion or a latent sharp image.
2. Learning a CNN for Motion Blur Estimation
The method predicts patch-level motion-kernel distributions with a CNN, expands its discrete candidates through rotated images, and estimates spatially varying blur kernels. An MRF then enforces motion smoothness across the image; extension reduces kernel-estimation MSE from 10.3 to 8.4.
- Patch-level motion-kernel estimation: The approach predicts motion-kernel probabilities for overlapping blurry patches and fuses them into a dense motion field with an MRF enforcing motion smoothness.It represents each local blur kernel through a motion vector and estimates spatially varying kernels without global parametric motion assumptions.
- Candidate motion-kernel set: The candidate set discretizes motion lengths from 1 to 25 at interval 2 and orientations from 0° to 150° at interval 30°, yielding 73 motion vectors.Length-one motions share the identity kernel, so orientation does not create additional distinct kernels in that case.
- Patch-level motion-kernel estimation: The CNN uses 30 × 30 patches and a six-layer C1−M2−C3−M4−F5−S6 architecture whose soft-max layer labels 73 candidate kernels.Training uses synthetically blurred patches generated from clean natural images and the candidate motion kernels.
- Motion-kernel set extension: 10.3 to 8.4: CNN motion-kernel set extension reduces motion-kernel MSE and removes blocky orientation artifacts in the illustrated region.The unextended model assigns the same quantized orientation to pixels in the blue rectangle, whereas extension produces more accurate kernels.
3. Dense Motion Field Estimation by MRF
The method fuses patch-level CNN motion-kernel probabilities into a dense field, then uses an MRF to favor confident and spatially smooth motion estimates.
- Patch-level estimation: Overlapping 30 × 30 × 3 patches produce motion-kernel probabilities that are spatially weighted and averaged into per-pixel confidences.Patches closer to a pixel’s center receive higher Gaussian weights.
- MRF inference: The MRF estimates a dense motion field by combining CNN confidence with a smoothness term that encourages neighboring pixels to have similar motion vectors.The dense field is M = {m_p = (l_p, o_p)} over image pixels.
- MRF inference: Each pixel uses the top 20 high-confidence kernels plus 30 sampled alternatives to reduce the computational cost of MRF optimization.Predicting dense motion blur for a 300 × 400 image takes around 80 seconds on a CPU, including CNN computation.
- Results: The full MRF model removes noisy unary-term estimates through its smoothness constraint, producing significantly improved quantitative motion results in the example.Figure 8 compares unary-only estimation, the full MRF model, and ground-truth motion blur.
4. Non-Uniform Motion Deblurring
The paper deconvolves the blurry image using spatially varying kernels and a natural-image patch prior, optimizing the resulting model with alternating subproblems.
- Model: Non-uniform deconvolution models the observed blurry image using spatially varying motion kernels and a patch-level natural-image prior.The prior is represented as a Gaussian mixture model learned from natural image patches.
- Optimization: Half-quadratic splitting introduces auxiliary patch variables and alternates optimization of the sharp image and those variables.The penalty parameter β is increased during the iterations.
- Optimization: The sharp-image subproblem represents non-uniform convolution as matrix-vector multiplication and solves the resulting linear equations with conjugate gradients.The matrix-vector operations are implemented efficiently using convolutions or local operations around each pixel.
- Implementation: The implementation uses 8×8 patches, λ = 2×10^5, and seven alternating iterations while increasing β from 50 to 3200 by a factor of two.These settings define the reported optimization schedule.
- Evaluation: Table 1 compares motion-kernel estimation on 15 test images with synthetic motion blur against BlurSpect and SLayerRegr.The supplied passage identifies the comparison setup and baselines but does not provide the table’s metric values.
5. Experiments
Experiments show that the CNN-based approach estimates challenging spatially varying motion kernels and produces effective non-uniform deblurring results. Kernel-set extension and motion smoothness improve estimation accuracy, while comparisons report stronger kernel estimation and blur removal than competing methods.
- 15 synthetic test images evaluate motion-kernel estimation under non-uniform camera motion with rotation and translation.
- Examples with strongly non-uniform blur and complex depth layers show that the CNN predicts spatially varying motion kernels effectively.The examples include three real blurry images and one synthetic image with camera rotation.
- The full method estimates motion kernels with average MSE motion 7.83 and PSNR motion 44.55.
- Motion-kernel set extension and the motion-smoothness constraint significantly improve motion-kernel estimation accuracy.
- Compared with available non-uniform deblurring methods, the approach estimates more accurate blur kernels and produces better final deblurring results.For comparison with, the paper reports visually more natural results because is commonly over-sharpened; source-code availability limits qualitative comparison.
- Table 2 reports mean motion-kernel estimation and deblurred-image PSNR values over 15 synthetic test images.The metrics are MSE ker for spatial-domain kernel error and PSNR deblur for the final restored image.
6. Conclusion
The paper proposes a CNN-based approach for non-uniform motion deblurring, combining local-patch kernel estimation with an MRF motion-blur field. It reports state-of-the-art motion deblurring results and identifies general non-uniform blur handling as future work.
- The proposed approach uses a CNN to estimate motion kernels from local patches and an MRF to predict the non-uniform motion-blur field.
- The resulting method achieves state-of-the-art motion deblurring results.
- Future work targets CNN-based estimation and removal of general non-uniform blurs within a single framework.