Source-linked AI summary
From Motion Blur to Motion Flow: a Deep Learning Solution for Removing Heterogeneous Motion Blur
Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton van den Hengel, Qinfeng Shi
TL;DR
Pixel-wise heterogeneous motion blur is difficult because the problem is ill-posed and existing approaches rely on restrictive assumptions, predefined priors, or costly optimization. The paper learns dense motion flow directly from blurred images with an FCN trained on simulated pairs, then recovers sharp images from the estimated flow. Experiments on synthetic and real-world data report strong performance and generalization, while the method restricts horizontal motion to be nonnegative to resolve directional ambiguity.
Problem
Single-image removal of pixel-wise heterogeneous motion blur is ill-posed, while existing methods face limited motion types, predefined priors, and complicated optimization or post-processing.
Method
An FCN learns an end-to-end mapping from a blurred image to dense motion flow using simulated blurred-image-motion-flow pairs, after which non-blind deconvolution recovers the sharp image.
Results
Experiments on synthetic and real-world data show the proposed method produces excellent heterogeneous-motion-blur estimation and removal results, with strong generalization across image datasets.
Takeaways & Limitations
Learning motion flow rather than latent-image content enables a flexible blur model with almost no underlying-image assumptions and avoids human labeling.
Takeaways & Limitations
The method restricts horizontal motion to be nonnegative because opposite motion vectors generate identical linear blur patterns.
Abstract
from arXiv · showhide
Removing pixel-wise heterogeneous motion blur is challenging due to the ill-posed nature of the problem. The predominant solution is to estimate the blur kernel by adding a prior, but the extensive literature on the subject indicates the difficulty in identifying a prior which is suitably informative, and general. Rather than imposing a prior based on theory, we propose instead to learn one from the data. Learning a prior over the latent image would require modeling all possible image content. The critical observation underpinning our approach is thus that learning the motion flow instead allows the model to focus on the cause of the blur, irrespective of the image content. This is a much easier learning task, but it also avoids the iterative process through which latent image priors are typically applied. Our approach directly estimates the motion flow from the blurred image through a fully-convolutional deep neural network (FCN) and recovers the unblurred image from the estimated motion flow. Our FCN is the first universal end-to-end mapping from the blurred image to the dense motion flow. To train the FCN, we simulate motion flows to generate synthetic blurred-image-motion-flow pairs thus avoiding the need for human labeling. Extensive experiments on challenging realistic blurred images demonstrate that the proposed method outperforms the state-of-the-art.
1. Introduction
The paper targets single-image removal of pixel-wise heterogeneous motion blur, which challenges methods built for uniform blur, restrictive motion models, or iterative priors. It learns dense motion flow directly with an FCN from simulated examples, then uses the estimate for blur removal.
- Heterogeneous motion blur requires recovering a sharp image when blur kernels may vary independently from pixel to pixel.
- Existing methods are limited by restricted motion types, predefined priors or features, and costly iterative optimization or post-processing.These limitations reduce practical applicability to real images and can produce artifacts.
- The proposed method learns a pixel-wise motion flow map directly from a single blurred image using a deep neural network trained on tens of thousands of examples.
- Training on simulated examples uses a flexible blur model and makes almost no assumptions about underlying images.
- The universal FCN estimates dense heterogeneous motion flow end to end on whole images and requires no post-processing.Whole-image training and testing use spatial context over a wider area than patch-level learning.
2. Related Work
Related work progresses from uniform-blur priors and motion models toward learning-based blur removal. Existing learning methods still commonly require known kernels, mild-blur restrictions, patch-level labels, or post-processing.
- Conventional blind image deblurring: Conventional blind deblurring commonly assumes spatially uniform blur and uses image priors or regularizers to constrain the ill-posed problem.
- Spatially-varying blur removal: Spatially-varying blur methods adopt flexible motion models, including projective paths, camera motion densities, and locally uniform overlapping patches.
- Learning based motion blur removing: Learning-based methods include kernel-dependent deconvolution, end-to-end reconstruction for mild Gaussian blur, and neural estimation of uniform motion-blur information.
- Learning based motion blur removing: The most relevant patch-based CNN assigns each small uniformly blurred patch a single motion label, ignores larger-area correspondence, and requires post-processing.
- The proposed scheme trains an FCN on simulated motion-flow maps, estimates flow end to end from a blurry image, and recovers the sharp image through non-blind deconvolution.
3. Estimating Motion Flow for Blur Removal
The paper models heterogeneous blur with a pixel-specific motion-flow field, estimates that field directly from a blurred image using an FCN, and recovers the sharp image through non-blind deconvolution.
- 3.1. A Heterogeneous Motion Blur Model: Heterogeneous blur is represented by a blur-kernel map whose kernels vary by pixel and operate on local image regions.The model also permits vectorized representation through a matrix whose rows correspond to pixel-located kernels.
- 3.1. A Heterogeneous Motion Blur Model: Each pixel’s motion is encoded by a two-dimensional vector, and the collection of horizontal and vertical motion maps forms the dense motion flow.The motion vector records horizontal and vertical movement over the exposure time.
- 3.2. Blur Removal via Motion Flow Estimation: Given the blurred image and estimated motion flow, the method recovers the sharp image by solving a non-blind deconvolution problem with a regularizer.The stated practical regularizer is based on a Gaussian mixture model.
- 3.3. Learning for Motion Flow Estimation: The network processes a blurred image through convolutional and up-sampling layers to produce a pixel-wise dense motion-flow map.Its structure includes skip connections that combine features at different resolutions.
- 3.3. Learning for Motion Flow Estimation: The FCN learns an end-to-end mapping from blurred images to motion flow using synthetic blurred-image and motion-flow pairs generated by simulated flows.Synthetic training avoids the infeasible task of obtaining dense ground-truth motion flow for many varied real images.
- 3.3. Learning for Motion Flow Estimation: Opposite motion vectors with equal length produce identical linear blur patterns, so the feasible motion domain restricts horizontal motion to be nonnegative.This restriction reduces ambiguity during learning.
4. Dense Motion Flow Estimation
The method uses a fully convolutional network to map an arbitrary-sized blurry RGB image to a same-sized dense motion-flow map, trained with simulated heterogeneous blur. Motion flows combine four camera-motion components, and recovered images are evaluated using PSNR and SSIM.
- Network Design: The FCN maps an arbitrary-sized RGB blurry image to a same-sized motion-flow map M = (U, V).The horizontal and vertical channels represent pixel movements in the two image directions.
- Network Design: The network uses convolutional and pooling layers, fractionally-strided convolutions for upsampling, and skip connections to combine information across layers.It contains 7 convolutional layers, 4 max-pooling layers, and 3 uconv layers.
- Network Design: Separate soft-max layers estimate posterior probabilities for the discrete horizontal and vertical motion labels.The final feature tensor is divided into slices corresponding to U and V before the two soft-max operations.
- Simulate Motion Flow for Data Generation: The training data target realistic motion flows by combining translations along x, y, and z axes with rotation around the z axis.The simulation projects motion directly onto the 2D image grid and models the flow as four additive components.
- Simulate Motion Flow for Data Generation: The simulator generates x/y translations, radial z-axis motion toward a vanishing point, and rotation-driven motion around a sampled center.The z-axis component uses distance from a vanishing point, while the rotational component varies with distance from its rotation center.
- Simulate Motion Flow for Data Generation: The dataset contains 10,200 blurred-image–motion-flow pairs generated from 200 BSD500 images and 10,000 independently simulated flow maps.Motion magnitudes are generated with umax = vmax = 36, and non-blurred images are also included for training.
- Evaluation: Table 1 compares PSNR and SSIM for recovered images using the estimated blur kernel.The caption states that the best results are bold-faced.
5. Experiments
Experiments on synthetic and real-world images evaluate motion-flow accuracy, generalization, runtime, and deblurring quality. The proposed method produces accurate flow without post-processing and recovers sharper images with fewer artifacts across varied blur scenarios.
- Motion flow estimation: The proposed method produces accurate motion-flow maps on BSD-S and BSD-M without MRF or CRF post-processing.Its performance remains accurate on the more challenging BSD-S dataset, where the patch-based method’s accuracy decreases significantly.
- Motion flow estimation: Figure 6 shows smoother flow close to ground truth, while Sun et al.’s method is more sensitive to image content and strong edges.The patch-level method generally underestimates motion values and produces errors near strong edges.
- Generalization ability: The method generalizes from BSD500-trained models to MC-S and MC-M, producing stable high-accuracy results on both COCO-based datasets.The experiment evaluates generalization on different images using synthetic datasets derived from Microsoft COCO.
- Real-world images: On real-world images, the method estimates smoother, more accurate flow on moving objects and recovers images with more details and fewer artifacts than comparison methods.The comparisons include camera motion, strong background blur, and large-scale object motion blur.
- Real-world images: For large-scale object motion blur, the method preserves details on the blurred moving foreground while keeping the sharp background unchanged.Competing results are described as oversmooth or losing details because of underestimated motion or segmentation error.
6. Conclusion
The paper presents a flexible deep-learning method that represents heterogeneous blur as pixel-wise linear motion blur and estimates dense motion flow with an FCN. Training data are generated automatically with simulated motion flows, and experiments on synthetic and real-world data demonstrate the method’s effectiveness.
- Conclusion: The method estimates and removes heterogeneous motion blur by representing it as pixel-wise linear motion blur and predicting dense motion flow with an FCN.Training data are automatically generated using simulated motion-flow maps.