Source-linked AI summary
Learning Blind Motion Deblurring
Patrick Wieschollek, Michael Hirsch, Bernhard Schölkopf, Hendrik P. A. Lensch
TL;DR
Motion blur from camera shake and moving objects makes sharp-frame reconstruction ill-posed, while realistic ground-truth data for neural blind deblurring is difficult to obtain. The paper introduces a recurrent deblurring network with temporal feature transfer and a synthetic-data generation method based on online videos. The approach processes arbitrary spatial and temporal input sizes, and the authors report improved results and run time on benchmark sets, while very fast motion remains difficult to recover.
Problem
Motion blur from camera shake and object motion makes sharp-frame reconstruction highly ill-posed, while realistic aligned training pairs are difficult to acquire.
Method
The paper combines a recurrent deblurring network with temporal feature transfer and synthetic blurry/sharp sequence generation from online videos.
Results
The approach handles arbitrary spatial and temporal input sizes and demonstrates improved results and run time on benchmark sets and challenging real-world examples.
Takeaways & Limitations
The method provides an efficient way to train and apply blind video deblurring across sequences of varying length and size without alignment preprocessing.
Takeaways & Limitations
Very fast motion remains difficult to recover, and optical-flow-based methods can produce slightly sharper results when camera motion is severe.
Abstract
from arXiv · showhide
As handheld video cameras are now commonplace and available in every smartphone, images and videos can be recorded almost everywhere at anytime. However, taking a quick shot frequently yields a blurry result due to unwanted camera shake during recording or moving objects in the scene. Removing these artifacts from the blurry recordings is a highly ill-posed problem as neither the sharp image nor the motion blur kernel is known. Propagating information between multiple consecutive blurry observations can help restore the desired sharp image or video. Solutions for blind deconvolution based on neural networks rely on a massive amount of ground-truth data which is hard to acquire. In this work, we propose an efficient approach to produce a significant amount of realistic training data and introduce a novel recurrent network architecture to deblur frames taking temporal information into account, which can efficiently handle arbitrary spatial and temporal input sizes. We demonstrate the versatility of our approach in a comprehensive comparison on a number of challening real-world examples.
1 University of T¨ubingen 2 Max Planck Institute for Intelligent Systems, T¨ubingen
Figure 1 compares the proposed learning-based blind burst-deblurring approach with FBA on blurry input sequences.
- The proposed approach reconstructs fine details that FBA does not recover, while both methods have similar run-time.
1. Introduction
Handheld videos commonly contain blur from camera shake and moving objects, making sharp-frame reconstruction highly ill-posed. The paper addresses this challenge with a recurrent network and realistically generated synthetic training data.
- Handheld videos combine camera shake and object motion, causing motion-blur artifacts during recording.
- Reconstructing a sharp frame from a blurry observation is highly ill-posed because the recording contains motion blur.
- The paper introduces a recurrent encoder-decoder with spatial residual connections and temporal feature transfer between successive iterations.
- The authors generate synthetic training data without acquisition cost and report improved results and run time across benchmark sets.
2. Related Work
Prior work includes patch-based priors, end-to-end multi-frame deblurring, and video deblurring with U-Net architectures.
- Traditional blind-deblurring methods use carefully chosen patch-based priors to predict sharp images.
- An end-to-end neural network processes input bursts patch-wise and can handle spatially varying blur.
- A related video-deblurring approach uses a U-Net with skip connections to regress a sharp image from an input burst.
3. Method
The method combines recurrent deblurring over arbitrary-length sequences with synthetic training-data generation from high-quality online videos. Its network fuses temporal features across deblur steps using encoder-decoder blocks and temporal skip connections.
- Recurrent deblurring: The recurrent deblur network iteratively improves a target frame using previous blurry observations and can process arbitrary spatial dimensions and frame counts.
- Training data: The training process uses synthetic-blur inputs, network predictions, and associated sharp ground truth.
- Training data: The method acquires online videos instead of manually recording training data, addressing the difficulty of obtaining large high-quality sharp-video datasets.
- Training data: Bidirectional optical flow generates arbitrary intermediate subframes, which are averaged to create plausible blurry versions of sharp frames.
- Training data: The resulting dataset uses sharp video segments and blurry counterparts produced offline, with different blur levels created by varying the averaging parameter.
- Handling the time dimension: The recurrent formulation avoids fixed burst lengths and padding or sliding-window workarounds required by channel-stacking approaches.
- Training: Training applies losses to successive deblur outputs produced after processing pairs of target and additional observations.
- Network architecture: Each deblur block uses an encoder-decoder with residual blocks, while temporal skip connections concatenate latent features from preceding blocks.
4. Experiments
Experiments show that the recurrent deblur network performs well across burst, spatially varying blur, low-light, forest, and video examples, while temporal and multi-scale processing improve recovery but introduce practical limits.
- Burst Deblurring: The network processes each burst observation recurrently and produces significantly better results than previous methods on challenging real-world scenes.It restores license-plate lettering and produces sharper results in the wood scene.
- Burst Deblurring: The recurrent approach recovers subtle license-plate writing and preserves original color tones from a random blurry shot.
- Burst Deblurring: For spatially varying blur, the network generalizes without patch-wise processing and exploits almost arbitrarily many input frames.After more than 10 input images, local contrast may saturate and cause a small color shift.
- Burst Deblurring: On real-world low-light static scenes, RDN recovers significantly more detail than FourierNet and FBA.
- Video Deblurring: Multi-scale input partly helps recover larger motion blur, but upsampling can create comic-style artifacts in static scene regions.The approach deblurs sequences at multiple resolutions and feeds upscaled predictions into higher-resolution processing.
- Architecture Analysis: Temporal skip connections propagate features that mark image regions likely to benefit from further deblurring.The features are visualized after projection to two dimensions and Hue encoding.
5. Conclusion
The paper presents a recurrent deblurring network for sequences of unaligned blurry frames and a scalable method for generating realistic blurry/sharp training pairs.
- The recurrent deblurring network removes blur from ego and object motion in unaligned blurry frame sequences.
- Temporal skip connections propagate information across several time steps while the network iteratively sharpens a target frame.
- The training-data method uses bidirectional optical flow to synthesize intermediate frames and averaging to emulate realistic motion blur.
- High-quality videos from YouTube enable generating an arbitrary amount of blurry/sharp training data.