Source-linked AI summary
Motion Deblurring in the Wild
Mehdi Noroozi, Paramanand Chandramouli, Paolo Favaro
TL;DR
Dynamic-scene deblurring is challenging when objects move arbitrarily relative to the camera and may be occluded or come into view. The paper trains a three-subgraph multiscale CNN to predict sharp images from single blurry frames, using central video frames and frame averages as training pairs. The authors report state-of-the-art performance for single-image space-varying motion blur and sharper images with fine details across evaluated datasets.
Problem
Dynamic-scene deblurring is challenging when objects move arbitrarily relative to the camera and may be occluded or come into view.
Source: §1 · p. 1Method
The paper trains a three-subgraph multiscale CNN to predict sharp images from single blurry frames, using central video frames and frame averages as training pairs.
Source: §1 · p. 1, §3 · p. 5, §5 · p. 10Results
Takeaways & Limitations
Frame averaging from high-frame-rate video provides reasonable blurred-sharp training pairs, and the authors report restoration under occlusions, motion parallax, and camera rotations.
Source: §5 · p. 10Shift-invariant blur evaluation methods disadvantage dynamic-scene methods because the image deformation is undefined for ambiguous shifts at every pixel.
Source: §4 · pp. 7, 9
Abstract
from arXiv · showhide
The task of image deblurring is a very ill-posed problem as both the image and the blur are unknown. Moreover, when pictures are taken in the wild, this task becomes even more challenging due to the blur varying spatially and the occlusions between the object. Due to the complexity of the general image model we propose a novel convolutional network architecture which directly generates the sharp image.This network is built in three stages, and exploits the benefits of pyramid schemes often used in blind deconvolution. One of the main difficulties in training such a network is to design a suitable dataset. While useful data can be obtained by synthetically blurring a collection of images, more realistic data must be collected in the wild. To obtain such data we use a high frame rate video camera and keep one frame as the sharp image and frame average as the corresponding blurred image. We show that this realistic dataset is key in achieving state-of-the-art performance and dealing with occlusions.
1 Introduction
The paper targets single-image deblurring in dynamic scenes, where motion, spatially varying blur, and occlusions complicate restoration. It proposes a data-driven CNN and realistic training pairs to handle this setting.
1 Introduction
- The method uses one blurry frame to produce a sharp image, aiming for high-quality results in dynamic scenes without user input.The authors report state-of-the-art performance for single-image space-varying motion blur.Source: §1 · pp. 1–2, §1.1 · p. 3
- Training pairs use a high-frame-rate camera: the central frame is sharp, while an average of nearby frames supplies the blur.Frames with optical flow above 1 pixel are discarded to avoid unrealistic motion blur.Source: §1 · pp. 1–2
1.1 Related work
- Earlier work often assumes uniform blur or specified motion models, while multi-frame methods address spatial variation.Dynamic scenes can also involve object motion, occlusion, or objects entering view.Source: §1 · p. 1, §1.1 · pp. 2–3
- Unlike prior deep-learning methods, the approach directly predicts the sharp image from one input, uses no user input, and targets natural dynamic scenes.The authors also distinguish their high-frame-rate-camera training set from prior deep-learning work.Source: §1.1 · p. 3
2 Blurry Images in the Wild
The WILD dataset creates paired blurred and sharp images from high-frame-rate outdoor video. Averaging is restricted to frames with small estimated motion to avoid unrealistic blur.
2 Blurry Images in the Wild
- WILD blurred images are averages of video frames, paired with the central frame as the sharp target.The videos were captured outdoors with a GoPro Hero5 Black at 240 frames per second and 1280×720 resolution.Source: §2 · p. 3, Figure 2 · p. 4
- The authors report that the resulting dataset generalizes to images from other camera models and scenes.Their example pair shows the average as blurry input and the central frame as the sharp target.Source: Figure 2 · p. 4
- The dataset uses an odd number of frames, randomly selected from 7 to 23, after downsampling each frame by a factor of 3.Only frames with estimated optical flow no greater than 1 pixel are retained for averaging.Source: §2 · p. 4
3 The Multiscale Convolutional Neural Network
DeblurNet uses three subgraphs in a multiscale CNN to reconstruct a sharp image from a blurry input. Residual skip connections and losses at multiple scales structure the reconstruction.
3 The Multiscale Convolutional Neural Network
- The architecture table notes that convolutional strides above one downsample, while deconvolutional strides above one upsample.Batch normalization and ReLU layers after convolutional layers are omitted from the table except after the last layer of N1.Source: Table 1 · p. 5
- DeblurNet has three subgraphs, each reconstructing the image at a different scale through convolution and fractional-stride deconvolution layers.The network progresses from a quarter-resolution residual to half-resolution and then full-resolution output.Source: §3 · pp. 4–5
- Each subgraph predicts a residual that is added to the blurry input, reducing the reconstruction burden and helping recover texture details.Downsampling reduces blur extent, so the multiscale stages address smaller amounts of blur at each scale.Source: §3 · p. 5
- The training objective minimizes reconstruction error at all three scales simultaneously.The loss is defined as L = L1 + L2 + L3.Source: §3 · p. 5
4 Experiments
The experiments evaluate DeblurNetWILD against prior methods and a synthetically trained shift-invariant network on WILD, Canon, and prior-work data. They report PSNR and qualitative comparisons, examine shift-invariant evaluation limits for dynamic scenes, and relate residual magnitude to blur size.
4 Experiments
- DeblurNetWILD uses 50K frame-averaged WILD pairs and outperforms DeblurNetSI despite its smaller training set.DeblurNetSI uses 105 synthetic shift-invariant blurs on sharp images that include the same WILD frames.Source: §4 · p. 6
- DeblurNetWILD achieves higher PSNR than DeblurNetSI on the WILD test set.Table 2 reports average PSNR; the test set includes images from locations and weather conditions different from training.Source: Table 2 · p. 6, §4 · p. 6
- Across WILD, Canon, and prior-work examples, the method returns sharper images with fine details.On the Canon test, DeblurNetWILD also produces better results than DeblurNetSI; Fig. 4 compares methods on WILD images.Source: §4 · pp. 6–7, Figure 4 · p. 7
- Dynamic scenes have ambiguous shifts at every pixel, making shift-based PSNR alignment unfeasible and disadvantaging these methods under shift-invariant evaluation.The paper notes that results may still look qualitatively good.Source: §4 · pp. 7, 9
- Across 700 images, normalized residual L1 magnitude correlates with normalized average estimated blur size.The blur estimate comes from integrating optical flow; Fig. 9 plots the two normalized quantities, while Fig. 10 shows the highest- and lowest-residual examples.Source: §4 · p. 9, Figure 9 · p. 10, Figure 10 · p. 10
5 Conclusions
DeblurNet restores sharp images from blurry inputs under challenging conditions, including occlusions, motion parallax, and camera rotations. The authors report state-of-the-art performance against prior dynamic-scene deblurring work using a realistic training dataset.
5 Conclusions
- The authors report state-of-the-art quantitative and qualitative performance against prior dynamic-scene deblurring work.The network handles challenging conditions including occlusions, motion parallax, and camera rotations.Source: §5 · p. 10
- DeblurNet chains three subgraphs for multiscale deblurring, each predicting a residual that is added to its input to yield the sharp image.The residual formulation lets each subgraph focus on small details, as confirmed experimentally.Source: §5 · p. 10
- Simple frame averaging from very high-frame-rate video produces reasonable blurred-sharp pairs for training DeblurNet.The authors identify a sufficiently realistic dataset as an important part of their solution.Source: §5 · p. 10
- The network may leave extreme blurs untouched, although the authors observe that it does not generate artifacts.Source: §5 · p. 10