Learning to Extract a Video Sequence from a Single Motion-Blurred Image

1. Introduction

• Exposure-time averaging makes motion blur both a visual degradation and a record of object texture and motion.Recovering those cues can help understand scene dynamics, such as in sports entertainment or traffic surveillance.Source: §1 · p. 1
• The paper introduces dynamic-blur blind deconvolution as recovering a sequence of sharp frames from one motion-blurred image.The target is a sequence of instantaneous frames rather than a single sharp image.Source: §1 · p. 1
• The task is difficult because deblurring is ill-posed and temporal averaging destroys frame order, creating many valid sequences.The method uses order-invariant losses based on frame-pair averages and absolute differences, allowing the network to choose outputs during training.Source: §1 · p. 2
• A large receptive field addresses deblurring, while a resampling layer avoids the memory and computational costs of large convolutional filters.The paper also uses adversarial training to make outputs more realistic and sharp.Source: §1 · p. 2

2. Prior Work

• Earlier uniform-motion and camera-shake deblurring methods use image priors, patch recurrence, or kernel-estimation techniques.Examples include combining bright and dark channel priors, exploiting patch recurrence across scales, and estimating kernels with gradient activation.Source: §2 · p. 2
• Non-uniform deblurring methods estimate motion, blur, segmentation, or sharp images, while dynamic-blur work also trains on averaged high-frame-rate video.Nah et al. [21] train a multi-scale CNN to restore a latent image directly.Source: §2 · p. 2
• Video deblurring methods use information from blurry video sequences, including temporal, stereo, flow, or scene-structure cues.The cited approaches restore sharp sequences or frames using multiple blurry frames or sequence information.Source: §2 · p. 2
• The paper identifies a gap: none of these approaches extracts a video sequence from a single motion-blurred image.It then introduces its loss functions and network design, tested on synthetic and real datasets.Source: §2 · pp. 2–3

3. From Video to Image

• Each discretized frame x[i] represents the integral of instantaneous irradiance over one time segment.The image y is captured over exposure τ, and g denotes the camera response function mapping irradiance to measured intensity.Source: §3 · p. 3
• The frame count T is set to the maximum pixel shift Δ, treating a one-pixel shift as negligible blur.This ensures only that each frame is unblurred on average; acceleration can still cause more than one pixel of blur in some frames.Source: §3 · p. 3
• Dynamic blur allows regions to shift independently and texture to appear or disappear through occlusion; the recovery target is the T frames satisfying the blur model.The model is not restricted to a single convolution of one sharp image.Source: §3 · p. 3
• Temporal averaging erases frame order, so even reversing the entire sequence can produce the same blurred input.Figure 2 illustrates four plausible two-object motion scenarios with identical average images.Source: §3 · p. 3, Figure 2 · p. 3

4. Unraveling Time

• Temporal ordering ambiguity prevents a unique sequence target, even though temporal smoothness may resolve some local ordering choices.Global forward-versus-backward motion remains ambiguous independently for each moving object.Source: §4 · p. 3
• For n independently moving objects, forward-or-backward direction choices create 2^n valid motion combinations with the same blurred average.The two-ball example has four equally valid sequences, so a network cannot be expected to learn one specific frame choice.Source: §4 · p. 3
• The middle frame of an odd-length sequence is unchanged across the illustrated alternatives, explaining prior methods' successful middle-frame prediction.The other frames can differ despite the shared blurry average.Source: §4 · pp. 3–4
• The paper introduces novel loss functions to handle temporal ordering ambiguity.The losses are intended to avoid requiring a uniquely specified temporal order.Source: §4 · p. 4

5. From Image to Video

• GoPro 240-fps training clips are averaged in groups of eight, but the method uses T = 7 so the middle frame can be predicted without ambiguity; its perceptual loss uses VGG16 relu2_2 and relu3_3.The middle-frame network is trained directly, while losses in the following subsections address the other frames.Source: §5 · p. 4

5.1. Globally Ordering-Invariant Loss

• The globally ordering-invariant loss lets networks choose their outputs, but works well only when the blurry frame averages no more than three frames.With more frames averaged, training may not converge well or produce a meaningful sequence.Source: §5.1 · p. 4

5.2. Pairwise Ordering-Invariant Loss

• For each symmetric frame pair, the proposed loss matches sums and absolute differences, imposing a stronger constraint while remaining invariant to motion direction.The second network receives both the blurry image and the first network’s output so it can generate a different frame.Source: §5.2 · p. 4

5.3. Learning a Temporal Direction

• Independent pairs retain a binary ordering ambiguity, so sequential training uses previous network outputs to establish direction for the middle-core frames and propagate it to outer frames.The non-core networks can share weights for i = 1, 2; Table 1 summarizes the networks, losses, and training procedure.Source: §5.3 · p. 4, Table 1 · p. 5

6. Implementation

• The middle-frame network uses resampling by a factor of four, 12 residual blocks, and six residual blocks with dilated convolutions to enlarge its receptive field.Feature extraction and refinement operate on grayscale images; three color-refined features are fused in color to compensate for misalignments.Source: §6 · p. 5
• Table 2 compares middle-frame prediction networks.The passage identifies the comparison’s scope but supplies no outcome values.Source: Table 2 · p. 5
• Table 3 compares execution time for the state-of-the-art single-image deblurring network [21] and the proposed model at three resolutions on a Titan X GPU.Further architecture details are in the supplementary material.Source: Table 3 · p. 5, §6 · p. 5

• Our network consistently performs better on the two datasets matching its training blur, while Nah’s network performs better on 1111 original images with larger blur.Source: §7 · pp. 5–6
• Figure 3 shows better visual results than Nah’s for moderate blur, while real reconstructions include a false car-motion direction and a rotating ball with complex motion.Source: Figure 3 · p. 6, §7 · p. 6, Figure 6 · p. 8
• The middle-frame prediction network is approximately 10 times faster than Nah’s and has half as many parameters; 4× resampling offers a better accuracy–execution-time trade-off.Source: §7 · pp. 6–7
• Independent predictions worsen farther from the center, global-loss outputs artifact, and plain pairwise outputs require manual reordering and still artifact away from the center.Source: §7 · p. 7
• Incorrect estimated middle frames from either method impair other frames, while supplying the ground-truth middle frame allows correct reconstruction.Source: Figure 5 · p. 7
• Sequential pairwise reconstruction yields fewer artifacts than pairwise reconstruction, especially at frames 1 and 7.Source: §7 · p. 7
• Teacher forcing converges faster, gives visually better predictions, and permits middle- and non-middle-frame networks to be trained in parallel.Source: §7 · p. 8
• The method is not robust to large blurs: when middle-frame deblurring fails, the non-middle-frame networks also fail.Source: §7 · p. 8

• The paper presents a first method to reconstruct a video sequence from a single motion-blurred image.Source: §8 · p. 8
• The task is more ambiguous than single-frame deblurring because motion blur loses the frames’ temporal ordering.Source: §8 · p. 8
• A data-driven CNN chooses a temporal ordering at its output and generalizes to real images captured with different cameras from the training cameras.Source: §8 · p. 8