PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Video Deblurring via Dynamic Temporal Blending Network

Tae Hyun Kim, Kyoung Mu Lee, Bernhard Schölkopf, Michael Hirsch

arXiv:1704.03285v1cs.CV

TL;DR

Video deblurring must handle ill-posed, spatially varying motion blur while most existing methods process complete frame sequences and are computationally costly. The paper introduces an online spatio-temporal recurrent network with dynamic temporal blending and a compact, enlarged-receptive-field design. It reports real-time sequential deblurring, speeds hundreds to thousands of times faster than existing methods, and strong performance on challenging videos, while its training data covers motion blur rather than defocus blur.

  • Problem

    Existing video-deblurring methods commonly require access to all recorded frames, making them computationally demanding and time consuming for practical use.

  • Method

    The paper proposes an online spatio-temporal recurrent network with dynamic temporal blending, a small computational footprint, and an enlarged receptive field.

  • Results

    The method performs sequential deblurring and is reported to be hundreds to thousands of times faster than existing state-of-the-art video-deblurring methods while removing general motion blur in real time for VGA video.

  • Takeaways & Limitations

    The approach provides real-time online deblurring for challenging dynamic-scene videos and reports superiority in comprehensive qualitative and quantitative comparisons.

  • Takeaways & Limitations

    The generated training datasets handle motion blur but not other blur types such as defocus blur.

Abstract

from arXiv · show

State-of-the-art video deblurring methods are capable of removing non-uniform blur caused by unwanted camera shake and/or object motion in dynamic scenes. However, most existing methods are based on batch processing and thus need access to all recorded frames, rendering them computationally demanding and time consuming and thus limiting their practical use. In contrast, we propose an online (sequential) video deblurring method based on a spatio-temporal recurrent network that allows for real-time performance. In particular, we introduce a novel architecture which extends the receptive field while keeping the overall size of the network small to enable fast execution. In doing so, our network is able to remove even large blur caused by strong camera shake and/or fast moving objects. Furthermore, we propose a novel network layer that enforces temporal consistency between consecutive frames by dynamic temporal blending which compares and adaptively (at test time) shares features obtained at different time steps. We show the superiority of the proposed method in an extensive experimental evaluation.

1. Introduction

The paper targets blind video deblurring in dynamic scenes with a fast online method that handles motion blur from both camera and object motion. It combines a compact spatio-temporal recurrent architecture, dynamic temporal blending, and a newly collected training dataset.

  • Motivation: Blind restoration of sharp video frames is highly ill-posed, while motion blur from moving objects and camera shake degrades video quality.The problem is especially pronounced in low-light recordings and hand-held action-camera videos.
  • Contribution: The proposed method is an online discriminative deblurring technique designed for real-time VGA processing of spatially varying blur from egomotion and object motion.The authors describe it as the first method of this type with real-time performance.
  • Architecture: The spatio-temporal recurrent layout extends the receptive field while keeping the network small, enabling large-blur removal with substantially faster execution.The paper reports runtimes orders of magnitude faster than previous methods without significantly sacrificing restoration quality.
  • Architecture: Dynamic temporal blending compares consecutive feature representations and performs input-dependent, pixel-specific information propagation during test time.This mechanism is part of a recurrent architecture operating across temporal steps.
  • Training Data: A high-speed-camera dataset of sharp and realistic blurry videos was collected by averaging consecutive sharp frames, enabling end-to-end network training.The dataset addresses the lack of publicly available video-deblurring training data.
  • Evaluation: Experiments report sequential processing and speeds hundreds to thousands of times faster than existing state-of-the-art video-deblurring methods.The evaluation covers challenging real-world dynamic scenes qualitatively and quantitatively.

2. Related Work

Earlier deblurring methods commonly assume stationary blur, use sharp-frame exemplars or segmentation, or estimate pixel-wise varying kernels and motion flows. These approaches struggle with spatially varying object motion or incur substantial computation, motivating the paper’s alternative.

  • Multi-frame Deblurring: Many multi-frame methods assume shift-invariant blur and therefore cannot directly handle spatially varying blur.Rotational camera motion can require additional non-trivial alignment of multiple images.
  • Sharp-frame Methods: Sharp-frame exemplar methods transfer information from registered neighboring frames but struggle to find correspondences for severely blurred, fast-moving objects.Local search and blur modeling improve alignment but do not remove this limitation.
  • Region-based Methods: Segmentation-based methods model differently blurred regions using foreground-background or layered-scene assumptions and constrained motion models.These approaches estimate region-specific blur rather than treating every pixel independently.
  • Pixel-wise Methods: Pixel-wise kernel and optical-flow methods address general motion blur but are time consuming because they estimate motion or spatially varying kernels.The cited approaches include flow-based registration and local homographies for varying object motion.

3. Training Datasets

The authors construct training pairs by averaging consecutive high-speed sharp frames to synthesize realistic motion-blurred videos, using the center short-exposure frame as the sharp reference. The process controls blur strength and synthesized frame rate while excluding non-motion blurs such as defocus.

  • Scope: The resulting dataset supports spatially and temporally varying motion blur from egomotion and object motion, but not defocus blur.This limitation follows from generating blur through temporal averaging of high-speed frames.
  • Dataset generation: Training pairs combine synthesized blurry frames with center short-exposure frames used as sharp references.Blurry frames are produced by averaging consecutive high-speed frames, while the center short-exposure image supplies the reference sharp frame.
  • Dataset generation: The synthesis models camera shake and object motion by averaging consecutive high-speed frames over an effective shutter interval.The parameter τ determines how many frames are averaged and therefore controls the simulated shutter speed.
  • Dataset controls: Larger τ values generate stronger motion blur, while the interval T controls the duty cycle and frame rate of the synthesized video.The dataset uses T ≥ τ, and varying τ and T changes blur strength and temporal sampling.
  • Acquisition: The authors recorded high-speed sharp frames with a GoPro HERO4 BLACK at 240 frames per second before downsampling them to 960x540.They selected τ ∈ {7, 9, 11, 13, 15} and τ ≤ T < 2τ to generate varied training examples.

4. Method Overview

The method combines a spatio-temporal recurrent network with dynamic temporal blending to handle large blur efficiently and maintain temporal consistency. Recurrence expands the effective receptive field without increasing parameter count, while adaptive feature blending uses information from consecutive time steps.

  • Spatio-temporal recurrent network: The recurrent architecture is designed to remove large motion blur while retaining a small computational footprint.A large receptive field is needed for large blur, but conventional deeper or wider networks increase runtime.
  • Spatio-temporal recurrent network: The spatio-temporal recurrent network repeatedly processes blurry inputs over time, increasing the effective receptive field without adding parameters.Previous feature maps are fed alongside current blurry inputs, so each frame experiences deeper spatial feature extraction.
  • Network architecture: The proposed model contains an encoder, dynamic temporal blending network, and decoder, with trainable variables shared across time.The encoder receives consecutive blurry frames and feature activations from the previous stage.
  • Dynamic temporal blending: Dynamic temporal blending compares feature maps from consecutive time steps and adaptively propagates information using an input-dependent weight map.The mechanism blends the current feature map with the previously filtered feature map to encourage temporal consistency.
  • Dynamic temporal blending: The blending weight lies between zero and one and favors the previously filtered feature when the current feature is degraded relative to it.The weight map is generated from learned linear filters and nonlinear activations based on the two feature maps.
  • Dynamic temporal blending: The dynamic temporal blending network adds only one convolutional layer and nonlinear activations, keeping its computation fast while improving deblurring quality.The complete recurrent model with this mechanism is referred to as STRCNN+DTB.

5. Implementation and Training

The model combines encoder, dynamic temporal blending, and decoder modules in a recurrent architecture, trained end-to-end with synthetic blurry/sharp video patches and a regularized MSE objective.

  • Architecture: The network comprises encoder, dynamic temporal blending, and decoder modules, with trainable variables shared across time.The architecture is described at a single time step while recurrence reuses the same parameters.
  • Architecture: Dynamic temporal blending compares consecutive feature maps and adaptively propagates pixel-specific information through learned weight maps.The layer receives consecutive feature representations, estimates weights with a convolution and squashing functions, and blends them.
  • Architecture: The decoder processes blended features through four residual blocks and outputs a latent sharp frame corresponding to the blurry input.Nearest-neighbor upsampling renders the predicted image at the input-frame resolution.
  • Objective function: Training uses mean squared error between latent frames and sharp ground truth, plus weight decay to prevent overfitting.The final objective combines data fidelity and regularization, with λ set to 10^-5 in the experiments.
  • Training procedure: Training samples 13 consecutive blurry frames, crops shared-location 128x128 patches, and optimizes with Adam using an exponentially decaying learning rate.The batch size is 8, the initial learning rate is 0.0001, and the decay rate is 0.96.

6. Experiments

Experiments show that STRCNN+DTB improves PSNR across blur strengths and input-frame settings, generalizes to challenging real videos, and enables substantially faster online processing than conventional methods.

  • Model comparison: STRCNN+DTB achieves a 0.37dB PSNR gain over the CNN baseline, while STRCNN achieves 0.2dB, averaged across blur strengths.STRCNN+DTB performs consistently better across the evaluated blur sizes.
  • Model comparison: STRCNN+DTB outperforms the other networks for every tested number of input blurry frames, and the five-frame model is selected as final.The selected configuration uses m = 2, corresponding to five input frames.
  • Online behavior: Online restoration quality is initially lower because temporal inputs are initialized to zero, then improves and stabilizes after one or two frames.The experiment tracks average PSNR as the number of processed frames increases.
  • Quantitative results: On 25 test videos, the proposed model outperforms the state-of-the-art methods in PSNR.The evaluation uses videos captured with the paper’s high-speed camera.
  • Qualitative and runtime results: The method restores challenging spatially varying blur at motion boundaries and remains competitive with offline deblurring while running orders of magnitude faster.The qualitative evaluation includes object motion, rotational camera shake, noise, and encoding artifacts.
  • Qualitative and runtime results: The method processes VGA video at approximately 24 frames per second and HD video at approximately 8 frames per second on an NVIDIA GTX 1080.These measurements demonstrate online processing at two video resolutions.
  • Ablation and temporal consistency: Dynamic temporal blending reduces artifacts compared with STRCNN by enforcing temporal consistency between consecutive frames.The comparison specifically reports fewer artifacts in regions affected by camera shake.

7. Conclusion

The paper concludes that its recurrent architecture and dynamic blending network support efficient video deblurring with large receptive fields and improved temporal consistency, validated on challenging real-world videos.

  • Conclusion: The spatio-temporal recurrent architecture preserves a large receptive field without computational overhead by reusing prior feature activations.This design supports near-real-time processing while addressing large blur.
  • Conclusion: The dynamic blending network enforces temporal consistency and provides a significant performance gain.The conclusion attributes this gain to the proposed blending mechanism.
  • Conclusion: Experiments on challenging real-world videos demonstrate the efficiency and superiority of the proposed method.The evaluation combines the newly acquired blurry/sharp video dataset with the proposed network.
Loading 1704.03285v1…