PaperlayerView exact revision on arXiv

Source-linked AI summary

Cascaded Deep Video Deblurring Using Temporal Sharpness Prior

Jinshan Pan, Haoran Bai, Jinhui Tang

arXiv:2004.02501v1cs.CV

TL;DR

Video deblurring must recover latent frames from blurred sequences despite motion and severe ill-posedness. The paper introduces a compact CNN that estimates flow from intermediate latent frames, restores those frames with a temporal sharpness prior, and trains cascaded stages end to end. It reports favorable accuracy and model size against state-of-the-art methods on benchmark and real-world videos, while noting a limitation when every frame position remains blurred.

  • Problem

    Recovering latent frames from blurred video is highly ill-posed, and existing alignment-based methods can be less effective for significant blur.

  • Method

    A compact CNN estimates optical flow from intermediate latent frames, restores latent frames using a temporal sharpness prior, and is jointly trained through cascaded end-to-end stages.

  • Results

    The method performs favorably against state-of-the-art methods in accuracy and model size on benchmark datasets and real-world videos.

  • Takeaways & Limitations

    Using video-deblurring domain knowledge makes the CNN model more compact and efficient while retaining favorable performance against state-of-the-art methods.

  • Takeaways & Limitations

    The temporal sharpness prior provides little significant improvement when blur exists at every position in all frames.

Abstract

from arXiv · show

We present a simple and effective deep convolutional neural network (CNN) model for video deblurring. The proposed algorithm mainly consists of optical flow estimation from intermediate latent frames and latent frame restoration steps. It first develops a deep CNN model to estimate optical flow from intermediate latent frames and then restores the latent frames based on the estimated optical flow. To better explore the temporal information from videos, we develop a temporal sharpness prior to constrain the deep CNN model to help the latent frame restoration. We develop an effective cascaded training approach and jointly train the proposed CNN model in an end-to-end manner. We show that exploring the domain knowledge of video deblurring is able to make the deep CNN model more compact and efficient. Extensive experimental results show that the proposed algorithm performs favorably against state-of-the-art methods on the benchmark datasets as well as real-world videos.

1. Introduction

Video deblurring estimates latent frames from blurred sequences, but motion, depth, and ill-posedness make recovery difficult. The paper proposes a compact CNN using intermediate-latent-frame optical flow, temporal sharpness, and cascaded end-to-end training.

  • Video deblurring is ill-posed because only blurred sequences are given despite blur from camera shake, object motion, and depth variation.
  • Conventional methods jointly estimate optical flow and latent frames under hand-crafted priors, but their assumptions produce complex, difficult-to-solve energy functions.
  • Deep CNN methods improve restoration but can be less effective for significant blur when they do not align input frames or use latent-frame warping.
  • Cascaded training jointly trains the model end to end, with experiments on benchmark datasets and real-world videos showing favorable accuracy and model size against state-of-the-art methods.
  • The proposed compact CNN estimates optical flow and latent frames while using a temporal sharpness prior to constrain restoration.

2. Related Work

Prior video-deblurring methods use sharp neighboring content, variational priors, or deep networks with temporal processing. The paper argues that large-capacity models limit real-world generalization and instead favors compact models grounded in established principles.

  • Hand-crafted interpolation methods exploit sharp adjacent contents but are less effective for moving-object blur and can generate smooth results.
  • Variational methods model motion blur with optical flow and constrain latent frames and flow using priors.
  • Deep methods use temporal and spatial attention with end-to-end training to produce promising deblurred results.
  • Large-capacity models drive much of these methods’ success, but their generalization ability on real applications is limited.

3. Motivation

The motivation combines variational video-blur modeling with deep restoration. The paper seeks compact CNNs that use optical-flow alignment and domain knowledge rather than relying only on direct prediction or larger capacity.

  • The blur model represents each blurred image using a clear image integrated over exposure time and warping functions defined by bidirectional optical flow.
  • Variational video-deblurring methods depend on constraints for latent images and optical flow, often producing highly non-convex objectives that are difficult to solve.
  • Many deep CNN methods estimate warping from blurred inputs rather than latent frames and are less effective for significant blur.
  • The proposed approach uses established variational principles and domain knowledge to make deep CNNs more compact for video deblurring.

4. Proposed Algorithm

The proposed video-deblurring model estimates optical flow between intermediate latent frames, aligns neighboring frames, and restores the reference latent frame with a CNN. A temporal sharpness prior and cascaded end-to-end training further guide restoration and flow estimation.

  • 4. Proposed Algorithm: The model jointly estimates optical flow and latent frames using an optical-flow module, latent-image restoration module, and temporal sharpness prior.The modules are jointly trained in a unified end-to-end framework.
  • 4.1. Optical Flow Estimation: The optical-flow module estimates motion between adjacent intermediate latent frames, using PWC-Net to provide motion information for image restoration.The flow estimator takes two images as input and shares parameters across frame pairs.
  • 4.2. Latent Frame Restoration: Adjacent frames are warped into the reference frame to provide aligned guidance, after which an encoder-decoder restoration network removes artifacts and blur.Direct averaging can produce artifacts from misalignment, so the warped frames are supplied to the restoration network instead.
  • 4.3. Temporal Sharpness Prior: The temporal sharpness prior identifies likely clear pixels when motion-compensated neighboring values are close to the reference pixel.A patch-based distance makes the sharpness estimate more robust, and the resulting prior helps the network distinguish clear from blurred pixels.
  • 4.4. Inference: Cascaded training learns stage-dependent optical-flow and restoration parameters by repeatedly applying the full network to blurred video sequences.Each stage updates the intermediate latent frames and estimates parameters by minimizing the training cost function.

5. Experimental Results

The method is evaluated on public benchmark and real-world video-deblurring datasets against variational and deep CNN baselines. It performs favorably on benchmark metrics and produces clearer structures and details in qualitative comparisons.

  • 5. Experimental Results: The evaluation compares the proposed method with variational and deep CNN-based algorithms on publicly available video-deblurring datasets.PSNR and SSIM are used to evaluate restored images on synthetic datasets.
  • 5.1. Quantitative Results: The benchmark results show that the proposed algorithm performs favorably against state-of-the-art methods in PSNR and SSIM.The comparison uses the benchmark dataset introduced by Su et al.
  • 5.2. Comparisons with the State of the Art: On the benchmark test dataset, the proposed method recovers finer image details and structures than the compared state-of-the-art algorithms.The comparisons describe residual blur or insufficient structure recovery for several competing methods.
  • 5.1. Quantitative Results: On the GOPRO dataset, the proposed algorithm generates deblurred videos with higher PSNR and SSIM values.Qualitative results also report clearer images and recognizable license numbers compared with state-of-the-art methods.
  • 5.2. Comparisons with the State of the Art: On real videos, the proposed algorithm produces clearer frames with better detailed structures, including clearer people and building boundaries.The reported comparison is qualitative and uses results from the real video deblurring dataset by Cho et al.

6. Analysis and Discussions

The analysis evaluates cascaded training, optical flow, the temporal sharpness prior, model size, and limitations across benchmark and real-world video deblurring settings.

  • Cascaded training: Cascaded training estimates optical flow from intermediate latent frames and iteratively updates latent frames, unlike the single-stage alternative using blurred inputs.The single-stage method does not generate high-quality results because optical flow relates to latent-frame information during exposure.
  • Cascaded training: Using more cascaded stages generates better deblurred images, but the improvement is not significant; two stages balance accuracy and speed.The comparison evaluates stage 1, stage 2, and stage 3 results.
  • Optical flow: The proposed optical flow contains sharper boundaries than flow estimated from blurred inputs, facilitating restoration of important image boundaries.Blurred flow boundaries affect boundary restoration, whereas the proposed flow preserves sharp boundaries.
  • Temporal sharpness prior: The temporal sharpness prior distinguishes sharp and blurred pixels in adjacent frames, improving deblurring accuracy and structural clarity.Its effectiveness is demonstrated through quantitative and qualitative comparisons on videos with significant blur.
  • Model size: The proposed algorithm uses a relatively smaller model while generating better results than baseline models without increasing model size.This supports the goal of improving accuracy without increasing model capacity through domain knowledge.
  • Limitations: The temporal sharpness prior is less effective when blur occurs at every position across all frames because clear pixels cannot be distinguished reliably.On three such videos, the prior does not significantly improve deblurring performance.

7. Concluding Remarks

The paper concludes with a compact deep CNN built from optical-flow estimation and latent-frame restoration, augmented by a temporal sharpness prior and cascaded training. End-to-end training yields a compact, efficient model that performs favorably on benchmark datasets and real-world videos.

  • Concluding Remarks: The proposed CNN combines optical flow estimation from intermediate latent frames with latent-frame restoration.These components form the main stages of the video-deblurring model.
  • Concluding Remarks: A temporal sharpness prior assists latent-image restoration, while cascaded training provides an end-to-end strategy for training the CNN.The paper presents both mechanisms as central parts of the proposed approach.
  • Concluding Remarks: The CNN is more compact and efficient and performs favorably against state-of-the-art methods on benchmark datasets and real-world videos.The conclusion attributes these outcomes to end-to-end training and the proposed design.

8. Supplemental Material

The supplemental material provides additional network details and experimental analysis for the proposed video-deblurring algorithm.

  • Supplemental Material: The supplemental material includes further network details and experimental results beyond the main paper.It directs readers to a supplementary source for the additional material.
Loading 2004.02501v1…