PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Spatio-Temporal Learning for Video Deblurring

Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Wei Liu, Hongdong Li

arXiv:1804.00533v2cs.CV

TL;DR

Video deblurring must model spatio-temporal information across neighboring frames while producing sharp results beyond pixel-wise fidelity. The paper proposes DBLRNet with 3D convolutions and DBLRGAN with content and adversarial losses, achieving reported improvements on two standard benchmarks, although temporal jumping artifacts remain incompletely solved.

  • Problem

    Video deblurring lacks effective modeling of neighboring-frame spatio-temporal dynamics, while pixel-wise metrics such as PSNR may not capture realistic sharpness.

  • Method

    DBLRNet uses 3D convolutions in a deep residual network, while DBLRGAN uses it as a generator with content and adversarial losses.

  • Results

    Experiments on two standard benchmarks show that DBLRNet and DBLRGAN outperform existing methods, with DBLRNet yielding 1%–5% PSNR improvement over DBLRNet (multi).

  • Takeaways & Limitations

    Joint spatial-temporal learning and adversarial training recover sharp, realistic video frames from blurred input sequences.

  • Takeaways & Limitations

    Jumping artifacts remain a challenge, and the method cannot solve them completely.

Abstract

from arXiv · show

Camera shake or target movement often leads to undesired blur effects in videos captured by a hand-held camera. Despite significant efforts having been devoted to video-deblur research, two major challenges remain: 1) how to model the spatio-temporal characteristics across both the spatial domain (i.e., image plane) and temporal domain (i.e., neighboring frames), and 2) how to restore sharp image details w.r.t. the conventionally adopted metric of pixel-wise errors. In this paper, to address the first challenge, we propose a DeBLuRring Network (DBLRNet) for spatial-temporal learning by applying a modified 3D convolution to both spatial and temporal domains. Our DBLRNet is able to capture jointly spatial and temporal information encoded in neighboring frames, which directly contributes to improved video deblur performance. To tackle the second challenge, we leverage the developed DBLRNet as a generator in the GAN (generative adversarial network) architecture, and employ a content loss in addition to an adversarial loss for efficient adversarial training. The developed network, which we name as DeBLuRring Generative Adversarial Network (DBLRGAN), is tested on two standard benchmarks and achieves the state-of-the-art performance.

I. INTRODUCTION

The paper addresses video deblurring challenges in modeling neighboring-frame spatio-temporal information and producing perceptually realistic sharp frames. It proposes DBLRNet and DBLRGAN, combining 3D spatio-temporal learning with content and adversarial losses, and evaluates them on two benchmarks.

  • Hand-held camera shake and object movement create blur that video deblurring seeks to remove for applications including 3D reconstruction, SLAM, and tracking.
  • Video deblurring is difficult because temporal dynamics across continuous frames are not fully understood and PSNR may not reflect perceptual sharpness or realism.
  • DBLRNet applies 3D convolutions within a deep residual network to learn joint spatial-temporal features across neighboring frames.
  • DBLRGAN uses DBLRNet as its generator and combines content loss with adversarial loss to balance pixel-wise measurement and realistic-looking restoration.
  • Experiments on the VideoDeblurring and Blurred KITTI datasets show that DBLRNet and DBLRGAN outperform existing methods.

II. RELATED WORK

Prior video-deblurring work includes geometry-based, aggregation, stereo, and deep-learning approaches, but existing frame-processing methods can preserve limited temporal information and prioritize pixel fidelity. DBLRNet addresses these issues by learning joint spatio-temporal representations from neighboring blurry frames with 3D convolutions.

  • Existing deblurring methods are broadly categorized as geometry-based or deep-learning methods.
  • Geometry-based methods: Single-image geometry-based methods estimate blur kernels and latent sharp images, but spatially varying real-world blur makes kernel estimation difficult.
  • Geometry-based methods: Aggregation methods combine information across low-quality images or frames to restore blurry regions and maintain spatial-temporal coherence.
  • Geometry-based methods: Stereo-video methods jointly use scene-flow motion cues and blur information, but require stereo cameras.
  • Deep learning methods: DBN stacks five neighboring frames along RGB channels and uses 2D convolutions, limiting temporal information, whereas DBLRNet uses 3D convolutions for joint spatial-temporal learning.DBLRNet takes five consecutive blurry frames as input and outputs the central deblurred frame.

III. OUR MODEL

DBLRNet learns joint spatial and temporal representations from neighboring frames using 3D convolutions within a residual architecture. It processes five consecutive grayscale luminance frames and outputs the deblurred central frame.

  • A. DBLRNet: DBLRNet applies 3D convolutions in deep residual networks to capture joint spatio-temporal features for video deblurring.
  • A. DBLRNet: The network uses 3D kernels over cubes formed from multiple neighboring frames, with the temporal dimension represented by the kernel’s third dimension.
  • A. DBLRNet: DBLRNet contains two 3 × 3 × 3 convolutional layers, residual blocks, and additional convolutional layers without spatial down-sampling or up-sampling.
  • A. DBLRNet: Table I specifies 14 residual blocks, while Table II reports the discriminator configuration using batch normalization and ReLU.
  • A. DBLRNet: The input is five consecutive frames converted to YCbCr, using the Y channel for deblurring before restoring color with the original Cb and Cr channels.

B. DBLRGAN

DBLRGAN places DBLRNet in a generative adversarial framework for video deblurring. The generator restores frames, while a CNN discriminator distinguishes deblurred outputs from real sharp frames.

  • B. DBLRGAN: DBLRGAN uses a generator to deblur images and a discriminator to distinguish deblurred images from real-world sharp images.
  • B. DBLRGAN: The adversarial formulation trains the generator to fool the discriminator and the discriminator to distinguish generated frames from real sharp frames.
  • B. DBLRGAN: The proposed DBLRNet serves as the generator, while the discriminator is a VGG-like CNN with 14 convolutional layers.

C. Loss Functions

DBLRGAN combines content and adversarial losses to balance pixel-wise fidelity with realistic sharpness. A weighting parameter controls the adversarial contribution during training.

  • C. Loss Functions: DBLRGAN uses content loss and adversarial loss during training.
  • C. Loss Functions: Content loss is based on mean square error and compares generated deblurred frames with sharp frames at corresponding pixel locations.
  • C. Loss Functions: Adversarial loss updates DBLRNet parameters to fool the discriminator, whose output estimates whether a recovered frame is real and sharp.
  • C. Loss Functions: The final loss combines content and adversarial terms using hyper-parameter α.
  • C. Loss Functions: When α = 0, DBLRGAN reduces to DBLRNet; larger α values increase adversarial influence, but large values can degrade performance.

IV. EXPERIMENTAL RESULTS

The experiments evaluate DBLRNet and DBLRGAN for video deblurring. The stated aim is to demonstrate the effectiveness of both proposed networks.

  • IV. EXPERIMENTAL RESULTS: The experiments target the task of video deblurring.
  • IV. EXPERIMENTAL RESULTS: The evaluation is designed to demonstrate the effectiveness of DBLRNet and DBLRGAN.
  • IV. EXPERIMENTAL RESULTS: Both proposed networks are included in the experimental evaluation.

A. Datasets

The evaluation uses the VideoDeblurring and Blurred KITTI benchmarks, covering real captured videos and synthetic blurred driving scenes.

  • VideoDeblurring Dataset: The VideoDeblurring benchmark contains videos from devices including iPhone 6s, GoPro Hero 4, and Nexus 5x, with about 100 frames per video.Its quantitative subset has 6,708 blurry and ground-truth sharp frames from 71 videos, while its qualitative subset contains 22 scenes without ground truth.
  • Blurred KITTI Dataset: The Blurred KITTI benchmark is based on KITTI stereo 2015 data and contains 199 scenes with images captured by left and right cameras.

B. Implementation Details and Parameters

Training uses extensive patch augmentation and staged learning-rate adjustments, while evaluation includes unaligned exemplar and PSNR comparisons for the proposed models.

  • Evaluation: Figure 4 and Table III compare proposed outputs and PSNR results without image alignment.
  • Training: DBLRNet is trained with content loss, starting at a learning rate of 10^-4 and reducing it to 10^-5 when the loss plateaus.
  • DBLRGAN: DBLRGAN uses α = 0.0002, selected empirically for best performance, and is initialized by fine-tuning DBLRNet as its generator.

C. Effectiveness of DBLRNet

DBLRNet benefits from both neighboring-frame inputs and 3D convolutions, while DBLRGAN produces sharper, more realistic details and operates without alignment.

  • DBLRNet Design: DBLRNet (multi) and DBLRNet use five neighboring frames, whereas DBLRNet (single) uses one frame; the baselines use only 2D convolutions.
  • DBLRNet Effectiveness: 3% ∼5% PSNR improvement from DBLRNet (single) to DBLRNet (multi) shows that stacking neighboring frames helps learn temporal features with 2D convolution.
  • DBLRNet Effectiveness: 1% ∼5% additional PSNR improvement from DBLRNet (multi) to DBLRNet is attributed by the authors to 3D-convolution-based spatio-temporal representations.
  • DBLRGAN: DBLRGAN yields sharper and more photo-realistic frames than DBLRNet, with finer textural details such as clearer letters.
  • Efficiency: The models generate high-quality results without alignment, avoiding a procedure described as computationally expensive and fragile.
  • Generalization: On bokeh-blurred Blurred KITTI videos, sharper edges after processing indicate generalization to another blur type.

E. Comparison with Existing Methods

The proposed models are compared with established methods on VideoDeblurring and Blurred KITTI, including methods using multiple frames or additional stereo information.

  • VideoDeblurring: On VideoDeblurring, DBLRGAN and DBLRNet are compared with PSDEBLUR, WFA, DBN, and DBN (single).DBN had previously achieved state-of-the-art performance on this dataset.
  • VideoDeblurring: DBLRGAN and DBLRNet significantly outperform DBN by 3.14 dB on the VideoDeblurring dataset.
  • Blurred KITTI: On Blurred KITTI, the proposed models are compared with methods including,, and geometry-based method.Method uses additional stereo information from image pairs and is described as the dataset’s state of the art.
  • Blurred KITTI: A DBLRNet trained on VideoDeblurring achieves comparable results with on Blurred KITTI, while DBLRGAN slightly outperforms.The authors note that their models are not specialized for the stereo setting.

F. Different Frames & Other Types of Blur

Experiments examine how input-frame count and blur type affect DBLRGAN, while the authors acknowledge unresolved jumping artifacts. Across two benchmarks, the proposed models outperform existing video-deblurring methods.

  • Different Frames: Five consecutive blurry frames produce the best PSNR among the tested three-, five-, seven-, nine-, and eleven-frame inputs.This comparison is conducted on the VideoBlurring dataset.
  • Other Types of Blur: Training on camera-shake blur, the model is also evaluated on Blurred KITTI videos captured from a high-speed car with bokeh-dominated blur.The experiment tests generalization to a different type of blurry video.
  • Limitations: The method cannot completely remove jumping artifacts that remain in some deblurred frames.The artifacts are shown in Figure 1, column 4&5, row 2.
  • Limitations: Jointly capturing spatial and temporal information helps alleviate unexpected temporal artifacts without post-processing or frame alignment.The authors also report better deblurred frames than prior methods when frames are severely blurred.
  • Results: Experiments on the VideoDeblurring and Blurred KITTI datasets show that DBLRNet and DBLRGAN outperform existing state-of-the-art video-deblurring methods.The reported benchmarks cover the proposed networks' effectiveness across standard datasets.
Loading 1804.00533v2…