PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth-Aware Video Frame Interpolation

Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang

arXiv:1904.00830v1cs.CV

TL;DR

Large motion and occlusions still challenge high-quality video frame interpolation. DAIN uses depth-aware flow projection and hierarchical contextual features, performing favorably against existing methods across diverse datasets while remaining compact and efficient.

  • Problem

    Large object motion and occlusions remain challenges for generating high-quality interpolated video frames despite advances in deep CNNs.

  • Method

    DAIN exploits depth-aware flow projection to prioritize closer objects, integrating optical flow, depth maps, local kernels, and learned hierarchical features.

  • Results

    The proposed model performs favorably against existing frame interpolation methods across diverse datasets and is reported as effective, efficient, and compact.

  • Takeaways & Limitations

    The results support exploiting depth cues for video frame interpolation, including explicit occlusion handling through depth-aware flow aggregation.

  • Takeaways & Limitations

    Depth estimation can fail in challenging cases, producing ambiguous object boundaries and blurred interpolation results.

Abstract

from arXiv · show

Video frame interpolation aims to synthesize nonexistent frames in-between the original frames. While significant advances have been made from the recent deep convolutional neural networks, the quality of interpolation is often reduced due to large object motion or occlusion. In this work, we propose a video frame interpolation method which explicitly detects the occlusion by exploring the depth information. Specifically, we develop a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones. In addition, we learn hierarchical features to gather contextual information from neighboring pixels. The proposed model then warps the input frames, depth maps, and contextual features based on the optical flow and local interpolation kernels for synthesizing the output frame. Our model is compact, efficient, and fully differentiable. Quantitative and qualitative results demonstrate that the proposed model performs favorably against state-of-the-art frame interpolation methods on a wide variety of datasets.

1. Introduction

The paper addresses quality degradation in video frame interpolation caused by large motion and occlusion by explicitly exploiting depth to detect occlusions. It introduces DAIN, which integrates depth-aware flow projection, optical flow, interpolation kernels, depth maps, and hierarchical contextual features for frame synthesis.

  • Motivation: Video frame interpolation supports applications including slow-motion generation, novel view synthesis, frame-rate up-conversion, and video-streaming frame recovery, but large motion and occlusion remain challenging.High-frame-rate videos can reduce temporal jittering and motion blurriness.
  • Core idea: The method explicitly detects occlusion using depth information, based on the observation that closer objects should be preferably synthesized in intermediate frames.It first estimates bidirectional optical flow and depth maps from the two input frames.
  • Depth-aware flow projection: The depth-aware flow projection layer aggregates colliding flow vectors according to depth, producing intermediate flows with clearer motion boundaries than simple flow averaging.Depth-based aggregation makes closer objects contribute preferentially over farther ones.
  • DAIN model: DAIN combines optical flow, local interpolation kernels, depth maps, and learnable hierarchical features to synthesize high-quality video frames.Hierarchical features provide contextual information from a large neighborhood without relying on a pre-trained recognition network such as ResNet.
  • Contributions: The proposed model is presented as more effective, efficient, and compact than state-of-the-art frame interpolation methods.The model uses an adaptive warping layer to warp the input frames.

2. Related Work

Recent learning-based video frame interpolation methods synthesize frames using direct CNNs, optical flow, adaptive kernels, or combinations thereof, but face blur, motion-related distortion, artifacts, memory, and computational costs. The proposed approach differs by explicitly using depth for occlusion detection and interpolation, while jointly estimating flow and depth for the interpolation task.

  • Video frame interpolation: Recent interpolation methods use direct CNN synthesis, optical flow, spatially-adaptive kernels, or end-to-end combinations of flow-based and kernel-based approaches.Long et al. directly synthesize intermediate frames with a generic CNN; AdaConv and SepConv estimate adaptive kernels, while Bao et al. combine flow and kernels.
  • Video frame interpolation: Flow-based methods reduce blurriness but remain challenging under large motion, where inaccurate flow causes severe distortion and visual artifacts.The passage contrasts reduced blurriness from flow with its vulnerability to difficult motion and flow-estimation errors.
  • Video frame interpolation: Kernel-based approaches synthesize pixels from large neighborhoods but typically require high memory footprint and heavy computational load.AdaConv and SepConv are identified as representative spatially-adaptive kernel methods.
  • Video frame interpolation: Existing methods implicitly handle occlusion with masks, contextual features, or large kernels, whereas this work explicitly detects occlusion using depth information in flow projection.The proposed method also incorporates depth maps with learned hierarchical features as contextual information for output synthesis.
  • Depth estimation: Depth estimation supports scene-geometry understanding, with conventional methods using stereo images and recent learning-based approaches estimating depth from a single image.The work uses Chen et al.’s hourglass model for depth estimation.
  • Depth estimation: Although some methods jointly estimate optical flow and depth using cross-task constraints, this model optimizes both specifically for frame interpolation rather than real pixel motion and scene depth.Its estimated flow and depth may therefore not resemble the true physical quantities.

3. Depth-Aware Video Frame Interpolation

The method synthesizes intermediate frames by estimating bidirectional optical flow and depth, using a depth-aware flow projection layer to handle occlusion. It then adaptively warps frames, depth maps, and contextual features before frame synthesis with residual learning.

  • Overview: The model estimates bidirectional optical flows and depth maps, projects intermediate flows with depth awareness, and synthesizes the frame through adaptive warping and a synthesis network.The model comprises flow estimation, depth estimation, context extraction, kernel estimation, and frame synthesis sub-modules.
  • Kernel Estimation and Adaptive Warping: An adaptive warping layer uses optical flows and local interpolation kernels to warp input frames, depth maps, and contextual features before synthesis.The layer samples each input image within a local window centered according to optical flow, while a U-Net estimates the kernels.
  • Depth-Aware Flow Projection: Depth-aware projection weights candidate flows by reciprocal depth so projected flows favor closer objects and reduce contributions from occluded, farther pixels.Unlike simple averaging, the method rewrites flows according to depth and points sampling toward the closer pixel.
  • Depth-Aware Flow Projection: The layer fills projection holes by averaging available four-neighbor flows, producing dense intermediate flow fields for warping.The depth-aware flow projection layer is fully differentiable, allowing joint optimization of the flow and depth estimation networks.
  • Context Extraction: The context extraction network learns interpolation-specific hierarchical features from scratch using a 7 × 7 convolution and two residual blocks without normalization.This replaces contextual features taken from a pre-trained classification network, which may be ineffective for video frame interpolation.
  • Frame Synthesis: The frame synthesis network concatenates warped inputs, depth maps, contextual features, projected flows, and kernels, then predicts residuals over linearly blended warped frames.Because optical flow already aligns the warped frames, synthesis focuses on enhancing details and sharpness.

4. Experimental Results

Experiments across multiple-resolution video datasets show that depth-aware flow projection and hierarchical contextual features improve interpolation quality. The proposed DAIN compares favorably with existing methods while using fewer parameters than MEMC-Net, but depth-estimation errors can blur boundaries in challenging cases.

  • Datasets and metrics: Evaluation covers Middlebury, Vimeo90K, UCF101, and HD datasets with differing resolutions and motion characteristics.Middlebury is around 640 × 480 pixels; Vimeo90K has 3,782 test triplets at 448 × 256 pixels; UCF101 has 379 triplets at 256 × 256 pixels; HD contains 11 high-resolution videos with typically larger motion.
  • Datasets and metrics: The experiments use IE and NIE on Middlebury and PSNR and SSIM on UCF101, Vimeo90K, and HD, with lower Middlebury errors indicating better performance.The study includes ablations, comparisons with state-of-the-art methods, and discussion of limitations and future work.
  • Ablation study: Pre-trained depth initialization improves over random initialization, while jointly optimized depth-aware projection produces sharper edges and shapes.DA-Scra performs worse than DA-None and cannot learn meaningful depth from random initialization; DA-Pret generates clear motion boundaries, and DA-Opti further optimizes the depth maps.
  • Ablation study: Learned contextual features substantially improve results, especially learned hierarchical features, and combining them with depth maps generates sharper and clearer content.Without contextual information, outputs are blurred; pre-trained conv1 features or depth maps improve performance, while learned hierarchical features further improve Vimeo90K and Middlebury results.
  • Comparison with existing methods: The proposed method performs favorably against existing methods across the evaluated datasets, including a 0.42dB PSNR gain over MEMC-Net on Vimeo90K.On Middlebury, it ranks 1st in NIE and 3rd in IE among published algorithms at submission; visual comparisons show clearer alignment and fewer ghosting or blurred results.
  • Efficiency: The model uses a similar parameter count to SepConv, runs faster, and uses 69% fewer parameters than MEMC-Net while achieving better performance.Execution time is measured on a 640 × 480 image.
  • Limitations: Depth-estimation errors can create ambiguous object boundaries and blurred results, although the method still reconstructs the skateboard better than ToFlow in the cited example.The limitation appears in challenging cases such as the boundary between a shoe and skateboard.

5. Conclusion

The paper proposes a compact, efficient depth-aware video frame interpolation algorithm that explicitly detects occlusion using depth information. It combines depth-aware flow projection with hierarchical features and depth maps as contextual information for synthesizing intermediate frames.

  • Conclusion: The method explicitly detects occlusion using depth information for video frame interpolation.This is the central design goal of the proposed algorithm.
  • Conclusion: Its depth-aware flow projection layer encourages sampling closer objects rather than farther ones.The layer is designed to incorporate depth ordering into intermediate-flow synthesis.
  • Conclusion: The model uses learned hierarchical features and depth maps as contextual information to synthesize intermediate frames.These inputs support the synthesis process alongside the proposed flow projection.
  • Conclusion: The proposed model is compact and efficient.The passage characterizes the overall model in these terms.
Loading 1904.00830v1…