PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Frame Interpolation via Adaptive Convolution

Simon Niklaus, Long Mai, Feng Liu

arXiv:1703.07514v1cs.CV

TL;DR

Video frame interpolation traditionally separates motion estimation from pixel synthesis, making results dependent on motion-estimation quality. This paper combines both operations through local convolution whose spatially adaptive kernels are estimated by a deep fully convolutional network, yielding high-quality interpolation that handles challenging cases such as occlusion and blur.

  • Problem

    Traditional frame interpolation separates motion estimation and pixel synthesis, but optical-flow estimation and flow-based synthesis are unreliable under occlusion, blur, and abrupt brightness changes.

  • Method

    A deep fully convolutional neural network estimates spatially adaptive kernels that convolve patches from two input frames, combining motion estimation and pixel synthesis in one process.

  • Results

    The method produces high-quality interpolation and handles challenging videos with occlusion, blur, and abrupt brightness change well.

  • Takeaways & Limitations

    Local convolution provides a robust interpolation formulation that can synthesize sharp results using edge-aware kernels and direct end-to-end training from widely available video data.

  • Takeaways & Limitations

    Motion handling is limited by the convolution kernel size: the system handles motion within 41 pixels well but not larger motion without downscaling or future multi-scale strategies.

Abstract

from arXiv · show

Video frame interpolation typically involves two steps: motion estimation and pixel synthesis. Such a two-step approach heavily depends on the quality of motion estimation. This paper presents a robust video frame interpolation method that combines these two steps into a single process. Specifically, our method considers pixel synthesis for the interpolated frame as local convolution over two input frames. The convolution kernel captures both the local motion between the input frames and the coefficients for pixel synthesis. Our method employs a deep fully convolutional neural network to estimate a spatially-adaptive convolution kernel for each pixel. This deep neural network can be directly trained end to end using widely available video data without any difficult-to-obtain ground-truth data like optical flow. Our experiments show that the formulation of video interpolation as a single convolution process allows our method to gracefully handle challenges like occlusion, blur, and abrupt brightness change and enables high-quality video frame interpolation.

1. Introduction

The paper replaces separate motion estimation and pixel synthesis with a single convolution-based process estimated by a deep fully convolutional network. This approach is trained end to end and targets robust, sharp interpolation in challenging video regions.

  • The single-process formulation avoids relying on optical flow, whose failures can cause artifacts in occluded, blurred, or abruptly illuminated regions.Traditional interpolation separates motion estimation and pixel synthesis, so errors in either step affect the interpolated frame.
  • A deep fully convolutional neural network estimates a spatially adaptive convolution kernel for each output pixel from receptive-field patches in the input frames.The estimated kernel is applied to co-centered input patches to synthesize the output pixel.
  • The method formulates pixel interpolation as convolution over patches from two input frames, unifying motion estimation and pixel synthesis.The spatially adaptive kernel captures both motion information and pixel-synthesis coefficients.
  • The method estimates edge-aware kernels and produces high-quality interpolation results for videos with occlusion, blur, and abrupt brightness change.
  • The network can be trained end to end with widely available video data without difficult-to-obtain ground-truth data such as optical flow.

2. Related Work

Prior video frame interpolation methods commonly estimate dense motion before synthesizing intermediate frames, while phase-based and deep-learning methods provide alternative strategies for motion and view synthesis.

  • Most existing methods estimate dense motion between consecutive input frames using stereo matching or optical flow, then synthesize intermediate frames from dense correspondences.Their performance depends on the quality of the estimated dense correspondence.
  • Phase-based methods encode motion in phase differences and manipulate phase information for video-processing applications including motion magnification and view expansion.A multi-scale pyramid strategy was used to accommodate large motion in phase-based interpolation.
  • Deep-learning approaches have addressed visual understanding, optical-flow estimation, style transfer, image enhancement, and view synthesis.
  • Existing deep view-synthesis algorithms render unseen views from input images but are not designed for video frame interpolation in general scenes.

3. Video Frame Interpolation

The method replaces flow-based motion estimation and pixel synthesis with a single local convolution process over two input frames. A spatially adaptive kernel captures both motion and resampling coefficients for each interpolated pixel.

  • Traditional interpolation estimates dense motion before synthesizing pixel colors, making results vulnerable to unreliable optical flow.Occlusion, blur, brightness changes, aliasing, and fixed-kernel resampling can produce artifacts.
  • The proposed method directly convolves co-centered patches from the two input frames to produce each target pixel.The kernel is estimated for the target pixel and applied to input patches P1(x, y) and P2(x, y).
  • The convolution kernel jointly represents local motion and pixel-synthesis resampling coefficients.This formulation avoids explicitly separating motion estimation from synthesis.
  • The convolution formulation provides flexibility for occlusion and can integrate edge-aware filtering for sharper interpolation.The paper describes direct kernel estimation as a data-driven alternative to heuristic flow handling in occluded regions.
  • A fully convolutional network estimates spatially varying kernels for individual output pixels.The network architecture uses convolutional layers, down-convolutions, Batch Normalization, and ReLU activations.

3.1. Convolution kernel estimation

The network estimates a kernel from large receptive-field patches, while training combines color and gradient losses. The gradient term addresses blur produced by color loss alone.

  • Convolution kernel estimation: The fully convolutional network takes receptive-field patches R1(x, y) and R2(x, y) and estimates the kernel used on smaller patches P1 and P2.All patches are centered at the corresponding output pixel location.
  • Convolution kernel estimation: The final convolutional layer uses spatial softmax so kernel coefficients are non-negative and sum to one.This enforces the stated constraint on each output convolution kernel.
  • Loss function: Color loss alone can produce blurry interpolation results, even with an L1 norm.The paper illustrates this effect in Figure 3 and motivates adding image-gradient supervision.
  • Loss function: The gradient loss is computed by convolving estimated kernels with gradients of the input patches, assuming kernels vary locally slowly.Eight finite-difference gradient versions are incorporated into the gradient loss.
  • Loss function: The final objective combines color and gradient losses as Ec + λ · Eg, with λ = 1 used in the experiments.The combined loss produces sharper interpolation results than color loss alone.

3.2. Training

The network is trained with standard optimization on video-derived samples, using only ground-truth frames rather than optical-flow annotations. The dataset emphasizes diverse motion.

  • Training uses Xavier initialization and AdaMax with β1 = 0.9, β2 = 0.999, learning rate 0.001, and mini-batches of 128 samples.
  • The loss requires only ground-truth video frames, allowing training from widely available videos without optical-flow ground truth.The authors use publicly available Flickr videos under a Creative Commons license.
  • The training collection contains 3,000 videos selected with keywords spanning activities such as driving, dancing, surfing, riding, and skiing.These categories provide a diverse selection of video content.
  • Training samples are consecutive triple-frame groups from which randomly centered triple-patch groups are extracted.The patches are 150×150 pixels and larger than the network receptive-field patches to support augmentation.
  • The authors sample 500,000 triple-patch groups with larger-flow patches more likely to be selected.This reduces the prevalence of samples with little or no motion and broadens the motion range.

3.3. Implementation details

The implementation accelerates dense kernel prediction with shift-and-stitch and uses zero-padding for boundaries. Kernel and receptive-field sizes trade motion coverage against computational complexity.

  • Shift-and-stitch implementation: Pixel-wise network application is redundant because neighboring output pixels reuse overlapping patch computations.The implementation addresses this redundancy with shift-and-stitch.
  • Shift-and-stitch implementation: The fully convolutional network processes larger inputs to produce sparsely distributed output pixels, which shifted inputs combine into a dense frame.
  • Shift-and-stitch implementation: For 1280 × 720 frames, shift-and-stitch reduces computation from 921,600 to 64 forward passes and runtime from 104 seconds to 9 seconds per frame.The comparison is reported for an Nvidia Titan X.
  • Implementation details: Zero-padding is used to synthesize boundary pixels, and experiments report that it usually avoids noticeable artifacts.
  • Hyper-parameters: Larger convolution kernels improve robustness to large motion but increase the number of values the network must estimate.The receptive field is selected separately to balance computation and motion sensitivity; 79×79 achieves the reported balance.

4. Experiments

Experiments show strong performance on real-world Middlebury scenes and visually appealing interpolation in challenging conditions, including blur and occlusion. The method also produces edge-aware kernels, scales to large images, and remains limited by kernel size and temporal flexibility.

  • Comparisons: On the Middlebury benchmark, the method ranks first on Evergreen and Basketball, second on Dumptruck, and third on Backyard among more than 100 methods.It performs less well on synthetic or laboratory scenes, partly because training uses real-world videos.
  • Comparisons: The method handles out-of-focus and motion-blurred videos better than flow-based methods, producing sharper results in regions with large motion.The phase-based method also handles blurry regions better, while the proposed method is sharper in the cited examples.
  • Comparisons: For occluded regions, learned convolution kernels produce visually appealing synthesis results instead of relying on unreliable optical flow and hole filling.In the examined leaf example, the kernel correctly obtains the occluded pixel’s color from Frame 1 only.
  • Edge-aware pixel interpolation: The estimated kernels are sparse and edge-aware, with anisotropic orientations aligned to image edges to support sharp interpolation.Flat-region kernels use two significant elements corresponding to relevant pixels in the input frames.
  • Discussion: Inference takes 2.8 seconds for 640 × 480, 9.1 seconds for 1280×720, and 21.6 seconds for 1920 × 1080 on a single Nvidia Titan X.The shift-and-stitch implementation supports parallel processing and reduces redundant kernel computation.
  • Discussion: Motion handling is limited by the current 41×82 convolution kernel: motion within 41 pixels works well, whereas larger motion can blur results.A KITTI stereo example with disparity over 41 pixels improves after downscaling the inputs by half.

5. Conclusion

The paper combines motion estimation and pixel synthesis into local convolution, using spatially adaptive kernels estimated by a deep fully convolutional network. Experiments report high-quality interpolation that handles occlusion, blur, and abrupt brightness changes well.

  • Conclusion: The method unifies motion estimation and pixel interpolation as a single local convolution over patches from two input frames.The convolution kernel captures both motion information and resampling coefficients.
  • Conclusion: A deep fully convolutional neural network estimates spatially adaptive, edge-aware kernels that produce sharp interpolated frames.The network can be trained directly from widely available video data.
  • Conclusion: Experiments show high-quality frame interpolation in challenging cases including occlusion, blur, and abrupt brightness change.
Loading 1703.07514v1…