Source-linked AI summary
Video Frame Interpolation via Adaptive Separable Convolution
Simon Niklaus, Long Mai, Feng Liu
TL;DR
Optical-flow interpolation is difficult in challenging regions, while large adaptive kernels create memory limits for high-resolution synthesis. The paper replaces 2D kernels with separable 1D-kernel pairs predicted densely by a fully convolutional network, enabling full-frame interpolation and perceptual-loss training. Experiments report favorable qualitative and quantitative comparisons with representative state-of-the-art methods.
Problem
Optical-flow-based interpolation is difficult in occluded, blurred, or abruptly changing regions, while large adaptive kernels demand too much memory for simultaneous high-resolution processing.
Method
A fully convolutional network estimates pixel-dependent pairs of vertical and horizontal 1D kernels, applies them through local convolution, and supports perceptual-loss training.
Results
The method compares favorably with representative state-of-the-art interpolation methods in both qualitative and quantitative experiments.
Takeaways & Limitations
Separable kernels reduce memory requirements and enable full-frame synthesis, allowing perceptual loss to improve visual quality.
Takeaways & Limitations
Motion handling remains limited by the system's 51-pixel kernel size, and the method currently interpolates only the midpoint t = 0.5.
Abstract
from arXiv · showhide
Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.
1. Introduction
Traditional interpolation depends on optical flow that is difficult to estimate accurately in challenging regions, while convolution-based methods face severe memory demands from large kernels. This paper addresses that limitation with separable convolution using 1D kernels, enabling full-frame synthesis and perceptual-loss training.
- Optical-flow-based interpolation struggles where occlusion, blur, or abrupt brightness changes make accurate flow estimation difficult.
- Separable convolution approximates each 2D kernel with horizontal and vertical 1D kernels, reducing an n × n kernel from n^2 to 2n variables.
- For 1080p frames, separable 41 × 41 kernels require 1.27 GB instead of 26 GB, enabling full-frame kernel generation and perceptual-loss training.
- The fully convolutional network is trained end-to-end on widely available video data and compares favorably with representative state-of-the-art methods qualitatively and quantitatively.
2. Related Work
Prior interpolation research includes optical-flow, phase-based, view-synthesis, and convolutional approaches. The paper is most closely related to adaptive-kernel methods that merge motion estimation and synthesis but remain constrained by memory for large kernels.
- Optical-flow methods interpolate frames using estimated dense motion, while phase-based methods modify per-pixel phase to generate intermediate frames.The phase-based approach can struggle to preserve high-frequency details under large temporal changes.
- Deep view-synthesis methods render unseen views using depth planes, disparity and color estimation, monocular-video synthesis, or appearance flow.
- A concurrent deep voxel flow method predicts dense voxel flows for frame interpolation, while other convolutional methods also synthesize frames between two inputs.
- Adaptive convolution methods estimate spatially varying kernels and convolve them with input frames, but large kernels prevent simultaneous synthesis of all high-resolution pixels under limited memory.
3. Video Frame Interpolation
The method estimates pixel-dependent separable kernels with a fully convolutional network and applies them through local convolution to synthesize the intermediate frame. Its design combines dense prediction, artifact-aware upsampling, perceptual loss, and video-based training.
- Video Frame Interpolation: The interpolation target is the frame temporally midway between two input frames, with convolution-based synthesis using pixel-dependent kernels and local patches.
- Separable Kernel Estimation: Each 2D kernel is approximated by vertical and horizontal 1D kernels, reducing parameters from n^2 to 2n and enabling one-pass high-resolution synthesis.
- Separable Kernel Estimation: A fully convolutional network estimates two pairs of 1D kernels for every output pixel and applies them equally across RGB channels through a local-convolution layer.
- Separable Kernel Estimation: The encoder-decoder uses contracting and expanding components with skip connections, while four sub-networks estimate the four 1D kernels.
- Separable Kernel Estimation: Bilinear interpolation is used in the decoder to address checkerboard artifacts associated with unsuitable upsampling layers.
- Loss Function: The model explores pixel-wise and perceptual losses; feature reconstruction using VGG-19 relu4_4 was empirically found effective for frame interpolation.
- Training and Implementation: Training uses consecutive video frames and randomly cropped, motion-selected patches with augmentation, while a custom CUDA layer renders interpolated frames efficiently.
4. Experiments
Experiments show that separable convolution performs strongly across quantitative and visual evaluations, while loss choice trades numerical accuracy against perceptual quality. The method also compares favorably with AdaConv and remains limited by kernel size and fixed temporal position.
- Experimental setup: The evaluation compares the proposed method with representative state-of-the-art interpolation methods using qualitative and quantitative experiments.The experiments include optical-flow-based, phase-based, and convolution-based methods.
- Visual comparison: Both proposed variants produce sharp, ghosting-free results on a difficult butterfly example, while AdaConv appears blurry.The example involves large motion and a delicate leg that challenges optical-flow estimation and phase-based interpolation.
- Visual comparison: Both methods handle opposing motion with significant occlusion better than the other evaluated methods.The authors attribute this result to the convolution approach and the use of fewer-parameter 1D kernels.
- Kernel analysis: Estimated kernels encode motion through offsets of non-zero values, average colors when brightness changes without motion, and select the visible frame in occluded regions.These behaviors are illustrated for butterfly, spotlight, and leaf-occlusion examples.
- Quantitative evaluation: On Middlebury, the L1 model performs particularly well in discontinuous-motion regions and achieves state-of-the-art overall average results without fine-tuning.The LF model is inferior quantitatively because its loss function optimizes perceptual quality.
- Comparison with AdaConv: Compared with AdaConv, the method is over 20 times faster at 1080p, achieves numerically better results, and often produces more appealing images.Its separable formulation also permits larger kernels, helping address larger motion, although the system remains limited to a 51-pixel kernel.
- Loss functions: L1 training gives better numerical results, whereas LF training produces higher visual quality.LF combines L1 and feature reconstruction loss; the study reports sharper images with more high-frequency details after incorporating LF loss.
- Limitations: The current system interpolates only the midpoint t = 0.5 and cannot produce a frame at an arbitrary time between the inputs.The authors propose adding temporal offset as a control variable in future work.
5. Conclusion
The paper presents a practical high-quality video frame interpolation method that combines motion estimation and synthesis through spatially adaptive separable kernels. One-dimensional kernels reduce parameters, enable full-frame synthesis, and support perceptual loss for improved visual quality.
- The method estimates spatially adaptive separable kernels for each output pixel and convolves them with input frames to render intermediate frames.
- Using 1D kernels to approximate full 2D kernels significantly reduces kernel parameters and enables synthesis of an entire high-resolution frame in one pass.
- Full-frame synthesis supports perceptual loss, which further improves the visual quality of interpolation results.
- Experiments show favorable quantitative and qualitative comparisons with state-of-the-art interpolation methods.