PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution

Xianhang Cheng, Zhenzhong Chen

arXiv:2006.08070v2cs.CVeess.IV

TL;DR

Kernel-based interpolation struggles with large motion and arbitrary temporal positions because it uses local kernels tied to the midpoint. EDSC learns deformable sampling components and temporal control to generate multiple intermediate frames. The method performs favorably against state-of-the-art approaches, while its unsupervised offsets remain weaker than those from pretrained flow estimators.

  • Problem

    Existing kernel-based methods are limited by local predefined kernels for large motion and cannot directly generate frames at arbitrary temporal positions.

  • Method

    EDSC jointly learns adaptive kernels, offsets, masks, and biases, while using intermediate time as a control variable for frame synthesis.

  • Results

    EDSC performs favorably against state-of-the-art methods and is the first kernel-based method reported to generate as many intermediate frames as needed between consecutive frames.

  • Takeaways & Limitations

    The approach extends kernel-based interpolation to non-local sampling and arbitrary-time multi-frame generation without relying on additional pretrained components.

  • Takeaways & Limitations

    Its unsupervised offsets and bidirectional flows do not reach the quality of methods initialized with off-the-shelf flow-estimation networks.

Abstract

from arXiv · show

Generating non-existing frames from a consecutive video sequence has been an interesting and challenging problem in the video processing field. Typical kernel-based interpolation methods predict pixels with a single convolution process that convolves source frames with spatially adaptive local kernels, which circumvents the time-consuming, explicit motion estimation in the form of optical flow. However, when scene motion is larger than the pre-defined kernel size, these methods are prone to yield less plausible results. In addition, they cannot directly generate a frame at an arbitrary temporal position because the learned kernels are tied to the midpoint in time between the input frames. In this paper, we try to solve these problems and propose a novel non-flow kernel-based approach that we refer to as enhanced deformable separable convolution (EDSC) to estimate not only adaptive kernels, but also offsets, masks and biases to make the network obtain information from non-local neighborhood. During the learning process, different intermediate time step can be involved as a control variable by means of an extension of coord-conv trick, allowing the estimated components to vary with different input temporal information. This makes our method capable to produce multiple in-between frames. Furthermore, we investigate the relationships between our method and other typical kernel- and flow-based methods. Experimental results show that our method performs favorably against the state-of-the-art methods across a broad range of datasets. Code will be publicly available on URL: \url{https://github.com/Xianhang/EDSC-pytorch}.

1 Introduction

The paper presents EDSC to overcome local-kernel limits in large-motion interpolation and to generate frames at arbitrary times. It combines deformable sampling with temporal control and relates kernel- and flow-based methods within one formulation.

  • 1 Introduction: EDSC provides a non-flow alternative that avoids explicit optical-flow estimation while addressing the local-neighborhood limitation of earlier kernel methods.Earlier kernel methods require larger predefined kernels for large motion, whereas EDSC learns offsets to reference more effective pixels.
  • 1 Introduction: EDSC learns adaptive kernels, offsets, masks, and biases to access non-local pixels and handle motion beyond a predefined kernel size.This extends separable convolution with deformable sampling and residual value adjustment.
  • 1 Introduction: Temporal information controls the estimated components, enabling direct generation of frames at arbitrary intermediate times without recursion.The controlled components include kernels, offsets, masks, and biases.
  • 1 Introduction: The paper theoretically shows that representative flow-based and kernel-based interpolation methods are special cases of EDSC.The relationship is established from the perspective of convolution and pixel reference.
  • 1 Introduction: The model performs favorably against state-of-the-art methods without using pre-calculated context, depth, flow, or edge information.Compared with the earlier model, HetConv reduces computation by 79.6% FLOPs and parameters by 59.6% without accuracy loss.

2 Related Work

Related work spans flow-based, adaptive-convolution, and alternative deep-learning approaches to interpolation. The paper emphasizes that existing non-flow kernel methods remain limited in arbitrary-time generation, motivating EDSC.

  • Flow-based interpolation: Flow-based methods estimate optical flow, warp and blend input frames, and often learn visibility masks for intermediate synthesis.Their flow estimation can be sophisticated and time-consuming, and results depend heavily on bidirectional-flow quality.
  • Other approaches: Other approaches generate intermediate frames using operations such as channel attention, PixelShuffle, deblurring, or space-time video processing without optical flow.The supplied related-work passage identifies these as alternative deep-learning directions.
  • Kernel-based interpolation: Kernel-based methods learn spatially adaptive kernels, with SepConv factorizing 2D kernels into two one-dimensional kernels to reduce memory demand.These methods avoid an explicit flow-estimation component but are constrained by predefined local sampling neighborhoods.
  • Multiple-frame interpolation: Existing multi-frame methods include recursive single-frame interpolation, flow-based schemes, phase-based methods, and video-frame-inpainting formulations.Recursive generation is described as inflexible because interpolation errors accumulate.
  • Arbitrary-time interpolation: Before EDSC, no non-flow kernel-based method could directly generate frames at arbitrary temporal positions because learned pixels were tied to a specific time step.This limitation is explicitly identified as an under-explored problem.

3 Proposed Method

EDSC extends kernel-based interpolation with deformable sampling and time-conditioned components, while relating kernel-, flow-, and adaptive-warping methods through shared pixel-reference conditions.

  • 3.1.1 Single Frame Interpolation: Standard kernel interpolation combines two learned kernels with local patches, but its motion coverage is limited by kernel size.Separable kernels reduce memory from O(n^2) to O(2n), while remaining constrained to motions up to n pixels.
  • 3.1.1 Single Frame Interpolation: EDSC learns adaptive separable kernels, offsets, masks, and biases to sample informative pixels beyond a fixed local neighborhood.Deformable sampling targets fewer, more relevant pixels instead of requiring a large regular kernel.
  • 3.1.1 Single Frame Interpolation: The deformable operation resamples patch locations using learned offsets and modulation masks, then adds a learned residual bias to the convolution result.Fractional offset locations are bilinearly sampled, and the bias addresses residual synthesis errors.
  • 3.1.2 Relationships with Kernel and Flow Based Methods: Earlier kernel-based interpolation is recovered when offsets are zero, masks equal one, and bias is zero; flow-based interpolation is recovered with a one-pixel kernel under corresponding conditions.These equivalences frame prior methods as special cases in terms of pixel reference.
  • 3.1.3 Multiple Frame Interpolation: For arbitrary-time interpolation, the intermediate time t controls kernels, masks, offsets, and biases rather than only rescaling offsets.This avoids the occlusion error that occurs when a midpoint-trained model reuses fixed masks and kernels.

3.2 Network Architecture

The EDSC network uses an encoder-decoder backbone with separate estimators for kernels, offsets, masks, and bias, and injects time information for multiple-frame generation.

  • Architecture: Given two input frames, the encoder-decoder extracts features used to estimate kernels, masks, offsets, and a bias value for each output pixel.The backbone is organized into an encoder-decoder plus four estimator types.
  • Kernel Estimator: The kernel estimator predicts adaptive vertical and horizontal 1D kernels for each input frame and pixel.It uses four parallel sub-networks whose output depth equals the kernel size n, set to 5 in this implementation.
  • Kernel Estimator: For multiple-frame interpolation, the intermediate time t is appended as an extra constant channel to condition the estimated kernels on the desired time step.This coord-conv extension ties learned kernel parameters to different temporal positions.
  • Other Estimators: The offset and mask estimators modify sampling locations and modulation, while the bias estimator predicts residual values for difficult occlusion regions.Masks use sigmoid outputs, and the bias estimator produces a three-channel tensor from encoder-decoder features.
  • Deformable Convolution: The deformable convolution combines estimated kernels, offsets, and masks to use information outside the local neighborhood.This operation is the network’s pixel-sampling mechanism for interpolation.

3.3 Training

Training uses color and perceptual losses, with separate single- and multiple-frame model variants trained on Vimeo90K datasets suited to their temporal outputs.

  • Loss Functions: The training objective combines a color reconstruction loss with a perceptual loss based on high-level VGG-19 features.The perceptual feature extractor is the relu4_4 layer of a pretrained VGG-19 network.
  • Model Variants: EDSC_s generates only the midpoint, whereas EDSC_m generates multiple arbitrary in-between frames using temporal information in the estimators.Both variants use the two stated loss-function types.
  • Datasets: EDSC_s is trained on Vimeo90K-Interp, while EDSC_m uses Vimeo90K-Septuplet because multiple time steps require more consecutive frames.The datasets contain 51,312 triplets and 91,701 seven-frame sequences, respectively.
  • Optimization: Optimization uses Adam with scheduled learning-rate reduction, followed by full-frame fine-tuning for ten additional epochs.Initial training uses 120 epochs, and the fine-tuning learning rate is 1.25e-5.

4 Experiments

The experiments evaluate datasets and metrics, compare EDSC with state-of-the-art algorithms, and use ablations to examine important components.

  • 4 Experiments: The evaluation section introduces datasets and metrics before comparing the proposed method with state-of-the-art algorithms.It also includes comprehensive ablation studies of important components.

4.1 Experimental Setup

The evaluation compares EDSC with recent interpolation methods across several datasets using standard image-quality metrics and Middlebury interpolation error. Baselines are grouped by their use of kernels, adaptive warping, optical flow, or neither.

  • Datasets: The evaluation covers UCF101, Vimeo90K, Middlebury, and SNU-FILM, spanning different resolutions and motion subsets.SNU-FILM includes Easy, Medium, Hard, and Extreme subsets, each with 310 triplets.
  • Metrics: Performance is measured with PSNR, SSIM, LPIPS, and Middlebury interpolation error, where PSNR and SSIM are higher-is-better while LPIPS and IE are lower-is-better.The study reports average Interpolation Error on Middlebury.
  • Baselines: Compared methods are divided into kernel-based, adaptive-warping, flow-based, and methods without optical flow or adaptive kernels.Kernel-based baselines include AdaConv, SepConv, IM-Net, DSepConv, and AdaCoF; adaptive-warping baselines include MEMC-Net* and DAIN.
  • Baselines: The comparison reports separate model versions when methods provide multiple configurations or training objectives.Examples include two loss functions for CtxSyn, SepConv, and SoftSplat, and two models for CyclicGen and AdaCoF.

4.2 Comparisons with state-of-the-arts

EDSC is evaluated as both a single-frame and arbitrary-position interpolator, using learned kernels together with offsets, masks, and biases. It achieves strong benchmark and qualitative results while requiring little additional cost for multi-frame interpolation.

  • Evaluation settings: EDSC models are evaluated separately for single-frame and arbitrary-position interpolation because most baselines target single-frame interpolation.The EDSC_s model is discussed first, followed by EDSC_m for multiple intermediate frames.
  • Network design: EDSC combines adaptive kernels with offsets, masks, and biases to sample non-local information and handle motion beyond a fixed kernel size.Its bias estimator provides residual values for pixel synthesis and is designed for a non-flow kernel-based method.
  • Quantitative comparisons: 0.12 dB and 0.13 dB PSNR gains over DAIN are achieved on UCF101 and Vimeo90K, respectively, without pre-trained PWC-Net or MegaDepth sub-models.The LC-trained model also exceeds AdaCoF+ by 0.23 dB on UCF101 and 0.37 dB on Vimeo90K while using 61% fewer parameters.
  • Quantitative comparisons: The LC-trained model ranks 3rd in Middlebury IE and 2nd in NIE, performs best on 5 of 8 sequences among kernel-based methods, and has the best average result.The comparison concerns kernel-based methods that do not rely on additional information.
  • Quantitative comparisons: EDSC is inferior to STAR-THR and SoftSplat, which use additional displacement, pretrained, or optical-flow information, but outperforms most other methods on reported metrics.The LC-trained model leads on most PSNR, SSIM, and IE comparisons, while the LF-trained model leads on LPIPS.
  • Qualitative comparisons: Qualitative examples show EDSC producing sharper and more complete results than several flow-based and kernel-based methods in scenes with occlusion, discontinuity, or complex motion.The reported examples include a man’s skin, a bottle, a sign, and a skateboarder.
  • Arbitrary-position interpolation: EDSC outperforms DAIN at every evaluated intermediate time step when generating frames 2 through 6 from frames 1 and 7 across 7,824 Vimeo90K-Septuplet sequences.The evaluation generates ×6 slow-motion frames.
  • Arbitrary-position interpolation: Adding temporal channels for multi-frame interpolation costs 0.072G FLOPs, 0.006M parameters, and 0.001 seconds per 1280×720 frame.These increases correspond to 0.52% FLOPs, 0.07% parameters, and execution on an Nvidia Titan X GPU.

4.3 Model Analysis

The analysis evaluates EDSC across motion, occlusion, perceptual control, and runtime, showing strong handling of difficult regions and flexible quality trade-offs. It also examines how learned sampling differs from local kernels and flow-guided warping.

  • Motion degrees: EDSC achieves the best PSNR and SSIM on SNU-FILM’s Easy, Medium, and Hard sets, except for a marginal disadvantage to AdaCoF+ on the comparison.The comparison concerns kernel-based methods evaluated across different motion degrees.
  • Occlusion handling: Learned offsets sample beyond the regular kernel neighborhood, while occluded pixels are selected from both patches or mainly from the visible patch under large motion.Figures 12 and 13 illustrate these two occlusion-handling patterns.
  • Occlusion handling: All three optical-flow-guided methods perform worse than EDSC on occluded and boundary interpolation error, with especially large gains in boundary error.The paper attributes this to inaccurate warped frames in occluded regions; Figure 14 shows less blur near heavily occluded boundaries.
  • Loss interpolation: Varying the interpolation coefficient smoothly balances perceptual quality against distortion: larger α improves LPIPS but worsens PSNR.The coefficient is adjusted through interpolated models trained with color and perceptual losses.
  • Runtime: The model runs faster than most existing methods in the reported runtime comparison.Runtime comparisons use the Urban sequence at 640 × 480, while component timings are reported on a Titan X GPU.

4.4 Ablation study

The ablations show that EDSC’s efficiency and quality depend on architecture, kernel size, offsets, masks, and biases. Larger kernels improve performance at greater computational cost, while the learned components improve sampling and synthesis.

  • Encoder-decoder architecture: HetConv reduces encoder-decoder computation and parameters as the 3 × 3 filter rate decreases.The encoder-decoder accounts for most parameters in typical kernel-based interpolation models.
  • Kernel size: Larger kernel sizes improve performance but increase FLOPs and runtime; n = 5 correctly samples complementary pixels and produces the best illustrated result.The authors do not recommend kernel sizes larger than 5.
  • Offsets: With n = 1, EDSC learns offsets equivalent to optical flow without explicit optical-flow training.This establishes the flow-based formulation as a special case of the proposed sampling framework.
  • Mask estimator: Mask estimation significantly improves performance, especially PSNR and interpolation error, by modulating offset-guided samples and their relative influence.Masks also reduce the burden on the estimated kernels.
  • Bias estimator: The bias estimator improves PSNR by 0.1 dB on UCF101 and 0.04 dB on Vimeo90K, while SSIM saturates.The learned biases model the linear relationship between sampled pixels and their kernels.

5 Discussions and limitations

The discussion identifies performance and flexibility limitations despite EDSC’s favorable results. It also notes that auxiliary information and pretrained sub-networks remain unexplored opportunities for improving quality.

  • Limitations: EDSC’s unsupervised offsets and simple network do not match the optical-flow quality of methods using pretrained flow estimators with good initialization.Although flow-based methods are theoretically represented when n = 1, their estimated flows and generated frames remain better in this setting.
  • Limitations: For multiple-frame interpolation, EDSC_m is less flexible than methods that explicitly warp pixels and features because it requires training from scratch with supervision at different time steps.The comparison concerns methods that generate outputs after explicit warping operations.
  • Future directions: The method does not use auxiliary reference frames, high-frame-rate inputs, or pretrained sub-networks, leaving these as possible routes to higher-quality interpolation.The authors identify extending the approach to broader video-processing tasks as future work.

6 Conclusion

The paper presents EDSC as a non-local kernel-based interpolator that learns adaptive sampling components and supports arbitrary intermediate times. Experiments report favorable performance against state-of-the-art methods, while the framework relates kernel- and flow-based interpolation as special cases.

  • Conclusion: EDSC improves kernel-based interpolation by learning offsets, kernels, masks, and biases over a non-local neighborhood with fewer parameters.The learned components address information outside the regular local neighborhood.
  • Conclusion: EDSC provides a kernel-based way to generate as many intermediate frames as needed between two consecutive frames.The method varies its learned components with intermediate temporal information.
  • Conclusion: Both kernel-based and flow-based interpolation methods can be regarded as special cases of the proposed framework.The relationship is demonstrated theoretically in the paper.
  • Conclusion: Comprehensive experiments show that EDSC performs favorably against state-of-the-art methods.The conclusion summarizes results across the paper’s evaluations.
Loading 2006.08070v2…