Source-linked AI summary
AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation
Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, Sangyoun Lee
TL;DR
Video frame interpolation needs warping operations that can handle complex motions despite existing Degrees of Freedom limitations. AdaCoF estimates adaptive weights and offsets to synthesize intermediate frames, and experiments report that it outperforms most competing algorithms, including challenging large-motion and occlusion cases.
Problem
Existing frame interpolation methods use warping operations with Degrees of Freedom limitations, making complex real-world motions difficult to address.
Method
AdaCoF estimates spatially adaptive kernel weights and multiple offset vectors for each target pixel, then linearly combines sampled values from the input frames.
Results
AdaCoF outperforms most competing algorithms, including in challenging cases involving large motion and occlusion.
Takeaways & Limitations
The learned warping operation provides a generalized frame-interpolation module whose parameters are produced by an end-to-end fully convolutional network.
Abstract
from arXiv · showhide
Video frame interpolation is one of the most challenging tasks in video processing research. Recently, many studies based on deep learning have been suggested. Most of these methods focus on finding locations with useful information to estimate each output pixel using their own frame warping operations. However, many of them have Degrees of Freedom (DoF) limitations and fail to deal with the complex motions found in real world videos. To solve this problem, we propose a new warping module named Adaptive Collaboration of Flows (AdaCoF). Our method estimates both kernel weights and offset vectors for each target pixel to synthesize the output frame. AdaCoF is one of the most generalized warping modules compared to other approaches, and covers most of them as special cases of it. Therefore, it can deal with a significantly wide domain of complex motions. To further improve our framework and synthesize more realistic outputs, we introduce dual-frame adversarial loss which is applicable only to video frame interpolation tasks. The experimental results show that our method outperforms the state-of-the-art methods for both fixed training set environments and the Middlebury benchmark.
1. Introduction
Video frame interpolation must handle large and complex real-world motions while selecting informative reference pixels for each output pixel. Existing kernel-, flow-, and hybrid-based warping methods impose Degrees of Freedom limitations, motivating AdaCoF.
- Video frame interpolation requires motion understanding because real-world videos contain large and complex motions.
- Existing approaches estimate each output pixel by finding informative reference locations in the input frames.
- Kernel-based methods cannot reach motions beyond their kernel size and inefficiently retain large kernels when motion is small.
- Flow-based methods reference only one location per input frame, limiting their suitability for complex motions and low-quality inputs.
- Hybrid kernel-flow methods add neighboring samples but retain relatively few reference points and fixed square-shaped kernels, limiting their Degrees of Freedom.
- AdaCoF estimates multiple offset vectors and spatially adaptive kernel weights, allowing target pixels to combine samples from arbitrary locations and numbers of pixels.
2. Related Work
Related work includes direct frame synthesis, optical-flow-based warping, and methods that augment interpolation with contextual, depth, consistency, or edge information. These approaches face challenges including blur, optical-flow dependence, occlusion, large motion, brightness changes, and limited reference areas.
- Classic methods estimate dense optical flow and warp input frames, making performance dependent on optical-flow algorithms.
- Optical-flow-based approaches have limitations with occlusions, large motion, and brightness changes, including methods without external flow modules.
- Direct CNN approaches using two input frames often produce blurry intermediate frames.
- Flow-guided small-kernel methods combine local kernels with flow locations, but square kernels restrict reference points to a small area and yield low Degrees of Freedom.
- Other methods use ResNet-18 context, estimated depth maps, cycle consistency loss, or additional edge maps to improve interpolation or address occlusions.
3. Proposed Approach
AdaCoF formulates interpolation as adaptive warping, estimating per-pixel weights, offsets, and occlusion to synthesize intermediate frames. Its architecture combines feature extraction, parameter prediction, differentiable sampling, and distortion- or perception-oriented objectives.
- Warping formulation: AdaCoF represents interpolation as forward and backward warping from consecutive input frames to an intermediate output.
- Adaptive Collaboration of Flows: For each target pixel, AdaCoF estimates adaptive kernel weights and multiple offset vectors to sample and combine values from arbitrary input locations.Offsets may point to non-grid locations, whose values are obtained by differentiable bilinear interpolation.
- Occlusion reasoning: An occlusion map V weights the contributions of the two input frames when the target pixel is visible in only one frame.V(i, j)=1 indicates visibility only in In, while V(i, j)=0 indicates visibility only in In+1.
- Adaptive Collaboration of Flows: Unlike classic and deformable convolution, AdaCoF uses pixel-dependent kernel weights and freely positioned offsets, while dilation expands the explored region.The network predicts weights, offsets, and the occlusion map from U-Net features through sub-networks.
- Network architecture: The fully convolutional architecture extracts features with a U-Net, predicts AdaCoF parameters, and synthesizes the intermediate frame from the input frames.Each sub-network preserves the input height and width, and the modules are differentiable for end-to-end training.
- Objective functions: Training combines an ℓ1-based distortion objective with optional perceptual and dual-frame adversarial losses for sharper, temporally compatible outputs.The adversarial loss conditions discrimination on the generated frame concatenated with an adjacent input frame; perception-oriented training fine-tunes after distortion training.
4. Experiments
Experiments evaluate AdaCoF through ablations, benchmark comparisons, and visual analyses. The results show benefits from flexible warping, dilation, adversarial fine-tuning, and handling large motion and occlusion.
- Ablation Study: AdaCoF’s warping operation outperforms alternative operations with lower Degrees of Freedom, while breaking square-shaped kernels is especially important.The PSNR gap between Ours-sdc and Ours exceeds the gap between Ours-kb and Ours.
- Ablation Study: Larger kernel sizes generally improve performance until PSNR saturation, with earlier saturation on UCF101’s smaller-motion, lower-resolution sequences.The evaluated kernel sizes are F ∈ {1, 3, 5, 7, 9, 11}, corresponding to F^2 offset vectors.
- Ablation Study: Larger dilation generally improves results, while offset vectors spread farther for large motion, providing better initialization.The dilation values evaluated are d ∈ {0, 1, 2}.
- Ablation Study: Adversarial fine-tuning increases sharpness; unlike WGAN-GP and TGAN, the proposed dual-frame loss preserves frame structures without introducing artifacts.The network is trained with Ld for 50 epochs and fine-tuned with Lp for 10 epochs.
- Quantitative Evaluation: AdaCoF outperforms competing algorithms on all fixed-training test datasets and ranks second in both IE and NIE on Middlebury.The comparison includes Phase Based, MIND, SepConv, DVF, and SuperSlomo.
- Visual Comparison: Visual comparisons show that AdaCoF better synthesizes fast-moving objects and handles occluded areas than competing approaches.The perception-oriented AdaCoF version mitigates motion blur, while other methods produce blur or artifacts in large-motion examples.
- Offset Visualization: Variance Flow maps assign larger values to challenging large-motion and occluded regions, supporting their use as uncertainty maps for motion estimation.The map reflects how widely offset vectors spread and is obtainable specifically through AdaCoF.
5. Conclusion
The paper identifies warping-operation flexibility as critical for complex video motions and proposes AdaCoF as a generalized, end-to-end trainable solution.
- AdaCoF addresses complex motions by treating warping-operation Degrees of Freedom as a critical factor in video frame interpolation.
- AdaCoF generalizes previous approaches, which are described as special versions of the proposed operation.
- A fully convolutional network obtains AdaCoF’s parameters through end-to-end training.
- Experiments report that AdaCoF outperforms most competing algorithms in challenging cases including large motion and occlusion.
- Visualized network outputs behave as intended and produce meaningful maps that can be used for other motion estimation tasks.