Source-linked AI summary
Video Frame Interpolation Transformer
Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, Ming-Hsuan Yang
TL;DR
Video interpolation methods based on CNNs are limited by content-agnostic kernels and inadequate long-range modeling. VFIT uses local spatial-temporal Transformer attention, space-time separation, and multi-scale synthesis, and reports favorable benchmark performance with efficient models.
Problem
CNN-based video interpolation is limited by content-agnostic kernels and restricted effective receptive fields for adaptive motion modeling and long-range dependencies.
Method
VFIT combines local spatial-temporal self-attention, space-time separation, and multi-scale kernel-prediction synthesis in a Transformer-based interpolation framework.
Results
VFIT-S improves over FLAVR by 0.18 dB using 17.7% of its parameters, while VFIT-B improves by 0.66 dB using 68.4%.
Takeaways & Limitations
Sep-STS reduces memory usage while improving interpolation performance, and VFIT is reported as parameter-, memory-, and run-time efficient.
Takeaways & Limitations
VFIT performs only 2× interpolation, while multi-frame or arbitrary-time interpolation is left for future work.
Abstract
from arXiv · showhide
Existing methods for video interpolation heavily rely on deep convolution neural networks, and thus suffer from their intrinsic limitations, such as content-agnostic kernel weights and restricted receptive field. To address these issues, we propose a Transformer-based video interpolation framework that allows content-aware aggregation weights and considers long-range dependencies with the self-attention operations. To avoid the high computational cost of global self-attention, we introduce the concept of local attention into video interpolation and extend it to the spatial-temporal domain. Furthermore, we propose a space-time separation strategy to save memory usage, which also improves performance. In addition, we develop a multi-scale frame synthesis scheme to fully realize the potential of Transformers. Extensive experiments demonstrate the proposed model performs favorably against the state-of-the-art methods both quantitatively and qualitatively on a variety of benchmark datasets.
1. Introduction
VFIT addresses limitations of CNN-based video interpolation by combining content-adaptive Transformer attention with spatial-temporal modeling. Its separable attention and multi-scale synthesis improve efficiency and reported performance.
- CNN-based interpolation uses content-agnostic kernels, limiting adaptive motion modeling for spatially variant, content-dependent motion compensation.
- Small convolution kernels inefficiently capture long-range dependencies, while larger kernels or stacked layers increase cost or fail to learn distant dependencies effectively.
- VFIT introduces local attention to avoid global self-attention’s quadratic cost while retaining fine-grained pixel interactions important for realistic details.
- STS extends local attention into the spatial-temporal domain, making the mechanism compatible with video interpolation.
- Sep-STS factorizes spatial-temporal self-attention, reducing memory usage and considerably improving video interpolation performance.
- VFIT’s multi-scale kernel-prediction framework handles multi-scale motion and structures through coarse-to-fine synthesis.
2. Related Work
Prior interpolation methods use flow-based, kernel-based, or direct-regression approaches, while VFIT introduces Transformer-based video interpolation with local spatial-temporal attention. The model is reported to outperform a CNN baseline with fewer parameters.
- Flow-based methods warp source pixels using predicted optical flow but commonly rely on linear or quadratic motion assumptions.
- Kernel-based methods learn adaptive or deformable spatially variant kernels without prescribed motion assumptions, but commonly predict kernels at one scale.
- Direct-regression CNN methods achieve state-of-the-art results, yet VFIT outperforms the cited FLAVR method with fewer parameters.
- The proposed architecture combines a Transformer encoder-decoder with multi-scale frame synthesis using three SynBlocks.
- VFIT applies local attention to retain long-range dependency modeling while avoiding global self-attention complexity for video inputs.
3. Proposed Method
VFIT combines video-compatible local self-attention with space-time separation for efficient deep feature learning, then synthesizes frames using multi-scale kernel prediction. Its Sep-STS design reduces memory usage, while hierarchical features support multi-scale motion and structure modeling.
- Architecture: VFIT processes video features with a Transformer encoder-decoder containing Sep-STS blocks and lightweight 3D convolutional resizing layers.The encoder downsamples through four stages, while the decoder upsamples with three 3D deconvolution layers.
- Spatial-temporal local attention: STS extends local self-attention to spatial-temporal cubes, using regular and shifted cube partitions to connect neighboring regions.Each cube has shape T × M × M, and multi-head self-attention is applied within each cube.
- Separation of space and time: Sep-STS factorizes attention into separate spatial-window and temporal-vector operations, combining spatial and cross-frame dependency modeling.Spatial attention uses regular and shifted 2D windows, while temporal attention operates inside vectors of length T; both operations are used together.
- Separation of space and time: Sep-STS reduces memory complexity from O((TMM) · THW) for STS to O((T + MM) · THW).The factorization reduces training GPU memory by 26.2% relative to the STS baseline.
- Efficiency: The model’s space-time separation improves efficiency without requiring a separate runtime benefit in the reported naive implementation.Although memory usage decreases, the PyTorch implementation has runtime similar to STS; customized CUDA kernels may improve efficiency.
- Frame synthesis: VFIT predicts spatially variant kernels from hierarchical features and fuses three scale-specific SynBlock predictions to synthesize the intermediate frame.The multi-scale synthesis proceeds from coarse to fine, with bilinear upsampling used when merging scales.
4. Experiments
Experiments evaluate VFIT across benchmark datasets, baselines, qualitative comparisons, runtime, and ablations. Results show strong quantitative and visual performance, efficient deployment characteristics, and benefits from Sep-STS and multi-scale design choices.
- Evaluation against the State of the Arts: VFIT-S outperforms FLAVR on all evaluated datasets with only 7.5M parameters, while VFIT-B improves over FLAVR by 0.66 dB on Vimeo-90K and 0.65 dB on DAVIS.The reported gains come solely from architecture design without external information such as pretrained optical-flow or depth models.
- Evaluation against the State of the Arts: VFIT produces clearer structures, fewer distortions, and overlapped predictions closer to ground truth than baseline interpolation methods.The overlap visualization treats clearer images as more accurate predictions and uses alignment with ground truth as the comparison criterion.
- Evaluation against the State of the Arts: VFIT runtime is on par with the best-performing CNN-based algorithms, supporting deployment in vision applications.Runtime is measured in seconds per frame on an Intel Core i7-8700K CPU and NVIDIA GTX 2080 Ti GPU using Vimeo-90K averages.
- Ablation Study: Sep-STS improves over convolutional and inseparable STS variants, despite being designed to reduce memory usage.VFIT-CNN uses more than twice the parameters of VFIT-STS while achieving similar results, whereas VFIT-B with Sep-STS performs better than VFIT-STS.
- Ablation Study: VFIT-B gains 0.43 dB on fast motion, 0.16 dB on medium motion, and 0.10 dB on slow motion over VFIT-CNN.The largest reported improvement occurs under fast motion, highlighting Sep-STS performance in challenging large-motion scenarios.
- Ablation Study: Transformer-based resizing variants perform slightly worse than VFIT-B and have degraded runtime, identifying current resizing designs as a limitation.The authors attribute this result to the lower effectiveness of current Transformer-based resizing operations for complex motion modeling.
5. Conclusion
VFIT is presented as an efficient, compact Transformer framework for video frame interpolation, with designs aimed at spatial-temporal attention and memory efficiency. The paper also identifies 2× interpolation as its current scope while noting possible extensions.
- VFIT provides a parameter-, memory-, and run-time-efficient framework for video frame interpolation with state-of-the-art performance.The architecture is described as simple and compact, with potential application to downstream vision tasks.
- Its spatial-temporal attention module can be integrated into other video processing tasks.
- A space-time separation scheme improves memory efficiency and supports the need for structured regularization in video Transformers.
- VFIT currently performs only 2× interpolation, while multi-frame and arbitrary-time interpolation are left for future work.The paper suggests predicting kernels tied to different time steps or using time as an additional input for these extensions.