Source-linked AI summary
Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation
Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, Limin Wang
TL;DR
VFI needs motion and appearance information, but prior mixed or separately designed extractors can be ambiguous or inefficient. This paper reuses inter-frame attention maps to extract both types of information within a hybrid CNN–Transformer pipeline, achieving state-of-the-art results for fixed- and arbitrary-timestep interpolation with lighter overhead than similarly performing models. The authors note that the method currently uses only two consecutive frames and restricts attention-based motion extraction to high-resolution appearance features.
Problem
Prior VFI methods either mix motion and appearance extraction or use separate modules, limiting representation clarity or efficiency.
Method
The method reuses inter-frame attention maps to enhance appearance features and extract motion, integrating this operation into a hybrid CNN–Transformer architecture.
Results
The model achieves state-of-the-art performance on fixed- and arbitrary-timestep interpolation across various datasets while offering lighter computation than models with close performance.
Takeaways & Limitations
A unified attention operation provides motion cues for arbitrary-timestep interpolation while the hybrid design balances performance, efficiency, and fine-grained information.
Takeaways & Limitations
The method is restricted to two consecutive input frames, preventing it from leveraging information from multiple consecutive frames.
Abstract
from arXiv · showhide
Effectively extracting inter-frame motion and appearance information is important for video frame interpolation (VFI). Previous works either extract both types of information in a mixed way or elaborate separate modules for each type of information, which lead to representation ambiguity and low efficiency. In this paper, we propose a novel module to explicitly extract motion and appearance information via a unifying operation. Specifically, we rethink the information process in inter-frame attention and reuse its attention map for both appearance feature enhancement and motion information extraction. Furthermore, for efficient VFI, our proposed module could be seamlessly integrated into a hybrid CNN and Transformer architecture. This hybrid pipeline can alleviate the computational complexity of inter-frame attention as well as preserve detailed low-level structure information. Experimental results demonstrate that, for both fixed- and arbitrary-timestep interpolation, our method achieves state-of-the-art performance on various datasets. Meanwhile, our approach enjoys a lighter computation overhead over models with close performance. The source code and models are available at https://github.com/MCG-NJU/EMA-VFI.
1. Introduction
VFI depends on extracting motion and appearance between consecutive frames, but prior mixed or separate extraction strategies create representation ambiguity or computational overhead. The paper unifies both operations through inter-frame attention and combines it with a hybrid CNN–Transformer pipeline for efficient interpolation.
- VFI generates intermediate frames by capturing motion between consecutive inputs and blending their corresponding appearance.
- Prior methods either mix motion and appearance in a shared extractor or use separate modules, creating ambiguity or high computational cost.
- Inter-frame attention reuses one attention map to enhance inter-frame appearance features and extract motion information simultaneously.The map represents temporal correlations, aggregates neighboring appearance features, and weights displacements to estimate approximate motion.
- A hybrid CNN–Transformer architecture uses CNNs for high-resolution low-level features and Transformer blocks for low-resolution motion and appearance features.This design reduces inter-frame-attention overhead while preserving fine-grained information.
- The model achieves state-of-the-art performance across various datasets while maintaining lighter computation than models with nearby performance.
2. Related Work
Related VFI work includes mixed methods that jointly process motion and appearance, while Transformer-based approaches improve correspondence modeling but incur greater computation and memory costs at high resolution. The paper situates its method within efforts to combine motion-aware representations with efficient CNN–Transformer designs.
- Mixed VFI methods concatenate input frames and process motion and appearance jointly without explicitly representing motion.They include directly generated and kernel-based approaches.
- Video-understanding studies extract motion and appearance jointly, including representations based on frame relations, disentangled dynamics, and expected displacement.
- Inter-frame attention uses a query region from one frame and neighboring keys/values from another to model motion and transfer appearance information.
- Transformer-based VFI methods capture long-range correspondence but require more computation and memory for high-resolution interpolation than CNN-based models.
3. Our Method
The method uses inter-frame attention to extract motion and appearance information together, then combines this module with a hybrid CNN-Transformer pipeline for efficient interpolation while preserving fine-grained features.
- Inter-frame motion-appearance extraction: Inter-frame attention reuses its attention map to enhance appearance features and extract motion information simultaneously.The attention map represents similarity between a patch and neighboring patches, enabling appearance transfer and motion estimation from one operation.
- Inter-frame motion-appearance extraction: Appearance features are aggregated from similar regions in the other frame and fused with the current-frame feature to model cross-frame appearance transformation.This blending supplies information about how appearance changes between frames for intermediate-frame generation.
- Inter-frame motion-appearance extraction: Motion vectors are estimated by weighting neighboring coordinates, subtracting the original position, and passing the result through a linear layer.Under local linear motion, the vector from I0 to It is approximated by multiplying the vector from I0 to I1 by t.
- Inter-frame motion-appearance extraction: The motion vector is timestep-invariant, so inter-frame attention needs to be computed only once for multiple arbitrary-timestep predictions.The vector can guide motion estimation for arbitrary timestep frame prediction after scaling by t.
- Overall pipeline: A hybrid CNN-Transformer pipeline uses hierarchical convolutions for high-resolution low-level features and reduced-resolution Transformer blocks for motion and inter-frame appearance features.The CNN features reduce Transformer input size, while cross-scale dilated convolutions restore fine-grained information before hierarchical motion-appearance extraction.
- Overall pipeline: The pipeline estimates bidirectional flows and masks, warps both input frames to timestep t, fuses them, and refines the fused frame with a RefineNet.The motion and appearance features support flow and mask estimation, while low-level and inter-frame appearance features are used for final refinement.
4. Experiments
Experiments evaluate the method across fixed- and arbitrary-timestep interpolation benchmarks, compare accuracy and efficiency, and test its components through ablations and visual analyses.
- Datasets: The model is evaluated on Vimeo90K, UCF101, Middlebury, SNU-FILM, Xiph, HD, and X4K1000FPS across fixed- and arbitrary-timestep settings.The datasets span varied resolutions, motion difficulties, and interpolation regimes.
- Implementation Details: Two model variants provide a time-friendly small model and a larger, more accurate model by adjusting Transformer depth and channel width.Ours-small uses two Transformer blocks per stage and 16 initial channels, while Ours uses four blocks and 32 channels.
- Comparison with the State-of-the-Art Methods: The method achieves state-of-the-art performance on almost all fixed-timestep test sets and is best on all reported arbitrary-timestep test datasets.It improves over the previous SOTA by 0.28 dB and 0.68 dB on Xiph-2K and Xiph-4K, and by 0.27 dB and 0.26 dB on SNU-FILM Hard and Extreme.
- Qualitative Comparison: Visual comparisons show better object-location estimation and texture preservation for large motions, plus better temporal consistency for complex multi-frame interpolation.These comparisons cover intermediate-frame and multi-frame generation.
- Ablation Studies: Ablations examine inter-frame attention, motion cues, scalable Transformer depth, and hybrid CNN/Transformer designs.Motion features used as cues achieve better results on multiple datasets while maintaining almost the same inference time.
5. Limitations and Future Work
The method’s efficiency-oriented hybrid design constrains high-resolution motion extraction, and the current model accepts only two consecutive frames.
- Limitations: The hybrid CNN and Transformer design relieves computational overhead but restricts inter-frame-attention motion extraction within high-resolution appearance features.This limitation reflects the trade-off introduced by the efficiency-oriented architecture.
- Future Work: The model is restricted to two consecutive input frames and therefore cannot leverage information from multiple consecutive frames.Future work aims to extend the approach to multi-frame inputs without excessive overhead.
- Future Work: The authors also plan to investigate inter-frame attention for action recognition and action detection.
6. Conclusion
The paper uses inter-frame attention to extract motion and appearance information jointly and combines it with a hybrid CNN and Transformer framework for efficient VFI.
- Inter-frame attention reuses correlation information in its attention map to enhance appearance information and model motion simultaneously.
- A hybrid CNN and Transformer framework is designed to trade off performance and efficiency.
- Experiments report state-of-the-art performance for fixed- and arbitrary-timestep interpolation with effectiveness compared with the previous SOTA method.
A. 1. Motion Estimation
Motion estimation combines multi-stage motion and appearance features with warped images, then uses lightweight convolutional updates to refine flow and fusion outputs.
- The motion-estimation pipeline follows RIFE’s iterative updates of optical flow F and fusion map O using convolutional layers.
- Motion and appearance features from each Transformer stage are combined with intermediate quantities before resolution upsampling.Two PixelShuffle operations with r = 2 quadruple feature resolution.
- Warped original images and previous-stage estimates are concatenated with the two-stream features and passed through three convolutional layers to generate residual updates.The residual is bilinearly upsampled and added to the previous flow and fusion estimates.
A. 2. RefineNet
RefineNet refines warped interpolation results by incorporating acquired low-level and inter-frame enhanced appearance features at corresponding stages.
- RefineNet uses a simplified U-Net architecture to refine warped results obtained with F and O.
- Low-level features L and inter-frame enhanced appearance features A are added to corresponding stages for better appearance refinement.
A. 3. Loss Functions
The training objective combines warp supervision with reconstruction supervision, balancing the two losses through a fixed weight.
- The warp loss directly supervises the warped result and implicitly supervises motion estimation.
- The reconstruction loss supervises the quality of the final synthesized frame.
- The full loss function combines warp and reconstruction losses.
- λ = 0.5 balances the warp and reconstruction losses.
A. 4. Detailed Runtime/Memory Comparisons
Runtime and memory comparisons position the method as efficient relative to high-performance models, while its small variant is comparable to real-time models.
- The method is efficient compared with high-performance models VFIFormer and ABME.
- Ours-small is comparable to the real-time model AdaCoF.
- Window size 7 is reported as a decent choice for the attention mechanism.