Source-linked AI summary
FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, Du Tran
TL;DR
Optical-flow-based interpolation is costly, difficult to deploy, and limited in handling complex motions and occlusions. FLAVR addresses this with flow-free 3D convolutions, reporting strong interpolation trade-offs and reusable representations for several downstream video tasks, while retaining stated scope limitations.
Problem
Optical-flow-based interpolation introduces computational and deployment costs, and its performance can be limited by noisy flow estimates around complex occlusions.
Method
FLAVR is a flow-free 3D CNN architecture that models spatio-temporal relations and predicts multiple intermediate frames end to end in one forward pass.
Results
FLAVR is reported as superior or comparable across multiple video-interpolation benchmarks while offering a favorable inference-speed versus accuracy trade-off, and its representations support several downstream tasks.
Takeaways & Limitations
Frame interpolation with FLAVR can serve as a self-supervised pretext task for action recognition, optical flow estimation, video object segmentation, and motion magnification.
Takeaways & Limitations
FLAVR requires retraining for each interpolation factor k and has limited generalization to data outside the training distribution.
Abstract
from arXiv · showhide
A majority of methods for video frame interpolation compute bidirectional optical flow between adjacent frames of a video, followed by a suitable warping algorithm to generate the output frames. However, approaches relying on optical flow often fail to model occlusions and complex non-linear motions directly from the video and introduce additional bottlenecks unsuitable for widespread deployment. We address these limitations with FLAVR, a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video frame interpolation. Our method efficiently learns to reason about non-linear motions, complex occlusions and temporal abstractions, resulting in improved performance on video interpolation, while requiring no additional inputs in the form of optical flow or depth maps. Due to its simplicity, FLAVR can deliver 3x faster inference speed compared to the current most accurate method on multi-frame interpolation without losing interpolation accuracy. In addition, we evaluate FLAVR on a wide range of challenging settings and consistently demonstrate superior qualitative and quantitative results compared with prior methods on various popular benchmarks including Vimeo-90K, UCF101, DAVIS, Adobe, and GoPro. Finally, we demonstrate that FLAVR for video frame interpolation can serve as a useful self-supervised pretext task for action recognition, optical flow estimation, and motion magnification.
1. Introduction
FLAVR targets the speed, deployment, and modeling limitations of optical-flow-based interpolation with a flow-free architecture that predicts multiple frames end to end. The paper reports strong accuracy–speed trade-offs and usefulness of learned representations for downstream video tasks.
- Background: Frame interpolation is motivated as a way to generate coherent intermediate video frames for limited acquisition frame rates and exposure times.Prior work is characterized as treating interpolation predominantly as a complicated, hard-coded graphics problem.
- Motivation: Optical-flow-based methods can be computationally inefficient and may require custom CUDA kernels, limiting seamless deployment across edge devices.QVI, DAIN, and BMBC take seconds to generate 8× interpolated frames.
- FLAVR: FLAVR uses spatio-temporal convolutions without external flow or depth maps to make end-to-end multiple-frame predictions in one forward pass.The authors state that the architecture implicitly handles complex motions and occlusions through learning from large-scale video data.
- Representation learning: Frame interpolation can provide a more generic self-supervised representation-learning objective than tasks tailored to specific downstream applications.The paper connects interpolation representations to motion, objects, and actions in raw videos.
- Contributions: FLAVR is presented as a scalable, flow-free 3D CNN architecture for single-shot multiple-frame interpolation.The paper describes it as the first approach combining optical-flow-free operation with single-shot multiple-frame prediction.
- Evaluation: The method is evaluated on Vimeo-90K, UCF101, DAVIS, Adobe, and GoPro, where it is reported as superior or comparable to current approaches.The paper emphasizes a favorable trade-off between interpolation accuracy and inference speed.
3. Frame Interpolation using FLAVR
FLAVR constructs interpolation training examples from unlabeled videos and uses a 3D U-Net to model temporal dynamics while predicting multiple frames in one pass. Its architecture combines temporal convolutions, gated features, multiscale decoding, and end-to-end pixel-level training.
- Problem setup: For k× interpolation, FLAVR predicts k−1 intermediate frames between each pair of original frames.The framework is designed for single-shot k× prediction for any value of k.
- Sampling training data: Training data are generated by subsampling raw videos with stride k and pairing temporal input windows with the intervening original frames as ground truth.Each example contains 2C input frames and an output clip of k−1 frames.
- Architecture overview: FLAVR is a 3D U-Net that replaces 2D encoder and decoder convolutions with 3D convolutions to model temporal dynamics and motion trajectories.The added temporal dimension also captures actions and frame correspondences.
- Decoder and prediction: The decoder uses 3D transpose convolutions, skip connections, and a final 2D projection that outputs H×W×3(k−1) values for the predicted frames.A 3D convolution is added after the final transpose convolution to address checkerboard artifacts, and temporal features are fused before prediction.
- Feature gating: Channel gating follows each convolutional layer and learns to emphasize feature dimensions containing useful interpolation cues such as motion boundaries.The gating operation applies learned channel weights after spatio-temporal pooling.
- Training objective: The network is trained end to end with a pixel-level L1 loss between predicted and ground-truth intermediate frames.The loss is defined over the predicted and ground-truth frames in each training clip.
- Representation learning: Pretraining FLAVR on frame interpolation is used to reuse learned video representations for action recognition, optical flow estimation, and motion magnification.The paper frames this as both an analysis of learned motion representations and a way to improve downstream performance over random initialization.
4. Experimental Setup
FLAVR is evaluated against established video-interpolation baselines using standardized datasets, metrics, and retraining procedures. The setup covers single- and multi-frame interpolation with RGB-only and methods using additional flow or depth inputs.
- Datasets: Training uses Vimeo-90K septuplets, with evaluation on its test split and generalization tests on UCF101 and DAVIS.Single-frame interpolation uses models trained on Vimeo-90K and evaluates generated quintuples from the additional datasets.
- Evaluation Metrics: PSNR and SSIM measure quantitative quality, while multi-frame interpolation additionally reports Temporal Change Consistency and a user study.For multi-frame interpolation, metric values are averaged over all predicted frames.
- Baselines: FLAVR is compared with DAIN, QVI, DVF, SepConv, SuperSloMo, and AdaCoF across single- and multi-frame interpolation settings.The baselines represent depth-aware, flow-based, volumetric, kernel-based, and adaptive-flow approaches.
- Comparison Protocol: For 2× interpolation, FLAVR consistently outperforms prior methods using RGB alone and methods requiring optical-flow or depth inputs.The comparison covers Vimeo-90K, UCF101, and DAVIS.
- Comparison Protocol: For 8× interpolation on Adobe and GoPro, FLAVR outperforms all previous work using only RGB as input.The multi-frame comparison is summarized in Table 2.
- Comparison Protocol: Baseline models were retrained to convergence on Vimeo-90K septuplets to establish a unified comparison testbed.Prior papers used different training and testing setups, including triplet splits and private datasets.
5. How does FLAVR compare with the state-of-the-art?
FLAVR matches or exceeds prior methods across single- and multi-frame interpolation benchmarks while offering strong speed–accuracy trade-offs. It also performs robustly across task difficulty and qualitative user evaluations.
- Single-frame interpolation: FLAVR sets the state of the art on Vimeo-90K single-frame interpolation and exceeds prior methods by a significant margin.
- Generalization: 33.33 PSNR on UCF101 and 27.44 PSNR on DAVIS outperform RGB-input baselines on challenging motion and occlusion settings.
- Multi-frame interpolation: 31.31 PSNR and 0.94 SSIM on GoPro 8× interpolation exceed all prior frame-interpolation approaches.
- User study: In a DAVIS user study, FLAVR significantly outperforms Super-SloMo and performs comparably to QVI.
- Speed–accuracy trade-off: 2.7×, 6.2×, and 12.7× runtime improvements over QVI occur for 8×, 16×, and 32× interpolation, respectively.
- Speed–accuracy trade-off: FLAVR is at least 6× faster than similarly accurate baselines and provides at least a 3dB accuracy gain over similarly fast baselines.
- Robustness to task difficulty: 1.28dB and 1.62dB gains over the next-best approach occur on SNU-Film hard and medium settings, respectively.
6. How useful is FLAVR in enabling downstream applications?
FLAVR’s learned representations and interpolated outputs support downstream action recognition, optical flow estimation, motion magnification, and low-frame-rate object segmentation.
- Action recognition: Pretraining FLAVR’s encoder on frame interpolation provides representations evaluated for downstream action recognition on UCF101 and HMDB51.
- Optical flow estimation: FLAVR pretraining yields lower optical-flow endpoint error than random initialization on MPI Sintel and KITTI.
- Video object segmentation: FLAVR improves video object-segmentation label propagation on low-frame-rate videos when used to upsample them before standard mask propagation.
- Motion magnification: With FLAVR pretraining, a motion-magnification network achieves 0.801 SSIM on the synthetic CoCo-Synth dataset.
7. Discussion
FLAVR provides flow-free, end-to-end video frame interpolation with a favorable speed–accuracy trade-off and representations useful for multiple downstream tasks. Its main limitations are retraining for each interpolation factor and limited generalization outside the training distribution.
- Discussion: FLAVR uses 3D convolutions for flow-free, end-to-end interpolation and offers a strong inference-speed versus accuracy trade-off.The learned representations support action recognition, optical flow estimation, video object segmentation mask propagation, and motion magnification.
- Discussion: FLAVR representations are useful across action recognition, optical flow estimation, video object segmentation mask propagation, and motion magnification.
- Limitations: FLAVR requires retraining for each interpolation factor k.The paper notes that the desired factor is usually known beforehand in practical applications.
- Limitations: FLAVR has limited generalization capability to data outside the training distribution.The paper identifies this as a limitation shared with other data-driven end-to-end deep-learning approaches.
8. Ablations
The ablations show that temporal context and 3D spatiotemporal modeling are important for interpolation, while fusion, gating, stride, and loss choices materially affect performance and appearance.
- Backbone Architecture: 3D-R18-4I outperforms 2D-R18-4I and R2D-18-2I by 1.3 and 2.3dB, respectively.The comparison supports using 3D convolutions and a larger temporal input context.
- Input Context: For 2× interpolation, C = 2 performs best; for 8× interpolation, C = 3 performs best, while larger contexts can hurt accuracy.C = 2 corresponds to four input frames, two from the past and two from the future.
- Choice of Fusion: Fusion-concat performs better than fusion-add and is used in the final model.The fusion methods combine features across corresponding encoder and decoder layers.
- Temporal Striding: Larger temporal striding decreases performance from 36.3 to 35.2, so all 3D convolution layers use temporal stride 1.
- Channel Gating: Removing spatiotemporal gating reduces PSNR from 36.3 to 36.1, while gating increases activation around motion boundaries.
- Loss Function: Additional VGG or Huber losses do not improve PSNR or SSIM; L1 loss also produces visually sharper images in this setting.
9. Experiment Settings for downstream applications
The downstream experiments test FLAVR outputs and pretrained representations on low-frame-rate object segmentation, motion magnification, action recognition, and optical flow estimation.
- Low-fps video object segmentation: Low-fps video object segmentation compares CRW on downsampled videos with CRW applied to videos interpolated by FLAVR.DAVIS videos are downsampled to 15FPS or 8FPS, then interpolated by 2× or 4×.
- Low-fps video object segmentation: FLAVR interpolation improves low-fps video object tracking when used before CRW label propagation.The label propagation mechanism remains the same as in CRW.
- Motion magnification: Motion magnification is evaluated by finetuning FLAVR pretrained on frame interpolation using the synthetic CoCo-Synth dataset.The setup uses no temporal or spatial post-processing filters.
- Action recognition: For action recognition, the decoder is removed and the pretrained encoder is paired with a classifier and temporal stride 4.Experiments use UCF101 and HMDB51.
- Optical flow estimation: For optical flow, the same encoder and decoder are reused, with the final prediction layer changed to output two flow channels.The two channels correspond to the x and y flow values at each pixel.
10. Qualitative Results
Qualitative experiments apply FLAVR to high-frame-rate insect videos and Middlebury single-frame interpolation, including settings where four-frame inputs must be constructed.
- Insect motion: Insect take-off and flight videos are downsampled to 15FPS and restored with FLAVR to obtain higher-frame-rate videos.The 8× model is applied once to the downsampled videos.
- Middlebury: Middlebury evaluation duplicates two available input frames because FLAVR requires four-frame inputs.The paper describes duplication as sub-optimal for two-frame sequences such as teddy.
11. User study
The user study compares FLAVR with representative linear- and quadratic-warping methods for 8× interpolation using DAVIS videos.
- The study compares FLAVR against SuperSloMo and QVI, representing linear and quadratic warping, respectively.
- The evaluation uses all 90 HD videos from the DAVIS dataset to generate 8× interpolated videos.
- Each FLAVR video is compared separately with videos generated by SuperSloMo and QVI.
12. Training details
The models are trained on Vimeo-90K for 2× interpolation and GoPro for 4× and 8× interpolation, using dataset-specific crops and video augmentations.
- Training datasets: 2× interpolation is trained on Vimeo-90K, while 4× and 8× interpolation are trained on GoPro using official train and validation splits.
- Augmentation and crops: Training uses 256×256 crops for Vimeo-90K and 512×512 crops for GoPro, with random frame reversal and horizontal flipping.
- Qualitative evaluation: The 8× qualitative example takes frames at t = 0 and t = 1 as inputs and predicts the seven intermediate frames.
13. Benchmarking inference time
Inference-time benchmarking measures forward-pass time on an NVIDIA-2080Ti using Adobe-240FPS samples and 512×512 crops, while the paper notes broader misuse risks for generated video outputs.
- Benchmark setup: Inference time is measured on an NVIDIA-2080Ti GPU with 12GB memory, excluding data preprocessing and CPU/GPU transfer.
- Benchmark setup: Results are averaged over 100 Adobe-240FPS samples using 512×512 crops.
- Potential risks: The paper warns that generated interpolation outputs could be used to forge or tamper with videos or support nefarious surveillance.