Source-linked AI summary
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz
TL;DR
Video interpolation seeks spatially and temporally coherent intermediate frames, while existing methods mainly target single-frame interpolation. This paper presents a time-independent end-to-end CNN that refines intermediate optical flow and models visibility, achieving consistently better performance than existing methods across datasets. A scope boundary is that the paper notes a rare case where an object appears and disappears between the two input frames.
Problem
Existing methods mainly focus on single-frame interpolation, while generating multiple coherent frames requires accurate motion interpretation and occlusion reasoning.
Method
The method estimates bidirectional flow, approximates and refines intermediate flows, predicts soft visibility maps, and fuses visibility-weighted warped inputs at arbitrary time steps.
Results
The approach consistently outperforms state-of-the-art single-frame methods across datasets and outperforms the recent DVF method on the KITTI 2012 optical-flow benchmark.
Takeaways & Limitations
A single time-independent network can produce as many intermediate video frames as needed while jointly handling motion interpretation and occlusion reasoning.
Takeaways & Limitations
The synthesis discussion notes a rare case in which an object appears and disappears between the two input frames.
Abstract
from arXiv · showhide
Given two consecutive frames, video interpolation aims at generating intermediate frame(s) to form both spatially and temporally coherent video sequences. While most existing methods focus on single-frame interpolation, we propose an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled. We start by computing bi-directional optical flow between the input images using a U-Net architecture. These flows are then linearly combined at each time step to approximate the intermediate bi-directional optical flows. These approximate flows, however, only work well in locally smooth regions and produce artifacts around motion boundaries. To address this shortcoming, we employ another U-Net to refine the approximated flow and also predict soft visibility maps. Finally, the two input images are warped and linearly fused to form each intermediate frame. By applying the visibility maps to the warped images before fusion, we exclude the contribution of occluded pixels to the interpolated intermediate frame to avoid artifacts. Since none of our learned network parameters are time-dependent, our approach is able to produce as many intermediate frames as needed. We use 1,132 video clips with 240-fps, containing 300K individual video frames, to train our network. Experimental results on several datasets, predicting different numbers of interpolated frames, demonstrate that our approach performs consistently better than existing methods.
1. Introduction
Video interpolation can create high-quality slow-motion video from ordinary recordings, but generating multiple coherent frames requires accurate motion interpretation and occlusion reasoning. The proposed time-independent end-to-end method refines intermediate flows, predicts visibility, and outperforms existing methods across datasets.
- Motivation and challenge: Generating multiple intermediate frames requires spatial and temporal coherence, accurate motion interpretation, and occlusion reasoning to avoid artifacts near motion boundaries.Producing 240-fps video from 30-fps input requires seven intermediate frames between consecutive inputs.
- Motivation and challenge: Recursive single-frame interpolation is slow, accumulates errors, and only efficiently produces 2^i − 1 intermediate frames.These limitations make arbitrary higher frame-rate conversion inefficient, such as generating 41 frames for 1008-fps video from 24-fps input.
- Proposed approach: The method estimates bidirectional flow, linearly approximates intermediate flows, then refines them and predicts soft visibility maps in an end-to-end network.Visibility-weighted warped inputs exclude occluded pixels, reducing artifacts around motion boundaries.
- Proposed approach: Time-independent network parameters allow interpolation at arbitrary time steps and generation of as many intermediate frames as needed.The approach warps both input images to each requested time step and adaptively fuses them.
- Evaluation: The network is trained on 1.1K 240-fps clips containing 300K frames and significantly outperforms existing methods across several datasets.Evaluations include datasets requiring different numbers of interpolations, plus unsupervised optical-flow evaluation on KITTI 2012.
2. Related Work
Prior interpolation methods use optical flow, optimization, phase propagation, or learned convolutions, but motion boundaries, occlusions, computational cost, and high-frequency motion remain challenging. Related work also includes CNN-based supervised and unsupervised optical-flow learning.
- Video interpolation: Classical optical-flow interpolation supports arbitrary intermediate times, but motion boundaries and severe occlusions can still produce artifacts.A state-of-the-art optical-flow method with occlusion reasoning provides a strong frame-interpolation baseline.
- Video interpolation: Poisson-based gradient propagation generates multiple frames but is computationally expensive because it requires complex optimization.Phase propagation performs well overall but can fail on high-frequency content with large motions.
- Deep interpolation methods: Learned interpolation methods include optical-flow supervision, spatially adaptive convolutions, and reference-based light-field video interpolation.The proposed method differs from reference-based light-field interpolation by targeting plain video without reference images.
- Learning optical flow: CNN-based optical-flow research includes FlowNet architectures, later FlowNet2 improvements, and methods that embed classical optical-flow principles into network designs.These approaches aim to learn mappings from input images to optical flow with varying computation and performance trade-offs.
- Learning optical flow: Unsupervised optical-flow learning uses image warping and reconstruction error as supervision, with extensions that preserve temporal information across video sequences.Related approaches differ in whether they model only two frames or use a memory module for sequence context.
3. Proposed Approach
The approach synthesizes arbitrary-time intermediate frames by estimating and refining time-specific bidirectional flow, predicting visibility, and adaptively fusing warped inputs. Its differentiable U-Net pipeline is trained for sets of intermediate frames.
- 3.1. Intermediate Frame Synthesis: The method estimates intermediate images by backward-warping both input frames and combining them with content-aware temporal and visibility weights.The warping function is differentiable, and visibility maps suppress contributions from occluded pixels.
- 3.2. Arbitrary-time Flow Interpolation: Intermediate bidirectional flows are approximated by linearly combining the forward and backward flows between the two input images at each time step.The approximation uses time-dependent scaling of the two input flow fields.
- 3.2. Arbitrary-time Flow Interpolation: The initial flow approximation works in smooth regions but fails near motion boundaries, so a second network learns flow residuals to reduce those artifacts.The refinement is concentrated mainly around motion boundaries.
- 3.2. Arbitrary-time Flow Interpolation: The flow interpolation network also predicts soft visibility maps that favor the visible input and adaptively combine both inputs when pixels remain visible in each.The visibility constraint prevents both inputs from being treated as fully occluded at the same intermediate pixel.
- 3.2. Arbitrary-time Flow Interpolation: A first U-Net jointly predicts forward and backward optical flow, while U-Net architectures are used for both flow computation and flow interpolation.The flow computation network takes the two input images and predicts F0→1 and F1→0.
- 3.3. Training: Training uses a linear combination of reconstruction, perceptual, warping, and smoothness losses for predicted intermediate frames.The reconstruction term measures RGB-space error, while the perceptual term is intended to preserve details and sharpen predictions.
4. Experiments
Experiments evaluate the model on multiple datasets and interpolation settings, including ablations of training strategy and architectural components. The approach achieves strong performance across single- and multi-frame interpolation tasks, with benefits from joint multi-frame training and additional data.
- 4.1. Dataset: The training data comprise 1,132 video clips and 376K frames, spanning varied indoor, outdoor, static-camera, moving-camera, activity, and sports scenes.The clips come from 240-fps videos collected from hand-held cameras and YouTube.
- 4.1. Dataset: The model is evaluated on Middlebury, UCF101, slowflow, and high-frame-rate Sintel using different numbers of interpolated frames.Evaluation includes eight Middlebury sequences, 379 UCF101 sequences, 46 slowflow videos, and high-frame-rate Sintel sequences.
- 4.2. Ablation Studies: Predicting more intermediate frames during training improves performance when each variant is tested to generate seven in-between frames.The ablation compares models trained to predict one, three, or seven evenly distributed intermediate frames.
- 4.2. Ablation Studies: Removing flow refinement, visibility maps, or another model component harms performance, while visibility maps reduce artifacts around motion boundaries.The ablation supports jointly learning motion interpretation and occlusion reasoning.
- 4.2. Ablation Studies: More training data improves performance, and the warping loss is the most important among the studied loss terms.Smoothness terms slightly reduce quantitative performance but help produce visually appealing optical flow.
- 4.3. Comparison with state-of-the-art methods: On UCF101, the model consistently outperforms both nonneural and CNN-based interpolation approaches on challenging motion regions.Metrics are computed using the motion masks provided by the prior work.
- 4.3. Comparison with state-of-the-art methods: On slowflow, the approach achieves the best PSNR and SSIM scores, while FlowNet2 achieves the best SSIM and L1 error scores.The experiment predicts seven in-between frames at half resolution.
- 4.3. Comparison with state-of-the-art methods: The approach achieves state-of-the-art results across datasets for both single- and multi-frame interpolation and can be applied across scenarios without modification.On Sintel, it is best at every intermediate time step except the last, where SepConv performs slightly better.
5. Conclusion
The proposed end-to-end CNN generates arbitrary numbers of intermediate frames and consistently outperforms state-of-the-art single-frame methods across four datasets. It also improves unsupervised optical-flow learning over DVF on KITTI 2012.
- The CNN produces as many intermediate video frames as needed between two input images.
- Flow interpolation refines approximated optical flows and predicts soft visibility maps, whose ablation benefits interpolation.
- The multi-frame approach consistently outperforms state-of-the-art single-frame methods on Middlebury, UCF101, slowflow, and high-frame-rate Sintel.
- The network outperforms DVF on the KITTI 2012 benchmark for unsupervised optical-flow learning.
A. Network Architecture
The flow computation and flow interpolation CNNs use similar U-Net architectures.
- Both CNNs share a similar U-Net architecture.
B. Visual Comparisons on UCF101
Figures 10 and 11 provide visual comparisons of single-frame interpolation results on UCF101.
- Figures 10 and 11 show visual comparisons of single-frame interpolation results on UCF101.
(a) (b) (c)
Figure 11 compares the actual in-between frame with interpolation results from five methods, including the proposed method, on UCF101.
- Figure 11 compares the actual in-between frame with results from PhaseBased, FlowNet2, SepConv, DVF, and the proposed method.