Source-linked AI summary
Quadratic video interpolation
Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, Ming-Hsuan Yang
TL;DR
Video interpolation must synthesize intermediate frames despite complex motion that linear, uniform-motion models do not capture well. This paper uses an acceleration-aware quadratic model with flow reversal and refinement, and reports favorable performance against existing linear models across video datasets.
Problem
Existing interpolation methods often assume uniform motion and use linear models, while relying mainly on two consecutive frames and underusing higher-order information such as acceleration.
Method
The method combines quadratic flow prediction using neighboring frames with flow reversal and learned flow refinement to synthesize intermediate frames.
Results
The method performs favorably against state-of-the-art video interpolation methods on different video datasets and better approximates curvilinear motion in examples.
Takeaways & Limitations
Acceleration-aware interpolation can model nonlinear video motion more accurately than existing linear models within the evaluated datasets.
Takeaways & Limitations
The reversal approach can produce holes where objects visible in the target frame are occluded in one source frame, requiring pixels from the other input frame to fill them.
Abstract
from arXiv · showhide
Video interpolation is an important problem in computer vision, which helps overcome the temporal limitation of camera sensors. Existing video interpolation methods usually assume uniform motion between consecutive frames and use linear models for interpolation, which cannot well approximate the complex motion in the real world. To address these issues, we propose a quadratic video interpolation method which exploits the acceleration information in videos. This method allows prediction with curvilinear trajectory and variable velocity, and generates more accurate interpolation results. For high-quality frame synthesis, we develop a flow reversal layer to estimate flow fields starting from the unknown target frame to the source frame. In addition, we present techniques for flow refinement. Extensive experiments demonstrate that our approach performs favorably against the existing linear models on a wide variety of video datasets.
1 Introduction
Video interpolation synthesizes intermediate frames to overcome camera temporal limits, but linear models assuming uniform motion struggle with complex, accelerating movement. The paper proposes an acceleration-aware quadratic model, flow reversal, and flow refinement for more accurate synthesis.
- Video interpolation synthesizes intermediate frames between original images, enabling temporal upsampling and applications including motion deblurring, video editing, virtual reality, and medical imaging.
- Existing methods commonly assume uniform, constant-speed motion and use linear models, while real-world motion can be complex and non-uniform.They also primarily use two consecutive frames, leaving higher-order information such as acceleration underexploited.
- The proposed quadratic model exploits acceleration from additional input frames, supporting curvilinear trajectories and variable velocity.It integrates CNNs with quadratic models for motion estimation and image synthesis.
- A flow reversal layer converts forward flow into backward flow from the unknown target frame to the source frame for image synthesis.The method also introduces techniques for filtering and refining estimated flow maps.
- The method performs favorably against state-of-the-art video interpolation methods across different video datasets.
2 Related Work
Prior interpolation methods largely rely on linear or two-frame formulations, limiting their handling of nonlinear motion. The paper relates quadratic interpolation to earlier spline-based work while targeting high-dimensional video data.
- Many state-of-the-art methods assume uniform motion and synthesize intermediate frames using optical flow, forward warping, or CNNs with linear motion models.
- Phase-based interpolation also models phase as a linear function of time, retaining an implicit uniform-motion assumption.The passage reports that such linear approaches do not exploit higher-order video information and produce less accurate interpolation results.
- Kernel-based methods are not constrained by uniform-motion models, but using only two consecutive frames limits their handling of nonlinear motion in complex scenarios.
- McAllister and Roulier use quadratic splines to preserve input convexity, but their method applies only to low-dimensional data.The present work addresses video interpolation in much higher dimensions.
- The proposed algorithm estimates input-frame flow, predicts intermediate flows quadratically, reverses flow directions, and synthesizes frames by warping and fusing inputs.
3 Proposed Algorithm
The method predicts intermediate frames with acceleration-aware quadratic motion, then reverses, filters, and fuses flows for synthesis. It uses neighboring frames to model nonlinear motion while refining artifacts in the backward flow.
- Quadratic flow prediction: Unlike linear interpolation, the method uses additional neighboring frames to estimate acceleration-aware motion.Flows f0→1 and f0→−1 are used to predict f0→t, while the opposite direction uses I0, I1, and I2.
- Quadratic flow prediction: The quadratic model uses acceleration information to predict intermediate motion with variable velocity and curvilinear trajectories.It assumes constant acceleration and derives flow as a quadratic function of time from forward flows to neighboring frames.
- Flow reversal layer: A flow reversal layer converts forward flow into backward flow by projecting, reversing, and averaging nearby flow values.The layer is differentiable, so gradients can reach the flow estimation module during end-to-end training.
- Frame synthesis: The system synthesizes frames by backward-warping I0 and I1 and fusing them with a learned CNN mask weighted by temporal distance.I−1 and I2 support acceleration-aware motion estimation but are not directly used as synthesis sources.
- Adaptive flow filtering: Adaptive flow filtering learns neighborhood sampling offsets and residuals to reduce thin streak artifacts and outliers in reversed flow maps.The filter acts like a learnable median filter with spatially variant, nonlinear refinement.
- Frame synthesis: End-to-end training combines an ℓ1 loss with a perceptual loss based on VGG16 conv4_3 features.All processing steps are differentiable, enabling joint optimization of the interpolation model.
4 Experiments
Experiments evaluate the quadratic interpolation model across multiple datasets, visualizations, metrics, and ablations. Results consistently favor the quadratic approach, especially for complex motion, while confirming the importance of acceleration-aware flow prediction and flow refinement.
- Implementation details: The model combines a fixed-flow training stage with whole-system finetuning and reports PSNR, SSIM, and interpolation error against ground-truth intermediate frames.Training uses 200 epochs with the flow estimation module fixed, followed by 40 finetuning epochs; interpolation error is the RMS image difference.
- Comparison with the state-of-the-arts: Table 1 evaluates center-frame and seven-frame averages on GOPRO and Adobe240, with the quadratic model consistently outperforming linear interpolation methods.The center frame is the fourth frame, while “whole” averages all seven synthesized frames.
- Comparison with the state-of-the-arts: Feature-tracking visualizations show that the quadratic model follows sharp curvilinear motion and aligns its interpolated center frame more closely with ground truth than linear methods.Linear methods fail on a sharply curved trajectory and produce severely blurred overlaps, whereas the quadratic results approximate the trajectory and align well.
- Comparison with the state-of-the-arts: The quadratic method remains more accurate when motion is nearly straight, because small deviations from strict uniform motion can produce noticeable shifts in synthesized images.The second GOPRO example shows better trajectory estimation and interpolation accuracy despite being closer to the linear-motion assumption.
- Ablation study: Ablations on DAVIS show that removing quadratic flow prediction, adaptive flow filtering, or flow reversal degrades all metrics, with filtering also reducing flow-field artifacts.The quadratic component has the largest reported role, while adaptive filtering provides a smaller quantitative gain but improves visual quality.
5 Conclusion
The paper presents quadratic video interpolation that uses acceleration information to model nonlinear motion and synthesize high-quality intermediate frames. It reports more favorable results than existing linear models across video datasets.
- The quadratic algorithm exploits acceleration information for non-linear video frame interpolation and supports end-to-end training.
- The method models complex real-world motion more accurately than existing linear models.
- The framework can extend beyond quadratic interpolation to higher-order models such as cubic interpolation.
- The proposed framework may also apply to multi-frame optical flow and novel view synthesis.