Source-linked AI summary
Context-aware Synthesis for Video Frame Interpolation
Simon Niklaus, Feng Liu
TL;DR
Video frame interpolation still struggles to blend warped frames accurately when optical flow encounters occlusion and large motion. This paper warps both frames and per-pixel context maps, then uses a synthesis network instead of pixel-wise blending; it reports higher-quality results, including a PSNR of 34.62 versus 34.12 for voxel flow on the DVF dataset.
Problem
Pixel-wise blending of flow-warped frames remains challenging because optical flow can be inaccurate under occlusion and large motion.
Method
The method extracts per-pixel context maps, pre-warps them with both input frames using bidirectional flow, and feeds them to a frame synthesis neural network.
Results
34.62 PSNR for the proposed method versus 34.12 for voxel flow on the DVF dataset, while experiments report higher-quality results than state-of-the-art approaches.
Takeaways & Limitations
Flexible synthesis can use neighboring pixels and contextual information to handle occlusion and optical-flow errors in difficult interpolation cases.
Takeaways & Limitations
Checkerboard artifacts can occur with uneven overlaps, motivating a modified GridNet architecture and bilinear upsampling.
Abstract
from arXiv · showhide
Video frame interpolation algorithms typically estimate optical flow or its variations and then use it to guide the synthesis of an intermediate frame between two consecutive original frames. To handle challenges like occlusion, bidirectional flow between the two input frames is often estimated and used to warp and blend the input frames. However, how to effectively blend the two warped frames still remains a challenging problem. This paper presents a context-aware synthesis approach that warps not only the input frames but also their pixel-wise contextual information and uses them to interpolate a high-quality intermediate frame. Specifically, we first use a pre-trained neural network to extract per-pixel contextual information for input frames. We then employ a state-of-the-art optical flow algorithm to estimate bidirectional flow between them and pre-warp both input frames and their context maps. Finally, unlike common approaches that blend the pre-warped frames, our method feeds them and their context maps to a video frame synthesis neural network to produce the interpolated frame in a context-aware fashion. Our neural network is fully convolutional and is trained end to end. Our experiments show that our method can handle challenging scenarios such as occlusion and large motion and outperforms representative state-of-the-art approaches.
1. Introduction
Video frame interpolation depends heavily on optical flow, yet occlusion, large motion, and inaccurate correspondences make blending warped frames difficult. The paper addresses this with context-aware neural synthesis that uses warped frames and per-pixel context maps.
- Optical-flow quality strongly affects interpolation, but accuracy decreases as motion increases and remains challenged by occlusion.
- Bidirectional-flow methods and adaptive weight maps improve handling of motion inaccuracies and occlusion, but pixel-wise blending still requires accurate warped-frame correspondences.
- The proposed network directly synthesizes an intermediate frame from pre-warped inputs instead of relying on pixel-wise blending.
- Per-pixel context maps extracted from a pre-trained image-classification network are warped with the input frames and supplied to the synthesis network.
- The method handles challenging cases such as significant occlusion and motion while producing higher-quality results than representative state-of-the-art approaches.
2. Related Work
Prior interpolation methods commonly combine optical-flow estimation with frame interpolation, but optical flow remains error-prone under occlusion, large motion, lack of texture, and blur. Related approaches address these issues through bidirectional flow, hole filling, adaptive blending, phase modification, or direct convolutional synthesis.
- Classic video frame interpolation methods first estimate optical flow and then use it for frame interpolation.
- Optical flow remains difficult and error-prone in significant occlusion, large motion, textureless regions, and blur.
- Some methods fill optical-flow holes, estimate bidirectional flow, or use per-pixel weights to improve handling of missing motion and warped-frame blending.
- Flow-free alternatives include phase-based interpolation and direct convolutional synthesis, although direct synthesis can sometimes produce blurry results.
3. Video Frame Interpolation
The method estimates bidirectional flow, extracts and pre-warps contextual features with the input frames, and uses a fully convolutional synthesis network rather than pixel-wise blending. It trains with alternative reconstruction losses and evaluates architectural and blending choices.
- The three-stage pipeline estimates bidirectional flow and context maps, pre-warps both, then synthesizes the intermediate frame with a neural network.
- PWC-Net supplies bidirectional flow, while forward warping preserves holes so the synthesis network can identify occluded regions.
- The synthesis network directly generates output from pre-warped frames, tolerating optical-flow inaccuracies better than weighted pixel-wise blending.
- Context-aware Frame Synthesis: Context maps from ResNet-18 conv1 encode each pixel's 7 × 7 neighborhood and are warped alongside the corresponding input frame.
- Context-aware Frame Synthesis: The modified GridNet processes features across multiple scales, with rows maintaining resolution and columns exchanging information through downsampling and upsampling.
- Loss functions: The feature loss produces more visually pleasing results, whereas Laplacian and ℓ1-based losses produce better quantitative results.
- Implementation: Training uses 256 × 256 examples, AdaMax optimization, and 50,000 examples over 50 epochs, with random crops and flips for augmentation.
4. Experiments
Experiments evaluate loss functions, contextual information, optical flow, synthesis design, and comparisons with state-of-the-art methods. The approach performs strongly on quantitative benchmarks and challenging visual cases, while LF loss improves perceptual quality.
- Evaluation setup: The evaluation combines quantitative and qualitative comparisons with baselines and representative state-of-the-art interpolation methods.The experiments include ablations, benchmark comparisons, and visual studies.
- Loss functions: Laplacian and color losses outperform feature loss quantitatively, while feature loss produces more visually pleasant results.The feature-loss model is trained by refining a Laplacian-loss model.
- Contextual information: Contextual information significantly improves the synthesis network over using only two pre-warped frames.Different context-extraction choices perform similarly, with ResNet-18 conv1 working overall best.
- Optical flow and motion compensation: Motion compensation improves synthesis quality, and lower optical-flow accuracy also reduces the quality of the final interpolation.Using SPyNet instead of PWC-Net worsens results, while omitting pre-warping significantly worsens synthesis quality.
- Quantitative evaluation: The method establishes a new state of the art on Middlebury and ranks first among more than 100 listed algorithms.The comparison reports a notable margin over the previously best-performing method.
- Visual comparison: In challenging motion and occlusion examples, the LF model retains more high-frequency details, whereas competing methods introduce artifacts or fail on large motion.Participants generally preferred the LF approach, while LLap performed better quantitatively.
5. Conclusion
The paper presents a context-aware synthesis approach for video frame interpolation, combining bidirectional flow, flexible synthesis, contextual information, and optical-flow initialization to produce high-quality results that outperform state-of-the-art methods.
- The approach combines bidirectional flow with a flexible frame synthesis network to handle occlusions and inaccuracies in motion estimation.
- Contextual information enables the synthesis network to perform informative interpolation.
- Optical flow provides a useful initialization for frame interpolation.
- The experiments show high-quality interpolation results that outperform state-of-the-art methods.