Source-linked AI summary
Asymmetric Bilateral Motion Estimation for Video Frame Interpolation
Junheum Park, Chul Lee, Chang-Su Kim
TL;DR
Video frame interpolation must estimate motion to synthesize unavailable intermediate frames, but conventional approximations can degrade near motion boundaries. This paper proposes ABME, which refines symmetric bilateral motion into asymmetric fields and combines backward warping with local-global synthesis; it reports state-of-the-art performance across various datasets.
Problem
Video frame interpolation synthesizes intermediate frames between successive inputs, but approximated intermediate motion can degrade interpolation near motion boundaries.
Method
ABME predicts symmetric bilateral fields for an anchor frame, refines them into asymmetric fields, and uses backward warping with dynamic filters and residual reconstruction.
Results
The proposed ABME achieves state-of-the-art performance across various datasets and meaningfully outperforms state-of-the-art video interpolators.
Takeaways & Limitations
The framework combines asymmetric motion modeling with synthesis using both local and global information.
Takeaways & Limitations
Tailoring ABME to 4K sequences with extreme motions remains future work, and Vimeo90K may be inadequate for that purpose.
Abstract
from arXiv · showhide
We propose a novel video frame interpolation algorithm based on asymmetric bilateral motion estimation (ABME), which synthesizes an intermediate frame between two input frames. First, we predict symmetric bilateral motion fields to interpolate an anchor frame. Second, we estimate asymmetric bilateral motions fields from the anchor frame to the input frames. Third, we use the asymmetric fields to warp the input frames backward and reconstruct the intermediate frame. Last, to refine the intermediate frame, we develop a new synthesis network that generates a set of dynamic filters and a residual frame using local and global information. Experimental results show that the proposed algorithm achieves excellent performance on various datasets. The source codes and pretrained models are available at https://github.com/JunHeum/ABME.
1. Introduction
Video frame interpolation synthesizes intermediate frames for applications such as enhancement, compression, slow-motion generation, and view synthesis. Existing backward-warping methods can suffer from inaccurate intermediate motion, motivating ABME’s asymmetric refinement and combined local-global synthesis.
- Motivation: Video frame interpolation increases video frame rates by synthesizing intermediate frames between successive inputs.It is used in video enhancement, compression, slow-motion generation, and view synthesis.
- Limitations of existing methods: Backward warping avoids the holes and overlaps associated with forward warping but requires motion fields from an unavailable intermediate frame to the inputs.Conventional methods therefore approximate these intermediate motion fields from input-frame optical flows.
- Limitations of existing methods: The symmetric bilateral model can misestimate motion when its linear trajectory constraint does not hold, especially for nonlinear object motion.Figure 1 reports that loosening this constraint yields more accurate bilateral vectors and fewer reconstruction artifacts around the head.
- Proposed approach: ABME first predicts symmetric bilateral fields, interpolates an anchor frame, and estimates asymmetric fields from that anchor to both input frames.The input frames are then backward-warped using the asymmetric fields.
- Proposed approach: The synthesis network combines FilterNet’s dynamic local filters with RefineNet’s residual reconstruction using global information.The reported experiments show meaningful outperformance of state-of-the-art video interpolators across various datasets.
2. Motion-Based Frame Warping
Motion-based interpolation can use forward or backward warping, but intermediate-frame motion is unavailable and common approximations degrade near motion boundaries. The paper motivates symmetric bilateral estimation and proposes ABME to relax its linear-motion assumption.
- Forward and backward warping: Forward warping maps source frames toward the target, whereas backward warping samples input frames using motion fields defined from the intermediate frame.The backward-warped estimate combines the two input frames with weights (1 − t) and t.
- Forward and backward warping: Forward warping can create holes when no flow reaches a target pixel and overlaps when multiple flows reach the same pixel.Softmax splatting alleviates these issues, while the proposed method uses backward warping instead.
- Motion approximation: The required intermediate-to-input motion fields are unavailable because the intermediate frame itself must be interpolated.Conventional methods approximate them by aggregating or borrowing flows between the input frames.
- Motion approximation: Simple approximations assume neighboring pixels have similar motion, but this assumption fails near motion boundaries and can produce poorly interpolated frames.The cited approximation forms include scaled input-frame flows and combinations of candidate fields.
- Symmetric bilateral motion estimation: Symmetric bilateral estimation assumes linear trajectories, making the two intermediate bilateral fields symmetric with respect to the intermediate frame.Occlusion and acceleration can break this symmetry, motivating ABME’s asymmetric refinement.
3. Proposed Algorithm
The proposed algorithm refines symmetric bilateral motion through an anchor frame into asymmetric fields, then uses complementary warping and a synthesis network to reconstruct the intermediate frame.
- ABME: ABME refines symmetric bilateral motion fields into asymmetric fields that can represent motion more reliably around occlusions and nonlinear motion.The method first approximates an anchor frame, then estimates motion independently from it to each input frame.
- ABME: The anchor frame is generated from symmetric bilateral fields and reconstructed with masks to reduce occlusion errors, especially near frame boundaries.A zero mask indicates pixels moving outside the frame, enabling occlusion-aware anchor reconstruction.
- ABME: ABMR-Net hierarchically refines motion using warped target features, reliability-masked anchor features, correlation costs, and residual motion fields.The reliability mask suppresses anchor features affected by symmetric matching errors, while refinement proceeds across multiple resolution levels.
- Frame Synthesis: Four candidate warped frames and feature pyramids are produced from symmetric and asymmetric fields, then combined complementarily for reconstruction.The input images are backward warped using the bilateral fields, yielding four frame candidates and corresponding feature candidates.
- Frame Synthesis: RefineNet addresses errors that arise when local neighbors lack reliable information because of motion errors or severe occlusion.It generates a residual frame to refine the dynamically filtered result.
4. Experiments
ABME is evaluated across multiple video-interpolation datasets and difficulty settings, using Vimeo90K-only training and comparisons with established methods. It generally achieves the strongest reported results, while ablations show benefits from asymmetric motion, complementary candidates, and contextual features.
- Comparison protocol: ABME is compared with ToFlow, SepConv, CyclicGen, DAIN, CAIN, AdaCoF, and BMBC using PSNR/SSIM results.The reported comparisons execute the available source codes for the conventional algorithms.
- Quantitative comparison: On all datasets except FILM (Easy), ABME provides the best results; on Vimeo90K, it achieves about 1 dB higher PSNR than BMBC.On FILM (Hard), ABME reaches 30.58 dB versus 30.17 dB for DAIN.
- Runtime: ABME is more than three times faster than BMBC on the reported Middlebury runtime test.The runtime is measured for interpolating an intermediate frame in the Urban sequence using an RTX 2080 Ti GPU.
- Quantitative comparison: ABME outperforms conventional algorithms across all Xiph difficulty classes and exceeds DAIN by 0.94 dB in D2 and 0.84 dB in D3.Xiph includes challenging texture, fast motion, and severe occlusion; its classes are ordered from D1 easiest to D5 most difficult.
- Model analysis: ABME reduces boundary errors caused by disocclusion by using asymmetric motion vectors, improving warped-frame quality around moving objects.The comparison describes accurate car motion estimation but boundary errors for symmetric bilateral motion fields.
- Model analysis: Using all four candidate warped frames gives the best PSNR, while asymmetric-only candidates outperform symmetric-only candidates.The candidates complement one another; symmetric fields are more reliable in static backgrounds, whereas asymmetric fields help around motion boundaries.
- Model analysis: Removing the filtered feature map lowers average Vimeo90K PSNR by 0.1 dB because contextual information helps remove noise and restore the frame.RefineNet receives both the filtered feature map and filtered frame as inputs.
5. Conclusions
The paper presents a video frame interpolation algorithm combining ABME with a synthesis network, and reports state-of-the-art performance across various datasets.
- ABME refines symmetric bilateral motions by loosening the linear motion constraint.
- The synthesis network generates dynamic filters and a residual frame using local and global information.
- Extensive experiments report state-of-the-art performance on various datasets.