Source-linked AI summary
BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation
Junheum Park, Keunsoo Ko, Chul Lee, Chang-Su Kim
TL;DR
Video interpolation must synthesize intermediate frames despite difficult motion, occlusion, and lighting conditions. BMBC estimates bilateral motions with a bilateral cost volume, approximates additional motions, and blends warped frames dynamically. The authors report better performance than state-of-the-art interpolation algorithms on multiple benchmark datasets.
Problem
Video interpolation requires accurate motion estimation to synthesize high-quality intermediate frames under large nonlinear motions, occlusions, and lighting variation.
Method
BMBC combines a bilateral motion network with bilateral cost volume, motion approximation, frame warping, context maps, and dynamic blending filters.
Results
The proposed algorithm outperforms state-of-the-art video interpolation algorithms on four benchmark datasets.
Takeaways & Limitations
Bilateral motion estimation and dynamic blending provide the paper's approach for generating intermediate video frames.
Takeaways & Limitations
The bilateral motion network may fail to estimate accurate motions in occluded regions, motivating an additional motion-approximation scheme.
Abstract
from arXiv · showhide
Video interpolation increases the temporal resolution of a video sequence by synthesizing intermediate frames between two consecutive frames. We propose a novel deep-learning-based video interpolation algorithm based on bilateral motion estimation. First, we develop the bilateral motion network with the bilateral cost volume to estimate bilateral motions accurately. Then, we approximate bi-directional motions to predict a different kind of bilateral motions. We then warp the two input frames using the estimated bilateral motions. Next, we develop the dynamic filter generation network to yield dynamic blending filters. Finally, we combine the warped frames using the dynamic blending filters to generate intermediate frames. Experimental results show that the proposed algorithm outperforms the state-of-the-art video interpolation algorithms on several benchmark datasets.
1 Introduction
Video interpolation improves temporal resolution by synthesizing intermediate frames, but accurate motion estimation remains difficult under complex motion, occlusion, and lighting variation. The proposed network combines bilateral motion estimation, bilateral cost volume matching, motion approximation, and dynamic blending, outperforming state-of-the-art methods on benchmark datasets.
- Low temporal resolution causes aliasing, abrupt motion artifacts, and degraded video quality.
- Video interpolation synthesizes intermediate frames between actual frames for temporal-resolution enhancement and applications such as quality improvement, compression, slow motion, and view synthesis.
- Large nonlinear motions, occlusions, and lighting variation make accurate motion or optical-flow estimation challenging.
- The proposed network combines a bilateral motion network using bilateral cost volume with motion approximation and dynamic filter generation.It estimates six bilateral motions, warps input frames and context maps, and combines warped frames using generated blending filters.
- The proposed algorithm performs better than state-of-the-art algorithms on various benchmark datasets.
2 Related Work
Related work progresses from direct CNN-based frame synthesis toward motion-aware interpolation methods that use warping and cost volumes. These approaches address limitations of direct generation and support explicit correspondence estimation.
- Video interpolation synthesizes intermediate frames between actual frames to increase temporal resolution.
- Direct CNN frame generation can produce severe blurring because it does not use motion information.
- Flow-based methods use warping to exploit motion, but forward warping suffers from holes and overlapped pixels, motivating backward warping.
- Backward warping requires estimating intermediate motions, meaning motion vectors for the synthesized frames.
- A cost volume records pairwise similarity scores and facilitates pixel matching for optical-flow estimation.
3 Proposed Algorithm
The proposed algorithm estimates bilateral motions directly and supplements them with approximated motions, then warps inputs and context maps to synthesize an intermediate frame with dynamic blending filters.
- Overview: The pipeline estimates two bilateral motions directly, approximates four more from bi-directional motions, and synthesizes an intermediate frame at time t.It takes successive frames I0 and I1 as input and outputs It for t ∈(0, 1).
- Bilateral motion estimation: The bilateral motion network treats the intermediate frame as reference and warps features from both input frames toward it.Spatial transformer networks perform the feature warping using motion vector fields.
- Bilateral motion estimation: The bilateral cost volume replaces the conventional cost volume because the intermediate reference frame is unavailable and must be predicted from two target frames.It stores bilateral matching costs over a displacement search window; its multilevel design reduces memory usage.
- Bilateral motion estimation: The method supports arbitrary time instances t ∈[0, 1] by incorporating t into bilateral-cost-volume computation and sharing trained network weights.At t = 0 or t = 1, the bilateral cost volume becomes identical to a conventional cost volume for estimating bi-directional motions.
- Motion approximation: Motion approximation uses bi-directional motions to produce four additional bilateral-motion candidates, retaining all four candidates for selecting reliable motions.The approximation assumes locally smooth motion and linear trajectories between consecutive frames.
- Motion approximation: The two motion schemes are complementary: approximation avoids noticeable artifacts in some cases, while direct bilateral estimation is more effective in others.The comparison attributes artifacts to motion inaccuracies in the direct-estimation results.
4 Experimental Results
The algorithm is evaluated against state-of-the-art methods on four benchmark datasets, with quantitative, qualitative, and ablation studies examining its performance and components.
- Experimental setup: The evaluation covers Middlebury, Vimeo90K, UCF101, and Adobe240-fps using comparisons with conventional and state-of-the-art interpolation algorithms.The study also includes ablations of the bilateral motion network and dynamic filter generation network.
- Comparison with the state-of-the-arts: The proposed algorithm outperforms state-of-the-art methods on Middlebury in average IE and NIE.Lower IE or NIE indicates better performance.
- Comparison with the state-of-the-arts: The proposed algorithm outperforms conventional algorithms on UCF101 and Vimeo90K, including a 0.3 dB higher PSNR than DAIN on Vimeo90K.The comparisons use average PSNR and SSIM scores.
- Comparison with the state-of-the-arts: The proposed algorithm outperforms state-of-the-art methods for ×2, ×4, and ×8 interpolation on Adobe240-fps, with larger gains as frame rate increases.It also reconstructs rotating-wheel details more faithfully in the qualitative comparison.
- Ablation studies: Ablations show that bilateral motions outperform four approximated candidates, while adding more approximated candidates further improves interpolation performance.The compared configurations are Appx4, BM, BM+Appx2, and BM+Appx4.
- Ablation studies: Adding input frames and context maps to the dynamic filter generation network improves interpolation performance, while kernel size has insignificant impact.Context maps provide additional gains of 0.07 and 0.05 dB on UCF101 and Vimeo90K, respectively.
5 Conclusions
The paper presents a deep-learning video interpolation algorithm based on bilateral motion estimation, combining bilateral motion and dynamic filter generation networks. Experiments show that it outperforms state-of-the-art algorithms on four benchmark datasets.
- The algorithm combines a bilateral motion network with a dynamic filter generation network for video interpolation.The bilateral cost volume estimates bilateral motions, while generated blending filters combine warped frames.
- Experimental results show that the proposed algorithm outperforms state-of-the-art video interpolation algorithms on four benchmark datasets.