Source-linked AI summary
MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement
Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang
TL;DR
Existing learning-based interpolation methods typically emphasize either optical flow or compensation kernels, limiting the balance between efficiency and interpolation accuracy. MEMC-Net combines both through a differentiable adaptive warping layer that jointly optimizes motion and compensation. It performs favorably across interpolation and enhancement tasks while using less memory for compensation-filter prediction, though its flow projection assumes locally linear motion and sequential processing is limited by imperfect flow.
Problem
Existing learning-based methods typically estimate either optical flow or compensation kernels, limiting computational efficiency and interpolation accuracy.
Method
MEMC-Net jointly estimates motion vectors and compensation filters, integrating them in a fully differentiable adaptive warping layer.
Results
MEMC-Net performs favorably against state-of-the-art algorithms for frame interpolation and for super-resolution, denoising, and deblocking, while requiring less memory to predict compensation filters.
Takeaways & Limitations
The generalized motion-compensated alignment enables MEMC-Net to extend from frame interpolation to multiple video enhancement tasks.
Takeaways & Limitations
The flow projection layer assumes locally linear motion, and a sequential variant is degraded by imperfect optical flow that corrupts warped frames.
Abstract
from arXiv · showhide
Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.
1 INTRODUCTION
Existing interpolation methods trade computational efficiency against accuracy because flow-based models can blur results while kernel-based models require large kernels for large motion. MEMC-Net jointly learns motion estimation and compensation through a differentiable adaptive warping layer, achieving favorable interpolation and enhancement results.
- Conventional MEMC methods are computationally efficient but typically use hand-crafted features and do not achieve state-of-the-art results.
- Flow-focused learning methods can produce blurry results from bilinear interpolation, whereas kernel-based methods are sensitive to large motion.
- MEMC-Net jointly estimates motion vectors and compensation filters using an adaptive warping layer that integrates optical-flow warping with learned compensation filters.
- The adaptive warping layer is fully differentiable, allowing gradients to optimize the motion-estimation and compensation networks jointly.
- MEMC-Net performs favorably against state-of-the-art interpolation algorithms on Middlebury, UCF101, and Vimeo90K while requiring less memory for compensation-filter prediction and executing efficiently.
- The model extends to super-resolution, denoising, and deblocking, obtaining more favorable results than state-of-the-art algorithms on each task.
2 RELATED WORK
Related methods include conventional MEMC systems, flow-based neural models, and kernel-based neural models. Kernel methods improve memory use through separable filters but remain limited by predefined motion ranges, whereas MEMC-Net integrates kernels with optical-flow warping.
- Conventional MEMC-based Methods: Conventional MEMC interpolation estimates motion vectors, warps reference-frame pixels along motion trajectories, and often uses post-processing to reduce artifacts.
- Learning-based Methods: Flow-based methods predict bidirectional flow, bilinearly warp input frames, and commonly blend them using estimated occlusion masks.
- Learning-based Methods: Kernel-based methods estimate spatially adaptive convolutional kernels over local patches, using large kernels to accommodate large motion.
- Learning-based Methods: SepConv reduces kernel memory from O(R2) to O(2R) by approximating 2D kernels with one vertical and one horizontal 1D kernel.
- Learning-based Methods: AdaConv and SepConv cannot handle motion larger than their predefined kernel size, while MEMC-Net integrates optical-flow warping and uses smaller kernels with lower memory requirements.
3 MOTION ESTIMATION AND MOTION COMPENSATION DRIVEN NEURAL NETWORK
MEMC-Net jointly estimates optical flow and compensation kernels, integrating them in a differentiable adaptive warping layer for frame synthesis. Its flow projection layer handles intermediate-frame motion and fills projection holes using neighboring available flows.
- The sequential MEMC-Net design was reconsidered because imperfect optical flow produces low-quality warped frames that limit later kernel estimation and post-processing.
- MEMC-Net simultaneously estimates flow and compensation kernels relative to the original reference frames and optimizes both networks through a shared warping layer.
- 3.2 Adaptive Warping Layer: The adaptive warping layer combines optical-flow alignment with learned interpolation kernels to synthesize output pixels from the input frame.
- 3.2 Adaptive Warping Layer: Bilinear coefficients are multiplied by learned kernel coefficients to form local adaptive kernels, enabling differentiable gradients to flow to optical-flow estimation.
- 3.2 Adaptive Warping Layer: Spatially varying kernels use a broader neighborhood than bilinear interpolation and improve robustness to inaccurate flow and occlusion.
- 3.3 Flow Projection Layer: The flow projection layer transforms reference-frame flows into intermediate-frame flows, averages multiple projections, and fills holes from four-directional neighboring flows.
4 VIDEO FRAME INTERPOLATION
MEMC-Net combines learned motion, interpolation kernels, occlusion masks, contextual features, and post-processing to synthesize intermediate video frames. It provides variants with and without contextual information.
- Motion estimation: MEMC-Net estimates forward and backward optical flows, then projects them to flows between the intermediate frame and both reference frames.
- Kernel estimation: The kernel estimation network predicts per-pixel coefficient maps that are reshaped into convolutional interpolation kernels.
- Mask estimation: An occlusion-mask network selects valid pixels from the two warped reference frames despite depth variation and relative object motion.
- Context extraction: Contextual maps from pretrained ResNet18 features are warped with the optical flow and interpolation kernels for post-processing.
- Post-processing: The post-processing network refines blended frames using images, flows, kernels, masks, and optionally warped context features; the variants are MEMC-Net and MEMC-Net*.
5 IMPLEMENTATION DETAILS
The implementation trains MEMC-Net using Vimeo90K data, robust losses, and separate learning-rate settings for major subnetworks. The paper also provides code, models, and generated interpolation results online.
- Loss Function: The loss combines a robust Charbonnier reconstruction term with regularization encouraging the two masks to sum to 1.0.
- Datasets: Training uses 51,312 Vimeo90K triplets, each containing 448×256 images, with flipping and temporal-order reversal augmentation.
- Hyper-parameter settings: The kernel, mask, and post-processing networks start at learning rate 0.001, while flow fine-tuning uses 0.00001.
- Hyper-parameter settings: Learning rates are reduced by 0.2 after validation loss fails to decrease for five epochs, using batch size 4 and Adam optimization.
- The authors make source code, trained models, and evaluated-method interpolation results available on the project website.
6 VIDEO FRAME ENHANCEMENT
The MEMC-Net framework extends from interpolation to super-resolution, denoising, and deblocking by using multiple frames and task-specific enhancement networks. These variants adjust flow and mask processing to match each task.
- Generalization: The framework generalizes to video super-resolution, denoising, and deblocking, using consecutive frames to extract texture cues for reducing task-specific distortions.
- Video super-resolution: For super-resolution, consecutive low-resolution frames are bicubically upsampled before motion and kernel estimation.
- Video super-resolution: Adaptive warping aligns neighboring upsampled frames using estimated optical flows and compensation kernels, while also warping ResNet18 context information.
- Video super-resolution: The enhancement network receives generated motions, kernels, contexts, warped frames, and the upsampled blurry frame for restoration.
- Model variants: MEMC-Net SR removes flow projection and mask estimation because flow is directly estimated for the target frame and every target pixel has a valid flow vector.
- Training: The enhancement variants train on Vimeo90K-SR, Vimeo90K-DN, and Vimeo90K-DB, each containing 91,701 seven-frame sequences at 448×256 resolution.
7 EXPERIMENTAL RESULTS
The experimental section analyzes sub-module contributions, processing speed, and model parameters, then evaluates frame interpolation and three video enhancement tasks.
- Experiments cover sub-module contributions, processing speed, model parameters, video frame interpolation, and three video frame enhancement tasks.
7.1 Analysis and Discussion
Experiments analyze MEMC-Net components, model size, and runtime across interpolation datasets. Integrating flow and learned kernels improves performance, while model size and runtime involve distinct trade-offs.
- Datasets: The evaluation covers Middlebury, UCF101, Vimeo90K, and HD videos using quantitative and qualitative interpolation analyses.Middlebury includes OTHER and EVALUATION sets with IE and NIE metrics.
- Flow-based analysis: Mask estimation, post-processing, and spatially adaptive kernel estimation each improve the flow-based baseline, with the complete model achieving state-of-the-art performance on three benchmarks.The baseline uses optical flow and bilinear interpolation; replacing the fixed bilinear kernel produces a significant gain.
- Model parameters: MEMC-Net s has 7,204,367 trainable parameters, 89.3% fewer than the full network, but performs less well on Middlebury with large displacement.It performs slightly better on Vimeo90K, where SPyNet is stronger on small motion.
- Execution speed: MEMC-Net processes 1920 × 1080p video in 0.41 second per frame, while four-GPU patch processing reaches 30 frames per second.The small model is suited to memory-constrained devices, whereas the full model is preferable when prompt response is required.
- Efficiency comparison: MEMC-Net s outperforms the larger 21.7M-parameter SepConv model, while parameter count alone does not determine runtime.ToFlow+Mask uses fewer parameters but runs slower than MEMC-Net and SepConv.
7.2 Video Frame Interpolation
MEMC-Net is evaluated against flow-, kernel-, direct-, and sequential-interpolation methods across standard and high-motion datasets. It performs favorably overall and is particularly effective for complicated or large motion.
- Comparative evaluation: The comparison includes MEMC-Net and MEMC-Net* alongside SepConv, DVF, ToFlow, CtxSyn, MIND, and SuperSlomo.Evaluations use UCF101, Vimeo90K, Middlebury, and HD videos.
- Middlebury: MEMC-Net performs favorably on Middlebury overall, including sequences with complicated motion such as dancing feet, moving arms, and fingers.CtxSyn performs well on smaller motions or fine textures, while SuperSlomo is best on the synthetic Urban sequence.
- Benchmark results: The proposed methods perform favorably across UCF101, Vimeo90K, and Middlebury, demonstrating generalization to diverse motion scenarios.The cited evaluation identifies best and second-best results using red and blue numbers, respectively.
- HD videos: On HD videos, MEMC-Net performs consistently across resolutions, with a larger advantage over SepConv on 1080p videos.The results indicate that fixed 51 × 51 kernels are unsuitable for large motion.
- Qualitative results: MEMC-Net reconstructs structures such as a lamppost and falling ball with fewer artifacts than competing flow- and kernel-based methods.The examples concern inaccurate optical flow and kernels too small for the motion.
7.3 Video Frame Enhancement
MEMC-Net extends motion-compensated alignment to video super-resolution, denoising, and deblocking. Across the reported datasets, the adapted models perform favorably against established alternatives and preserve sharper or finer details.
- Data generation: The enhancement experiments use bicubic downsampling for super-resolution and Gaussian noise with σ = 20 for denoising.Deblocking inputs are generated with H.264 encoding at qp 37 while in-loop deblocking is disabled.
- Super-resolution: MEMC-Net SR obtains higher PSNRs than EDSR on both Vimeo90K and BayesSR despite using fewer residual blocks and filters.It also performs favorably against existing video super-resolution methods, especially on BayesSR.
- Denoising: MEMC-Net DN exceeds the second-best denoising method by 1.24dB on Vimeo90K and 1.95dB on V-BM4D.The qualitative examples show better preservation of fine textures on clothes and streets.
- Deblocking: MEMC-Net DB performs favorably against all evaluated deblocking algorithms.Both MEMC-Net DB and V-BM4D reduce blockiness around hands and eyes, whereas ToFlow and EDSR DB do not reduce blocky pixels well.
8 CONCLUSIONS
MEMC-Net combines motion estimation and compensation with data-driven feature learning through end-to-end trainable adaptive warping and flow projection layers. Its generalized alignment supports interpolation and several video enhancement tasks, with favorable benchmark results.
- MEMC-Net is a motion estimation and motion compensation driven neural network for video frame interpolation and enhancement.
- Adaptive warping and flow projection layers integrate the subnetworks into an end-to-end trainable model.The framework combines MEMC’s large-motion handling with data-driven feature extraction.
- Generalized motion-compensated alignment extends the framework to video super-resolution, denoising, and deblocking.Quantitative and qualitative evaluations report favorable performance against state-of-the-art algorithms across interpolation and enhancement tasks.