Source-linked AI summary
IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation
Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, Jie Yang
TL;DR
Video frame interpolation needs accurate intermediate frames but existing flow-based pipelines can require separate refinement and synthesis modules, increasing delay and parameters. IFRNet combines flow and feature refinement in one encoder-decoder and adds task-oriented supervision, with experiments reporting state-of-the-art accuracy and fast inference. Its distillation design assumes task-oriented flow generally agrees with true optical flow while differing in some details.
Problem
Existing flow-based VFI pipelines separate flow and feature refinement, increasing inference delay and model parameters for real-time applications.
Method
IFRNet jointly refines bilateral intermediate flow fields and an intermediate feature through coarse-to-fine decoders, using task-oriented flow distillation and geometry consistency losses.
Results
Benchmark results report state-of-the-art VFI accuracy with fast inference speed and lightweight model size.
Takeaways & Limitations
A single encoder-decoder joint-refinement design can support efficient frame interpolation without cascaded synthesis or refinement modules.
Takeaways & Limitations
The task-oriented flow distillation approach assumes task-oriented flow generally agrees with true optical flow while differing in some details.
Abstract
from arXiv · showhide
Prevailing video frame interpolation algorithms, that generate the intermediate frames from consecutive inputs, typically rely on complex model architectures with heavy parameters or large delay, hindering them from diverse real-time applications. In this work, we devise an efficient encoder-decoder based network, termed IFRNet, for fast intermediate frame synthesizing. It first extracts pyramid features from given inputs, and then refines the bilateral intermediate flow fields together with a powerful intermediate feature until generating the desired output. The gradually refined intermediate feature can not only facilitate intermediate flow estimation, but also compensate for contextual details, making IFRNet do not need additional synthesis or refinement module. To fully release its potential, we further propose a novel task-oriented optical flow distillation loss to focus on learning the useful teacher knowledge towards frame synthesizing. Meanwhile, a new geometry consistency regularization term is imposed on the gradually refined intermediate features to keep better structure layout. Experiments on various benchmarks demonstrate the excellent performance and fast inference speed of proposed approaches. Code is available at https://github.com/ltkong218/IFRNet.
1. Introduction
Video frame interpolation converts low-frame-rate sequences into high-frame-rate video but remains difficult in complex dynamic scenes and costly for real-time use. IFRNet addresses these limitations by jointly refining intermediate flows and features in one efficient encoder-decoder, supported by task-oriented supervision.
- VFI converts low frame rate image sequences to high frame rate videos and supports applications including slow-motion generation, novel view synthesis, and cartoon creation.
- Complex dynamic scenes challenge VFI through large displacement, severe occlusion, motion blur, and abrupt brightness changes.
- Conventional flow-based VFI estimates intermediate flow, warps inputs or features, and uses a synthesis network to generate the target frame.
- Separate flow and feature refinement encoder-decoders increase inference delay and model parameters while missing their mutual promotion.
- IFRNet jointly refines bilateral intermediate flows and an intermediate feature through coarse-to-fine decoders, eliminating separated processing and supporting sharper motion and texture details.
- Task-oriented flow distillation and feature-space geometry consistency losses guide multi-scale motion estimation and intermediate feature reconstruction.
- Benchmark results report state-of-the-art VFI accuracy together with fast inference speed and lightweight model size.
2. Related Work
Related VFI methods use flow, kernels, or hallucination to synthesize frames, with flow-based methods relying on correspondence, warping, and refinement. IFRNet instead integrates intermediate-flow refinement and feature reconstruction in a single encoder-decoder.
- VFI methods are broadly classified as flow-based, kernel-based, and hallucination-based approaches with different strengths and weaknesses.
- Flow-based methods use optical flow for explicit frame correspondence, then warp input images and refine them with an image synthesis network.
- IFRNet differs from prior flow-based paradigms by jointly refining intermediate flows and an intermediate feature to generate the target frame in one encoder-decoder.
- Optical-flow research combines encoder-decoder networks with pyramid features, backward warping, and real-time estimation, while knowledge distillation supports flow prediction.
3. Proposed Approach
IFRNet uses a compact encoder-decoder that jointly refines bilateral intermediate flows and intermediate features coarse-to-fine. Task-oriented flow distillation and geometry consistency losses provide targeted supervision for motion estimation and feature structure.
- IFRNet Architecture: IFRNet extracts pyramid features, then jointly refines bilateral intermediate flow fields and intermediate features through coarse-to-fine decoders.The architecture merges flow estimation and feature refinement into one encoder-decoder model.
- IFRNet Architecture: Intermediate features provide missing reference information for flow estimation, while improved flows align source features more precisely for subsequent feature reconstruction.The two representations are progressively refined because each improves the other across pyramid levels.
- IFRNet Architecture: Each decoder processes stacked pyramid features with six 3×3 convolutions and one 4×4 deconvolution, selectively updating channels to maintain efficiency.The decoder uses PReLU activations and partial-channel updates in selected convolutions.
- IFRNet Architecture: The final decoder synthesizes the frame by blending warped inputs with a sigmoid merge mask and adding a residual for details unreliable under flow-based generation.The merge mask uses bidirectional occlusion information, while the residual compensates for details such as doubly occluded regions.
- Task-Oriented Flow Distillation Loss: Task-oriented flow distillation adaptively changes generalized Charbonnier robustness across locations to emphasize useful teacher knowledge for frame synthesis.Robustness is controlled by spatial masks derived from differences between relaxed flow predictions and pseudo-label flow.
- Feature Space Geometry Consistency Loss: Geometry consistency regularizes reconstructed intermediate features against encoder features from the ground-truth frame, preserving local geometry without constraining channel-wise representation.The loss restricts neighboring local-patch geometry at multiple feature-map locations.
4. Experiments
Experiments evaluate IFRNet across established VFI benchmarks, state-of-the-art comparisons, and ablations of its intermediate feature and proposed losses. Results show strong accuracy, efficient inference, and complementary benefits from the feature and loss designs.
- Quantitative Evaluation: IFRNet is evaluated on Vimeo90K, UCF101, SNU-FILM, and Middlebury using PSNR, SSIM, IE, and NIE metrics.The experiments also measure inference speed and computation complexity on a Tesla V100 at 1280×720 resolution.
- Quantitative Evaluation: IFRNet large achieves the best PSNR and SSIM results on Vimeo90K and UCF101, while running 11.5 × faster than ABME.
- Qualitative Evaluation: On SNU-FILM Hard, IFRNet synthesizes fast-moving objects with sharp boundaries and preserves contextual details.
- Quantitative Evaluation: IFRNet large outperforms SoftSplat on Middlebury’s average IE and NIE metrics while running 2.5 × faster.The reported accuracy improvement is limited, but the speed advantage comes from avoiding cascaded VFI architecture.
- Quantitative Evaluation: IFRNet improves on RIFE by about 0.2 dB on Vimeo90K and achieves better results with half the parameters.IFRNet small further improves speed by 31% and reduces parameters and computation complexity by 44%, with only a slight accuracy decrease.
- Ablation Study: Ablations show that the intermediate feature provides flow-reference information and global context, improving flow estimation and scene-detail compensation.The study concludes that residual compensation from intermediate context features is necessary because intermediate flow prediction is unreliable.
- Ablation Study: Task-oriented flow distillation improves Vimeo90K PSNR by 0.2 dB, while feature-space geometry consistency provides a further complementary improvement.Fixed robustness performs best at p = 0.3, whereas robustness near 1.0 can introduce harmful pseudo-label knowledge.
5. Conclusion
The paper concludes that IFRNet is an efficient single encoder-decoder for video frame interpolation, jointly refining intermediate flow and features without cascaded synthesis or refinement. Its benchmark performance and inference speed support use in frame-rate up-conversion and intermediate view synthesis systems.
- Conclusion: IFRNet jointly refines intermediate flow and a powerful intermediate feature without an additional cascaded synthesis or refinement module.
- Conclusion: The intermediate feature boosts flow estimation and represents global context for generating sharp motion boundaries and vivid motion details.
- Conclusion: Task-oriented flow distillation and feature-space geometry consistency losses further release IFRNet’s potential.
- Conclusion: Experiments demonstrate state-of-the-art performance and fast inference speed across various benchmarks.
- Conclusion: The authors expect IFRNet to serve as a component for frame-rate up-conversion and intermediate view synthesis systems.
6. Multi-Frame Interpolation
IFRNet supports 8× interpolation by conditioning one decoder on a temporal encoding mask for each desired intermediate time. On GoPro and Adobe240, it outperforms other state-of-the-art methods while retaining a speed advantage.
- Multi-Frame Interpolation: IFRNet predicts multiple intermediate frames using a one-channel temporal encoding mask rather than recursively interpolating frames or scaling optical flow.For 8× interpolation, t takes values from 1/8 through 7/8.
- Multi-Frame Interpolation: The temporal encoding is a conditional input to decoder D4, with all values set to the selected intermediate time t.
- Multi-Frame Interpolation: The proposed distillation and geometry-consistency losses remain applicable at every intermediate time instance.
- Results: IFRNet outperforms all other state-of-the-art methods with two input frames on both GoPro and Adobe240 in PSNR and SSIM.
- Results: IFRNet is 0.84 dB better than DAIN on GoPro and exceeds SuperSloMo by 1.27 dB on Adobe240.
- Efficiency: The encoder requires one forward pass while decoders run seven times with different temporal embeddings for conversion from 30 fps to 240 fps.
7. Video Comparison
Video comparisons on SNU-FILM show that IFRNet faithfully reproduces motion boundaries and texture details. The gradually refined intermediate feature is identified as supporting this visual quality.
- Video Comparison: On SNU-FILM, IFRNet generates motion boundaries and texture details faithfully compared with other open-source VFI methods.
- Video Comparison: The reported visual quality is attributed to the powerful gradually refined intermediate feature.
8. Network Architecture
IFRNet uses a pyramid encoder and four coarse-to-fine decoders, with shared architectural details across its large and small variants.
- The architecture comprises a pyramid encoder E and coarse-to-fine decoders D4, D3, D2, and D1.
- The two input frames are encoded by the same Siamese pyramid encoder network.
- Figures 13–16 provide detailed diagrams of the bottom, middle, and top decoder components.
- The coarse-to-fine decoders use 64 channels in IFRNet large and 24 channels in IFRNet small at their third and fifth convolution layers.
9. Visualization and Discussion
Visualizations illustrate IFRNet’s flow distillation, geometry-regularized intermediate features, and shared processing of motion and synthesis components. The reported design is associated with smoother flow predictions and lightweight, fast inference.
- Flow distillation: Task-oriented flow distillation produces robustness masks whose darker regions approach 1 and brighter regions approach 0.The visualization shows ground-truth frames, pseudo-label and predicted intermediate flows, and the corresponding masks.
- Flow distillation: With task-oriented flow distillation loss, IFRNet’s intermediate flow prediction appears smoother and less artifact-prone than the pseudo-label flow prediction.
- Feature geometry: Geometry consistency loss regularizes the refined intermediate feature to preserve better structure layout.Figure 18 compares mean feature maps with and without the geometry consistency loss.
- Intermediate components: IFRNet visualizations show input frames, predicted flows, warped frames, a merge mask, a merged frame, and a residual used in frame interpolation.The residual is multiplied by 10 and shifted by 0.5 for visualization.
- Efficiency discussion: Sharing one encoder-decoder across merge mask, residual, and intermediate optical flow makes IFRNet more lightweight and fast than cascaded designs.
- Comparison with PWC-Net: IFRNet is distinguished from PWC-Net by reconstructing its anchor feature in the decoder and incorporating occlusion, texture, and temporal information.
- Middlebury rankings: Figures 20 and 21 show screenshots of IFRNet’s IE and NIE rankings on the Middlebury benchmark.
10. Screenshots of the Middlebury Benchmark
On the Middlebury benchmark, IFRNet large achieves the best reported IE and NIE results among published methods trained on Vimeo90K and runs several times faster than prior state-of-the-art algorithms.
- IFRNet large achieves the best IE and NIE results among published VFI methods trained on Vimeo90K.The comparison is summarized using average interpolation error and average normalized interpolation error.
- IFRNet large runs several times faster than previous state-of-the-art algorithms on the reported Middlebury benchmark comparison.