PaperlayerView exact revision on arXiv

Source-linked AI summary

XVFI: eXtreme Video Frame Interpolation

Hyeonjun Sim, Jihyong Oh, Munchurl Kim

arXiv:2103.16206v2cs.CV

TL;DR

VFI lacks high-resolution, high-frame-rate data and methods capable of handling extreme motion. The paper introduces X4K1000FPS and XVFI-Net, which combines scalable recursive multi-scale processing with complementary flow reversal. XVFI-Net achieves state-of-the-art performance on the new high-resolution benchmark and remains robust on a lower-resolution benchmark, subject to the dataset’s single-camera collection.

  • Problem

    Existing VFI datasets lack rich 4K high-frame-rate videos, while extreme motion and occlusions make high-resolution interpolation challenging.

  • Method

    The paper constructs X4K1000FPS and proposes XVFI-Net, a recursive multi-scale shared network with BiOF-I, BiOF-T, and complementary flow reversal.

  • Results

    XVFI-Net achieves state-of-the-art performance on X4K1000FPS with a significant margin over previous VFI methods and performs robustly on lower-resolution benchmarks.

  • Takeaways & Limitations

    The dataset and scalable framework extend VFI evaluation and modeling toward 4K video with extremely large motions.

  • Takeaways & Limitations

    X4K1000FPS was obtained using one single camera, although the authors consider the resulting extreme HFR 4K dataset valuable because such cameras are few.

Abstract

from arXiv · show

In this paper, we firstly present a dataset (X4K1000FPS) of 4K videos of 1000 fps with the extreme motion to the research community for video frame interpolation (VFI), and propose an extreme VFI network, called XVFI-Net, that first handles the VFI for 4K videos with large motion. The XVFI-Net is based on a recursive multi-scale shared structure that consists of two cascaded modules for bidirectional optical flow learning between two input frames (BiOF-I) and for bidirectional optical flow learning from target to input frames (BiOF-T). The optical flows are stably approximated by a complementary flow reversal (CFR) proposed in BiOF-T module. During inference, the BiOF-I module can start at any scale of input while the BiOF-T module only operates at the original input scale so that the inference can be accelerated while maintaining highly accurate VFI performance. Extensive experimental results show that our XVFI-Net can successfully capture the essential information of objects with extremely large motions and complex textures while the state-of-the-art methods exhibit poor performance. Furthermore, our XVFI-Net framework also performs comparably on the previous lower resolution benchmark dataset, which shows a robustness of our algorithm as well. All source codes, pre-trained models, and proposed X4K1000FPS datasets are publicly available at https://github.com/JihyongOh/XVFI.

1. Introduction

The paper addresses VFI under extreme motion and high-resolution settings by introducing the X4K1000FPS dataset and XVFI-Net. The proposed framework combines scalable multi-scale inference with complementary flow reversal and achieves strong performance on the new benchmark.

  • Existing VFI methods face challenges from occlusions, large motions, and illumination changes, especially for high-resolution fast-motion video.VFI synthesizes intermediate frames to increase frame rate and reduce motion judder.
  • X4K1000FPS provides 4K high-frame-rate videos containing extremely large motions and occlusions for extreme VFI research.The dataset was directly photographed to address the lack of suitable high-resolution, high-frame-rate data.
  • XVFI-Net uses a recursive multi-scale shared structure with cascaded BiOF-I and BiOF-T modules for bidirectional optical-flow learning.BiOF-I learns flow between input frames, while BiOF-T estimates flow from the target time to the inputs.
  • XVFI-Net can begin inference from different down-scaled inputs and adjust its number of scales according to input resolution or motion magnitude.This scalability supports computationally efficient inference.
  • Complementary flow reversal generates stable optical-flow estimates from the target time to the input frames.The method is designed to improve qualitative and quantitative VFI performance.
  • XVFI-Net achieves state-of-the-art performance on X4K1000FPS with a significant margin over previous VFI methods while using relatively few filter parameters.Source code, pretrained models, and the dataset are publicly available.

2. Related Work

Prior VFI approaches use optical flow, kernels, hallucination, or multi-scale structures, but often struggle with high-resolution computation, fixed scale capacity, or arbitrary target times. XVFI-Net addresses these constraints through scalable shared parameters and separate input-to-input and target-to-input flow modules.

  • VFI methods include optical-flow-based, kernel-based, and pixel-hallucination-based approaches.Flow-based methods estimate motion or target-to-input flows, while other methods synthesize pixels directly.
  • Many existing methods cannot efficiently process entire high-resolution frames because of heavy computational complexity.XVFI-Net is designed to operate on entire 4K inputs with fewer parameters while capturing large motions.
  • Fixed, independently parameterized pyramid levels limit the adaptability of conventional multi-scale methods to different displacement sizes.RRPN shares parameters recurrently but only predicts centered frames and accumulates errors when generating arbitrary intermediate times recursively.
  • XVFI-Net uses shared parameters across scales and separates BiOF-I from BiOF-T to support arbitrary target times efficiently.BiOF-T can be skipped at down-scaled inference levels, reducing computation while retaining target-time flexibility.

3. Proposed X4K1000FPS Dataset

The paper introduces X4K1000FPS because existing VFI datasets lack substantial 4K high-frame-rate video with rich extreme motion. Its selected training and test splits emphasize occlusion, motion magnitude, and scene diversity, and exhibit larger motion than prior benchmarks.

  • Existing VFI datasets do not contain rich amounts of 4K high-frame-rate video, limiting evaluation for very high-resolution applications.This motivates a dedicated extreme VFI dataset.
  • X4K1000FPS was photographed at 4096×2160 resolution and 1,000 fps, producing 175 scenes with 5,000 frames each.The dataset was captured using a Phantom Flex4K camera over five seconds per scene.
  • Occlusion maps and optical flows were estimated to guide selection of challenging samples.Occlusion makes optical-flow estimation and frame interpolation difficult.
  • The dataset statistics compare occlusion and optical-flow magnitudes across Vimeo90K, Adobe240fps, X-TEST, and X-TRAIN.The table includes 3,782 Vimeo90K test triplets, 200 Adobe240fps clips, 15 X-TEST clips, and 4,408 X-TRAIN clips.
  • X-TEST contains 15 manually selected scenes chosen for occlusion, optical-flow magnitude, and scene diversity.Each test sample uses input frames separated by 32 frames and interpolates seven intermediate frames.
  • X-TEST and X-TRAIN contain comparable occlusion but significantly larger motion than previous VFI datasets.This establishes the proposed data as a more demanding benchmark for extreme motion.

4. Proposed Method : XVFI-Net Framework

XVFI-Net uses a recursive, shared multi-scale architecture to estimate bidirectional input and target-to-input optical flows for extreme-motion interpolation. Its scale-adaptive inference can begin at different input resolutions while refining target-time flows only at the original scale.

  • BiOF-I module: XVFI-Net recursively extracts contextual features across scales and estimates bidirectional flows between the two input frames, passing them from coarser to finer levels.At each scale, flows are upscaled and used to initialize the next finer scale.
  • Frame reconstruction: Warped input and contextual features, estimated flows, an occlusion mask, and a residual image are combined to reconstruct the intermediate frame at each scale.The reconstruction uses the occlusion mask to blend warped frames with a residual image.
  • BiOF-T module: The BiOF-T module combines negative anchor flows with complementary flows to stabilize target-to-input optical flow estimation.Complementary normalized flows fill holes produced by flow reversal, while normalized anchor flows represent the intermediate time t.
  • Adjustable scalability: The shared multi-scale structure supports adjustable inference depth, allowing XVFI-Net to start from any down-scaled input and adapt the number of scales to resolution or motion magnitude.Multi-scale reconstruction loss is applied across the selected training scale depth to generalize inference from different starting levels.
  • Efficient scalability: During inference, BiOF-I operates recursively across selected scales, while BiOF-T runs only at the original scale to reduce computation.Only the bidirectional input-frame flows are passed between scale levels before target-time flows are refined at s = 0.
  • Training objectives: Training combines multi-scale reconstruction and edge-aware optical-flow smoothness losses, with the total loss weighted as Ltotal = Lr + λs · Ls.The edge weighting factor e2 and spatial coordinate x are explicitly defined in the loss formulation.

5. Experiment Results

Experiments evaluate XVFI-Net on extreme 4K interpolation and compare its flow approximation, scalability, visual quality, and efficiency with prior methods. XVFI-Net achieves strong results on X-TEST while retaining robustness on lower-resolution benchmarks.

  • Experimental Setup: X-TRAIN contains 4,408 clips of 768×768 resolution and 65 consecutive frames, with training triplets sampled at temporal distances from 2 to 32 frames.Samples use two input frames and one intermediate target frame, with target time randomly selected between the inputs.
  • Comparison to Previous Methods: XVFI-Net is compared with DAIN, FeFlow, and AdaCoF using retrained models on X-TRAIN and original pretrained models, with multi-frame interpolation evaluated on X-TEST.The evaluation interpolates seven frames per scene for ×8 interpolation.
  • Comparison to Previous Methods: 2.6dB, 0.049, and 1.32 are the XVFI-Net (Stst=5) gains over DAINf in PSNR, SSIM, and tOF, respectively, while using 22.9% of DAIN’s parameters.XVFI-Net infers intermediate 4K frames at once without patch-wise iteration.
  • Comparison to Previous Methods: XVFI-Net captures complex structures under extremely fast motion at t = 0.5, whereas previous methods fail on the same X-TEST examples.FeFlow is described as computationally heavy for 4K inputs and inappropriate for large-motion feature alignment, producing blurry outputs.
  • Ablation Studies: CFR produces the lowest flow-approximation EPE because bidirectional flows complement each other and fill holes left by flow reversal.Linear approximation causes anchor-frame misalignment, while flow reversal introduces holes that are not projected from any flow vector.
  • Ablation Studies: Increasing Stst generally improves X-TEST performance by enlarging receptive fields and refining flows, while the smaller Stst=3 performs better on lower-resolution Adobe240fps.On Vimeo90K, XVFI-Netv uses 5.5 million parameters and runs about ×7 faster than FeFlow while achieving lower performance.

6. Conclusion

The paper introduces X4K1000FPS and XVFI-Net for extreme high-resolution video frame interpolation, reporting strong performance on high-resolution datasets and robustness on a lower-resolution benchmark. The dataset uses one camera but remains valuable because extreme high-frame-rate 4K cameras are rare.

  • X4K1000FPS is a high-quality high-frame-rate dataset with 4K resolution and a wide range of motions.
  • XVFI-Net handles large pixel displacements with inference scalability adjustable to input resolution or motion magnitude.
  • XVFI-Net achieves state-of-the-art performance on high-resolution datasets compared with previous methods and shows robustness on a lower-resolution benchmark.
  • The dataset was obtained with one camera, while its extreme high-frame-rate 4K setting remains valuable because such cameras are scarce.

Appendices

The dataset collection rules were designed to capture diverse object and camera motions across speeds and locations.

  • The shooting rules include independently moving objects with a stationary camera and videos captured from a moving car.
  • Additional recordings involve walking, irregular camera trajectories at non-uniform speeds, and zooming or panning.

A.2. Test Dataset: X-TEST

The X-TEST construction selects consecutive frames using occlusion, optical-flow magnitude, and scene diversity, then forms nonuplets at an approximately 960 fps rate.

  • Thirty-two consecutive frames are manually selected for each test scene based on occlusion, optical-flow magnitudes, and diversity among 5,000 frames.
  • Nonuplets are composed by sampling every 4 frames from the selected 32-frame sequences.
  • The test videos are approximated at 960 fps, calculated as 32×30 fps, rather than exactly 1,000 fps.

A.3. Train Dataset: X-TRAIN

The X-TRAIN dataset uses occlusion-guided sampling and overlapping spatial-temporal crops, retaining diverse samples with high bidirectional occlusion.

  • Training samples are selected using occlusion maps estimated by IRR-PWC on spatially and temporally down-sampled frames.
  • Each 4K frame is divided into overlapping 768 × 768 patches arranged in an 81 × 31 grid, excluding boundary patches affected by large occlusion values under translation.
  • About 386K candidate samples per scene are generated from 81 × 31 spatial patches and 154 temporal clips of 65 frames.
  • Only candidates in the top 10% of bidirectional occlusion values are retained, followed by sparse selection of 4,408 training samples to preserve diversity.

B. Details of Architecture of XVFI-Net

XVFI-Net uses shared multi-scale sub-networks with separate flow-estimation and refinement components, while inference selectively skips BiOF-T at reduced scales to improve efficiency. Qualitative results show this design handles complex extreme motion and fine structures better than prior methods.

  • Architecture: The detailed architectures describe sequential layer operations, channel-wise tensor concatenation, residual blocks, and convolution filters selected by stride.A residual block consists of conv2d, ReLU, conv2d, and identity addition.
  • Architecture: XVFI-Net uses feature extraction, auto-encoder-based flow estimation, and U-Net-based refinement blocks within a recursive multi-scale architecture.Parameters are shared across scale levels except for the BiFlownet at the lowest scale depth.
  • Inference efficiency: During inference, BiOF-T can be skipped at down-scaled levels while BiOF-I continues recursively, accelerating 4K processing by about 22% versus full recursion.The added runtime of reduced-scale BiOF-I processing is negligible relative to original-scale BiOF-I and BiOF-T processing.
  • Qualitative behavior: XVFI-Net captures complex propeller motion and fine structures such as electric wires and car edges where prior methods produce interpolation failures or structural distortions.The examples include zoom-out, fast-car, and hand-shaken scenes with large pixel displacements.

C. Additional Qualitative Results

Additional experiments visualize XVFI-Net components and compare VFI methods across 4K, HD, and Vimeo90K results. The visualizations show progressively finer flows and support the role of coarse-to-fine refinement in final interpolation.

  • Visual comparisons: Additional qualitative comparisons cover X-TEST at 4K, Adobe240fps at HD resolution, and Vimeo90K.The comparisons include each setting described in the main paper.
  • Component visualization: XVFI-Net’s visualization shows finer estimated flows at upper levels than at lower levels.The component visualization includes optical flows and occlusion masks.
  • Component visualization: The coarse-to-fine structure improves original-scale interpolation using occlusion masks and iteratively updated flows learned from scratch.These flows and masks support the final VFI result at scale s = 0.

D. Failure Cases

The failure analysis identifies challenging 4K scenes where all compared methods, including XVFI-Net, fail on some patches. Difficult cases involve extreme camera motion, tiny structures, rapid rotation, blur, and abrupt appearance changes.

  • Failure cases: Even XVFI-Net fails on some patches in the extremely challenging X-TEST scenes with very large optical-flow magnitudes.The reported flow magnitude mean is 196.5 in scenes involving camera shaking and fast-moving cars.
  • Failure cases: Tiny electric lines that blend into static backgrounds are difficult for all compared methods to interpolate accurately.The failure analysis identifies this as a specific cropped failure patch.
  • Failure cases: Fast-moving car-wheel rotations, input blurriness, and abrupt brightness or color changes also challenge every evaluated VFI method.These factors are described as additional sources of difficulty in the failure examples.
Loading 2103.16206v2…