PaperlayerView exact revision on arXiv

Source-linked AI summary

FILM: Frame Interpolation for Large Motion

Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, Brian Curless

arXiv:2202.04901v4cs.CV

TL;DR

Large temporal spacing between near-duplicate photos creates large scene motion that challenges existing frame-interpolation methods. FILM addresses this with a unified, scale-agnostic network and Gram matrix loss, outperforming other methods on large motions while producing sharp frames and extending interpolation to near-duplicate photos.

  • Problem

    Near-duplicate photos can have second-or-more temporal spacing and large scene motion, posing a major challenge for existing frame-interpolation methods.

  • Method

    FILM uses a unified single-stage network with weight-shared multi-scale features, scale-agnostic bidirectional motion estimation, and Gram matrix loss for large disocclusions.

  • Results

    FILM outperforms other methods on large motions and significantly improves frame realism and sharpness in large disoccluded regions.

  • Takeaways & Limitations

    The approach extends frame interpolation from regular video frames to near-duplicate photos, generating sharp and appealing videos despite large motion.

  • Takeaways & Limitations

    Under extreme in-between motion, FILM can produce unnatural deformations even when the resulting videos remain appealing.

Abstract

from arXiv · show

We present a frame interpolation algorithm that synthesizes multiple intermediate frames from two input images with large in-between motion. Recent methods use multiple networks to estimate optical flow or depth and a separate network dedicated to frame synthesis. This is often complex and requires scarce optical flow or depth ground-truth. In this work, we present a single unified network, distinguished by a multi-scale feature extractor that shares weights at all scales, and is trainable from frames alone. To synthesize crisp and pleasing frames, we propose to optimize our network with the Gram matrix loss that measures the correlation difference between feature maps. Our approach outperforms state-of-the-art methods on the Xiph large motion benchmark. We also achieve higher scores on Vimeo-90K, Middlebury and UCF101, when comparing to methods that use perceptual losses. We study the effect of weight sharing and of training with datasets of increasing motion range. Finally, we demonstrate our model's effectiveness in synthesizing high quality and temporally coherent videos on a challenging near-duplicate photos dataset. Codes and pre-trained models are available at https://film-net.github.io.

1 Introduction

FILM targets frame interpolation with large motion, especially between near-duplicate photos, using a unified architecture trained from regular frames. It combines scale-agnostic motion estimation with Gram matrix optimization to improve generalization, sharpness, and training simplicity.

  • Motivation: Near-duplicate photos can be separated by a second or more, creating large scene motion that challenges methods designed for consecutive video frames.Interpolating these photos can produce engaging videos revealing scene and camera motion.
  • Evaluation: FILM is compared qualitatively with ABME on near-duplicate photos, where the cited figure reports large artifacts for ABME and improvements with FILM.The figure illustrates the method's intended advantage on the challenging photo-interpolation setting.
  • Method: FILM uses a shared-weight multi-scale feature extractor and a scale-agnostic bi-directional motion estimator to handle both small and large motion.The approach is trained on regular frames by treating large motion at finer scales as analogous to small motion at coarser scales.
  • Image quality: Gram matrix loss matches high-level VGG feature auto-correlation, improving the realism and sharpness of frames, especially in large disoccluded regions.The loss is intended to address blur arising where large motion reveals previously unseen content.
  • Scope: The work expands frame interpolation to near-duplicate photos, a setting identified as a novel application area.The paper presents this extension as a new space for interpolation research.
  • Training: FILM introduces a unified single-stage architecture that removes reliance on separately pretrained optical-flow, depth, or other prior networks.The model is trainable from regular frame triplets alone, avoiding scarce prior-network training data.

2 Related Work

Prior work addresses video interpolation, large motion, image quality, and occlusion handling with specialized or multiple networks. FILM instead uses shared multi-scale features and scale-agnostic motion estimation to generalize across motion ranges within a unified design.

  • Frame interpolation: Prior frame-interpolation methods primarily target video frame pairs with small motion, while near-duplicate photo interpolation had no prior work to the authors' knowledge.The paper positions near-duplicate photos as a distinct application with larger temporal spacing and motion.
  • Large motion: Extreme-motion training can weaken performance on regular footage, and other methods perform poorly when test motion differs from the training range.FILM addresses this range mismatch with shared multi-scale features and scale-agnostic motion estimation.
  • Image quality: Prior approaches improve image quality with per-pixel kernels, perceptual losses, or adversarial losses, but some struggle with large motion or require complex training.FILM adopts Gram matrix loss as an extension of perceptual-loss-based image-quality optimization.
  • Single-stage networks: Recent systems add depth, motion-estimation, or pretrained feature networks, making training complex and sometimes requiring scarce pretraining data.FILM introduces a single unified network to simplify this training setup.

3 Method

FILM is a unified, three-stage interpolator for large motion, combining shared multi-scale features, scale-agnostic bidirectional motion estimation, and fusion-based frame synthesis. Its training uses RGB, VGG, and Gram matrix losses to balance benchmark accuracy with sharpness.

  • Feature Extraction: Sharing convolution weights across scales addresses disappearing small objects and reduced supervision pixels in standard pyramids.The design treats large fine-scale motion as analogous to small coarse-scale motion, increasing pixels available for supervision.
  • Architecture: FILM has three stages: shared feature extraction, scale-agnostic motion estimation, and fusion that outputs the interpolated color image.The final decoder fuses warped features and bidirectional motions to synthesize the mid-frame.
  • Feature Extraction: FILM builds scale-agnostic feature pyramids by applying shared encoders at multiple pyramid levels and concatenating features with matching spatial dimensions.The same depth-specific convolution weights are reused across pyramid levels, producing compatible multiscale features.
  • Motion Estimation: At each pyramid level, the motion estimator predicts task-oriented bidirectional flows from the intermediate frame to the input images.Flow is recursively refined by adding a predicted residual to the upsampled estimate from the coarser level, with shared residual weights across levels 3–7.
  • Loss Functions: The loss combines pixel-wise L1 reconstruction, VGG feature differences, and Gram matrix differences between interpolated and ground-truth frames.The Gram matrix loss matches VGG-feature autocorrelation and is reported to improve sharpness and realism in large disoccluded regions.

4 Implementation Details

The model is implemented and trained in TensorFlow 2 using Vimeo-90K or large-motion datasets, with substantial augmentation and distributed GPU training. Two Vimeo-90K variants separate benchmark-focused L1 training from image-quality-focused style-loss training.

  • Training Setup: FILM is implemented in TensorFlow 2 and trained using either Vimeo-90K or the paper’s large-motion datasets.The training setup uses different dataset choices depending on the experiment.
  • Training Setup: Vimeo-90K training uses batch size 8, 256×256 random crops, eight NVIDIA V100 GPUs, augmentation, and Adam optimization.Augmentation includes rotations, horizontal flips, and reversing triplets.
  • Training Variants: Two Vimeo-90K models use either L1 loss for higher benchmark scores or the combined style loss for image quality.The style loss combines L1, VGG, and Gram losses.
  • Evaluation Implementation: SoftSplat was reimplemented in TensorFlow 2 for qualitative evaluation because pretrained models were unavailable at the time.The authors report that their renderings were comparable to those in the original paper.

5 Results

FILM achieves strong quantitative and qualitative results across small-to-medium and large-motion benchmarks, while its ablations show benefits from perceptual training, weight sharing, and motion-matched data. It also offers faster inference with modest memory overhead, but can produce unnatural deformations under extreme motion.

  • Small-to-Medium Motion: FILM outperforms SoftSplat when perceptually-sensitive losses are used on Vimeo-90K, Middlebury, and UCF101.The paper reports the highest scores on Middlebury and UCF101 and outperformance of SoftSplat on Vimeo-90K.
  • Large Motion: FILM outperforms competing models on large-motion benchmarks, including both PSNR and SSIM against SoftSplat-LF on Xiph-4K.All methods in the comparison are trained on Vimeo-90K; custom large-motion training adds +0.5dB on Xiph-4K.
  • Ablations: Weight sharing improves PSNR, generalization beyond the training motion range, and practical memory fit, while the highest-quality unshared model could not be trained stably.The ablation attributes stronger visual generalization to shared features.
  • Qualitative Comparisons: The Gram Matrix-based loss significantly improves FILM’s frame sharpness and reduces blurry artifacts compared with L1 and combined L1/VGG training.Qualitative comparisons also report crisp facial details, preserved fingers, and more faithful large-disocclusion inpainting.
  • Ablations: Training-data motion range matters: Xiph-4K performs best with 0-100 pixels, whereas larger ranges reduce PSNR on Vimeo-90K.The best model trained with the Xiph-matched range gains +0.5dB over the state of the art.
  • Efficiency: FILM is 3.95× faster than ABME and 9.75× faster than SoftSplat, using 1.27× and 1.01× more memory, respectively.These averages come from 100 inference runs on an NVIDIA V100 GPU.
  • Limitations: Under extreme in-between motion, FILM can produce unnatural deformations despite appealing resulting videos.The authors provide failure examples in supplementary video material.

6 Conclusions

FILM is a unified, single-stage frame interpolation model designed for large-motion near-duplicate photos while retaining performance on small motion.

  • 6 Conclusions: FILM interpolates near-duplicate photos using a unified single-stage model trained from regular frames without optical-flow or depth prior networks.Its core design shares feature-pyramid weights across scales and uses a scale-agnostic bidirectional motion estimator.
  • 6 Conclusions: The Gram matrix loss helps FILM handle wide disocclusions from large motion while generating sharp frames.The loss matches feature correlations during optimization.
  • 6 Conclusions: FILM outperforms other methods on large motions, handles small motions well, and generates high-quality temporally smooth videos.

Supplementary Materials FILM: Frame Interpolation for Large Motion

The supplementary materials provide implementation details, a supplementary video, and additional visual results for FILM.

  • Supplementary Materials FILM: Frame Interpolation for Large Motion: The paper's authors are affiliated with Google Research and the University of Washington.
  • Supplementary Materials FILM: Frame Interpolation for Large Motion: The supplementary materials include additional implementation details for FILM.
  • Supplementary Materials FILM: Frame Interpolation for Large Motion: A supplementary video presents a quick overview of the paper's motivations and more visual results.

A.1 Loss Combination Weights

The style loss combines L1, VGG, and Gram losses with different weights during two successive 1.5-million-iteration training phases.

  • A.1 Loss Combination Weights: The style loss optimally combines L1, VGG, and Gram losses using a piecewise linear weight schedule.
  • A.1 Loss Combination Weights: For the first 1.5M iterations, the loss weights are (1.0, 1.0, 0.0) for L1, VGG, and Gram losses.
  • A.1 Loss Combination Weights: For the final 1.5M iterations, the weights are (1.0, 0.25, 40.0), selected so each loss contributes equally to the combined style loss.

A.2 SoftSplat Implementation

The supplementary implementation reproduces SoftSplat benchmark scores and produces renderings comparable to those in the original paper.

  • A.2 SoftSplat Implementation: The authors implemented SoftSplat in TensorFlow 2 using the original authors' tuned hyperparameters.
  • A.2 SoftSplat Implementation: The TensorFlow 2 implementation replicated SoftSplat's published benchmark scores.
  • A.2 SoftSplat Implementation: On a DAVIS image sample, the implementation's renderings were comparable to those provided in the original paper.

B Supplementary Video

The supplementary materials provide a video with method motivations, illustrations, visual results, and failure samples, alongside a DAVIS flamingo interpolation comparison. In that example, FILM-LS produces a crisp frame, while competing outputs show color distortions or transparencies.

  • The supplementary video includes motivations, method illustrations, visual results, and failure samples.
  • In a DAVIS walking-flamingo example, FILM-LS produces a crisp frame compared with the shown interpolation methods.The comparison includes SoftSplat-L1, SoftSplat-LF, ABME, FILM-L1, and FILM-LS.
  • The comparison shows color distortions in ABME and transparencies in SoftSplat renderings.
Loading 2202.04901v4…