PaperlayerView exact revision on arXiv

Source-linked AI summary

AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation

Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, Ming-Ming Cheng

arXiv:2304.09790v1cs.CV

TL;DR

Video frame interpolation needs task-oriented flows that remain effective for large motions and occluded regions. AMT combines bidirectional all-pairs correlation with multi-field refinement to update flows and features and warp inputs through multiple flow groups. It achieves state-of-the-art performance across benchmarks with high efficiency, including comparable accuracy to a transformer-based method at nearly 23× lower computational cost.

  • Problem

    Task-oriented flow estimation for video frame interpolation remains difficult because large motions and occlusions challenge existing flow predictions.

  • Method

    AMT builds bidirectional all-pairs correlation volumes for joint flow-feature updates and derives multiple fine-grained flow groups from updated coarse bilateral flows.

  • Results

    AMT achieves state-of-the-art performance on various benchmarks with high efficiency and comparable performance to VFIFormer at nearly 23× less computational cost.

  • Takeaways & Limitations

    The convolution-based AMT architecture provides an efficient alternative for frame interpolation while handling large motions and occluded regions.

  • Takeaways & Limitations

    The correlation lookup uses a locally smooth motion assumption to scale estimated bilateral flows for coordinate alignment.

Abstract

from arXiv · show

We present All-Pairs Multi-Field Transforms (AMT), a new network architecture for video frame interpolation. It is based on two essential designs. First, we build bidirectional correlation volumes for all pairs of pixels, and use the predicted bilateral flows to retrieve correlations for updating both flows and the interpolated content feature. Second, we derive multiple groups of fine-grained flow fields from one pair of updated coarse flows for performing backward warping on the input frames separately. Combining these two designs enables us to generate promising task-oriented flows and reduce the difficulties in modeling large motions and handling occluded areas during frame interpolation. These qualities promote our model to achieve state-of-the-art performance on various benchmarks with high efficiency. Moreover, our convolution-based model competes favorably compared to Transformer-based models in terms of accuracy and efficiency. Our code is available at https://github.com/MCG-NKU/AMT.

1. Introduction

Video frame interpolation synthesizes intermediate frames to increase temporal resolution, but accurate task-oriented flows remain difficult under large motions and occlusions. AMT addresses these issues with bidirectional all-pairs correlations and multi-field refinement, achieving strong accuracy with reduced computational cost.

  • Video frame interpolation synthesizes intermediate frames from reference frames to increase temporal resolution and supports tasks such as slow-motion generation and video compression.
  • Existing flow-based methods face unresolved challenges in estimating task-oriented flows for large motions and occluded regions.Pretrained optical-flow models can produce flows unqualified for VFI usage, while end-to-end estimation still has major difficulties.
  • AMT uses bidirectional all-pairs correlation and retrieved correlations to jointly update bilateral flows and interpolated content features across scales.The design targets dense correspondence and improves flow fidelity across scales, especially for large motions.
  • AMT derives multiple fine-grained flow groups from updated coarse bilateral flows, providing diverse pixel-wise solutions that alleviate ambiguity in occluded areas.The input frames are separately backward warped using these flow groups.
  • +0.17dB PSNR: AMT-S outperforms IFRNet-B on Vimeo90K using 60% of its FLOPs and parameters.
  • AMT achieves comparable performance to transformer-based VFIFormer while requiring nearly 23× less computational cost.The comparison uses a large AMT model against VFIFormer.

2. Related Work

VFI research includes kernel-based, hallucination-based, and flow-based methods, with flow-based approaches becoming mainstream because of optical-flow robustness. AMT builds on cost-volume and flow-estimation ideas while combining coarse-scale flow consistency with fine-scale task-specific diversity.

  • VFI methods are commonly categorized as kernel-based, hallucination-based, or flow-based approaches.Kernel-based methods use dynamic kernels or offsets, while hallucination-based methods generate frames directly from input features.
  • Flow-based methods have become mainstream because of optical-flow robustness and use pretrained or jointly trained modules to estimate motion.
  • Backward and forward warping are standard ways to use estimated flows, while UNet-like architectures commonly synthesize the final frame.
  • AMT combines all-pairs correlation for coarse-scale flow consistency with multi-field refinement for fine-scale task-specific diversity.

3. Method

AMT is a one-stage flow-based interpolator that jointly updates bilateral flows and intermediate features using bidirectional all-pairs correlations, then refines the result with multiple flow groups. Its design targets large motions and occluded regions by combining coarse flow consistency with fine-grained, task-specific alternatives.

  • AMT synthesizes an intermediate frame from input frames I0 and I1 at target time t, with 0 < t < 1.
  • The encoder extracts features and initial bilateral flows, while multi-scale correlation volumes jointly update flows and intermediate features.A separate correlation encoder produces dense features at 1/8 input resolution; the context encoder provides the initial intermediate feature and bilateral flows.
  • Bidirectional correlation volumes represent asymmetric correspondences in both directions and are queried through estimated bilateral flows across correlation-pyramid levels.The volumes are built once, while scaled lookup addresses the coordinate mismatch between visible input frames and the invisible intermediate frame.
  • Retrieved correlations are concatenated with bilateral flows and intermediate features to predict flow residuals through efficient convolutional update blocks.The update blocks operate at low resolution after downscaling flows and features, then upsample predicted residuals for flow updates.
  • Multi-field refinement derives multiple fine-grained bilateral flow groups from updated coarse flows, separately warps the inputs, and merges candidate frames.Each group also predicts an occlusion mask and residual content; the final decoder enlarges output channels to produce the groups efficiently.

4. Experiments

AMT is evaluated across public benchmarks, ablations, and qualitative comparisons, showing strong accuracy and efficiency across model scales. Its all-pairs correlation and multi-field refinement designs improve flow modeling, content updates, and occlusion handling.

  • Benchmarks and setup: AMT is trained on Vimeo90K for 300 epochs using AdamW and evaluated with PSNR and SSIM across diverse benchmarks.The evaluation includes Vimeo90K, UCF101, and SNU-FILM settings, with inference latency measured on 1280 × 720 inputs using an NVIDIA RTX 3090 GPU.
  • Quantitative comparison: 0.17dB PSNR: AMT-S outperforms IFRNet-B on Vimeo90K while using about 60% of its parameters and FLOPs.The performance gap becomes more apparent on the Hard and Extreme SNU-FILM partitions, indicating strong large-motion modeling in those settings.
  • Quantitative comparison: AMT-G outperforms Transformer-based methods in most cases, achieves about 5× faster inference than VFIFormer, and uses half EMA-VFI’s parameters.VFIFormer uses two-stage training and 600 epochs, whereas AMT uses 300 epochs; AMT performance saturates after scaling except on Vimeo90K, suggesting possible overfitting.
  • Qualitative comparison: Qualitatively, AMT produces sharper motion boundaries, fewer artifacts, and more plausible reconstruction of objects occluded by foreground regions.These comparisons are reported on the Hard partition of SNU-FILM and include hallucination-based, kernel-based, and flow-based methods.
  • Ablation studies: Bidirectional correlation volumes outperform unidirectional and partial alternatives, while scaled lookup preserves the shared coordinate system needed for faithful flow updates.The ablations also show that correlation features from an exclusive correlation encoder are necessary, and cross-scale updates are effective.
  • Ablation studies: Three flow pairs provide a notable gain, performance improves through seven pairs before saturation, and multi-field refinement yields more consistent textures in occluded backgrounds.The smaller AMT-S uses three pairs for efficiency, whereas larger models use five pairs for better performance; adaptive fusion and per-candidate residual compensation are also beneficial.

5. Conclusion

AMT combines all-pairs correlation with multi-field refinement to handle large motions and occluded regions while achieving efficient state-of-the-art frame interpolation.

  • AMT introduces all-pairs correlation and multi-field refinement as its two essential designs.
  • The two designs enable AMT to handle large motions and occluded regions during frame interpolation.
  • AMT achieves state-of-the-art performance on various benchmarks with high efficiency.

A. Architecture Details

The implementation provides three AMT model scales and specifies shared architectural components and settings for reproducibility.

  • AMT is implemented in three model sizes: AMT-S, AMT-L, and AMT-G.Their architecture details are provided for reproducibility.
  • The correlation encoder uses standard residual blocks and instance normalization.
  • The lookup radius is set to 3, and upper-level update heads use bilinear upsampling.The upper levels are specified as l>1.

B. Multi-Frame Interpolation

For multi-frame interpolation, AMT synthesizes seven intermediate frames from two inputs at 8× interpolation and produces visually more consistent results with fewer artifacts and clearer edges.

  • AMT performs 8× interpolation by synthesizing 7 intermediate frames from two input frames.GoPro is used for training, with evaluation on GoPro and Adobe240 test data.
  • The quantitative comparison for 8× interpolation is reported in Table 3.
  • AMT produces more temporally consistent results than IFRNet-B on Adobe240, with fewer artifacts and clearer edges.The visual comparison examines the 1/4 and 1/2 time steps.

C. Limitation

AMT’s all-pairs 4D correlation volume limits adaptation to very high-resolution inputs in resource-constrained environments because construction scales quadratically with image resolution.

  • The 4D correlation volume makes AMT difficult to adapt to very high-resolution inputs under resource constraints.
  • Constructing the all-pairs correlation volume has computational complexity quadratic in image resolution.
  • Potential mitigations include computing correlations only when looked up or factorizing the 4D volume into two 3D volumes.

D. Discussions with RAFT

AMT extends RAFT’s correlation-based design for frame interpolation with bidirectional matching, scaled lookups, and multi-field flow refinement. These changes target large motions and occlusions while improving efficiency over methods that require multiple inference steps.

  • Discussion: AMT’s RAFT-like design is presented as underexplored for frame interpolation despite RAFT’s influence on modern optical-flow estimation.The discussion positions AMT as a task-specific extension rather than a direct reuse of RAFT.
  • Volume Design: AMT adapts RAFT’s correlation framework by modeling dense correspondences in both directions for frame interpolation.RAFT uses a unidirectional correlation volume for one-way optical flow, whereas AMT requires bilateral correspondences.
  • Correlation Lookup: Scaled bilateral-flow lookups address coordinate mismatch caused by the invisible intermediate frame.AMT also retrieves bidirectional correlations and initializes lookup from bilateral flows rather than zero.
  • Predict and Update Manner: AMT jointly updates bilateral flows and the interpolated content feature across scales instead of updating a single-resolution flow alone.Its update blocks use separate heads for content and flow residuals, with convolutional layers handling motion and content features.
  • Multi-Field Refinement: Multi-field refinement derives multiple flow pairs from updated coarse bilateral flows, enabling separate backward warping and diverse candidates in occluded regions.This differs from approaches that use forward warping or multiple inference steps to prepare intermediate candidates.

F. More Visual Results

Additional visual comparisons on Vimeo90K and SNU-FILM show AMT’s behavior across methods grouped by computational cost. AMT more faithfully reconstructs large-motion objects and produces plausible textures with fewer artifacts.

  • Benchmark Comparisons: AMT is compared with CAIN, AdaCoF, ABME, RIFE, IFRNet variants, and VFIFormer on Vimeo90K and SNU-FILM.The methods are divided into low- and relatively high-computational-cost groups for fair comparison.
  • Visual Quality: AMT synthesizes objects undergoing large motions more faithfully than the comparison methods.The visual results cover additional comparisons on both benchmark datasets.
  • Visual Quality: AMT generates plausible textures with fewer artifacts in the reported visual comparisons.This qualitative conclusion is stated for the additional Vimeo90K and SNU-FILM results.

G. Broader Impact

The supplementary material presents architecture details and qualitative comparisons for AMT variants across datasets and computational-cost settings. It also notes that faithful frame synthesis could enable video forgery or tampering.

  • Broader Impact: Faithful synthesis of non-existent frames may be abused to forge or tamper with videos.This is the paper’s stated broader-impact concern.
  • Qualitative Results: The AMT-G comparison with VFIFormer reports clearer recovered structure and edges for AMT.The figure caption identifies this as a qualitative comparison between the two methods.
  • Architecture Details: AMT-G and AMT-L architecture figures specify output channels, output-group count N, and use the IFRBlock decoder.The figures provide architecture details for the large and lightweight AMT variants.
  • Qualitative Results: Qualitative comparisons include AMT-S against IFRNet-B on Adobe240 at time steps 1/4 and 1/2.Additional visual comparisons cover Vimeo90K and SNU-FILM, including low- and relatively high-complexity settings.
Loading 2304.09790v1…