PaperlayerView exact revision on arXiv

Source-linked AI summary

CDFI: Compression-Driven Network Design for Frame Interpolation

Tianyu Ding, Luming Liang, Zhihui Zhu, Ilya Zharkov

arXiv:2103.10559v2cs.CV

TL;DR

DNN-based frame interpolation often uses models too large for resource-limited devices. CDFI prunes AdaCoF and adds multi-resolution warping to improve the compact model. The resulting model outperforms AdaCoF on three benchmark datasets while using a quarter of its initial size.

  • Problem

    DNN-based frame interpolation models are often large, limiting their suitability for resource-limited systems such as mobile devices.

    Source: §1 · p. 2

  • Method

    CDFI uses sparsity-inducing pruning to compress AdaCoF, then adds a multi-resolution warping module to improve the compact model.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    More than 1 dB of PSNR on the Middlebury dataset: the final model outperforms AdaCoF on three benchmark datasets while using a quarter of its initial size.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The compression-driven framework is described as generic and transferable to other DNN-based frame interpolation algorithms.

    Source: §5 · p. 8

Abstract

from arXiv · show

DNN-based frame interpolation--that generates the intermediate frames given two consecutive frames--typically relies on heavy model architectures with a huge number of features, preventing them from being deployed on systems with limited resources, e.g., mobile devices. We propose a compression-driven network design for frame interpolation (CDFI), that leverages model pruning through sparsity-inducing optimization to significantly reduce the model size while achieving superior performance. Concretely, we first compress the recently proposed AdaCoF model and show that a 10X compressed AdaCoF performs similarly as its original counterpart; then we further improve this compressed model by introducing a multi-resolution warping module, which boosts visual consistencies with multi-level details. As a consequence, we achieve a significant performance gain with only a quarter in size compared with the original AdaCoF. Moreover, our model performs favorably against other state-of-the-arts in a broad range of datasets. Finally, the proposed compression-driven framework is generic and can be easily transferred to other DNN-based frame interpolation algorithm. Our source code is available at https://github.com/tding1/CDFI.

1. Introduction

DNN-based frame interpolation models are often too large for resource-limited deployment. CDFI compresses AdaCoF, then improves the compact model with multi-resolution warping, reporting stronger benchmark performance at smaller size.

1. Introduction

  • Many DNN-based interpolation models exceed 20 million parameters; MEMC-Net [3] has more than 70 million and requires around 280 megabytes in 32-bit floating point.Large models are described as difficult to train, inefficient at inference, and unlikely to be deployable on mobile devices.Source: §1 · p. 2
  • A 10× compressed AdaCoF maintains similar benchmark performance, and a multi-resolution warping module uses a feature pyramid to help image synthesis.The feature pyramid addresses occlusion and preservation of finer details, which AdaCoF handles less well than large motion.Source: §1 · p. 2
  • More than 1 dB of PSNR on the Middlebury dataset: the final model outperforms AdaCoF on three benchmark datasets while using a quarter of its initial size.It also performs favorably against other state-of-the-art methods.Source: §1 · p. 2
  • CDFI’s compression-driven design is presented as transferable to other DNN-based frame interpolation algorithms.The framework first compresses AdaCoF and then improves the compact model.Source: §1 · p. 2

2. Related work

Prior interpolation methods use flow, local kernels, or combinations of both, with limitations in complex scenes, large motion, or model size. This work uses AdaCoF as a baseline and treats pruning-based compression as a tool for understanding and redesigning the architecture.

2.1. Video frame interpolation

  • Flow-based methods use pixel-wise information, while kernel-based methods [43] [44] convolve local patches and cannot handle motion beyond the kernel region.Hybrid methods [2] [3] combine flow and local kernels; [2] uses depth information to detect occlusion.Source: §2.1 · p. 3
  • AdaCoF [32] uses spatially-adaptive separable deformable convolution to synthesize output pixels; this work explores over-parameterization and compression to build a smaller, stronger model.The authors describe the compression-driven design as transferable to other DNN-based frame interpolation algorithms.Source: §2.1 · p. 3

2.2. Pruning-based model compression

  • The paper adopts optimization-based pruning to induce sparse connections, using compression to understand the architecture and make room for further improvements.Its three steps are training with an ℓ1 sparsity constraint, reformulating a small dense network from the identified sparse structure, and retraining it.Source: §2.2 · p. 3
  • Figure 3 visualizes the difference between an interpolation and the ground-truth image.The figure caption specifies the comparison but does not state an outcome.Source: Figure 3 · p. 3

3. The proposed approach

CDFI first compresses AdaCoF through sparsity-inducing pruning, then adds feature-space warping and synthesis components to address limitations in handling occlusion and contextual details. The approach trains the network using Vimeo-90K data and evaluates it on Vimeo-90K, Middlebury, and UCF101.

3.1. Motivation

  • AdaCoF struggles with severe occlusion and finer details, while its 21.8-million-parameter encoder-decoder is cumbersome to improve directly.The cited configuration is F = 5, d = 1 and requires 83.4 megabytes when stored with PyTorch.Source: §3.1 · p. 4

3.2. First stage: compression of the baseline

  • Fine-grained ℓ1-regularized pruning promotes unstructured zeros; layer density then guides redesign of a smaller dense network trained from scratch.The orthant-based stochastic solver is used to solve the sparsity-regularized problem.Source: §3.2 · pp. 4–5
  • 35.46 PSNR on Middlebury after retraining, with 10× compressed AdaCoF performing on par with the original model.The compressed model’s training was about 5× faster; Table 1 compares model statistics.Source: §3.2 · p. 5, Table 1 · p. 5

3.3. Second stage: improve upon the compression

  • Feature pyramids are warped at multiple scales and synthesized into an RGB candidate, while path selection blends it with AdaCoF’s warped-frame candidate.The design adds a second occlusion module to synthesize the final result from the two candidates.Source: §3.3 · pp. 5–6

3.4. Training

  • Training uses Vimeo-90K triplets with flips and temporal-order perturbations; evaluation uses Vimeo-90K, Middlebury, and UCF101 with PSNR, SSIM, and LPIPS.The architecture is trained with AdaMax for up to 100 epochs.Source: §3.4 · pp. 6–7

4. Experiments

The experiments show that a 10× compressed AdaCoF retains similar performance, while feature-pyramid and path-selection additions improve results. CDFI compares favorably with other methods and remains compact.

4.1. Ablation study

  • A 10× compressed AdaCoF sacrifices little performance across three benchmark datasets and different settings of (F, d).The authors interpret this as evidence of redundancy in AdaCoF.Source: §4.1 · p. 7
  • Adding the feature-pyramid module to compressed AdaCoF improves PSNR, SSIM, and LPIPS on Vimeo-90K and Middlebury, and LPIPS on all three benchmarks.Filtering multi-scale feature maps with 1-by-1 convolutions further improves PSNR and SSIM while slightly reducing model size.Source: §4.1 · p. 7
  • Path selection helps the completed model handle large motion that the feature-pyramid-only model struggles to capture, while retaining fine details.The examples include sharper ball edges and a preserved finger shape compared with AdaCoF.Source: §4.1 · p. 7

4.2. Quantitative evaluation

  • CDFI performs favorably against compared methods on SSIM and LPIPS and on PSNR except for SoftSplat.The authors also report that the model is significantly smaller than competitors.Source: §4.2 · pp. 7–8

4.3. Qualitative evaluation

  • On DAVIS 2016, CDFI produces sharper, more realistic interpolations with clearer details than AdaCoF+ and the compared recent methods.BMBC occasionally produces sharp results but is described as less consistent; the authors identify CDFI as the smallest model among them.Source: §4.3 · p. 8, Figure 7 · p. 8

5. Conclusions

CDFI uses compression to guide efficient frame-interpolation architecture design, then improves on the compressed model. The conclusion presents this approach as applicable beyond AdaCoF and identifies iterative refinement of compression and design as future work.

  • CDFI uses model compression as a guide for determining an efficient frame-interpolation architecture, then improves upon it.Source: §5 · p. 8
  • A much smaller AdaCoF performs similarly to the original, and simple modifications let it outperform the baseline and other state-of-the-art methods.Source: §5 · p. 8
  • Because optimization-based compression does not depend on a baseline's particular design, the authors consider CDFI generic to other models.Source: §5 · p. 8
  • Future work could better associate compression and design stages so they iteratively refine the underlying architecture.Source: §5 · p. 8

Related papers

Loading 2103.10559v2…