PaperlayerView exact revision on arXiv

Source-linked AI summary

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

Duolikun Danier, Fan Zhang, David Bull

arXiv:2303.09508v3eess.IVcs.CV

TL;DR

Conventional VFI objectives can poorly reflect perceptual quality, motivating alternatives to pixel- and feature-distance training. LDMVFI treats interpolation as conditional generation with latent diffusion and reports favorable perceptual performance against the state of the art, including at high resolution.

  • Problem

    Common VFI methods optimize pixel or feature distances that correlate poorly with perceptual quality, leaving a gap between these objectives and human judgments.

    Source: §1 · p. 1

  • Method

    LDMVFI formulates VFI as conditional generation with latent diffusion and adds VFI-specific components, including the VQ-FIGAN autoencoding model.

    Source: §1 · p. 1

  • Results

    LDMVFI performs favorably against the state of the art on three perceptual metrics, and a user study supports its perceptual performance on full HD videos.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported evaluations demonstrate the potential of latent diffusion for perception-oriented VFI across test sets covering content up to 4K.

    Source: §6 · p. 7

    LDMVFI has relatively slow inference due mainly to iterative denoising and a large parameter count associated with its adapted generic image-generation U-Net.

    Source: §5.1 · p. 6

Abstract

from arXiv · show

Existing works on video frame interpolation (VFI) mostly employ deep neural networks that are trained by minimizing the L1, L2, or deep feature space distance (e.g. VGG loss) between their outputs and ground-truth frames. However, recent works have shown that these metrics are poor indicators of perceptual VFI quality. Towards developing perceptually-oriented VFI methods, in this work we propose latent diffusion model-based VFI, LDMVFI. This approaches the VFI problem from a generative perspective by formulating it as a conditional generation problem. As the first effort to address VFI using latent diffusion models, we rigorously benchmark our method on common test sets used in the existing VFI literature. Our quantitative experiments and user study indicate that LDMVFI is able to interpolate video content with favorable perceptual quality compared to the state of the art, even in the high-resolution regime. Our code is available at https://github.com/danier97/LDMVFI.

1 Introduction

VFI methods often optimize pixel or feature distances that poorly reflect perceptual quality, particularly for dynamic textures and complex motion. LDMVFI recasts interpolation as conditional generation with latent diffusion and reports favorable perceptual results against the state of the art.

1 Introduction

  • Pixel- and feature-distance objectives correlate poorly with human judgments of VFI quality, and existing methods can underperform perceptually on dynamic textures with complex motion.VFI generates intermediate frames between consecutive video frames and is used in applications including frame-rate increase, compression, view synthesis, medical imaging, and animation.Source: §1 · p. 1
  • LDMVFI formulates VFI as conditional image generation using latent diffusion, with an autoencoder and a denoising U-Net operating in latent space.The paper introduces VQ-FIGAN, a vector-quantization-based VFI autoencoder, as a task-specific component.Source: §1 · pp. 1–2
  • LDMVFI is reported to perform favorably against the state of the art on three perceptual metrics, with a user study also supporting its perceptual performance.The evaluations span test sets with content up to 4K; the user study compares full HD interpolations with several competitive methods.Source: §1 · p. 2
  • The paper presents LDMVFI as the first use of latent diffusion models to address VFI as conditional generation and demonstrate its potential for perception-oriented interpolation.Its stated contributions include VFI-specific latent-diffusion components and quantitative and qualitative comparisons with the state of the art.Source: §1 · p. 2

2 Relate Work

Prior VFI work is organized around flow-based, kernel-based, and hybrid or end-to-end approaches, while most methods train against pixel or feature distances. Diffusion models offer a generative alternative, but prior VFI use was limited and latent diffusion had not been applied to VFI.

Video Frame Interpolation

  • Flow-based VFI estimates intermediate motion, while kernel-based methods predict adaptive kernels; other approaches combine flows and kernels or synthesize frames end to end.The cited flow-based methods either infer intermediate flows from input-frame flows or estimate them directly; kernel-based methods range from fixed-size to deformable kernels.Source: §2 · p. 2
  • Most existing VFI methods optimize L1/L2 or VGG feature losses, which have been reported to correlate poorly with subjective quality, especially for dynamic textures with complex motion.Some methods combine VGG feature loss with L1 loss, but the passage reports that these distances do not fully reflect perceptual quality.Source: §2 · p. 2

Diffusion Models

  • Diffusion models had been used for VFI in one cited low-resolution approach, while latent diffusion had not previously been exploited for VFI.The prior VFI diffusion model is described as lacking VFI-specific innovations and showing limited interpolation performance.Source: §2 · p. 2

3 Proposed Method: LDMVFI

LDMVFI models interpolation as conditional generation: it encodes frames into a compact latent space and denoises a target-frame representation conditioned on the two input frames. Its VQ-FIGAN autoencoder is adapted for VFI with neighboring-frame features, efficient attention, and deformable-kernel synthesis.

3 Proposed Method: LDMVFI

  • LDMVFI learns a conditional distribution for the intermediate frame from two consecutive inputs, using VQ-FIGAN to encode and decode frames and a denoising U-Net for latent-space reverse diffusion.The decoder reconstructs the target using feature pyramids extracted from the two input frames.Source: §3 · p. 2

3.1 Latent Diffusion Models

  • Latent diffusion encodes images into lower-dimensional representations and performs forward and reverse diffusion in latent space, concentrating computation on a compact representation.The passage states that compact latent projection supports more computationally efficient sampling.Source: §3.1 · p. 3

3.2 Autoencoding with VQ-FIGAN

  • VQ-FIGAN uses neighboring-frame feature pyramids in MaxViT-based cross-attention during decoding, replacing full self-attention with more efficient MaxViT blocks.Its decoder predicts adaptive deformable-convolution kernels, spatial offsets, a visibility map, and a residual; the resulting per-frame samples are combined using visibility and residual maps.Source: §3.2 · pp. 3–4

3.3 Conditional Generation with LDM

  • The denoising U-Net learns to predict noise added to the target-frame latent, conditioned on the input-frame latents, and inference iteratively denoises sampled latent noise before decoding.The network takes the noisy target latent and diffusion step as input alongside the two conditioning latents.Source: §3.3 · p. 4

4 Experimental setup

The experiments train LDMVFI on Vimeo90k and BVI-DVC and evaluate it on VFI benchmarks spanning low resolutions to 4K. Perceptual metrics and a human study assess quality, alongside standard PSNR and SSIM reporting.

4 Experimental setup

  • VQ-FIGAN uses downsampling factor f = 32, kernel size K = 5, and an 8192-entry codebook; diffusion sampling uses DDIM for 200 steps.VQ-FIGAN and the denoising U-Net are trained to convergence, around 70 and 60 epochs respectively, on RTX 3090 GPUs.Source: §4 · p. 4
  • Training uses 64,612 Vimeo90k and 17,600 BVI-DVC frame triplets, with random 256×256 crops, flips, and temporal-order reversal.The paper also reports results for a model trained on Vimeo90k alone for reference.Source: §4 · p. 4
  • Evaluation covers Middlebury, UCF-101, DAVIS, SNU-FILM, and VFITex at resolutions from 225×225 to 4K; BVI-HFR supports the user study.Table 1 compares Middlebury, UCF-101, DAVIS, and VFITex, while Table 2 covers SNU-FILM; VFIformer is omitted from Table 2 because of GPU memory limits.Source: §4 · pp. 4–5, Table 1 · p. 5, Table 2 · p. 5
  • LPIPS, FloLPIPS, and FID assess perceptual performance; PSNR and SSIM are reported for completeness, and a psychophysical experiment compares human judgments.The authors note that PSNR and SSIM are limited indicators of perceptual quality.Source: §4 · p. 5

5 Experiments

Across benchmark tests, LDMVFI performs favorably against state-of-the-art methods, and a user study finds higher preference for its videos in every comparison. Ablations examine VQ-FIGAN design, latent downsampling, and the diffusion process.

5.1 Quantitative Evaluation

  • LDMVFI outperforms other VFI methods in most benchmark cases, with an approximately 20% gain over the second-best method on VFITex; Figure 3 shows high-frequency detail retention under complex motion.MCVD, the other diffusion-based VFI method tested, does not perform satisfactorily overall.Source: §5.1 · pp. 5–6, Figure 3 · p. 6

5.2 Subjective Experiment

  • In all three user-study comparisons, LDMVFI has higher preference ratios, with statistically significant advantages at 95% confidence.The study used 22 full-HD source sequences and 2AFC judgments from 20 participants.Source: §5.2 · pp. 6–7, Figure 4 · p. 6

5.3 Ablation Study

  • Removing deformable-convolution synthesis in V2 decreases performance, while V1’s additional removal of neighbor-frame features during reconstruction causes a more severe drop.V1 also replaces decoder MaxCABlocks with ResNetBlocks and resembles the original VQGAN.Source: §5.3 · p. 7, Table 3 · p. 7
  • Increasing the downsampling factor from 8 to 32 generally improves performance, especially on DAVIS, but factor 64 degrades LPIPS and FloLPIPS.The authors describe a trade-off between using neighboring-frame information in decoding and retaining sufficient information for latent diffusion.Source: §5.3 · p. 7
  • Removing the denoising U-Net and reverse diffusion process in V6 significantly reduces performance.V6 feeds the concatenated z0 and z1 directly to the decoder.Source: §5.3 · p. 7, Table 3 · p. 7

6 Conclusion

LDMVFI frames video frame interpolation as conditional generation with latent diffusion, combining a compact autoencoding space and a denoising U-Net. Its diffusion-training appendix describes a forward noise process and a learned reverse process, with conditioning on the two input-frame encodings.

6 Conclusion

  • LDMVFI is presented as the first approach to formulate VFI as conditional generation using latent diffusion models.Its two main components are an autoencoding model for a compact latent space and a denoising U-Net that reverses diffusion on latent representations.Source: §6 · p. 7

A Details of LDMVFI Loss Function (Diffusion Part)

  • The forward diffusion process gradually adds noise, while a θ-parameterized Gaussian approximates the intractable reverse conditionals.The reverse-chain approximation is described under the condition that βt is sufficiently small at each forward step.Source: A Details of LDMVFI Loss Function… · p. 11
  • The training objective focuses on the tractable reverse-process term after the prior term is ignored and the decoder handles the final term.The prior can be set to a standard normal distribution; the cited passage says the decoder handles the final term separately.Source: A Details of LDMVFI Loss Function… · p. 11
  • LDMs perform diffusion in the lower-dimensional latent space provided by an encoder and decoder, and LDMVFI conditions noise prediction on both input-frame encodings.The conditional encodings are z0 = E(I0) and z1 = E(I1).Source: A Details of LDMVFI Loss Function… · p. 12

B Details of LDMVFI Training and Inference

The appendix summarizes LDMVFI’s training and inference algorithms and describes DDIM sampling for interpolation. It also specifies U-Net design choices, including MaxViT blocks and a base channel size of 256.

B Details of LDMVFI Training and Inference

  • The full training and inference algorithms are summarized in Algorithms 1 and 2.The appendix states that their symbols and equations refer to the derivations in Section A.Source: B Details of LDMVFI Training and… · p. 12

C Details of DDIM Sampling Process

  • DDIM sampling starts from latent Gaussian noise, encodes the two input frames, iteratively predicts noise, and decodes the final latent representation into the interpolated frame.The decoder also receives features extracted from both input frames.Source: C Details of DDIM Sampling Process · pp. 12–13

D Denoising U-Net Architecture

  • The denoising U-Net replaces vanilla self-attention layers with MaxViT blocks, and its base channel size c is 256 in LDMVFI.The architecture figure uses each block’s arrow to indicate its input and output channel counts.Source: D Denoising U-Net Architecture · p. 14, Figure 5 · p. 14

E Full Quantitative Evaluation Results

The appendix reports quantitative comparisons across standard VFI test sets and metrics. Its tables also identify evaluation exclusions and provide results for models trained only on Vimeo90k-triplet.

E Full Quantitative Evaluation Results

  • Table 4 covers Middlebury, UCF-101, DAVIS, VFITex, and four SNU-FILM subsets across PSNR, SSIM, LPIPS, FloLPIPS, and FID.The passage describes the table as the full evaluation results for LDMVFI and compared VFI methods.Source: E Full Quantitative Evaluation Results · p. 14, Table 4 · p. 15
  • ST-MFNet and FLAVR are not evaluated on Middlebury, while VFIformer is omitted on SNU-FILM because of GPU out-of-memory.The Table 4 caption gives these evaluation constraints.Source: Table 4 · p. 15
  • Table 5’s caption identifies quantitative comparisons on the same named test sets for models trained only with Vimeo90k-triplet.The caption lists Middlebury, UCF-101, DAVIS, VFITex, and four SNU-FILM subsets.Source: Table 5 · p. 16

F Additional Quantitative Evaluation with Vimeo90k-triplet Training

The appendix discusses results for models trained only on Vimeo90k-triplet and compares them with models retrained on Vimeo90k and BVI-DVC. It also points to ablation results for variants of LDMVFI.

F Additional Quantitative Evaluation with Vimeo90k-triplet Training

  • For some methods, Vimeo90k+BVI-DVC retraining lowers PSNR on Middlebury and UCF-101 but raises overall PSNR on DAVIS and VFITex.The authors suggest the difference may relate to greater training-distribution overlap for Middlebury and UCF-101 with Vimeo90k-triplet, while BVI-DVC adds larger, more complex motion.Source: F Additional Quantitative Evaluation… · p. 16
  • The Vimeo90k-triplet-only results are limited to methods with publicly available pretrained versions, according to the appendix description.That description identifies Table 5 as the location of these additional results.Source: F Additional Quantitative Evaluation… · p. 16
  • Tables 6 and 7 report ablation experiment results for variants of proposed LDMVFI; Table 6 shows variant performance, and Table 7 reports results on SNU-FILM.Table 7 notes that full details of variants V1–8 can be found in the main paper.Source: Table 6 · p. 17, Table 7 · p. 17

G Full Ablation Study Results

The ablations examine how model capacity and conditioning choices affect LDMVFI, alongside the variability of its sampled interpolations. Capacity reduction has little effect at a smaller latent size but a clearer performance cost at a larger one.

Effect of Model Size

  • With f = 32, reducing c from 256 to 64 leaves performance without significant change, while at f = 8 decreasing c shows a clearer decline.The default c = 256 model has 439.0M parameters; c = 128 and c = 64 reduce this to 126.8M and 48.7M, respectively.Source: G Full Ablation Study Results · p. 17

Conditioning Mechanism

Variability of VFI results

  • Repeated DDPM samples vary little for regular motion but vary for irregular dynamic textures, such as water.Figure 6 visualizes the difference between two samples as a residual image.Source: G Full Ablation Study Results · p. 17, Figure 6 · p. 18
  • LDMVFI’s sampled results are reported as sharper than IFRNet’s, and its large performance gain over existing methods on VFITex is linked to this texture setting.The passage contrasts generative single samples with L1-trained methods that average possible results, producing blur and flickering.Source: G Full Ablation Study Results · pp. 17–18

H Details of Subjective Experiment

The subjective experiment was conducted in a controlled lab setting, and the appendix reports t-test p-values for preference ratios. A separate DAVIS experiment evaluates ×4 interpolation and reports the best perceptual-metric scores for LDMVFI.

H Details of Subjective Experiment

I Multi-Frame (×4) Interpolation

J Limitations and Societal Impact

The paper identifies slow inference, larger model size, and extremely large motion as limitations, and notes potential societal harms from generative misuse and energy consumption. The appendix also points to dataset and code license information and provides additional visual examples.

J Limitations and Societal Impact

  • LDMVFI has slower inference and more parameters than other methods, and can produce unsatisfactory results under extremely large motion, a challenge shared by many state-of-the-art VFI methods.The authors attribute slow inference mainly to iterative reverse diffusion and identify the denoising U-Net as the main source of model size.Source: J Limitations and Societal Impact · p. 19
  • The authors flag misuse of generative models for manipulated content and higher carbon footprint from energy-intensive training and evaluation as potential societal harms.Source: J Limitations and Societal Impact · p. 19

K Attribution of Assets: Code and Data

L Additional Qualitative Examples

Loading 2303.09508v3…