PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Conditional VRNNs for Video Prediction

Lluis Castrejon, Nicolas Ballas, Aaron Courville

arXiv:1904.12165v1cs.CVcs.LG

TL;DR

Video prediction must model multiple plausible futures without producing blurry frames. The paper improves VRNNs with higher-capacity likelihood networks and hierarchical, more flexible latent distributions, and reports favorable performance across three datasets, including strong BAIR Push gains over prior VAE-based models.

  • Problem

    VAE and VRNN video predictors can represent uncertainty but often produce blurry predictions, while limited latent distributions can lead to poor model fitting.

  • Method

    The paper extends VRNNs with a higher-capacity likelihood model and hierarchical latent variables that provide more flexible prior and posterior distributions.

  • Results

    The method performs favorably across three datasets; on BAIR Push, it improves FVD by 44% and LPIPS by 9.8% over SVG-LP, while improving LPIPS by 11.2% over SAVP.

  • Takeaways & Limitations

    Video prediction models benefit from increased likelihood and latent capacity, with the results pointing toward further gains from bigger and more flexible generative models.

Abstract

from arXiv · show

Predicting future frames for a video sequence is a challenging generative modeling task. Promising approaches include probabilistic latent variable models such as the Variational Auto-Encoder. While VAEs can handle uncertainty and model multiple possible future outcomes, they have a tendency to produce blurry predictions. In this work we argue that this is a sign of underfitting. To address this issue, we propose to increase the expressiveness of the latent distributions and to use higher capacity likelihood models. Our approach relies on a hierarchy of latent variables, which defines a family of flexible prior and posterior distributions in order to better model the probability of future sequences. We validate our proposal through a series of ablation experiments and compare our approach to current state-of-the-art latent variable models. Our method performs favorably under several metrics in three different datasets.

1. Introduction

Video prediction must represent multiple plausible futures while preserving coherent dynamics, yet VAE-based models often produce blurry predictions. The paper addresses this underfitting by increasing likelihood capacity and using hierarchical latent distributions, achieving favorable results across three datasets.

  • Video prediction is self-supervised but requires models to capture complex real-world dynamics to generate coherent future sequences.
  • Multiple plausible futures make uncertainty central to video prediction, motivating probabilistic latent-variable models such as VAEs and VRNNs.
  • VAE and VRNN predictions are often blurry, commonly attributed to pixel-space similarity objectives such as Mean Squared Error.
  • The paper argues that limited latent expressiveness and decoder capacity can impair fitting, so it scales the likelihood network and uses more flexible latent distributions.
  • The proposed hierarchical VRNN uses multiple latent levels per timestep to make prior and posterior distributions more flexible.
  • Across three datasets, the approach performs favorably against state-of-the-art models, including 44% better FVD and 9.8% better LPIPS than SVG-LP on BAIR Push.

2. Related Work

Prior video-prediction methods span deterministic, object-factorized, autoregressive, GAN-based, and VAE/VRNN-based approaches. This work builds on latent-variable models while avoiding object-decomposition assumptions and addressing the optimization difficulty of hierarchical VAEs.

  • Early video-prediction systems used deterministic patch-based or recurrent models to predict future frames from prior observations or actions.
  • ConvLSTMs extend LSTMs to spatial feature maps by replacing matrix multiplications with convolutions and are widely used for video prediction.
  • Object-disentangling approaches separate motion and context, whereas this paper models future uncertainty without assuming such a scene decomposition.
  • Autoregressive models capture complex pixel dependencies but have slow, poorly scalable inference at high resolutions, while pure GAN video training remains unstable.
  • VAE and VRNN approaches model sequence uncertainty with latent variables, including per-step latents and learned priors in SVG and hybrid VAE-GAN modeling in SAVP.
  • Hierarchical VAEs provide more flexible latent distributions but are difficult to train because of multiple layers of conditional latents.

3. Preliminaries

The paper formulates video prediction as conditional generation of future frames and uses VRNN latent variables to represent timestep-specific variation. Because exact latent marginalization is intractable, training uses an amortized posterior and ELBO optimization.

  • Given D context frames c and T future frames x, the goal is to learn a generative model maximizing p(x|c).
  • VRNN introduces latent variables z at each timestep to capture variation in observed future frames through a conditional likelihood model.
  • The VRNN prior is learned autoregressively, conditioning each timestep’s observed and latent variables on prior latent samples and observed frames.
  • Exact computation of p(x|c) requires marginalizing over z, which is computationally intractable.
  • VRNN therefore uses an amortized approximate posterior q(z|x,c) and derives an evidence lower bound on the marginal log-likelihood.
  • The model is trained by maximizing the ELBO with stochastic backpropagation and the reparameterization trick.

4. Hierarchical VRNN

The model extends VRNNs with multiple latent levels, factorized hierarchical prior and posterior distributions, and dense connectivity to ease optimization. Recurrent prior, posterior, and decoder networks use context and frame representations to generate future frames.

  • Hierarchical latent variables: The model uses L latent levels per timestep and factorizes both the prior and approximate posterior across the hierarchy.Each level conditions on previous-time variables at that level and lower-level variables at the current timestep.
  • Dense latent connectivity: Dense connections link each stochastic latent variable to the previous input and all subsequent latent levels, easing hierarchical optimization.The prior and posterior also connect latent variables directly to output variables.
  • Training and generation: During training, latent samples come from the approximate posterior and the ELBO is maximized; generation samples from the learned prior using ancestral sampling across levels.Sampling proceeds from the top latent level to lower levels conditioned on previously sampled values.
  • Model parametrization: Context-frame features initialize recurrent prior, posterior, and decoder states, while the decoder predicts each next frame from the previous frame, latent samples, and its hidden state.The architecture encodes frames with a CNN and uses recurrent networks throughout the probabilistic model.
  • Distribution parametrization: The prior and posterior use hierarchical diagonal Normal distributions whose parameters are recurrent functions of frame features and samples from previous latent levels.The prior uses the previously generated frame, whereas the posterior uses the frame being generated.
  • Model parametrization: The likelihood decoder combines previous-frame representations and latent samples, using ConvLSTMs and transposed convolutions to reconstruct the next frame.The decoder upsamples feature maps back to the input resolution.

5. Experiments

The models use Adam with teacher forcing and learning-rate warmup, alongside beta warmup that gradually increases KL regularization during training.

  • Optimization: Training uses Adam with batch size b = 128, learning-rate warmup from λ = 2e-5 to λ = 1.6e-4 over 5 epochs, and teacher forcing.The optimization also uses β1 = 0.5, β2 = 0.9, and weight decay δ = 1e-4.
  • KL regularization: Beta warmup gradually increases the KL-divergence weight in the ELBO, transitioning training from an unregularized autoencoder toward a VAE.The paper states that this procedure usually encodes more information in the latent variables.

5.1. Ablation Study

The ablations show that increasing likelihood capacity improves data fit, while hierarchical latents help when combined with training techniques that encourage their use. More latent levels improve fit up to three levels, but deeper hierarchies create diminishing returns and optimization difficulty.

  • Likelihood capacity: Adding more ConvLSTM layers to the likelihood network monotonically improves model performance, and larger hidden states further improve data fit.The experiment varies decoder recurrent-layer capacity, including 1, 3, and 6 ConvLSTM layers and a higher-capacity 6-ConvLSTM variant.
  • Hierarchy of latents: Hierarchical latent models with more levels generally achieve lower train and test ELBOs, indicating better data fit.The comparison includes latent configurations 1, 1-8, 1-8-32, and 1-8-16-32.
  • Hierarchy of latents: The four-level 1-8-16-32 model does not outperform the three-level model, showing diminishing returns beyond three latent levels.The authors attribute this result to the difficulty of training deep latent hierarchies.
  • Training hierarchical latents: Without beta warmup and dense connectivity, only a few top-level latents are used; with them, most latents are active across hierarchy levels.Figure 4 measures normalized KL per latent channel and uses KL thresholds to identify active and maximally activated units.

5.2. Comparisons to Previous Approaches

The evaluation compares hierarchical and single-level VRNNs with prior video-prediction models across synthetic and real-world datasets, using complementary perceptual, structural, and realism metrics. Across datasets, increased likelihood capacity and latent hierarchy improve reported metrics and visual quality, while metric suitability varies by dataset and human judgment.

  • Evaluation protocol: The evaluation samples 100 predictions per test sequence and selects the best sample using SSIM or LPIPS, while FVD compares generated and ground-truth populations for realism.LPIPS and FVD are intended to correlate better with human judgment than pixel-space metrics, although no single metric captures plausibility, realism, and coverage completely.
  • Stochastic Moving MNIST: Both VRNN variants significantly outperform SVG-LP on all Stochastic Moving MNIST metrics, while the hierarchical model gains mainly on FVD.The two variants obtain comparable results on this relatively simple dataset, and CNN-based LPIPS and FVD may be less suitable for it.
  • BAIR Push: 44% lower FVD and 9.8% better LPIPS than SVG-LP are reported for the hierarchical VRNN on BAIR Push.The hierarchical model also reaches similar FVD to SAVP while exceeding it by 11.2% in LPIPS; SAVP remains sharper but can produce inconsistent interactions.
  • BAIR Push: On BAIR Push, the single-level VRNN already improves over SVG-LP, while adding hierarchical latents further improves FVD and LPIPS.This pattern supports benefits from both a higher-capacity likelihood model and more expressive latent structure.
  • Cityscapes: On Cityscapes, increasing likelihood capacity and adding hierarchical latents improve all reported metrics, while SVG-LP often predicts a static image.SVG-LP can match SSIM at later timesteps, but this does not translate into better-looking samples, indicating limitations of SSIM for this setting.

6. Conclusions

The paper proposes a hierarchical VRNN with an improved likelihood model and hierarchical latent variables for video prediction. It reports favorable comparisons with state-of-the-art models across FVD, LPIPS, and SSIM, alongside visually appealing and coherent samples.

  • Conclusion: The proposed hierarchical VRNN combines an improved likelihood model with a hierarchy of latent variables for video prediction.The approach is evaluated against current state-of-the-art models using FVD, LPIPS, and SSIM.
  • Conclusion: The results suggest that current video-prediction models benefit from increased capacity, motivating larger and more flexible generative models.The stated conclusion connects the favorable metric and sample-quality results to increased model capacity.
Loading 1904.12165v1…