Source-linked AI summary
Stochastic Video Generation with a Learned Prior
Remi Denton, Rob Fergus
TL;DR
Future video prediction must capture uncertain outcomes while preserving sharp frames, a challenge for deterministic and adversarial approaches. The paper introduces an unsupervised model combining deterministic prediction with samples from a learned, time-varying uncertainty prior. It reports varied, sharp generations that compare favorably with existing approaches and can match complex future distributions on synthetic data.
Problem
Video prediction must represent the inherent uncertainty of future world dynamics without producing blurry frames or failing to capture the full outcome distribution.
Method
SVG combines a deterministic frame predictor with time-dependent stochastic latent variables, using a recurrent inference network and a learned prior to model uncertainty.
Results
The model produces high-quality generations comparable to or better than existing approaches and matches complex future distributions on synthetic data.
Takeaways & Limitations
The framework is sufficiently general for application to more complex datasets given appropriate encoder and decoder modules.
Takeaways & Limitations
Random latent samples can ignore temporal dependencies present between frames.
Abstract
from arXiv · showhide
Generating video frames that accurately predict future world states is challenging. Existing approaches either fail to capture the full distribution of outcomes, or yield blurry generations, or both. In this paper we introduce an unsupervised video generation model that learns a prior model of uncertainty in a given environment. Video frames are generated by drawing samples from this prior and combining them with a deterministic estimate of the future frame. The approach is simple and easily trained end-to-end on a variety of datasets. Sample generations are both varied and sharp, even many frames into the future, and compare favorably to those from existing approaches.
1. Introduction
Future video prediction must represent uncertainty in world dynamics without sacrificing sharpness. The proposed SVG model combines deterministic frame prediction with time-dependent stochastic latents and a learned uncertainty prior.
- Motivation: Future frame prediction is difficult because world dynamics are inherently uncertain, such as when surface imperfections or ball spin alter a bouncing ball’s trajectory.Deterministic pixel-level losses can blur predictions by accommodating multiple possible futures.
- Motivation: Adversarial losses attempt to model distributions but often face training difficulties and mode collapse, limiting coverage of possible futures.
- Approach: SVG combines a deterministic frame predictor with time-dependent stochastic latent variables to generate future frames.
- Approach: SVGLP learns a predictive uncertainty prior that stays low during predictable motion and rises at events such as ball-ground contact.The resulting high-variance event makes sampled future frames differ substantially.
- Training and evaluation: A recurrent inference network estimates each time step’s latent distribution, enabling end-to-end training across real-world and stochastic Moving MNIST datasets.Sample generations remain varied and sharp many frames into the future.
2. Related work
Related video-generation approaches model uncertainty, motion, content, or high-level structure in different ways. SVG instead separates deterministic and stochastic components, uses continuous direct frame generation, and avoids action labels and adversarial pixel losses.
- Predictive representations: Predictive representation methods learn features for recognition or control, rather than primarily generating future video frames.
- Video generation models: Several video generators use low-dimensional representations, discrete vector quantization, autoregressive pixels, or pixel transformations to produce future frames.
- Factorized representations: Content-motion factorization methods separate static and dynamic elements, whereas SVG factorizes deterministic and stochastic components to target uncertainty in motion.
- Supervision and conditioning: Hierarchical generation can model complex scenes but may require annotated pose information, unlike the paper’s unsupervised approach.
- Supervision and conditioning: Action-conditional models assume known actions at each frame, while SVG does not use action information.
- Uncertainty modeling: GAN-based video methods can sharpen generations but introduce training instability and mode collapse; SVG uses only an ℓ2 pixel-reconstruction loss.
- Stochastic sequence models: Other stochastic sequence models use recurrent latent variables for music, motion capture, speech, handwriting, or language, while related video methods differ in latent inference and sampling.
3. Approach
The model predicts future frames with a deterministic recurrent predictor augmented by time-dependent latent variables, using either a fixed or learned prior. A recurrent inference network enables end-to-end variational training, while the learned prior conditions uncertainty on previous frames and supports sequential generation.
- Model components: SVG combines a deterministic frame predictor with latent variables carrying stochastic information that the predictor cannot capture.The predictor uses previous frames and latent variables to generate the next-frame estimate.
- Training: The inference network estimates qφ(zt|x1:t) from the current and previous frames, and training balances reconstruction with a KL constraint against the prior.The likelihood term becomes an ℓ2 penalty, while β controls the trade-off between prediction error and prior fitting.
- Prior models: SVG-FP uses the fixed Gaussian prior N(0, I), whereas SVG-LP learns a time-varying conditional prior pψ(zt|x1:t−1) from past frames.The learned prior network outputs the parameters of a conditional Gaussian and is trained jointly with the rest of the model.
- Generation: At test time, SVG samples each latent variable from its prior and generates the next frame from the predictor, feeding generated frames back into the recurrent inputs.SVG-FP samples from N(0, I), while SVG-LP samples from its learned conditional prior.
- Comparison: Compared with the related variational model, both SVG variants produce sharper frames and are trained end-to-end in a single phase with a unified loss.The comparison attributes the training difference to the time-variant posterior used by SVG, while the learned-prior model additionally samples from a time-varying distribution.
4. Experiments
Experiments evaluate SVG-FP and SVG-LP on stochastic Moving MNIST, KTH actions, and BAIR robot videos using quantitative and qualitative measures. The learned-prior model captures when uncertainty arises, supports varied and sharp long-horizon predictions, and generally outperforms deterministic or prior stochastic baselines.
- Evaluation setup: SVG-FP and SVG-LP were evaluated on one synthetic dataset and two real-world datasets using SSIM, PSNR, and qualitative comparisons.The evaluation used Stochastic Moving MNIST, KTH actions, and BAIR robot videos.
- Stochastic Moving MNIST: On SM-MNIST, both stochastic models outperform the deterministic baseline, with SVG-LP performing best overall, especially at later time steps.Models were conditioned on 5 frames, trained to predict 10, and evaluated using the best of 100 generated samples per test sequence.
- Stochastic Moving MNIST: The learned prior predicts collision points that produce significant randomness in SM-MNIST trajectories.Across 500 synchronized test sequences, predicted uncertainty rises near the true collision times.
- Stochastic Moving MNIST: SVG-LP accurately captures complex, time-varying MNIST trajectory distributions, remaining accurate many time steps after wall collisions.Its predicted distribution is peaked before collisions, broadens afterward, and is reshaped by subsequent collisions.
- KTH Action Dataset: On KTH, SVG-FP and SVG-LP outperform the deterministic baseline while producing samples that reflect variability in future limb locations.Different stochastic samples represent pose uncertainty that the deterministic model cannot resolve precisely.
- BAIR robot pushing dataset: On BAIR, both SVG models outperform Babaeizadeh et al. in SSIM, while SVG-LP leads early PSNR and produces sharper long-horizon generations.The evaluation generated 28 frames after conditioning on 2, despite training to predict only 10 future frames; Babaeizadeh et al. became marginally better in later PSNR.
5. Discussion
The model combines deterministic frame prediction with time-varying stochastic latent variables, enabling high-quality generations and matching complex future distributions.
- The model combines deterministic next-frame prediction with stochastic latent variables drawn from a learned, time-varying distribution.A recurrent inference network estimates the latent distribution at each time step, supporting end-to-end training.
- On real-world sequences, the model produces high-quality generations comparable to or better than existing approaches.
- On synthetic data, the model matches complex distributions of possible futures.
- The framework can be applied to more complex datasets when appropriate encoder and decoder modules are available.
A. Variational bound
The variational bound decomposes the video model’s likelihood objective across recurrent time steps, using time-dependent inference distributions and independent latent variables.
- The derivation begins from the variational lower bound on the data likelihood and then simplifies it using the recurrent model structure.
- The frame predictor is recurrent, taking x_t−1 and z_t as inputs while depending on earlier frames and latent variables through recurrence.
- The inference network outputs a distinct distribution q_φ(z_t|x_1:t) for each time step.
- The collection z = [z_1, ..., z_T] represents latent variables across all time steps, with q_φ(z|x) denoting their joint distribution.
- Independence across time allows the KL-divergence term to decompose into individual time steps.