PaperlayerView exact revision on arXiv

Source-linked AI summary

Stochastic Adversarial Video Prediction

Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, Sergey Levine

arXiv:1804.01523v1cs.CVcs.AIcs.LGcs.RO

TL;DR

Video prediction must represent ambiguous futures without averaging them into blurry frames, while existing latent-variable and adversarial approaches each have limitations. The paper combines variational latent variables with adversarial video losses, and reports predictions that are more realistic and diverse than prior and concurrent work. Its evaluations also show that human judgments and diversity measures are important because PSNR and SSIM poorly track qualitative rankings.

  • Problem

    Ambiguous future frames are multimodal: latent-variable models can remain unrealistic, while adversarial models can ignore latent variables and fail to produce diverse predictions.

  • Method

    The model combines a variational latent-variable video predictor with adversarial losses that model the spatiotemporal joint distribution of pixels.

  • Results

    The VAE component improves prediction diversity, while adversarial loss produces substantially more realistic predictions than prior and concurrent work.

  • Takeaways & Limitations

    Combining latent variables and adversarial training yields predictions that both cover possible futures and provide realistic images.

Abstract

from arXiv · show

Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior and concurrent work in these aspects.

1 Introduction

Video prediction is difficult because ambiguous futures become multimodal, while pixelwise losses blur predictions and GANs can collapse diversity. The paper combines latent variables with adversarial losses to produce realistic, diverse stochastic predictions.

  • Challenge: Ambiguous futures diverge beyond a few frames, making deterministic pixelwise objectives average possibilities into blurry predictions.VAE-based models represent stochasticity with latent variables but can remain blurry when those variables do not capture all uncertainty.
  • Challenge: GANs model joint pixel distributions without independence assumptions, but conditional models can ignore latent variables and collapse to few futures.Prior and concurrent GAN video-prediction approaches often use deterministic models, unconditional generation, or input noise alone.
  • Approach: The proposed VAE-GAN combines a variational lower bound with adversarial video losses, using time-varying latent variables to sample plausible futures.An inference network estimates latent distributions, while video discriminators distinguish generated videos from real videos.
  • Evaluation: PSNR and SSIM often disagree with human preference rankings, motivating combined evaluation with human judgments and automated diversity and accuracy measures.The paper reports that standard metrics rarely correctly order prediction quality across methods.
  • Results: VAE components improve generated-image diversity, while adversarial losses make predictions substantially more realistic than prior and concurrent methods.The paper evaluates realism, prediction diversity, and accuracy through systematic comparisons and ablations.

2 Related Work

Prior video-prediction work varies in generator design and training objective, but real-world ambiguity makes deterministic pixel losses blurry. VAE latent variables support diversity, whereas adversarial losses support realism, motivating their combination.

  • Generative models: Video-prediction methods differ in both generative architecture and training objective, including direct pixel generation and alternative transformations.The literature includes feedforward and recurrent architectures among earlier pixel-generation approaches.
  • Training objectives: On ambiguous real-world videos, minimizing mean squared error averages multiple possible futures and produces blurry predictions.MSE can still work well for deterministic synthetic videos such as video-game frames.
  • Stochastic modeling: Pixel-autoregressive models can produce sharp images but make training and inference impractically slow, while latent-variable models address uncertainty through sampling.Conditional VAEs have been applied to optical-flow, single-frame, and video prediction settings.
  • Adversarial modeling: GAN adversarial losses improve realism but are difficult to tune and prone to mode collapse, limiting coverage of possible predictions.Related approaches include unconditional video generation and conditional generation using input noise.
  • Combined approach: GANs with input noise generate realistic videos but fail to cover possible futures, whereas VAE-GAN predictions are both visually plausible and highly diverse.The comparison directly motivates combining VAE latent-variable modeling with adversarial loss.
  • Evaluation: Because PSNR and SSIM do not match human qualitative rankings, the paper evaluates realism, diversity, and accuracy with multiple complementary measures.The authors identify latent variables as critical for diversity and adversarial losses as most important for realism.

3 Video Prediction with Stochastic Adversarial Models

The model predicts diverse, perceptually realistic futures by sampling latent codes through a recurrent generator and training with variational and adversarial objectives. Its latent inference process encourages stochastic information use, while adversarial training matches predicted and real video distributions.

  • Model: A recurrent generator maps an initial image and time-varying latent codes to predicted future frames, with test-time codes sampled from a unit Gaussian prior.The latent codes represent ambiguous or stochastic events that may affect the future.
  • Model: The generator predicts each frame from the previous frame and latent code, using a fixed-variance Laplacian distribution centered at its predicted frame.Initial frames are ground truth, while later inputs may be previously predicted frames.
  • Variational objective: Because marginalizing latent variables is generally intractable, training maximizes a variational lower bound using a recognition model to approximate the posterior.The recognition model is parameterized by a deep network and conditionally Gaussian latent distributions.
  • Variational objective: The encoder observes adjacent frames so latent variables can represent ambiguous transition information between them.During training, latent codes are sampled from the approximate posterior and used to reconstruct the next frame through the generator.
  • Variational objective: A regularization term keeps the approximate posterior close to the prior, enabling prior-based sampling at test time.The prior used for sampling is fixed rather than learned.
  • Variational objective: The VAE objective weights reconstruction and KL terms, with their relative weighting determined by the fixed variance of the Laplacian distribution.This objective is the variational component of the combined training procedure.
  • Adversarial objective: Adversarial training matches predicted and real video distributions to keep predictions on the video manifold and improve sharpness.A discriminator distinguishes generated videos from real videos, providing the adversarial signal.

D Lgan(G, D). (5)

SAVP combines adversarial training with variational latent variables for conditional video prediction, using separate objectives and discriminators for prior- and posterior-sampled codes. Its recurrent generator predicts frames autoregressively while retaining information through LSTM layers.

  • Testing: During testing, the generator samples a time-varying latent code and uses the previous frame to synthesize each next frame.Predictions are fed back autoregressively, so later frames can be generated from the model’s own earlier outputs.
  • Training: During training, an inference network estimates latent variables while video discriminators distinguish generated videos from real videos.The adversarial component operates on generated videos, and the inference network supplies posterior latent codes for the variational component.
  • Training: The model uses separate discriminators and losses for latent codes sampled from the prior and from the learned posterior.The posterior-based objective uses q(z_t|x_t:t+1) and discriminator Dvae, whereas the prior-based conditional GAN uses a reconstruction term analogous to L1.
  • Method: SAVP combines GAN and VAE objectives to model realistic and stochastic video predictions.The model uses adversarial losses together with latent-variable learning, drawing on the complementary strengths of GANs and VAEs.
  • Architecture: A convolutional LSTM generator predicts pixel-space transformations while conditioning on the current frame and latent code.Skip connections from the first frame are also used; after the initial frames, conditioning shifts to the generator’s own predictions.

4 Experiments

The experiments evaluate realism, diversity, and best-sample accuracy on BAIR and KTH using human judgments, VGG similarity, and standard metrics. Results show that combining VAE and GAN objectives improves realism while retaining diversity, with dataset stochasticity shaping the accuracy–diversity relationship.

  • Evaluation: Realism was evaluated with real-versus-fake Amazon Mechanical Turk judgments, while diversity used VGG feature distances and accuracy used the best of 100 sampled predictions.VGG cosine similarity was additionally used because PSNR and SSIM can poorly reflect human perceptual judgments.
  • Realism and diversity: On BAIR, SAVP produces more realistic predictions than VAE-based baselines while retaining diversity, whereas GAN-only models trade substantially lower diversity for higher realism.The GAN variant achieves the highest fooling rate, while SAVP lies roughly halfway between the GAN and VAE variants on realism.
  • Realism and accuracy: On KTH, SAVP achieves the highest realism score, while the VAE-only model outperforms the other variants on PSNR, SSIM, and VGG similarity.The deterministic model is not far behind in similarity, consistent with KTH being less stochastic when conditioned on 10 frames.
  • Metric interpretation: PSNR and SSIM can favor blurry predictions, whereas L1-trained models perform better on VGG cosine similarity and GAN-based models prioritize joint pixel distributions.SV2P achieves high PSNR and SSIM despite producing blurry, unrealistic images.
  • Accuracy and stochasticity: On stochastic BAIR videos, greater diversity is associated with higher best-sample similarity, but this relationship is not observed on the less stochastic KTH dataset.SAVP is both more diverse and more similar than the GAN-only variant on the robot dataset.
  • Overall finding: The combined VAE-GAN objective is reported as the best choice for realistic coverage of diverse stochastic futures.Adding GAN training to the VAE increases realism without sacrificing diversity on stochastic datasets.

5 Conclusion

The proposed model combines variational latent variables with adversarial training to produce realistic, diverse stochastic video predictions. Experiments evaluate realism, diversity, and accuracy against prior methods and ablations.

  • Combining latent variables with adversarial loss produces visually and physically realistic stochastic video predictions.
  • VAE-style training enables diverse stochastic predictions, while adversarial loss improves visual realism according to human raters.
  • The evaluation compares the full model with VAE-only and GAN-only variants using human ratings, diversity, accuracy, and other quantitative and qualitative measures.
  • The approach produces more realistic predictions than prior methods while preserving the sample diversity of VAE-based methods.

A.1 Network Details

The generator predicts future frames autoregressively with a convolutional LSTM, combining copied, transformed, and newly synthesized pixels. Time-varying latent codes provide stochastic conditioning throughout the network.

  • The convolutional LSTM repeatedly predicts future frames and feeds each prediction back into the model.The model initializes from 2 or 10 ground-truth frames for BAIR or KTH and predicts 10 future frames.
  • Each prediction can copy previous-frame pixels, use kernel-transformed pixels, or synthesize pixels from scratch.A compositing mechanism combines these sources, including the first frame where applicable.
  • Architectural improvements include skip connections, instance normalization, and ReLU activations in the generator network.
  • The generator uses time-varying 8-dimensional latent codes to enable stochastic sampling during training and testing.A fully connected LSTM models temporal correlations before the codes enter the main convolutional network.
  • The encoded latent code is concatenated along the channel dimension into all convolutional layers of the main network.
  • The encoder maps image pairs at each time step to latent-distribution parameters, while separate 3D convolutional discriminators process complete videos.

A.2 Training Details

Training uses scheduled sampling and Adam optimization, with separate optimizer settings for GAN-based versus deterministic and VAE models. KL weighting is annealed for VAE variants.

  • Scheduled sampling progresses from one-step training to fully autoregressive training over time.
  • All models are trained with Adam for 300,000 iterations, with the learning rate linearly decayed during the final 100,000 iterations.SVG uses a training schedule provided by its author.
  • GAN variants use β1 = 0.5, β2 = 0.999, learning rate 0.0002, and batch size 16.
  • Deterministic and VAE models use β1 = 0.9, β2 = 0.999, and learning rate 0.001.
  • For VAE variants, the KL-divergence weight is linearly annealed from zero to its final value between iterations 50,000 and 100,000.
  • The relative KL-to-GAN-loss weighting is 0.001 for BAIR robot-pushing datasets and 0.00001 for KTH.

B.1 Comparison of Pixel-Wise L1 and L2 Losses

The study compares L1 and L2 reconstruction losses and examines GAN-based baselines on BAIR robot-pushing videos. L1 generally yields less blurry, more perceptually similar predictions than L2.

  • L1 and L2 losses are compared for deterministic and VAE variants using full-reference similarity metrics.L1 corresponds to a factorized Laplacian model, while L2 corresponds to a factorized Gaussian model.
  • L2-trained models tend to generate blurry predictions but achieve higher PSNR than equivalent L1-trained models.PSNR is closely related to L2 because it is a logarithmic function of that loss.
  • L1-trained models significantly outperform equivalent L2-trained models on VGG cosine similarity, which better reflects human perceptual judgments.
  • On SSIM, L1-trained models achieve roughly the same or better similarity than L2-trained models.
  • MoCoGAN is evaluated as a representative unconditional GAN and struggles on the robot dataset because multiple entities move simultaneously.Its strengths are associated with human-action videos focused on an actor in the center of the frame.

B.3 Results on Action-Conditioned BAIR robot pushing dataset

On action-conditioned BAIR robot pushing, the model predicts future frames from initial frames and robot actions. The VAE and combined models improve realism, diversity, and accuracy relative to prior methods.

  • Setup: The model predicts the next 10 frames from the first 2 frames while conditioning on a sequence of robot actions.Each action is a 4-dimensional vector describing Cartesian translation and gripper state.
  • Accuracy: The VAE model significantly outperforms prior methods on full-reference accuracy metrics.
  • Realism and diversity: The VAE-based models produce higher-diversity videos, and the models outperform Babaeizadeh et al. in both prediction diversity and realism.Diversity is lower here than on other datasets because the action-conditioned task involves less stochasticity.
  • Realism and diversity: Our GAN, VAE, and SAVP models fool human judges at rates around 35-40%, while SV2P produces less realistic images.The SAVP model improves prediction realism compared with the VAE model.
  • Accuracy: Our VAE and deterministic models both outperform SV2P on similarity to the ground-truth video.The performance gap between the VAE and deterministic models is small in this less-stochastic dataset.
Loading 1804.01523v1…