PaperlayerView exact revision on arXiv

Source-linked AI summary

Stochastic Latent Residual Video Prediction

Jean-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, Patrick Gallinari

arXiv:2002.09219v4cs.CVcs.LGstat.ML

TL;DR

Stochastic video prediction requires modeling uncertain futures, yet existing approaches often rely on costly image-autoregressive recurrent models, while fully latent alternatives have been difficult to design and train. The paper introduces a stochastic latent model with residual updates, separating temporal dynamics from frame synthesis. It reports improved performance over prior state-of-the-art methods and supports interpretable, temporally efficient dynamics.

  • Problem

    Stochastic video prediction must represent uncertain future dynamics, but existing image-autoregressive recurrent models are costly and tightly couple frame synthesis with temporal modeling.

  • Method

    The model uses a fully latent stochastic temporal process with residual updates, decoupling latent dynamics from frame synthesis and incorporating a static content variable.

  • Results

    The proposed model outperforms prior state-of-the-art methods for stochastic video prediction on representative challenging benchmarks.

  • Takeaways & Limitations

    Residual latent dynamics provide a simpler, more interpretable, and temporally efficient approach that scales to complex video prediction.

  • Takeaways & Limitations

    The authors identify future work in replacing existing VRNN dynamics, adding video-specific priors, and leveraging the state-space structure for model-based reinforcement learning.

Abstract

from arXiv · show

Designing video prediction models that account for the inherent uncertainty of the future is challenging. Most works in the literature are based on stochastic image-autoregressive recurrent networks, which raises several performance and applicability issues. An alternative is to use fully latent temporal models which untie frame synthesis and temporal dynamics. However, no such model for stochastic video prediction has been proposed in the literature yet, due to design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model whose dynamics are governed in a latent space by a residual update rule. This first-order scheme is motivated by discretization schemes of differential equations. It naturally models video dynamics as it allows our simpler, more interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.

1. Introduction

Stochastic video prediction must represent uncertain dynamics, but prevalent image-autoregressive approaches are computationally costly and tightly couple frame synthesis with temporal modeling. The paper instead introduces a stochastic latent residual dynamic model that separates these components and outperforms state-of-the-art methods.

  • Deterministic video prediction models inadequately capture intrinsically stochastic video dynamics.Predicting future frames from conditioning frames has applications in reinforcement learning and robotics.
  • Image-autoregressive recurrent models re-encode each generated frame, increasing computational cost and tying synthesis to temporal dynamics.These dependencies may hurt generation performance and limit applicability.
  • Latent temporal models decouple state dynamics from frame decoding, offering computational and interpretability advantages in low-dimensional spaces.They can also represent a complete system state for applications such as reinforcement learning.
  • The proposed stochastic dynamic model uses residual latent-state updates conditioned on learned stochastic variables.Its design supports efficient training and interpretable processing of high-dimensional videos, while connecting residual dynamics to ODE discretization.
  • The proposed approach outperforms current state-of-the-art stochastic video-prediction models on representative benchmarks.

2. Related Work

Prior video-synthesis research spans several tasks, while video prediction methods include deterministic recurrent models, stochastic image-autoregressive systems, and latent state-space models. The paper positions its fully latent residual model as a scalable approach for stochastic prediction on complex videos.

  • Video synthesis includes translation, super-resolution, interpolation, generation, and video prediction.
  • Deterministic models: Deterministic video-prediction methods commonly use LSTMs or ConvLSTMs for sequence modeling.
  • Stochastic and image-autoregressive models: Pixel-level autoregressive and flow-based stochastic methods require complex temporal generation over high-dimensional data, causing prohibitive generation costs.
  • Stochastic and image-autoregressive models: Stochastic image-autoregressive methods use learned priors and predictor recurrent networks, with adversarial training improving sharpness but reducing diversity.
  • State-space models: Existing latent state-space models generally target low-dimensional data or control tasks, whereas this work applies a fully latent residual dynamic model to complex videos.The residual updates are positioned within work connecting differential equations and neural networks.

3. Model

The model represents video frames through latent states whose stochastic residual dynamics are separated from frame synthesis. A static content variable captures non-temporal information, while variational inference trains the latent model efficiently from conditioning frames and video sequences.

  • Stochastic video prediction estimates the distribution of possible future frames from conditioning frames.
  • Latent dynamics: Each frame is generated only from its latent state, making temporal dynamics independent of previously generated frames.
  • Latent residual dynamics: The latent transition uses a stochastic residual update: the next state depends deterministically on the previous state given an auxiliary random variable.The auxiliary variables have a learned factorized Gaussian prior conditioned on the previous state.
  • Latent residual dynamics: The residual update follows a first-order movement inspired by Euler discretization, making the temporal model lighter and more interpretable than a regular RNN.The model permits step sizes smaller than 1 to approach continuous dynamics.
  • Latent residual dynamics: Keeping the auxiliary variable constant between integer time steps allows different training or testing step sizes and arbitrary video frame rates.Intermediate latent states can be decoded into the observation space.
  • Content variable: A content variable remains constant during generation and is supplied to the frame generator alongside the dynamic latent state.It lets the dynamic component focus on movement and supports skip connections for more realistic frames.
  • Content variable: The content variable is a permutation-invariant deterministic function of a fixed number of conditioning frames, sampled across the sequence during training to limit temporal leakage.Its unconstrained information is separated from the temporally focused variables y and z.
  • Content variable: The content variable can be removed, yielding a more classical deep state-space model.

4. Experiments

The experiments evaluate the stochastic latent model against state-of-the-art baselines on one synthetic and three real-world video datasets. Results show strong performance across datasets, advantages for residual latent dynamics, and stable generation at higher frame rates.

  • Evaluation setup: The evaluation covers four stochastic video prediction datasets and compares the method with state-of-the-art baselines using PSNR, LPIPS, FVD, and related metrics.The study also includes qualitative samples and latent-dynamics analyses.
  • Moving MNIST: On Moving MNIST, the model outperforms SVG on PSNR and SSIM while preserving temporal consistency when digits cross.Its performance decreases substantially less than SVG’s at long prediction horizons, especially through t + 95.
  • Ablations and dynamics: Residual dynamics outperform the model’s MLP and GRU alternatives, particularly on long-term dynamics and FVD, while halving ∆t preserves performance and enables higher-frame-rate generation.The model generates twice as many frames with stable overall prediction performance and demonstrates double- and quadruple-rate generation.
  • KTH: On KTH, the method substantially outperforms every considered baseline on each metric and better predicts subject poses while retaining diverse motion samples.SVG sometimes fails to generate a person, whereas SAVP and the proposed model do; the proposed model’s best sample is closer to the subject poses.
  • Human3.6M: On Human3.6M, the model significantly outperforms StructVRNN on all considered metrics and produces more realistic frames with better captured subject movements.StructVRNN is identified as the prior state of the art on this dataset.
  • BAIR: On BAIR, the method achieves similar or better PSNR, SSIM, and LPIPS than state-of-the-art models and obtains the second-best FVD score.SAVP remains better on FVD, while SV2P’s good PSNR is associated with blurry samples and prohibitive LPIPS scores.

5. Conclusion

The paper introduces a latent stochastic video-prediction model that decouples frame synthesis from dynamics through residual updates of a small latent state. It reports better performance than prior state-of-the-art methods and identifies broader applications for the model’s principles.

  • The proposed model decouples frame synthesis and dynamics using residual updates of a small latent state.This design is presented as a latent dynamic model for stochastic video prediction.
  • The model is reported to perform better than RNN-based models for stochastic video prediction.
  • The model is described as the first latent dynamic model that scales to video prediction and the first residual model to scale to complex stochastic video data.
  • The authors suggest applying the model’s state-space, residual-dynamic, and static-content principles to other models and model-based reinforcement learning.

B. Datasets Details

The experiments use one simulated dataset and three real-world video datasets, with dataset-specific motion, appearance, conditioning, and prediction settings. Frames are represented as normalized greyscale or RGB pixels.

  • All dataset frames use greyscale or RGB pixel values normalized to [0, 1] by dividing original values by 255.
  • The study uses Moving MNIST, KTH, Human3.6M, and BAIR datasets with distinct sources of stochasticity and video characteristics.
  • Moving MNIST: Moving MNIST contains one or two digits moving linearly in 64 × 64 frames, with direction and velocity resampled at border bounces.Models are trained on 15 timesteps, conditioned on 5 frames, and predict 20 stochastic or 95 deterministic frames.
  • KTH: KTH contains 64 × 64 videos of 25 people performing six actions, with uncertainty in subject appearance, action, and execution.Models use 20 training timesteps, 10 conditioning frames, and predict 30 frames.
  • Human3.6M: Human3.6M includes more actions and details than KTH but fewer training subjects, less varied backgrounds, and subjects that remain within the frames.The reported setup follows Minderer et al. (2019).
  • BAIR: BAIR contains 64 × 64 videos of a Sawyer robotic arm pushing tabletop objects, with stochastic direction changes during motion.Models are trained on 12 timesteps, conditioned on 2 frames, and predict 28 frames.

C.1. Specifications

The implementation uses shared encoder-decoder design choices, dataset-specific latent sizes and training schedules, and standard optimization settings. Additional specifications define content-variable construction, decoder behavior, and evaluation references.

  • Architecture: The encoders and decoders use mirrored architectures, with DCGAN-style components for Moving MNIST and mirrored VGG16 components for the other datasets.The decoder output is a 128-dimensional vector.
  • Content variable: The content variable is computed from k input frames, using k=5 for Moving MNIST, k=3 for KTH and Human3.6M, and k=2 for BAIR.
  • Network specifications: The model uses a single-layer LSTM with hidden state size 256 and MLPs with dataset-independent hidden sizes specified separately for inference and forward models.Inference MLPs have three 256-unit layers, while forward-model MLPs have four 512-unit layers.
  • Latent variables: For Moving MNIST, y and z have size 20; for KTH, Human3.6M, and BAIR, both latent variables have size 50.
  • Optimization: Models are trained with Adam at learning rate 3 × 10^-4, β1=0.9, and β2=0.999.
  • Optimization: Optimization schedules range from 150,000 to 1,000,000 main steps across datasets, with additional linearly decayed-learning-rate steps.The stochastic and deterministic Moving MNIST schedules are specified separately.

D. Additional Numerical Results

The additional numerical results report PSNR, SSIM, and LPIPS averaged across time on four datasets, using the learned-prior SVG variant except on KTH. These metrics are presented for the tested methods and baselines.

  • Tables 2 to 5 report time-averaged PSNR, SSIM, and LPIPS results for the Moving MNIST, KTH, Human3.6M, and BAIR datasets.
  • The experiments use learned-prior SVG on all datasets except KTH, where they use the fixed-prior version following its authors.

E. Pendulum Experiments

On Pendulum, the proposed model outperforms DVBF and is only surpassed by KVAE, approaching a method that uses exact latent-space inference.

  • Pendulum comparison: Our model outperforms DVBF on the Pendulum dataset but is merely beaten by KVAE.KVAE uses a Kalman filter for exact latent-space inference, whereas both DVBF and our model use sequential VAEs.
  • Interpretation: The comparison indicates that the proposed sequential-VAE dynamics are better adapted to Pendulum than DVBF's dynamics.The authors attribute the difference to the proposed dynamic model rather than to a change in the sequential-VAE training framework.

F. Influence of the Euler step size

The model can generalize across Euler step sizes, but transfer depends on the step size used during training: smaller training steps preserve performance under refinement, whereas larger ones can cause drops.

  • Effect of training step size: Training KTH with ∆t = 1 can reduce performance when testing uses smaller Euler step sizes.The authors attribute this to the rough training approximation adapting the model to an overly discretized dynamic.
  • Testing beyond training settings: Testing with a higher ∆t than used in training causes an expected performance loss from loosening the Euler approximation, while state-of-the-art performance is maintained.The learned dynamic can still be further discretized when a reasonable performance drop is acceptable.

G. Autoregressivity and Impact of Encoder and Decoder Architecture

On KTH, the proposed model outperforms SVG for fixed encoder-decoder architectures and is more robust to reducing architectural capacity, especially for PSNR-sensitive dynamics.

  • Architecture comparison: Our model outperforms SVG for each fixed choice of encoder and decoder architecture on KTH.The comparison uses DCGAN and VGG architectures for both methods.
  • Architecture comparison: VGG performs better than DCGAN for each method because DCGAN is the less powerful architecture.The figure and table report PSNR, SSIM, and LPIPS results with confidence intervals.
  • Robustness to capacity: Our model loses less performance than SVG when switching from VGG to DCGAN, showing greater robustness to encoder and decoder capacity.The difference is particularly pronounced for PSNR, which most penalizes dynamics errors.
  • Robustness to capacity: Reducing SVG's encoder and decoder capacity lowers both realistic-frame quality and its ability to learn a good dynamic.The authors connect this sensitivity to SVG's autoregressive dependence on encoder and decoder performance.

H. Additional Samples

Additional samples illustrate the proposed model's temporal consistency, realistic and diverse predictions, content manipulation, frame-rate flexibility, and latent-space interpolation, while also showing a content-swap failure case.

  • Stochastic Moving MNIST: On Stochastic Moving MNIST, our model maintains digit shape and temporal consistency even in a worst-1% PSNR example.SVG changes a digit's shape during prediction, whereas the proposed model preserves it.
  • KTH: On KTH, the proposed model can preserve actions and subject appearance in examples where competing methods produce inconsistent or indistinct predictions.In a shadow-foreshadowing example, only SAVP and the proposed model make the subject appear, but SAVP predicts the wrong action and inconsistent subject.
  • Human3.6M: On Human3.6M, the proposed model better captures subject movements and diversity while producing more realistic subjects and fewer artefacts.An additional example likewise reports better action capture and diverse realistic predictions.
  • Temporal resolution: The model generates videos at doubled or quadrupled frame rates by using a halved ∆t relative to training.Intermediate frames are explicitly shown in the generation examples.
  • Content manipulation: Content swapping combines dynamic variables inferred from one video with a content variable computed from another video's conditioning frames.The background and subject appearance can transfer successfully, although one additional example is a failure case.
  • Latent-space interpolation: The latent-space visualizations show interpolation results between two trajectories, including their reconstructed and intermediate-frame representations.Figures 32 and 33 provide additional examples of trajectory interpolation.
Loading 2002.09219v4…