PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Video Transformer

Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, Evgeny Burnaev

arXiv:2006.10704v1cs.CVcs.LGeess.IV

TL;DR

Video generation can require substantial computing resources, with some models needing up to 512 TPUs for training. The Latent Video Transformer predicts future frames autoregressively in discrete latent space, reporting comparable quality on BAIR Robot Pushing and Kinetics 600 while using an 8-V100-GPU research setup.

  • Problem

    Some video-generation models require up to 512 TPUs for short, low-resolution videos, while generated-video quality remains low.

    Source: §1 · p. 1

  • Method

    The Latent Video Transformer encodes frames into discrete latent representations and predicts future latent frames autoregressively rather than generating in pixel space.

    Source: §1 · p. 1

  • Results

    The model achieves quality comparable to state-of-the-art methods on BAIR Robot Pushing and Kinetics 600, though Kinetics-600 results are inferior to others and can show visual artifacts.

    Source: §1 · p. 2, §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The authors report comparable video-prediction results using the usual research setup of 8 V100 GPUs.

    Source: §5 · p. 9

    On Kinetics-600, results are inferior to other approaches; approximately half of the generated videos have artifacts.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

The video generation task can be formulated as a prediction of future video frames given some past frames. Recent generative models for videos face the problem of high computational requirements. Some models require up to 512 Tensor Processing Units for parallel training. In this work, we address this problem via modeling the dynamics in a latent space. After the transformation of frames into the latent space, our model predicts latent representation for the next frames in an autoregressive manner. We demonstrate the performance of our approach on BAIR Robot Pushing and Kinetics-600 datasets. The approach tends to reduce requirements to 8 Graphical Processing Units for training the models while maintaining comparable generation quality.

1 Introduction

Video generation remains computationally demanding, with some low-resolution, short-video models requiring up to 512 TPUs while still producing low-quality results. The Latent Video Transformer generates autoregressively in discrete latent space and reports comparable quality on two datasets with reduced computational requirements.

1 Introduction

  • Some models require up to 512 TPUs to train short, low-resolution videos, yet generated-video quality remains low.The paper frames high computational cost and limited quality as central challenges.Source: §1 · p. 1
  • The Latent Video Transformer combines representation learning with recurrent video generation, predicting frames in latent rather than pixel space.The paper presents this shift as a way to reduce computational requirements without significant quality deterioration.Source: §1 · pp. 1–2
  • Its discrete latent space represents each frame as indices, enabling autoregressive generative models and approaches from natural language processing.This discrete representation is identified as the model's key novelty.Source: §1 · p. 1
  • The model achieves quality comparable to state-of-the-art methods on BAIR Robot Pushing and Kinetics 600.The paper evaluates the approach on both datasets.Source: §1 · p. 2

2 Related work

Video-generation research spans several model families, while autoregressive approaches face slow generation and high resource requirements. The paper places its work in autoregressive modeling and describes a discrete-latent pipeline that encodes frames, generates latent frames, and decodes them.

2.1 Video Generation

  • Video generation includes recurrent, VAE-based, autoregressive, normalizing-flow, GAN, and optical-flow approaches.VAE-based models can produce blurry results, while GANs can suffer from mode dropping.Source: §2.1 · p. 2
  • Video Transformer uses autoregressive generation with subscaling and attention, but generation remains slow and training resource demands are high.The paper reports 128 TPUs and 1M training steps for Video Transformer.Source: §2.1 · p. 2
  • The paper follows the autoregressive Video Transformer setup but moves generation into discrete latent space to mitigate GPU memory use and accelerate inference.The cited passage frames this as the work's key novelty in relation to Video Transformer [49].Source: §2.1 · p. 2

2.2 Discrete latent space

  • An autoencoder learns an encoder and decoder that map inputs to and from a compact latent representation while keeping reconstruction close to the input.Discrete latent spaces, such as those used by VQ-VAE, use a categorical distribution; autoregressive models can learn the latent prior.Source: §2.2 · p. 2
  • The video pipeline encodes conditioning frames into discrete latents, generates new latent frames autoregressively, then decodes them to pixel space.The paper notes a parallel application of a similar pipeline to audio generation in Jukebox [12].Source: §2.2 · pp. 2–3

3 Latent Video Transformer

LVT predicts future frames from initial frames using a frame autoencoder and autoregressive generation in latent space. Its discrete codes support autoregressive modeling, with VQ-VAE frame coding feeding a latent Video Transformer.

3 Latent Video Transformer

  • LVT transfers video modeling from pixels to compact discrete latent representations and uses an autoregressive model to generate future frames.The proposed system consists of a frame autoencoder and an autoregressive generative model.Source: §3 · p. 3

3.1 Frame Autoencoder

  • The frame autoencoder uses VQ-VAE to encode images as discrete codebook indices and decode their embeddings back into pixel space.Training combines reconstruction loss with regularization against volatile encodings and uses EMA codebook updates.Source: §3.1 · p. 3
  • To mitigate index collapse with large codebooks, encoder outputs are split across channels into parts, each quantized with its own codebook; Figure 1 shows n_c=4.Index collapse occurs when some codebook vectors attract many encoder outputs, limiting the vectors used.Source: §3.1 · pp. 3–4, Figure 1 · p. 3

3.2 Latent Video Generator

  • A Video Transformer generates the remaining latent frames autoregressively, conditioned on the discrete representations of the initial frames.The model adapts the Video Transformer from pixel space to latent space.Source: §3.2 · p. 4
  • Generation proceeds slice by slice, raster-scanning pixels within each slice, while padding and masks preserve the autoregressive order.Figure 2 illustrates generation order, already-generated pixels, zero-padding, and slice membership.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
  • After latent generation, the frame decoder maps discrete codes to learned embeddings and decodes them frame by frame into pixel-space video.Source: §3.2 · p. 5

4 Experiments

Experiments evaluate 16-frame video modeling on BAIR Robot Pushing and Kinetics-600 using 8 Nvidia V100 GPUs. Results are comparable to other methods on BAIR but inferior on Kinetics-600, while cheaper attention variants retain similar quality with a slight reduction.

4.1 Experimental setup

  • Experiments used 16-frame, 64 × 64 videos across BAIR and Kinetics; models trained on 8 Nvidia V100 GPUs for two days.The reported setup covers quantitative and qualitative results on both datasets.Source: §4.1 · pp. 5–6

4.2 BAIR Robot Pushing

  • BAIR video prediction achieved performance comparable to other methods, while four codebooks improved VQ-VAE and generator accuracy.Preliminary experiments indicated that increasing the number of codebooks further would lead to overfitting.Source: Table 3 · p. 6, §4.2 · p. 6

4.3 Kinetics-600

  • On Kinetics-600 reconstruction, four codebooks performed best on both MSE and FVD and were retained because evaluation was costly.The authors cite the large test set as the reason evaluation time was particularly significant.Source: Table 4 · p. 7, §4.3 · p. 7
  • On Kinetics-600, results were inferior to other methods, and about half of sampled videos had artifacts.The authors link the result to error accumulation and the dataset's complexity and diversity; increasing model size led to very slow convergence.Source: Table 6 · p. 7, §4.3 · p. 7

4.4 Ablation Study of Attention

  • Cheaper attention variants can replace the query-key term at the cost of a slight quality reduction, which the authors report does not make that term crucial.Table 7 evaluates attention variants on BAIR using single-frame subscaling and four codebooks.Source: §4.4 · p. 8, Table 7 · p. 8

5 Conclusion

The model moves video generation from pixel space to latent space to reduce computational requirements. It achieves comparable video-prediction results using 8 V100 GPUs, with visual artifacts observed in some Kinetics-600 cases.

5 Conclusion

  • Prior video-generation methods required up to 512 TPUs for parallel training.The paper frames reducing computational requirements as its focus.Source: §5 · p. 8
  • Using 8 V100 GPUs, the model achieves comparable video-prediction results, with decent BAIR results but visual artifacts in some Kinetics-600 cases.The authors report that increasing model size led to very slow convergence.Source: §5 · p. 9
  • The generation process moves from pixel space to latent space.The authors focus on reducing the model’s computational requirements.Source: §5 · p. 9

Broader Impact

The model is presented as potentially useful for future-frame prediction and image animation. The authors also flag reliability risks, especially when errors are costly or long-tail events lack training examples.

Broader Impact

  • Potential uses include predicting pedestrian movement, animating landscape photos, and creating GIF animations.The authors also mention animating pictures on mobile phones.Source: Broader Impact · p. 9
  • Wrong future predictions in self-driving pedestrian forecasts could carry high error costs.The authors identify self-driving cars as a high-stakes example.Source: Broader Impact · p. 9
  • The model could fail on long-tail events absent from its training data, motivating further reliability research.Source: Broader Impact · p. 9

A Adaptive Input and Adaptive Softmax.

On BAIR Robot Pushing, the authors examined latent-code usage and tested Adaptive Input and Adaptive Softmax. Neither method improved quality in the reported configuration.

A Adaptive Input and Adaptive Softmax

B BAIR visualizations

The BAIR visualizations pair generated video frames with codebook and change-mask views. The authors report realistic-looking results and relate stable versus changing codes to static backgrounds and moving objects.

B BAIR visualizations

  • The first row of each visualization contains five real and eleven generated frames.Source: B BAIR visualizations · p. 12
  • Rows 2–5 visualize codebooks using indexed colors, while rows 6–9 show binary masks where yellow marks a code change between consecutive frames.Each code is a matrix of indices mapped to specific colors.Source: B BAIR visualizations · p. 12
  • The authors describe BAIR results as realistic, with static-background codes staying stable and codes for non-static objects changing across frames.Source: B BAIR visualizations · p. 12

C Kinetics-600 visualizations

Kinetics-600 visualizations contrast good and bad samples through their latent codes and code-change patterns; the text reports that next-frame prediction is easier when the next code resembles the current one.

C Kinetics-600 visualizations

  • Good and bad Kinetics-600 samples have different latent codes and code differences; next-frame prediction is easier when the next code resembles the current one.Source: C Kinetics-600 visualizations · p. 12
  • Figure 12 presents additional videos with the first five frames real and the remaining frames generated.Source: C Kinetics-600 visualizations · p. 12, Figure 12 · p. 18
  • Figures 8 and 10 show the first five frames as real and later frames as generated, alongside real and generated codes and masks marking changes between consecutive-frame codes.Rows 2–5 show real-video codes, rows 6–9 their change masks, rows 10–13 generated-video codes, and rows 14–17 their change masks.Source: Figure 8 · p. 14, Figure 10 · p. 16
Loading 2006.10704v1…