PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Autoregressive Video Models

Dirk Weissenborn, Oscar Täckström, Jakob Uszkoreit

arXiv:1906.02634v3cs.CVcs.AIcs.LG

TL;DR

Video generation remains difficult because of its statistical complexity, stochasticity, and data demands. The paper uses scalable autoregressive models with three-dimensional self-attention and reports strong benchmark performance and occasional realistic continuations, while full Kinetics diversity remains challenging.

  • Problem

    Video generation remains challenging because videos are statistically complex, stochastic, and expensive to model at scale.

  • Method

    The paper introduces autoregressive video models using three-dimensional block-local self-attention and spatiotemporal subscaling.

  • Results

    The models achieve state-of-the-art results across video-generation benchmarks and occasionally produce encouraging continuations on Kinetics, especially cooking videos.

  • Takeaways & Limitations

    The results provide an initial scalable baseline for modeling unusually complex and diverse videos such as those in Kinetics.

  • Takeaways & Limitations

    The full diversity of Kinetics remains a major challenge, with failures including frozen movement, object distortions, and washout after a few frames.

Abstract

from arXiv · show

Due to the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural video remains a challenging task. State-of-the-art video generation models often attempt to address these issues by combining sometimes complex, usually video-specific neural network architectures, latent variable models, adversarial training and a range of other methods. Despite their often high complexity, these approaches still fall short of generating high quality video continuations outside of narrow domains and often struggle with fidelity. In contrast, we show that conceptually simple autoregressive video generation models based on a three-dimensional self-attention mechanism achieve competitive results across multiple metrics on popular benchmark datasets, for which they produce continuations of high fidelity and realism. We also present results from training our models on Kinetics, a large scale action recognition dataset comprised of YouTube videos exhibiting phenomena such as camera movement, complex object interactions and diverse human movement. While modeling these phenomena consistently remains elusive, we hope that our results, which include occasional realistic continuations encourage further research on comparatively complex, large scale datasets such as Kinetics.

1 INTRODUCTION

The paper scales conceptually simple autoregressive video models with three-dimensional self-attention and subscaling, achieving strong benchmark results and realistic continuations while exposing challenges on complex datasets.

  • Motivation: Video generation remains challenging because it requires producing large amounts of statistically complex, stochastic data.Autoregressive models additionally face slow generation, despite conceptual simplicity and tractable likelihood.
  • Motivation: The authors ask how far pixel-level autoregressive modeling can extend from images to video on modern hardware accelerators.Recent high-fidelity autoregressive image generation motivates this question.
  • Approach: The model generalizes Transformer self-attention to three-dimensional block-local attention implemented efficiently on TPUs.Three-dimensional subscaling further reduces memory requirements.
  • Approach: The techniques model videos as 3D volumes, enabling direct interactions across spatial and temporal pixel representations.This avoids treating video solely as a sequence of still image frames.
  • Results: The models obtain strong benchmark results and high-fidelity BAIR continuations with plausible object interactions.On another robot-pushing dataset, perplexity is reduced by almost 50% compared with prior autoregressive models.
  • Results: On Kinetics-600, encouraging continuations appear for a limited cooking-video subset, while the full dataset remains challenging.The subset includes camera movement, complex object interactions, and diverse subjects, motivating evaluation on challenging datasets.

2 RELATED WORK

The paper situates its approach among deterministic, latent-variable, explicitly motion-focused, convolutional, and sparse-attention video-generation methods, emphasizing simplicity and the slow-sampling trade-off of autoregression.

  • Autoregressive video models: Compared with earlier autoregressive video work, this model does not distinguish temporal and spatial dimensions and relies mainly on multi-head self-attention within the 3D volume.Earlier work separately modeled these dimensions with dilated convolutions and convolutional LSTMs.
  • Limitations: Autoregressive models have notoriously slow generation speed, although parallelized sampling and future accelerators may alleviate this drawback.The paper frames speed as a major practical limitation for extremely high-dimensional data such as video.
  • Efficiency: Block-local self-attention reduces the generally quadratic space complexity of self-attention for 3D volumes.Sparse attention after linearizing images is presented as a concurrent alternative.
  • Efficiency: Subscaling is generalized to video to reduce memory requirements, alongside hierarchical multi-scale generation as an alternative.Both image- and video-generation work had explored hierarchical multi-scale approaches.
  • Prior video-generation approaches: Earlier deterministic video-generation methods fail to capture the high stochasticity inherent in video.Generative latent-variable models emerged partly in response to this limitation.
  • Prior video-generation approaches: Prior approaches often encode video-specific intuitions such as explicit motion modeling or optical-flow generation.The authors instead align their conceptual simplicity with 3D convolutions and spatiotemporal self-attention.

3 VIDEO TRANSFORMER

The Video Transformer models videos as three-dimensional spatiotemporal volumes using block-local self-attention and subscale autoregressive generation. Its encoder conditions each smaller slice decoder on previously generated slices while varying block sizes across layers to improve connectivity and efficiency.

  • 3 VIDEO TRANSFORMER: The model represents videos as 3D spatiotemporal volumes and applies pixel-channel-level autoregressive factorization without sequentially linearizing spatial positions.The factorization orders pixels with a combined subscale and raster-scan ordering.
  • 3.1 BLOCK-LOCAL SELF-ATTENTION: Each self-attention layer divides the video volume into non-overlapping 3D blocks and computes attention independently within each block.Block representations contain t·h·w positions, with hidden size d.
  • 3.1 BLOCK-LOCAL SELF-ATTENTION: The attention mechanism projects inputs into queries, keys, and values, adds per-dimension relative position biases, aggregates values, and uses residual multi-head transformations.Layer normalization is applied before each block, which the authors found improves training.
  • 3.1 BLOCK-LOCAL SELF-ATTENTION: Varying block sizes across layers connects pixel positions across the encoder while avoiding the expensive data copying required by overlapping blocks.Decoder masking still omits some dependencies, although experiments showed no visible systematic artifacts.
  • 3.2 SPATIOTEMPORAL SUBSCALING: Subscaling divides a video into smaller sub-sampled slices that are generated sequentially, reducing the number of pixels held in memory to Np/s.The current slice is decoded after an encoder processes a partially masked video containing preceding slices, with optional auxiliary conditioning.
  • 3.3 CHANNEL PREDICTION & LOSS FUNCTION: The slice decoder uses masked convolutions and autoregressive channel prediction, conditioning current pixels on the encoded preceding slices and previously predicted channels.The decoder representation is conditioned on the encoder state before final per-channel predictions are produced.

4 EXPERIMENTS

The experiments evaluate Video Transformer variants on BAIR Robot Pushing and Kinetics using intrinsic bits/dim and extrinsic FVD metrics. Results show strong benchmark performance, with spatiotemporal subscaling improving continuation robustness and enabling occasional modeling of complex Kinetics phenomena, while full Kinetics diversity remains challenging.

  • Models & Setup: The evaluation focuses on BAIR Robot Pushing and Kinetics, with additional Moving MNIST and robot-pushing results in the appendix.The setup reports bits/dim for intrinsic evaluation and FVD for perceptual quality, temporal coherence, and sample diversity.
  • Models & Setup: Spatiotemporal subscaling models videos across temporal and spatial dimensions, whereas spatial-subscaling and single-frame variants restrict or omit temporal subscaling.The single-frame model conditions on the previous three frames; spatiotemporal subscaling can represent longer temporal dependencies.
  • Models & Setup: Sampling four 30x64x64 videos takes approximately eight minutes on a Tesla V100, leaving autoregressive generation potentially impractical.The authors expect parallel sampling strategies and future hardware to reduce this disadvantage.
  • BAIR Robot Pushing: 25% lower perplexity than prior VideoFlow results is achieved by the large model on BAIR Robot Pushing, while every model variant obtains lower FVD than evaluated prior work.The comparison is reported in bits/dim and FVD; prior work used longer priming sequences, so the results are not strictly comparable.
  • BAIR Robot Pushing: Almost 50% lower bits/dim than prior work is obtained on an earlier robot-pushing dataset, while the models match the lower bound on Moving MNIST.These appendix results are summarized as evidence of improvement over prior autoregressive video models.
  • BAIR Robot Pushing: Spatiotemporal subscaling improves robustness and continuation quality despite slightly worse bits/dim, including FVD 170 for the large model on Kinetics.At temperature 1.0, average FVD is 122±4 for spatiotemporal subscaling, versus 134±4 for spatial subscaling and 153±7 for single-frame models.
  • Kinetics: On a cooking subset of Kinetics, continuations capture camera movement, object movement, and some stochastic motion, but human fingers and faces remain difficult.The model occasionally generates scene jumps and can produce somewhat believable human-motion continuations.
  • Kinetics: The full diversity of Kinetics remains a major challenge, with failures including frozen movement, object distortions, and washout after a few frames.The authors state that larger datasets and/or models will be required to capture the complexity of short YouTube clips.

5 CONCLUSION

The paper presents a scalable autoregressive video model using block-local self-attention and spatiotemporal subscaling. It reports strong benchmark results and an initial, occasionally encouraging attempt on complex Kinetics videos, while emphasizing that full Kinetics modeling remains difficult.

  • Conclusion: The model uses TPU-efficient block-local self-attention and spatiotemporal subscaling to capture longer-range spatiotemporal dependencies while scaling substantially.The architecture models videos as 3D volumes rather than sequences of still frames.
  • Conclusion: State-of-the-art benchmark results support an initial attempt to model unusually complex and diverse Kinetics videos.The model produces occasional encouraging continuations, especially for a subset of cooking videos.
  • Conclusion: Modeling the full range of Kinetics videos remains a major challenge despite the reported encouraging continuations.The conclusion preserves the paper’s scope boundary for complex, diverse YouTube videos.

A.1 MOVING MNIST

On Moving MNIST, the single-frame prediction model nearly reaches the loss lower bound, whereas spatial subscaling introduces aliasing artifacts and worsens future-frame prediction.

  • A.1 MOVING MNIST: Moving MNIST contains 100K training and 10K validation/test videos of two deterministically moving digits, with crossing-induced partial occlusion.The evaluation primes on ten frames and predicts the subsequent ten.
  • A.1 MOVING MNIST: The deterministic loss and nats-per-frame metric enable direct comparison with Kalchbrenner et al. (2016).The loss uses grayscale targets z_i and predicted scalar intensities y_i.
  • A.1 MOVING MNIST: The single-frame prediction model virtually solves Moving MNIST by nearly matching the loss lower bound.This model does not use subscaling.
  • A.1 MOVING MNIST: Spatial subscaling produces aliasing artifacts that make future frames harder to predict on Moving MNIST.The authors limit this finding to Moving MNIST but suggest spatial subscaling can potentially hurt generation.

A.2 ROBOTIC PUSHING.

On Robotic Pushing, the large spatiotemporal subscaling model substantially outperforms prior autoregressive video modeling results on both seen and novel objects.

  • A.2 ROBOTIC PUSHING: The task contains robotic-arm pushing and grasping videos with roughly 50K training videos and 1500 test videos covering seen and novel objects.The evaluation uses the initial two frames for priming and conditions on the robot-arm action for each frame.
  • A.2 ROBOTIC PUSHING: The model achieves 0.51 bits/dim on seen objects and 0.47 bits/dim on novel objects.Kalchbrenner et al. (2016) reported 0.92 bits/dim, or 0.64 nats/dim, on each test split with two frames of priming.
  • A.2 ROBOTIC PUSHING: These results correspond to an almost 50% reduction in perplexity compared with prior autoregressive video modeling.The model is a large 2048-dimensional spatiotemporal subscaling model.

B HYPER-PARAMETER SWEEPS

Hyper-parameter sweeps show that wider networks matter most, while sampling temperature 0.9 consistently gives more robust predictions and better extrinsic metrics.

  • B HYPER-PARAMETER SWEEPS: Hidden size has the largest impact on performance, followed by the number of layers and heads.The authors argue that wider networks suit modern hardware because they require more parallel compute.
  • B HYPER-PARAMETER SWEEPS: Increasing depth is less preferred for scaling because computation time, memory, and parameter growth are more favorable for wider networks.For wider networks, computation time grows sub-linear, memory linearly, and parameters partially quadratically; deep networks grow linearly in these aspects.
  • B HYPER-PARAMETER SWEEPS: Shuffling block-size arrangements between layers changes results by at most 0.01 bits/dim, while the reported setup performs best overall.All shuffled configurations were within 0.01 bits/dim.
  • B HYPER-PARAMETER SWEEPS: Sampling temperature 0.9 consistently produces more robust predictions and better results on all extrinsic metrics than temperature 1.0.This comparison was performed only on the BAIR Robot Pushing validation set.

C CONNECTIVITY IN BLOCK-LOCAL SELF-ATTENTION

Block-local self-attention efficiently connects video positions, but masking creates decoder blind spots that can require prohibitively large full-attention blocks; the observed samples show no systematic visible artifacts.

  • C CONNECTIVITY IN BLOCK-LOCAL SELF-ATTENTION: A masked connection between positions in consecutive frames may require one block spanning the full frame width and height across at least two time steps.Full self-attention in such blocks can become prohibitive for large h and w.
  • C CONNECTIVITY IN BLOCK-LOCAL SELF-ATTENTION: The authors report no simple solution that completely eliminates blind spots.Suggested remedies include larger masked-convolution kernels and combining block-local with dilated self-attention.
  • C CONNECTIVITY IN BLOCK-LOCAL SELF-ATTENTION: Masking-induced independence assumptions do not produce systematic visible artifacts in the reported samples.The authors view this as evidence of potential for further parallelization of autoregressive video generation.
  • C CONNECTIVITY IN BLOCK-LOCAL SELF-ATTENTION: Blocks spanning only one time, row, or column dimension perform substantially worse than blocks extending at least somewhat in all directions.The authors attribute this likely to stronger independence assumptions imposed by future masking.

E KINETICS COOKING

Because many Kinetics prefixes make continuation difficult, the authors create a cooking-video subset where these problems occur less often while retaining full-set training.

  • E KINETICS COOKING: Many Kinetics prefixes are difficult because objects are too small, movement is too fast, or movement is absent.These conditions produce blurry frames or little to no movement in predicted continuations.
  • E KINETICS COOKING: The authors create a cooking-video subset intended to exhibit these continuation problems to a lesser degree.The subset is formed by filtering videos whose labels match cooking-related categories.
  • E KINETICS COOKING: The models are still trained on the full Kinetics training set, while the cooking subset is used to showcase samples in some cases.The subset therefore affects sample presentation rather than the stated training-set scope.

F SAMPLES

The samples illustrate fidelity, realism, and diversity on BAIR Robot Pushing, alongside diverse Kinetics continuations and artifacts in the single-frame model.

  • BAIR Robot Pushing: Figures 5 and 6 show BAIR Robot Pushing samples from the spatiotemporal subscaling models, illustrating fidelity and realism.The samples use 30 future frames from 12 test videos, with one prime frame and temperature 0.9.
  • BAIR Robot Pushing: Figures 7 and 8 show diverse BAIR Robot Pushing samples from the spatiotemporal subscaling models.Each figure presents 11 samples of 30 future frames for one test video.
  • Kinetics Cooking: Figures 9–11 show 16 diverse Kinetics cooking continuations from the spatiotemporal subscaling model, each primed with 5 frames and generating 11 frames.The passage describes these continuations as retaining fidelity while varying across samples.
  • Kinetics Cooking: The single-frame Kinetics model produces strange color artifacts attributed to its standard raster-scan generation order.Figure 12 depicts these exploding-color artifacts.
  • Kinetics Continuation Limits: Random Kinetics videos do not always provide good prefixes for generating continuations.Figure 13 shows ground-truth videos and two 30-frame continuation samples illustrating this limitation.
Loading 1906.02634v3…