PaperlayerView exact revision on arXiv

Source-linked AI summary

Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer

Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, Devi Parikh

arXiv:2204.03638v4cs.CVcs.AI

TL;DR

Video synthesis has made limited progress on generating long videos with consistent themes and storylines. TATS combines a time-agnostic 3D-VQGAN with a hierarchical time-sensitive transformer, enabling thousands of frames from short training clips. It achieves diverse, coherent, high-quality long-video generation and supports meaningful text- and audio-conditioned videos, while its time-agnostic encoder has an efficiency-related padding limitation.

  • Problem

    Existing video-synthesis studies mainly improve frame quality and transitions, leaving consistent themes and meaningful storylines across long videos insufficiently explored.

  • Method

    TATS combines a time-agnostic 3D-VQGAN with a hierarchical time-sensitive transformer to generate long videos from short clips using sliding-window inference.

  • Results

    TATS achieves state-of-the-art short- and long-video generation results across multiple benchmarks and produces meaningful videos conditioned on text and audio.

  • Takeaways & Limitations

    The approach demonstrates that training on clips with tens of frames can support generation of videos with thousands of frames while preserving visual quality and temporal coherence.

  • Takeaways & Limitations

    Using realistic padding to make the encoder fully time agnostic can require as many as O(Ld) extra real frames and become expensive for deep networks or large temporal compression rates.

Abstract

from arXiv · show

Videos are created to express emotion, exchange information, and share experiences. Video synthesis has intrigued researchers for a long time. Despite the rapid progress driven by advances in visual synthesis, most existing studies focus on improving the frames' quality and the transitions between them, while little progress has been made in generating longer videos. In this paper, we present a method that builds on 3D-VQGAN and transformers to generate videos with thousands of frames. Our evaluation shows that our model trained on 16-frame video clips from standard benchmarks such as UCF-101, Sky Time-lapse, and Taichi-HD datasets can generate diverse, coherent, and high-quality long videos. We also showcase conditional extensions of our approach for generating meaningful long videos by incorporating temporal information with text and audio. Videos and code can be found at https://songweige.github.io/projects/tats/index.html.

1 Introduction

Long video synthesis requires modeling long-range temporal dependence while preserving frame quality and coherence beyond the training length. TATS addresses this with a time-agnostic VQGAN and a hierarchical time-sensitive transformer, generating coherent long videos from short training clips.

  • Motivation: Long video synthesis remains underexplored because existing work largely targets frame quality and short-range transitions rather than consistent themes and storylines across many frames.Modeling long-range temporal dependence is required for meaningful videos with more frames.
  • Efficiency: Training on short clips avoids the prohibitively high training and inference costs associated with extending GAN-based methods directly to longer videos.Prior examples required 32–512 TPU replicas and 12–96 hours when trained on 16-frame videos.
  • Approach: TATS removes undesired temporal dependence from a 3D-VQGAN and enables a transformer to capture longer-range temporal dependence.These are the paper’s two core design insights for generating thousands of frames from short clips.
  • Time-agnostic VQGAN: Zero padding corrupts latent tokens under sliding-window inference, while replicate padding better approximates real frames and preserves visual quality beyond the training length.The corruption varies with temporal position and is addressed without additional computational overhead.
  • Time-sensitive transformer: A hierarchical transformer first generates sparse latent frames for global structure, then an interpolation transformer fills skipped frames while attending to sparse frames on both sides.TATS demonstrates generated samples of 1024 frames.
  • Results: The model achieves state-of-the-art short- and long-video results across UCF-101, Sky Time-lapse, Taichi-HD, and AudioSet-Drum, with text and audio conditioning producing meaningful videos.The paper also generalizes popular metrics to evaluate long-term quality, diversity, and coherence.

2 Methodology

The methodology combines a time-agnostic VQGAN for position-independent spatiotemporal tokenization with a time-sensitive hierarchical transformer for long-range temporal modeling. Replicate padding mitigates boundary-token corruption, enabling sliding-window generation beyond the training length, while hierarchical prediction supports video-wide coherence and conditioning can add temporal information.

  • Extending the VQGAN framework for video generation: 3D-VQGAN replaces 2D convolutions with 3D convolutions to encode videos into discrete spatiotemporal tokens and reconstruct them with perceptual and adversarial objectives.The model uses spatial and temporal discriminators to encourage frame quality and plausible motion, alongside feature matching for training stability.
  • Extending the VQGAN framework for video generation: Sliding-window inference repeatedly conditions on the most recent token slices, drops the earliest ones, and samples longer sequences from the transformer.The vanilla video VQGAN and transformer degrade beyond the training length, motivating the time-agnostic encoder design.
  • Time-Agnostic VQGAN: Zero temporal padding makes the encoder time-agnostic, avoiding position-dependent token corruption that otherwise breaks sliding-window generation.Zero padding affects tokens unevenly near temporal boundaries, so removing it is crucial for matching inference sequences to training representations.
  • Time-Agnostic VQGAN: Replicate padding approximates real-frame padding without computational overhead, allowing encoded tokens to preserve visual quality beyond the training video length.Real-frame padding can require O(Ld) extra frames and computations, whereas replicate padding reuses boundary slices under the assumption that videos remain frozen beyond their given length.
  • Time-Sensitive Transformer: A hierarchical time-sensitive transformer first generates sparse latent frames for global structure and then models intervening tokens to capture longer temporal dependence.Temporal information remains necessary for consistent themes, storylines, and long-range dependence; conditioning on audio or text can provide additional temporal information.

3 Experiments

The experiments evaluate standard and long video generation across multiple datasets, metrics, conditions, and qualitative properties. TATS achieves strong short-video results and generates long videos with delayed quality degradation, coherent themes, recurrent actions, smooth transitions, and controllable text-conditioned meanings.

  • Experimental Setups: The evaluation covers unconditional, class-conditioned, audio-conditioned, and text-conditioned video generation across UCF-101, Sky Time-lapse, Taichi-HD, AudioSet-Drum, and MUGEN.Metrics include FVD, KVD, IS, SSIM, and PSNR, with long-video evaluation focused on 1024-frame generations.
  • Short Video Generation: TATS-base achieves state-of-art FVD and KVD on UCF-101 and Taichi-HD, state-of-art KVD on Sky Time-lapse, and improved audio-conditioned quality.On UCF-101, class conditioning improves IS from 57.63 to 79.28; TATS-base also improves IS by 76.2% over a previous method with synthetic initial frames.
  • Long Video Generation: TATS-base delays long-video quality degradation relative to Vanilla VQGAN, MoCoGAN-HD, and CCVS, while TATS-hierarchical further improves long-term quality.The comparison uses FVD changes across non-overlapping 16-frame clips extracted from generated long videos.
  • Long Video Generation: TATS-hierarchical outperforms baselines on CCS and ICS, with more than half of videos still classified consistently in the final 16 frames of 1024-frame videos.ICS captures class shifts more accurately than CCS, whereas CCS is more intuitive and supports area-under-the-curve analysis.
  • Qualitative Evaluation: On qualitative evaluation, generated 1024-frame videos contain realistic recurrent actions with unique, non-identical motions rather than copied short loops.The model also produces natural transitions across different weather and timing conditions in long sky videos.
  • Qualitative Evaluation: Temporal conditioning enables controllable meaningful synthesis, including changing a video's storyline by modifying its text.The paper demonstrates this capability by replacing “killed by” with “kills” in a MUGEN video.

4 Related Work

Related work uses GANs, recurrent models, and transformers to generate videos, but these approaches face challenges when extended to long sequences.

  • GAN-based video generator: GAN-based methods can generate plausible short videos, but longer-video training and inference require prohibitively high memory and time costs.
  • Autoregressive methods: Autoregressive methods reduce training-cost constraints through sequential prediction using recurrent temporal noise, pixels, or latent tokens.
  • Long video generation: These prior strategies handle temporal dynamics differently, but their potential for long video generation remains associated with important challenges.

5 Conclusion

TATS is trained on clips with tens of frames yet generates thousands of frames using a sliding window, including meaningful text- and audio-conditioned videos.

  • TATS combines a time-agnostic VQGAN with a time-sensitive transformer to generate thousands of frames from clips with tens of frames.The model uses a sliding window during inference and supports text and audio conditioning.

A Implementation Details

The appendix details TATS's training stabilization, padding analysis, ablations, and interpolation attention design.

  • The implementation details cover stabilizing 3D-VQGAN training with GAN losses.
  • The section analyzes temporal dependence from zero padding and compares potential solutions through ablations.
  • It also describes the model's interpolation attention mechanism.

A.1 Training video VQGAN with GAN losses

Directly adding GAN losses to VideoGPT causes discriminator collapse, while architectural changes are used to stabilize video VQGAN training.

  • Figures 10 and 11 document discriminator collapse and blob-shaped artifacts associated with normalization layers in VQGAN.
  • Directly applying GAN losses to VideoGPT causes severe discriminator collapse and destabilizes decoder and reconstruction losses.Discriminator losses saturate quickly and provide nonsensical gradients to the decoder.
  • GAN-loss instability is attributed partly to axial-attention layers interacting poorly with the training objective.The proposed alternative uses a pure convolution architecture similar to VQGAN for video compression.
  • A more powerful decoder helps reconstruction follow the discriminator during GAN-loss training.The decoder doubles feature maps whenever resolutions are halved, unlike VideoGPT's described design.

A.2 Zero paddings inhibit sliding attention window

Zero padding makes sliding-window generation difficult because inference token sequences differ from training sequences; padding choices trade temporal consistency against computational cost.

  • A.2 Zero paddings inhibit sliding attention window: Zero padding corrupts latent tokens during sliding-window inference, producing sequences that differ substantially from those observed during training.
  • A.2 Zero paddings inhibit sliding attention window: The transformer cannot generalize to sequences beginning with shifted latent tokens because such sequences are absent from its training data.
  • A.2 Zero paddings inhibit sliding attention window: The padding problem affects VQVAE-transformer generative models and is especially severe near borders and in long videos with short temporal lengths.The issue is less visible in 2D-VQGAN because generation selects spatially centered tokens, whereas long-video tokens remain close to temporal borders.
  • A.2 Zero paddings inhibit sliding attention window: Removing more padding makes the encoder more time-agnostic, while removing all padding and adding real frames achieves equivariance score = 1 at substantially higher memory and computational cost.The study measures this trade-off with an equivariance score across padding strategies.
  • A.2 Zero paddings inhibit sliding attention window: Interpolation causal attention lets generated tokens attend to preceding tokens and anchor tokens at both ends, without leaking information from the final anchor frame.

B.1 Dataset and evaluation details

The approach is validated across five video datasets spanning actions, sky time-lapses, Taichi, drums with audio, and gameplay. Evaluation uses generated samples for short- and long-video metrics, with repeated runs for variability estimates.

  • Datasets: The evaluation covers UCF-101, Sky Time-lapse, Taichi-HD, AudioSet-Drum, and MUGEN.These datasets cover action recognition, sky conditions, Taichi performance, drum videos with audio, and gameplay recordings.
  • Datasets: UCF-101 contains 101 classes and 13,320 videos, with 9,537 training videos following the official split.Frame counts by class are also reported.
  • Datasets: Sky Time-lapse officially releases 2,647 videos, despite a reported collection size of 5,000.Training and testing follow the split used by prior work.
  • Datasets: AudioSet-Drum has 6,000 training and 1,000 test videos, each 90 frames, with STFT audio features and image quality measured at frame 45.The evaluation follows the referenced prior setting.
  • Evaluation: For evaluation, the study generates 2,048 short videos, 512 long videos, and 10,000 videos for IS, repeating evaluations 10 times to report standard deviations.CCS and ICS for long videos also use 512 generated videos.

B.2 Training and inference details

Training uses fixed VQGAN and transformer configurations, while inference comparisons measure the time required to generate 1024-frame videos. TATS-Hierarchical is reported as the fastest VQVAE-transformer method in that comparison.

  • VQGAN training: The VQGAN uses a codebook vocabulary of K = 16,384 and embedding size c = 256.GAN losses begin after reconstruction generally converges after 10K steps.
  • Transformer training: Both autoregressive and interpolation transformers use 24 layers, 16 heads, and embedding size 1024.They are trained on 8 NVIDIA V100 32GB GPUs with batch size 3 per GPU.
  • Inference cost: The comparison measures the time required to generate a 1024-frame video across different methods.The results are reported in Table 3.
  • Inference cost: TATS-Hierarchical generates videos in 1/3 the time of CCVS and 1/5 the time of VideoGPT among VQVAE-transformer methods.The paper notes that sparse attention could provide further improvements.

C Additional results on long video generation

Additional results show repeated actions and smooth transitions in long generated videos, while also exposing degradation and failure modes. These include periodic artifacts, accumulated sliding-window errors, unrealistic content, and degeneration after completed events.

  • Repeated actions and smooth transitions: Long-video examples include repeated actions across UCF-101 and smooth transitions in UCF-101, Sky Diving, and Taichi-HD videos.The examples also include smoothly transforming pants color in a Taichi video.
  • Repeated actions and smooth transitions: Some smooth UCF-101 and Taichi-HD transitions still contain unrealistic content.The UCF-101 Sky Diving example is attributed to limited data availability per class.
  • Implications: The paper suggests separating content and motion generation to address unrealistic content in some long-video examples.This proposal is presented for cases with limited per-class data and smooth but unrealistic transitions.
  • Failure cases: Sliding-window errors can accumulate, producing repeated tokens and deteriorating video quality, especially in regions with large motions.A Boxing Punching Bag example begins degrading around 300 frames.
  • Failure cases: Videos can degrade quickly when a thematic event ends, as in Long Jump videos that transition to another scene after the action finishes.The reported result is degeneration following the completed action.
  • Comparison with DIGAN: DIGAN exhibits periodic artifacts on the Sky dataset, although this quality degradation is not reflected in FVD.The paper attributes the artifacts to sinusoidal positional encodings and conjectures that FVD misses them because of a Kinetics-domain gap.
Loading 2204.03638v4…