Source-linked AI summary
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, Nan Duan
TL;DR
Text-to-video generation is computationally demanding and difficult to evaluate, while prior work has limited generalization from simple or small datasets. GODIVA uses VQ-VAE, three-dimensional sparse attention, and large-scale pretraining to generate open-domain videos autoregressively; experiments show fine-tuning and zero-shot capability, alongside the proposed Relative Matching metric.
Problem
Text-to-video generation requires diverse, spatially and temporally coherent videos, while existing work has limited generalization and evaluation remains challenging.
Method
GODIVA converts video pixels into discrete VQ-VAE tokens and autoregressively generates them from language with three-dimensional sparse attention.
Results
GODIVA supports downstream fine-tuning and good zero-shot generation on unseen texts, while Relative Matching evaluates visual quality and semantic match.
Takeaways & Limitations
GODIVA provides an open-domain text-to-video pretrained model with demonstrated fine-tuning and zero-shot generation capabilities.
Takeaways & Limitations
Generating long, high-resolution videos and automatically evaluating text-to-video generation remain challenges.
Abstract
from arXiv · showhide
Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation. Existing works typically experiment on simple or small datasets, where the generalization ability is quite limited. In this work, we propose GODIVA, an open-domain text-to-video pretrained model that can generate videos from text in an auto-regressive manner using a three-dimensional sparse attention mechanism. We pretrain our model on Howto100M, a large-scale text-video dataset that contains more than 136 million text-video pairs. Experiments show that GODIVA not only can be fine-tuned on downstream video generation tasks, but also has a good zero-shot capability on unseen texts. We also propose a new metric called Relative Matching (RM) to automatically evaluate the video generation quality. Several challenges are listed and discussed as future work.
1 Introduction
Text-to-video generation must produce semantically coherent, diverse videos while handling substantial spatial and temporal complexity. GODIVA addresses this task with a VQ-VAE representation and three-dimensional sparse attention for autoregressive generation.
- 1 Introduction: Real-world video generation remains challenging because videos require diverse content with both spatial and temporal coherence.
- 1 Introduction: Text-to-video generation requires understanding a natural description and creating a semantically consistent video.The task must account for both the large number of generated pixels and coherence among them.
- 1 Introduction: GODIVA uses a VQ-VAE auto-encoder to represent continuous video pixels as discrete video tokens before autoregressive generation.
- 1 Introduction: Its three-dimensional sparse attention models temporal, row, and column information while conditioning generation on language.The design considers the input text, previous generated frames at the same position, and spatially aligned rows and columns.
- 1 Introduction: The model is pretrained for open-domain text-to-video generation and evaluated in both fine-tuning and zero-shot settings.The paper also introduces Relative Matching for evaluating visual quality and semantic match.
2 Related Works
Prior video-generation work largely focuses on video prediction, while text-to-video research remains relatively limited and often uses GAN-based methods. GODIVA instead applies VQ-VAE and sparse attention to text-conditioned open-domain video generation.
- 2.1 Video-to-video generation: Video-to-video generation predicts future frames from the first few frames of a video.
- 2.1 Video-to-video generation: Existing video-to-video methods include deterministic, GAN-based, and other approaches that model spatial and temporal video information.
- 2.2 Text-to-image and text-to-video generation: Text-to-video generation is more challenging than text-to-image generation because it must preserve both spatial and temporal visual coherence.
- 2.3 Text-to-video generation: Earlier text-to-video methods use GANs, 3D convolutions, conditional filters, or LSTM-based networks for fixed- or varying-length videos.
- 2.3 Text-to-video generation: GODIVA uses VQ-VAE for text-to-video generation and is presented as the first paper to use VQ-VAE for this task.It also incorporates three-dimensional sparse attention to model sparse relations between visual tokens.
3 The GODIVA Method
GODIVA represents videos as discrete VQ-VAE tokens and autoregressively generates those tokens from text. Three-dimensional sparse attention models temporal, row, and column dependencies while reducing computation.
- 3.1 Frame-wise video auto-encoder: GODIVA encodes each video frame into a lower-dimensional discrete latent representation using a VQ-VAE encoder, quantization codebook, and decoder.The encoder produces latent regions, which are quantized into indexes and embedded before decoding reconstructed frames.
- 3.2 GODIVA video generator: A decoder models P(z|t) autoregressively, using text embeddings as input and predicting the sequence of discrete visual tokens.A linear layer maps hidden states to the VQ-VAE vocabulary, and cross-entropy trains the predicted visual tokens.
- 3.2 GODIVA video generator: The ground-truth video becomes a sequence of M = L × h × w discrete visual tokens, each mapped into embeddings with video positional information.For 64 × 64 videos with 10 frames, the representation contains M = 2560 tokens.
- 3.2 GODIVA video generator: Three-dimensional sparse attention separately models temporal, row, and column relations rather than attending across the entire visual-token sequence.The sparse attention axes attend to indexes in the preceding axis, and the layers are stacked alternately.
- 3.2 GODIVA video generator: The sparse design reduces attention complexity from O((Lhw)^2) to O(Lhw(L + h + w)).This reduction addresses the large number of visual tokens required for real-world videos.
4 Experiments
Experiments evaluate GODIVA qualitatively on real-world and Moving MNIST data, and quantitatively with automatic and human metrics. Results indicate semantic alignment, visual quality, and the usefulness of Relative Matching.
- Automatic Evaluation Metrics: Relative Matching divides generated-video SIM by ground-truth-video similarity to reduce the influence of CLIP’s domain bias.SIM averages text-frame similarities over the video, while RM provides a relative score.
- Qualitative Results: GODIVA generates clear frames and coherent semantics in zero-shot real-world video generation, including changing scenes for the query “Play golf on grass.”Compared with prior methods, the examples show transitions from a grass field to a close-up athlete shot and ball-hitting action.
- Qualitative Results: GODIVA generates semantically correct unseen Moving MNIST videos despite no training example matching the exact described motion.The authors interpret this as learning text-video semantic alignment rather than retrieving the most similar training video.
- Quantitative Results: SIM and RM follow the same trend as human evaluation metrics across GODIVA ablations on MSR-VTT.The experiments compare automatic metrics with visual realisticity and semantic consistency judgments.
- Quantitative Results: 98.34 RM is achieved after 32 samples from the top 10 probabilities are ranked with CLIP during inference.The reported procedure significantly improves performance relative to the corresponding generation setting.
5 Conclusions
GODIVA combines VQ-VAE discrete visual tokens with three-dimensional sparse attention for open-domain text-to-video generation and introduces Relative Matching for automatic evaluation. Experiments support fine-tuning and zero-shot generation, while long high-resolution videos and automatic evaluation remain challenging.
- Conclusions: GODIVA uses VQ-VAE discrete visual tokens and three-dimensional sparse attention to generate open-domain videos from natural descriptions.The sparse attention mechanism is intended to reduce computation cost.
- Conclusions: Relative Matching is proposed as an automatic metric for video-generation quality.The metric is presented as part of the paper’s evaluation contribution.
- Conclusions: Experiments show that GODIVA supports downstream fine-tuning and has zero-shot capability on unseen texts.The conclusion states both capabilities without restricting them to a single evaluation dataset.
- Future Work: Generating long, high-resolution videos remains challenging because 64×64 videos with 10 frames already require 2560 visual tokens.The conclusion identifies this computational scale as a continuing challenge.
- Future Work: Automatic text-to-video evaluation remains challenging, and video-based CLIP is suggested as a possible future direction for semantic consistency.The conclusion also identifies open-domain GAN-based generation as a continuing research direction.
6 Appendix
The appendix presents additional GODIVIA-generated video samples, including prompts involving a suited man and a girl with judges.
- Figure 6 presents more samples generated by GODIVIA, including a suited man sitting on a chair and talking before a camera.The associated input text describes this scene.
- Figure 7 presents more samples generated by GODIVIA, including a girl and judges talking by voice.The associated input text describes this scene.