Source-linked AI summary
Video Generation From Text
Yitong Li, Martin Renqiang Min, Dinghan Shen, David Carlson, Lawrence Carin
TL;DR
Text-to-video generation remains difficult because text must specify both static scene structure and dynamic motion. The paper proposes a hybrid VAE-GAN framework that generates a text-conditioned gist and uses Text2Filter to model motion, along with an automatically constructed text-video corpus. The resulting framework generates plausible, diverse, text-consistent videos and outperforms baselines that directly adapt text-to-image procedures.
Problem
Text-to-video generation is challenging because the model must extract both static and dynamic information from text.
Method
A hybrid VAE-GAN framework generates a text-conditioned gist for background structure and applies a text-derived filter to model dynamic motion.
Results
The framework generates plausible and diverse videos that reflect the input text and significantly outperforms baselines adapting text-to-image procedures.
Takeaways & Limitations
Separating gist generation from motion generation helps enforce text-conditioned static backgrounds and capture dynamic motion information.
Takeaways & Limitations
The CNN video generator has difficulty preserving object shape while producing reasonable motion; human-pose or skeleton features are proposed as a future improvement.
Abstract
from arXiv · showhide
Generating videos from text has proven to be a significant challenge for existing generative models. We tackle this problem by training a conditional generative model to extract both static and dynamic information from text. This is manifested in a hybrid framework, employing a Variational Autoencoder (VAE) and a Generative Adversarial Network (GAN). The static features, called "gist," are used to sketch text-conditioned background color and object layout structure. Dynamic features are considered by transforming input text into an image filter. To obtain a large amount of data for training the deep-learning model, we develop a method to automatically create a matched text-video corpus from publicly available online videos. Experimental results show that the proposed framework generates plausible and diverse videos, while accurately reflecting the input text information. It significantly outperforms baseline models that directly adapt text-to-image generation procedures to produce videos. Performance is evaluated both visually and by adapting the inception score used to evaluate image generation in GANs.
1 Introduction
Text-to-video generation is challenging because the text must determine both a video's broad visual structure and object motion. The proposed framework separates these static and dynamic features through gist generation and text-conditioned motion modeling.
- Text-to-video generation is less explored than text-to-image generation and has more degrees of freedom because text must specify both appearance and motion.
- Directly adapting text-to-image methods can produce videos whose motion is not influenced by the input text.
- The framework decomposes generation into a conditional VAE that produces a text-conditioned gist and a video generator that adds content and motion from the gist and text.The gist represents background color and object layout, while the video generator extracts dynamic and detailed information.
- Text-derived filter kernels are applied to the gist to form a text-gist feature vector that models their interaction more effectively than simple feature concatenation.
- The contributions include a gist-based framework, Text2Filter for dynamic features, and automatic construction of a matched text-video corpus from YouTube videos.
2 Related Work
Prior work spans video prediction, image-conditioned video generation, unconditional GAN-based video generation, and conditional text-to-image generation. The paper addresses the difficulty of adapting these approaches to videos, whose spatiotemporal structure requires modeling both static and motion information.
- Video prediction learns to transfer given frames into subsequent frames, whereas complete video generation requires more than predicting future frames.
- Image-conditioned video methods must distinguish potentially moving objects in a supplied image, unlike text-to-video generation, which conditions on captions.
- GAN-based video-generation methods can synthesize videos from randomly sampled noise, with later work injecting noise at each time step through an RNN.
- VAE training maximizes a variational lower bound, while GAN training uses a minimax game between a generator and discriminator.
- Conditional generative models provide text or other side information to the generator and discriminator, but converting such methods to text-video pairs is non-trivial.
- Directly adapting text-to-image GANs performs poorly for videos because the generator must extract both static and motion information from a higher-dimensional 4D tensor.
3 Model Description
The model decomposes text-to-video generation into a text-conditioned gist for static structure and a GAN-based generator for motion and detail. Text2Filter combines text with the gist, while reconstruction, adversarial, and variational objectives train the system.
- The framework contains a conditional gist generator, a video generator, and a video discriminator.
- Gist Generator: The conditional VAE generates a gist representing the text-conditioned background and object layout, using the first video frame as its practical image target.
- Gist Generator: At test time, the video-frame encoder is discarded, so text alone produces a sketch that conditions subsequent video generation.
- Video Generator: The video GAN uses scene dynamic decomposition: a learned mask blends a moving component with a static background repeated across time.
- Text2Filter: Text2Filter converts encoded text into a 3D filter applied to the gist, producing a text-gist vector that better couples text with motion generation than concatenation.
- Objective and Training: Training combines CVAE and GAN losses with an L1 reconstruction loss, whose inclusion empirically improves performance.The loss weights are γ1 = γ2 = 1 and γ3 = 0.1 in the experiments.
4 Dataset Creation
The dataset-creation method automatically builds matched text-video data from online videos because no standard public text-to-video dataset exists. It collects metadata, removes outliers, screens titles and quality, and expands the corpus with Kinetics videos.
- The authors propose downloading videos with matching text descriptions to address the lack of a standard public text-to-video dataset.
- For each keyword, the process collects YouTube videos and metadata including titles, descriptions, durations, and tags before cleaning outliers.
- The curation method identifies the ten most frequent tags and matches selected tags to words in existing data.
- Kinetics videos are added to expand the dataset, although their visual and textual quality and consistency are mixed.
- After screening, the resulting collection contains about 400 videos per category, and Kinetics is considered cleaner than YouTube8M for this process.
5 Experiments2
The experiments compare four text-to-video models and show that combining gist generation with Text2Filter improves static-background and motion conditioning. Qualitative and classifier-based evaluations also expose limitations in motion realism and object-shape preservation.
- 5.1 Video Preprocessing: The dataset uses short, qualified clips to avoid sudden shot changes and fast-changing backgrounds that challenge current video-generation techniques.Videos are sampled at 25 frames per second and limited to 32 frames at 64 × 64 resolution.
- 5.2 Models for Comparison: The comparison includes direct text-to-video generation, pair-based conditioning, gist-only generation, and the complete gist-plus-Text2Filter model.DT2V omits gist generation; PT2V adds a discriminator for real, synthetic, or mismatched video-text pairs; GT2V omits Text2Filter.
- 5.2 Models for Comparison: The complete T2V model provides both background and motion features, whereas DT2V and PT2V fail to generate plausible videos and GT2V lacks adequate motion.The experiments attribute the full model’s behavior to the intermediate gist-generation step and the Text2Filter component.
- 5.3 Static Features: Gists encode static information from text, changing background colors and layouts for settings such as grass, sea, swimming pools, and snow.Gists for different motions at the same location are reported to be similar.
- 5.4 Motion Features: The model can generate different motions with similar backgrounds, but its CNN video generator struggles to preserve object shape while producing reasonable motion.The paper suggests using specific features such as human pose or skeleton generation as a possible improvement.
6 Conclusion
The paper proposes a hybrid VAE-GAN framework for generating videos from text, using gist generation for static backgrounds and Text2Filter for dynamic motion information.
- The framework uses an intermediate gist-generation step to enforce text-conditioned static video backgrounds.
- Text2Filter helps capture dynamic motion information from text.
- The authors plan to improve visual quality by generating human pose or skeleton features with a more powerful video generator.