Source-linked AI summary
Make-A-Video: Text-to-Video Generation without Text-Video Data
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, Yaniv Taigman
TL;DR
Text-to-video generation lacks the paired text-video data available for text-to-image learning, motivating a way to transfer image-generation knowledge while learning motion from unlabeled videos. Make-A-Video extends T2I models with factorized spatiotemporal modules and a video-resolution pipeline, and reports state-of-the-art T2V generation. The approach remains limited in learning video-only text associations and generating longer, multi-scene stories, and may reproduce social biases from web data.
Problem
T2V generation lacks large-scale paired text-video data, despite the availability of extensive text-image data and unlabeled video footage.
Method
Make-A-Video transfers text–visual knowledge from T2I models and learns motion from unlabeled videos using spatiotemporally factorized modules, frame interpolation, and super-resolution.
Results
Make-A-Video reports state-of-the-art T2V generation in quantitative and qualitative evaluations.
Takeaways & Limitations
Labeled images combined with unlabeled video footage can achieve T2V generation without relying on paired text-video data.
Takeaways & Limitations
The approach cannot learn text associations for phenomena inferable only from videos and does not yet generate longer, multi-scene, detailed stories; web training may also exaggerate social biases.
Abstract
from arXiv · showhide
We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V). Our intuition is simple: learn what the world looks like and how it is described from paired text-image data, and learn how the world moves from unsupervised video footage. Make-A-Video has three advantages: (1) it accelerates training of the T2V model (it does not need to learn visual and multimodal representations from scratch), (2) it does not require paired text-video data, and (3) the generated videos inherit the vastness (diversity in aesthetic, fantastical depictions, etc.) of today's image generation models. We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. First, we decompose the full temporal U-Net and attention tensors and approximate them in space and time. Second, we design a spatial temporal pipeline to generate high resolution and frame rate videos with a video decoder, interpolation model and two super resolution models that can enable various applications besides T2V. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.
1 INTRODUCTION
Make-A-Video transfers text–image knowledge to text-to-video generation and learns motion from unlabeled video, avoiding paired text-video data. Its factorized spatiotemporal design and separate interpolation and super-resolution models support high-quality, high-frame-rate video generation.
- Motivation and approach: Make-A-Video learns text–visual correspondences from T2I models and realistic motion from unpaired video, generating videos without paired text-video data.This avoids training T2V models from scratch and uses unsupervised video footage to learn how entities move and interact.
- Results: Make-A-Video is reported to achieve state-of-the-art T2V generation, with coherent motion across diverse visual concepts.Examples include realistic dog motion and camera-driven scene changes around nearly static books.
- Method: The method extends a diffusion-based T2I model to T2V through a spatiotemporally factorized diffusion model.The architecture adds temporal information while transferring knowledge from the pretrained image model.
- Method: Spatial and temporal super-resolution strategies generate high-definition, high-frame-rate videos from textual input.The pipeline includes frame interpolation and super-resolution components in addition to the video decoder.
- Evaluation: The model is evaluated against existing T2V systems using quantitative and qualitative measures, including a 300-prompt zero-shot human evaluation set.The authors plan to release the evaluation prompts.
2 PREVIOUS WORK
Prior T2V work is constrained by scarce paired text-video data and the complexity of video modeling. Make-A-Video instead adapts T2I priors with trainable temporal components and open-source data, while adding a high-resolution video pipeline.
- Text-to-video generation: T2V progress lags T2I because large-scale high-quality text-video pairs are scarce and video data is higher-dimensional.Earlier approaches often focused on simple domains such as moving digits or specific human actions.
- Text-to-video generation: GODIVA uses a 2D VQVAE and sparse attention, while NÜWA extends it through unified multitask representations.These systems target more realistic scenes and multiple generation tasks, respectively.
- Recent approaches: CogVideo builds on a frozen T2I model with temporal attention, whereas VDM uses a space-time factorized U-Net with joint image-video training.Both collected private text-video pairs, unlike Make-A-Video's use of open-source datasets.
- Leveraging image priors: Image-prior methods simplify video generation, but fixed image models can restrict T2V generation and some approaches do not exploit massive text-image datasets.Make-A-Video fine-tunes its T2I model rather than freezing it.
- Make-A-Video: Make-A-Video combines pseudo-3D convolution and temporal attention with a decoder, interpolation model, and super-resolution stages for video output.Its architecture is designed to improve temporal information fusion and produce higher-resolution, higher-frame-rate videos.
3 METHOD
Make-A-Video extends a pretrained text-to-image model into a spatiotemporal text-to-video system, learning motion from unlabeled videos while preserving spatial knowledge. Its pipeline combines factorized temporal layers, frame interpolation, and super-resolution to generate higher-quality, higher-frame-rate videos.
- 3 METHOD: Make-A-Video combines a pretrained T2I model, spatiotemporal convolution and attention layers, and frame interpolation for text-to-video generation.The system also includes spatial and spatiotemporal super-resolution components.
- 3.2.1 PSEUDO-3D CONVOLUTIONAL LAYERS: Pseudo-3D convolutions follow each pretrained 2D convolution with a 1D temporal convolution, sharing information across spatial and temporal axes without full 3D convolution cost.The 2D layers retain pretrained spatial weights, while the 1D layers are trained from scratch and initialized as identity functions.
- 3.2.2 PSEUDO-3D ATTENTION LAYERS: Pseudo-3D attention stacks temporal attention after spatial attention to approximate spatiotemporal attention while avoiding the memory cost of full 3D attention.The spatial attention is initialized from the pretrained T2I model, and temporal attention is initialized as an identity function.
- 3.2 SPATIOTEMPORAL LAYERS: Frame-rate conditioning provides both augmentation for limited video data and control over the generated video's frames per second.Training samples 16 frames with randomly varied frame rates from 1 to 30.
- 3.3 FRAME INTERPOLATION AND SUPER-RESOLUTION: A masked frame interpolation network increases frame count through interpolation or extrapolation, while super-resolution modules improve spatial and spatiotemporal detail.The spatiotemporal super-resolution module operates across frames to reduce flickering, whereas the highest-resolution module operates spatially because temporal extension is computationally constrained.
- 3.4 TRAINING: The T2I backbone is trained on text-image pairs, while newly added temporal layers are fine-tuned on unlabeled video data.The components are trained independently, and only the prior receives text as input.
4 EXPERIMENTS
The experiments evaluate Make-A-Video on public, unpaired video data using zero-shot and fine-tuning benchmarks, automatic metrics, human preferences, and qualitative applications. Make-A-Video shows strong zero-shot generalization and improved video quality, text correspondence, motion consistency, and semantic interpolation.
- 4.1 DATASETS AND SETTINGS: The models use a 2.3B English text-image subset, WebVid-10M, and a 10M HD-VILA subset, with only videos—not aligned text—used for video training.Automatic evaluation uses zero-shot UCF-101 and MSR-VTT settings, while the human evaluation uses 300 fixed prompts.
- 4.2 QUANTITATIVE RESULTS: MSR-VTT zero-shot performance is much better than GODIVA and NÜWA, and outperforms CogVideo with both Chinese and English inputs.The evaluation generates one 16 × 256 × 256 sample per prompt for Make-A-Video and CogVideo.
- 4.2 QUANTITATIVE RESULTS: On UCF-101, zero-shot performance is competitive with approaches trained on the benchmark, while fine-tuning achieves state-of-the-art results with a significant reduction in FVD.The authors associate the reduced FVD with more coherent generated videos.
- 4.2 QUANTITATIVE RESULTS: Human evaluations report better video quality and text-video faithfulness for Make-A-Video across comparisons with CogVideo and VDM.The comparisons cover DrawBench, the authors’ 300-prompt test set, and 28 examples from the VDM website.
- 4.3 QUALITATIVE RESULTS: Qualitative comparisons show richer content, motion consistency, and text correspondence, while additional applications include image animation, semantic image interpolation, and video variation.The interpolation model is described as producing more semantically meaningful transitions than FILM.
5 DISCUSSION
The discussion frames Make-A-Video as combining labeled images with large-scale unlabeled video to learn world dynamics without relying on labeled data. It also identifies limitations involving video-only text phenomena, long and multi-scene generation, visual cleanliness, and social bias.
- 5 DISCUSSION: Unsupervised learning from orders of magnitude more videos helps reduce reliance on labeled data when combined with labeled images.The discussion presents this combination as the demonstrated approach for learning world dynamics.
- 5 DISCUSSION: The approach cannot learn text associations for phenomena that can only be inferred from videos, such as waving direction.Generating longer videos with multiple scenes, events, and more detailed stories remains future work.
- 5 DISCUSSION: The models may have learned and exaggerated harmful social biases from large-scale web data.The authors note that NSFW images and toxic words were removed from the T2I training data, while the datasets are publicly available.