Source-linked AI summary
MagicVideo: Efficient Video Generation With Latent Diffusion Models
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, Jiashi Feng
TL;DR
Text-to-video generation remains difficult because paired data are scarce and video dynamics are complex, while existing diffusion pipelines are computationally expensive. MagicVideo uses latent diffusion with an efficient video U-Net, lightweight adaptor, and directed temporal attention to generate text-aligned videos. The paper reports realistic and smooth video generation with improved efficiency and applications beyond text-to-video.
Problem
Text-to-video generation is limited by scarce paired video-text data, complex temporal dynamics, and high computational cost.
Method
MagicVideo maps videos into a low-dimensional latent space and adapts pretrained image-generation components with a lightweight adaptor and directed temporal attention.
Results
MagicVideo generates realistic, smooth, temporally coherent videos with consistent object identity and is reported to be more efficient than recent methods.
Takeaways & Limitations
The framework supports efficient text-conditional video generation and applications including image-to-video, video variations, and video editing.
Takeaways & Limitations
Video generation can be used for malicious video editing, and pretrained Stable Diffusion weights may inherit ethical issues from the LAION dataset.
Abstract
from arXiv · showhide
We present an efficient text-to-video generation framework based on latent diffusion models, termed MagicVideo. MagicVideo can generate smooth video clips that are concordant with the given text descriptions. Due to a novel and efficient 3D U-Net design and modeling video distributions in a low-dimensional space, MagicVideo can synthesize video clips with 256x256 spatial resolution on a single GPU card, which takes around 64x fewer computations than the Video Diffusion Models (VDM) in terms of FLOPs. In specific, unlike existing works that directly train video models in the RGB space, we use a pre-trained VAE to map video clips into a low-dimensional latent space and learn the distribution of videos' latent codes via a diffusion model. Besides, we introduce two new designs to adapt the U-Net denoiser trained on image tasks to video data: a frame-wise lightweight adaptor for the image-to-video distribution adjustment and a directed temporal attention module to capture temporal dependencies across frames. Thus, we can exploit the informative weights of convolution operators from a text-to-image model for accelerating video training. To ameliorate the pixel dithering in the generated videos, we also propose a novel VideoVAE auto-encoder for better RGB reconstruction. We conduct extensive experiments and demonstrate that MagicVideo can generate high-quality video clips with either realistic or imaginary content. Refer to \url{https://magicvideo.github.io/#} for more examples.
1. Introduction
MagicVideo addresses the data and computational challenges of text-to-video generation by learning video distributions in a low-dimensional latent space and adapting image-generation components to video. Its designs target efficient training while producing temporally coherent, text-aligned videos.
- Challenges: Video generation is hindered by scarce paired video-text data, redundant video content, complex temporal dynamics, and high computational cost.A recent video diffusion model required 6–10 seconds and 75G GPU memory per diffusion iteration for 16-frame, 64 × 64 coarse clips.
- Approach: MagicVideo adopts latent diffusion to model video data in a lower-dimensional space rather than directly denoising raw RGB videos.A VAE maps videos into latent codes, reducing the spatial dimensions processed during diffusion denoising.
- Approach: Its architecture combines shared 2D convolutions with temporal computation operators, enabling initialization from pretrained text-to-image convolution weights.This design is intended to improve data efficiency while modeling spatial and temporal video features.
- Approach: A lightweight frame-wise adaptor adjusts feature distributions across frames, using a few scalar parameters while exploiting frame correlations.The adaptor helps preserve temporal consistency without the memory cost of dedicated per-frame temporal convolutions.
- Approach: Directed temporal attention computes each future frame from preceding frames while preventing future information from altering earlier frames.The authors state that this improves motion consistency over conventional bidirectional self-attention.
- Results: MagicVideo generates diverse, temporally coherent, text-aligned videos and is reported to produce realistic, smooth videos efficiently.The experiments also demonstrate applications to image-to-video and video-to-video generation.
2. Related works
Related work establishes diffusion models as effective image-generation tools and describes their extension to video through 3D U-Nets, autoregressive modeling, and cascaded or multi-stage pipelines. MagicVideo is positioned within controllable, text-conditional video generation.
- Diffusion-based generative models: Diffusion-based models have achieved strong results in image generation and editing, including text-conditioned generation with DDPMs and latent diffusion.Latent diffusion encodes images into a low-dimensional space before modeling their latent representations with a time-conditional U-Net.
- Video generation: Prior video-generation approaches include GAN-based, autoregressive, and diffusion-based methods for unconditional or controllable video synthesis.Diffusion approaches extended image DDPMs with 3D U-Nets, while autoregressive systems modeled subsequent frames or video sequences conditioned on text and prior frames.
- Video generation: Text-conditional video generation has used autoregressive models, cascaded diffusion pipelines, and multi-stage methods that combine text-to-image knowledge with video modeling.These approaches include CogVideo, Imagen Video, and Make-A-Video.
3. Method
MagicVideo models video distributions in a low-dimensional latent space and adapts image-model components for efficient video generation. Its architecture combines frame-wise distribution adjustment, directed temporal attention, motion-aware sampling, and VideoVAE decoding.
- Overall framework: MagicVideo generates key frames by diffusion in a low-dimensional latent space, then interpolates, decodes to RGB, and upsamples the resulting video.The inference pipeline repeatedly denoises latent representations before VAE decoding and high-resolution upsampling.
- Video distribution adaptor: The video distribution adaptor simplifies “2D+1D” processing to shared 2D convolutions plus lightweight frame-specific parameters.The adaptor adjusts intermediate feature statistics using learnable scale and bias parameters for each frame.
- Spatial and directed temporal attention: Directed temporal attention computes each future frame from preceding frames while preventing future information from affecting earlier frames.A lower-triangular mask enforces this directional dependency across temporal tokens at matching spatial locations.
- Training strategy: The model samples 16 frames from successive video segments and embeds ν = 16/Ls · FPS to represent the observed motion speed.The resulting embedding is added to the video frame features during training.
- Training strategy: Training uses frame-wise reconstruction loss, with unsupervised video-only pretraining followed by fine-tuning on annotated video-text pairs.Video-frame CLIP embeddings serve as proxy text conditions during the unsupervised stage, while both stages use the same objective.
- VideoVAE decoder: The VideoVAE decoder adds two temporal directed-attention layers while retaining low-dimensional latents to reduce pixel dithering without much extra computation.This design addresses artifacts observed when conventional VAE decoding reconstructs video frames independently.
4. Experiments
Experiments evaluate MagicVideo on standard video-generation benchmarks, ablations, qualitative comparisons, human ratings, and conditional-generation applications. Results indicate improved video quality, temporal consistency, and efficiency across these evaluations.
- Ablation studies: Directed attention substantially reduces FVD, while the adaptor saves computation and also benefits video-generation quality.The ablation uses 1,000 Webvid-10M test clips with 16 extracted frames each.
- Spatial and temporal attention: Spatial attention supports diverse frame contents, whereas temporal attention supports consistency across generated frames.Combining both attention branches is reported to provide content diversity and cross-frame consistency.
- Qualitative evaluation: Compared with CogVideo and VDM, MagicVideo produces richer details, including recognizable cars and clearer near-camera vehicles for “Busy freeway at night.”The comparison is qualitative and includes Make-A-Video as a strong baseline.
- Quantitative evaluation: MagicVideo is evaluated on MSR-VTT and UCF-101 using text prompts to generate 16 key frames without fine-tuning.The study compares these outputs with recent state-of-the-art methods using the datasets’ test descriptions or class labels.
- Applications: MagicVideo supports image-to-video generation, video variation, and text-guided video editing.Examples include generating a relevant video from an image, varying an input video, and adding candies falling onto a driving-bus clip.
- Human evaluation: Human evaluation on 200 DrawBench prompts compares MagicVideo and CogVideo on realism, faithfulness, smoothness, and generation efficiency.Table 3 reports the percentage of samples where MagicVideo is better than or equal to CogVideo.
5. Conclusions
MagicVideo addresses video-generation efficiency with a low-dimensional latent diffusion framework and image-model adaptations, while generating realistic, smooth text-conditioned videos. The authors also flag ethical risks from malicious video editing and inherited issues in Stable Diffusion’s LAION pretraining data.
- MagicVideo models video generation in a low-dimensional latent space and incorporates directional attention and an adaptor module to use pretrained image-generation models.
- The authors report that MagicVideo generates realistic and smooth videos from text descriptions efficiently.
- Video generation can be used for malicious video editing, although current deepfake detection technology can detect fake contents.
- Using pretrained Stable Diffusion weights may transfer ethical issues from the LAION dataset into MagicVideo.