Source-linked AI summary
Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, Ishan Misra
TL;DR
Text-to-video generation still trails image generation in quality and diversity because it must model complex spatiotemporal outputs from limited text-video data. Emu Video factorizes generation into an image step followed by video generation conditioned on the image and text, significantly outperforming prior and commercial systems in human evaluations.
Problem
Text-to-video generation remains lower-quality and less diverse than image generation while requiring complex spatiotemporal modeling from smaller video-text datasets.
Method
Emu Video first generates an image from text, then generates a video conditioned on that image and the text.
Results
91.8% average win rate for quality and 86.6% for faithfulness over prior work and commercial solutions in human evaluations.
Takeaways & Limitations
The factorized model also supports image-to-video generation, whose outputs were preferred 96% of the time over VideoComposer [78].
Takeaways & Limitations
Video realism, fine-grained details, physics, long-duration quality and consistency, and recovery from unrepresentative conditioning frames remain areas for improvement.
Abstract
from arXiv · showhide
We present Emu Video, a text-to-video generation model that factorizes the generation into two steps: first generating an image conditioned on the text, and then generating a video conditioned on the text and the generated image. We identify critical design decisions--adjusted noise schedules for diffusion, and multi-stage training that enable us to directly generate high quality and high resolution videos, without requiring a deep cascade of models as in prior work. In human evaluations, our generated videos are strongly preferred in quality compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing approach naturally lends itself to animating images based on a user's text prompt, where our generations are preferred 96% over prior work.
1 Introduction
Emu Video factorizes text-to-video generation into image generation followed by image-and-text-conditioned video generation, strengthening conditioning while leveraging pretrained text-to-image models. It significantly surpasses prior and commercial methods, achieving 91.8% quality and 86.6% text-faithfulness average human-evaluation win rates.
- Motivation: Video generation remains harder than image generation because it must model a higher-dimensional spatiotemporal output space while maintaining conditioning.The work hypothesizes that progressively stronger conditioning, analogous to autoregressive sequence generation in NLP [11], improves video quality.
- Method: Emu Video factorizes text-to-video generation into generating an image from text, then generating a video conditioned on that image and the text.The intermediate image strengthens conditioning and reduces the video model’s burden to predicting how the image evolves.
- Method: A frozen pretrained text-to-image model initializes the text-to-video model, helping retain visual diversity, style, and quality despite smaller video-text datasets.This factorized approach outperforms direct T2V methods without requiring the deep model cascade used by prior work.
- Empirical results: 91.8% quality and 86.6% text-faithfulness average win rates show Emu Video surpasses all prior and commercial methods in human evaluations.The comparison uses JUICE pairwise evaluations, released prior-work videos, and no cherry-picking of Emu Video generations.
2 Related Work
Prior text-to-image generation is dominated by diffusion models, while text-to-video methods commonly extend or leverage T2I models through training-free motion injection, temporal parameters, or autoregressive designs. Emu Video instead uses explicit image conditioning, treating the generated first frame as a stronger condition than an image embedding.
- Text-to-Image (T2I) diffusion models: Diffusion models are state-of-the-art for text-to-image generation, outperforming prior GAN and autoregressive methods.They generate outputs by gradually denoising normally distributed noise, either in pixel space or a lower-dimensional latent space [36] [37] [59].
- Text-to-Video (T2V) generation: Factorized text-to-video generation first produces an image conditioned on text and then generates a video conditioned on both the image and text.The image is temporally zero-padded and concatenated with a binary padding mask and the noised input.
- Text-to-Video (T2V) generation: Most prior text-to-video methods leverage text-to-image models, using training-free motion injection [44], one-shot adaptation such as Tune-A-Video, or added temporal parameters.These approaches include direct text-to-video mappings and are contrasted with Emu Video’s explicit image conditioning.
- Text-to-Video (T2V) generation: Make-A-Video trains text-to-video generation from a pretrained text-to-image model and its prior network without paired video-text data, while Imagen Video builds upon Imagen.The supplied passage identifies both as examples of methods leveraging text-to-image models.
- Text-to-Video (T2V) generation: Emu Video differs from Make-A-Video by explicitly conditioning training and inference on the first frame itself rather than a shared image-text embedding, providing a stronger condition.Make-A-Video finetunes all parameters from a pretrained text-to-image model, whereas Emu Video retains the pretrained model’s visual quality and diversity.
3 Approach
Emu Video factorizes text-to-video generation into image generation followed by video generation conditioned on both the text prompt and generated image. It uses latent diffusion with explicit frame conditioning, zero terminal-SNR scheduling, temporal layers, and interpolation for higher frame rates.
- Factorized generation: Emu Video first generates an image from the text prompt, then generates T video frames conditioned jointly on that prompt and the image.Both stages use a latent diffusion model initialized from a pretrained text-to-image model.
- Image conditioning: The starting frame is concatenated with the noised latent input and marked by a binary temporal mask, preserving its full image information.The image is represented as a single-frame video, zero-padded across time, unlike approaches conditioning only on semantic embeddings [78].
- Model architecture: The model adds learnable temporal convolutions and attention around frozen spatial layers initialized from the pretrained text-to-image model.This preserves image-model conceptual and stylistic diversity while adapting the architecture for video generation.
- Noise schedule: Zero terminal-SNR scheduling removes residual signal at the final training timestep, addressing train-test discrepancy when inference begins from random Gaussian noise.The approach uses zero terminal-SNR for 512px training, while 256px training uses the schedule from.
- Interpolation: An interpolation model initialized from the video model converts low-frame-rate outputs into higher-frame-rate videos using zero-interleaved frames and conditioning masks.It takes 8 input frames and produces 37 frames at 16fps.
- Inference and implementation: Inference generates an image without temporal layers, uses it with the prompt to produce high-resolution video, and avoids a deep cascade of models.Emu Video can be trained on standard video-text datasets and increases frame rate through the interpolation model.
4 Experiments
Experiments show that Emu Video’s factorized design, zero terminal-SNR schedule, and high-quality finetuning yield strong video quality, faithfulness, and image animation performance. Human evaluations find Emu Video superior to prior and commercial methods, while automated metrics show competitive IS and best FVD on UCF101.
- Design choices: Zero terminal-SNR is strongly preferred over the standard noise schedule for 512px generation, indicating that correcting train-test discrepancy is critical for high-resolution video.The benefit is stronger for factorized generation than for direct text-to-video generation.
- Human evaluation: Emu Video significantly outperforms all prior work, including commercial solutions, by 91.8% in Quality and 86.6% in Faithfulness on average.The evaluation uses 16-frame, 4-second videos with high-quality finetuning and interpolation to 16fps, with videos postprocessed to match resolution, aspect ratio, and frame rate.
- Human evaluation: Evaluators prefer Emu Video over Make-A-Video and Imagen Video primarily for pixel sharpness and motion smoothness, with preferences of 96.8% and 81.8%, respectively.The model uses a two-model cascade with 6.0B total parameters, compared with Imagen Video’s seven-model, 11.6B-parameter cascade and Make-A-Video’s five-model, 9.6B-parameter cascade.
- Automated metrics: On UCF101, Emu Video achieves competitive IS, the lowest FVD among prior work, and beats Make-A-Video by 90.1% in Quality and 80.5% in Faithfulness.The automated comparison uses the zero-shot text-to-video setting, and the human comparison uses 303 generated videos spanning three samples per class.
- Image animation: Emu Video’s image animations are strongly preferred over all six baselines across two prompt sets.Human evaluations compare generations initialized from the same images, while automated image-animation metrics are reported in the appendix.
- Nearest neighbor baseline: Emu Video is preferred over real nearest-neighbor retrieved videos by 81.1% in Faithfulness, indicating generations are not simply reproducing training examples.The retrieval baseline searches all 34M training videos using CLIP similarity between evaluation prompts and training prompts.
5 Limitations and ethical considerations
Emu Video still has limitations in realism, fine-grained artifacts, physics, long-duration consistency, and recovery from unrepresentative conditioning frames. The authors also emphasize that real-world applications require careful evaluation of safety, risk, bias, and societal impact.
- Limitations: Key limitations include realism, hand and face artifacts, physical modeling, and maintaining quality and consistency over long video durations.These factors are considered in the JUICE metric through object/scene consistency and pixel-quality evaluations.
- Limitations: For longer videos, a tested variant generates plausible extensions that respect the original video and future text prompts.
- Limitations: Emu Video should improve its ability to recover when conditioning frames are not representative of the prompt.
- Ethical considerations: Before real-world application, the data, model, intended uses, safety, risks, bias, and societal impacts require careful study.The authors note that generative methods support many use cases beyond this work’s scope.
Appendix · 1 Implementation Details
The appendix specifies the shared U-Net and frame-wise VAE architectures, training and inference settings, and classifier-free guidance used for text-to-video and interpolation. It also explains how guidance scales control motion and how interpolation constructs longer 16-fps videos.
- 1 Implementation Details: The shared U-Net has 4.3B parameters, including 2.7B frozen parameters initialized from a pretrained text-to-image model and 1.7B trainable parameters.Both text-to-video and interpolation models use this same U-Net architecture.
- 1 Implementation Details: Inference uses DDIM [70] with 250 diffusion steps, with CFG scales of wimg = 7.5 for image generation and wimg = 2.0, wtxt = 7.5 for video generation and interpolation.Training covers 256px, 512px, high-quality finetuning, and frame-interpolation stages.
- 1 Implementation Details: The frame-wise image VAE downsamples videos spatially by 8 × 8 and produces 8-channel latent representations.The VAE is applied independently to video frames.
- 1 Implementation Details: The selected ordered CFG formulation recovers the image-and-text-conditioned prediction when wi = wp = 1 and the unconditional prediction when wi = wp = 0.Alternate conditioning orderings and unordered formulas did not improve over Eq. 1, while the reversed ordering required substantially different guidance values.
- 1 Implementation Details: The ratio of text CFG scale wp to image CFG scale wi controls generated-video motion, measured by mean motion-vector energy in H.264 encodings.Higher wi keeps the video closer to the initial image and favors camera motion, whereas higher wp favors movement at the expense of object consistency; without HQ finetuning, motion is lower and CFG effects are stronger.
- 1 Implementation Details: The interpolation model takes 8 zero-interleaved frames generated by F at 4fps and produces 37 frames at 16fps, avoiding the duration loss of inserting three frames after each input frame.This design increases temporal resolution while preserving the intended video duration more effectively than the alternative described in the appendix.
- 1 Implementation Details: For 16-frame inputs, interpolation splits the video into two 8-frame clips, processes them independently, removes overlapping frames, and concatenates them into 65 frames at 16fps.The resulting video lasts 4.06 seconds.
2 Additional experiments
Additional experiments show that Emu Video benefits from explicit image conditioning, outperforms Gen2 variants and prior image-animation methods, and depends on carefully chosen low-resolution pretraining. The model remains effective with substantially less training data, although performance declines slightly at 10% data.
- Image conditioning for commercial T2V systems: Emu Video outperforms both Gen2 and the stronger Gen2 I2V API, including when compared using the same images for image animation.Gen2 I2V itself outperforms text-only Gen2, while Emu Video remains preferred to both variants.
- Effect of training steps and data: Around 70K low-resolution, high-FPS pretraining steps are optimal, while fewer or more iterations reduce performance after identical 512px finetuning.Reducing low-resolution pretraining to 75%, 50%, or 25% also causes a gradual performance drop, highlighting the stage’s importance.
- Effect of the amount of training data: At 10% of the training data, performance declines only slightly, reaching approximately 43% on both Quality and Faithfulness.Training steps remain constant while data is reduced for both the 256px and 512px stages.
- Automated metrics for image animation: Emu Video outperforms VideoComposer across both prompt sets and all three automated metrics, although static-video bias favors PikaLabs and Gen2.The metrics are Frame consistency (FC), Image similarity (IC), and Text consistency (TC).
3 Human evaluations
Human evaluations use JUICE, which requires evaluators to justify pairwise choices, improving agreement and reducing ambiguity compared with naive voting. Analyses identify the factors driving Emu Video’s wins and characterize evaluation reliability, fairness, and inter-rater agreement.
- Evaluation protocol: JUICE asks evaluators to select reasons for pairwise video-quality and text-faithfulness choices, using templates with defined factors and training examples.The factors were derived by having internal raters explain their choices, then categorizing those explanations into five Quality reasons and two Faithfulness reasons.
- Reliability and robustness: JUICE reduced split-agreement samples by 28%, increased complete-agreement samples by 24%, and improved Fleiss’ kappa from 0.004 to 0.31 versus naive evaluation.The comparison used Emu Video against Make-A-Video and measured split, partial, and complete agreement across five-rater evaluations.
- Evaluation protocol: Each comparison used the majority vote of five evaluators, randomized video positioning to reduce bias, and Fleiss’ kappa to quantify agreement beyond chance.Fleiss’ kappa applies to categorical ratings from any number of evaluators and ranges from −1 to 1, with positive values indicating agreement.
- Winning factors: Pixel sharpness and motion smoothness were the two most contributing factors when Emu Video beat Make-A-Video and Imagen Video.Across most baseline comparisons, evaluators primarily selected pixel sharpness and motion smoothness; amount of motion also mattered against PYOCO and PikaLabs.
4 Comparisons to Prior Work
The section details fair human evaluations of Emu Video against prior work using method-specific prompt datasets and postprocessing to equalize video dimensions. Without postprocessing, Emu Video achieves average win rates of 93.8% in quality and 93.1% in faithfulness, exceeding its postprocessed averages of 91.8% and 86.6%.
- Evaluation datasets: Human evaluations use five prompt datasets because closed-source methods provide different publicly released example videos, while CogVideo, Gen2, and Pika Labs use the Align Your Latents prompt set.Gen2 and Pika Labs are sampled through online interfaces, whereas CogVideo is open-source.
- Fairness and postprocessing: Postprocessing equalizes comparisons by center-cropping aspect ratio and duration, interpolating spatial dimensions, and adjusting frame rate and frame count.Videos are converted to Emu Video's 1:1 aspect ratio so models are indistinguishable aside from generated content.
- Human-evaluation protocol: Evaluators independently compare videos generated by two models for the same prompt and choose the preferred model while remaining unaware of model identities.The protocol is designed to reduce bias from knowing which model generated each video.
- Unprocessed human evaluations: 93.8% quality and 93.1% faithfulness average win rates for Emu Video without postprocessing exceed 91.8% and 86.6% in the postprocessed comparison.Table 10 reports that Emu Video significantly outperforms all prior work across settings and metrics; the authors attribute the larger unprocessed margins partly to evaluator bias favoring its superior generations.
5 Qualitative Results
Emu Video produces high-quality, text-faithful videos with smooth motion, temporal consistency, and sharp visuals for both natural and fantastical prompts, in both T2V and I2V settings. Qualitative comparisons show advantages over prior methods in text faithfulness, motion, consistency, and visual quality.
- 5.1 Additional Qualitative Results: Emu Video generates high-quality, text-faithful videos with sharp visuals, smooth motion, and consistent objects for both natural and fantastical prompts.These properties are shown for both text-to-video and image-to-video generation.
- 5.2 Qualitative Comparisons: Compared with prior text-to-video methods, Emu Video consistently produces more text-faithful videos with greater motion smoothness, temporal consistency, and pixel sharpness.The comparisons span Gen2, Pika Labs, Align Your Latents, CogVideo, Imagen Video, PYOCO, Make-A-Video, and Reuse & Diffuse.
- 5.1 Additional Qualitative Results: Emu Video’s image-conditioned generations remain natural-looking and exhibit smooth, consistent motion while following both the conditioning image and text prompt.Figure 10 presents examples across diverse prompts.
- 5.2 Qualitative Comparisons: Emu Video generates realistic, smooth, and consistent videos for challenging fantastical prompts, whereas PYOCO lacks motion smoothness, consistency, and fine-grained detail.The comparison is illustrated on two prompts, including a futuristic cyberpunk scene.
- 5.2 Qualitative Comparisons: On UCF101, Emu Video produces higher-quality and more coherent zero-shot text-to-video generations than Make-A-Video.The examples cover walking with a dog, biking, handstand pushups, and skiing.