Source-linked AI summary
Lumiere: A Space-Time Diffusion Model for Video Generation
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, Inbar Mosseri
TL;DR
Lumiere addresses the difficulty of learning globally coherent motion in text-to-video generation. It uses a Space-Time U-Net to generate full-frame-rate video clips at once, built on a pretrained text-to-image diffusion model, and reports state-of-the-art generation with broad editing and creation applications.
Problem
Large-scale text-to-video generation remains challenging because motion modeling adds temporal errors, memory and compute demands, and substantial training-data requirements.
Method
Lumiere uses a Space-Time U-Net with spatial and temporal down- and up-sampling to generate the full video duration at once, followed by temporally aggregated spatial super-resolution.
Results
The model demonstrates state-of-the-art video generation and supports image-to-video, video inpainting, stylized generation, and consistent video editing.
Takeaways & Limitations
Generating the full video at once enables globally coherent motion and facilitates a range of video content-creation and editing applications.
Takeaways & Limitations
Lumiere is not designed for multi-shot videos or transitions between scenes, which remain open challenges.
Abstract
from arXiv · showhide
We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.
1. Introduction
Existing T2V systems commonly generate distant keyframes and fill them with temporal cascades, limiting globally coherent motion. Lumiere instead generates the full video duration at once with a space-time architecture, then applies temporally aggregated spatial super-resolution and supports multiple creation tasks.
- Motivation: Existing T2V models commonly generate distant keyframes followed by cascaded temporal super-resolution models.This design is memory efficient but restricts globally coherent motion through temporal aliasing, limited context windows, and accumulated domain-gap errors.
- Lumiere’s approach: Lumiere generates the full temporal duration at once using a Space-Time U-Net that downsamples and upsamples in both space and time.The architecture performs most computation in a compact space-time representation and generates 80 frames at 16fps with a single base model.
- Lumiere’s approach: The pipeline processes all frames jointly without temporal super-resolution cascades, targeting globally coherent motion.High-resolution output is produced by applying spatial super-resolution on overlapping temporal windows and aggregating the results.
- Lumiere’s approach: Lumiere builds on a pretrained text-to-image diffusion model while using temporal windowing for its high-resolution spatial super-resolution stage.The pretrained model operates in pixel space, making full-duration high-resolution processing infeasible because of memory requirements.
- Applications: The framework demonstrates text-to-video, image-to-video, video inpainting, stylized generation, and consistent video editing applications.Figure 3 contrasts the common keyframe-and-cascade pipeline with Lumiere’s full-video generation and overlapping-window aggregation.
2. Related work
Related work spans diffusion and autoregressive approaches for text-to-image and text-to-video generation. Lumiere differs by generating the full video-frame duration at once and avoiding the temporal cascade common in T2V models.
- Text-to-Image Generation: Text-to-image generation commonly uses diffusion models, with cascaded pixel-space systems and compressed latent-space systems representing major approaches.Examples include DALL-E2 and Imagen for cascaded diffusion, and Stable Diffusion for latent-space generation.
- Text-to-Video Generation: Lumiere generates the full frame duration at once, avoiding the temporal cascade commonly used in text-to-video models.This distinguishes its design from related T2V approaches that generate keyframes and subsequently interpolate missing frames.
- Text-to-Video Generation: Recent text-to-video research includes both autoregressive Transformers and diffusion models trained on large-scale datasets.Lumiere extends an Imagen text-to-image model, while its architectural contributions are described as applicable to latent diffusion as well.
- Text-to-Video Generation: Lumiere’s architectural contribution is presented as orthogonal to improvements in diffusion noise scheduling and video-data curation.These are identified as separate possible directions in related work.
3. Lumiere
Lumiere combines diffusion generation with a Space-Time U-Net that processes videos across spatial and temporal scales, then uses MultiDiffusion for temporally coherent spatial super-resolution. The design supports full-clip generation and several video applications.
- Framework: Lumiere uses diffusion models that progressively denoise Gaussian noise into video samples from an approximated data distribution.The generative process consists of a series of denoising steps.
- Framework: The framework’s base model generates full clips at coarse spatial resolution, followed by a temporally aware spatial super-resolution model.The resulting high-resolution video is produced from the coarse full-clip output.
- Space-Time U-Net: The architecture performs most computation on a compact space-time representation to make full-video generation computationally tractable.This design follows the use of space-time pooling for efficient volumetric processing.
- Generation results: Figure 5 presents text-to-video and image-to-video samples, with the leftmost frame serving as the image-to-video condition.The examples include intricate object motion and coherent camera motion.
- Space-Time U-Net: STUNet interleaves temporal blocks into a pretrained T2I U-Net and downsamples and upsamples video in both space and time.Factorized space-time convolutions are used at most levels, while temporal attention is used at the coarsest level.
- MultiDiffusion: Temporal SSR segments are reconciled by linearly combining predictions over overlapping windows through MultiDiffusion.The SSR network operates only on short video segments because of memory constraints.
4. Applications
Lumiere extends full-video generation to conditional creation and editing tasks, including stylization, image-to-video, inpainting, and cinemagraphs. Its full-frame-rate generation supports coherent downstream video editing without a temporal super-resolution cascade.
- Video-to-video editing: SDEdit enables consistent video-to-video stylization because Lumiere generates full-frame-rate videos without a temporal super-resolution cascade.This provides an interface for applying an off-the-shelf editing method to complete videos.
- Stylized generation: Stylized generation linearly interpolates fine-tuned and original text-to-image spatial weights to balance style adherence with plausible motion.The interpolation coefficient α is manually selected from [0.5, 1] in the reported experiments.
- Stylized generation: The model matches distinct motion priors to vector-art and realistic spatial styles, producing style-specific motions such as pencil-stroke drawing and gradual scene construction.The line-drawing style produces pencil-stroke-like animation, while the cartoon style gradually pops out and constructs the scene.
- Conditional generation: Conditional video generation takes a noisy video, a masked conditioning video, and a binary mask as concatenated inputs, then learns to animate masked content while copying unmasked information.The modified input tensor is ⟨J, C, M⟩ ∈ R^T×H×W×7.
- Image-to-video: Image-to-video generation preserves the supplied first frame while producing intricate coherent motion across the full video duration.The conditioning signal contains the first frame followed by blank frames, with the first frame unmasked and later frames masked.
- Inpainting and cinemagraphs: Inpainting guides seamless completion of masked video regions with text, supporting object replacement, insertion, localized editing, and outpainting.Cinemagraphs use an input image and mask to animate only a selected region while keeping the rest static; the unmasked first frame helps preserve the conditioning appearance.
5. Evaluation and Comparisons
Lumiere is evaluated on diverse text-to-video prompts, downstream video editing, UCF101 zero-shot metrics, qualitative baseline comparisons, and user studies. The reported evidence combines strong visual and motion quality with competitive benchmark scores, while noting important metric and duration differences across methods.
- Evaluation setup: The evaluation uses 109 prompts describing diverse objects and scenes, alongside zero-shot text-to-video evaluation on UCF101.The prompt set includes 91 prompts from prior methods and additional prompts created by the authors.
- Qualitative evaluation: Lumiere generates high-quality text-to-video samples with intricate object motion and coherent camera motion.Examples include a walking astronaut and a moving car.
- Qualitative comparisons: Compared with baselines, Lumiere produces 5-second videos with higher motion magnitude while maintaining temporal consistency and overall quality.Gen-2 and Pika often produce near-static videos, while ImagenVideo has lower visual quality and AnimateDiff and ZeroScope show artifacts and shorter durations.
- User studies: In user studies, Lumiere is preferred over baselines for both text-to-video and image-to-video generation.The image-to-video comparison includes Pika, StableVideoDiffusion, and Gen2; the SVD comparison focuses on video quality because SVD is not text-conditioned.
- Quantitative evaluation: Lumiere achieves competitive FVD and IS scores on zero-shot UCF101 evaluation.The authors caution that these metrics may be influenced by low-level details, distribution shift, and the protocol’s use of only 16 generated frames.
6. Conclusion
The paper presents a text-to-video framework built on a pretrained text-to-image diffusion model and a space-time U-Net that directly generates full-frame-rate video. It reports state-of-the-art generation results and applicability to several content-creation and editing tasks, while noting important scope limitations.
- Lumiere combines a pretrained text-to-image diffusion model with a space-time U-Net that directly generates full-frame-rate video clips.
- The framework is demonstrated for image-to-video, video inpainting, and stylized generation.
- The method is not designed for videos containing multiple shots or transitions between scenes.
- The authors state that their design principles are applicable to latent video diffusion models and may motivate further text-to-video model research.
7. Societal Impact
The work aims to help novice users create visual content flexibly, while acknowledging risks of generating fake or harmful content. It emphasizes developing tools to detect bias and malicious use cases for safe and fair deployment.
- The technology is intended to enable novice users to generate visual content in a creative and flexible way.
- The authors identify potential misuse for creating fake or harmful content and call for tools that detect biases and malicious use cases.
A. Qualitative Comparison
Figure 11 provides a qualitative comparison of Lumiere with several prominent text-to-video diffusion models. The comparison includes Gen-2, Pika, ImagenVideo, AnimateDiff, and ZeroScope.
- Figure 11 presents qualitative samples comparing Lumiere with Gen-2, Pika, ImagenVideo, AnimateDiff, and ZeroScope.
B. Initialization
The appendix examines initialization choices for temporal down- and up-sampling and the use of MultiDiffusion in spatial super-resolution. Identity-style temporal initialization better preserves the pretrained image prior, while MultiDiffusion avoids temporal boundary artifacts.
- Initialization: The temporal modules use identity-initialized convolution, temporal striding for downsampling, and frame duplication followed by convolution for upsampling.
- Initialization: The initialization ablation trains on UCF-101 and compares loss functions for different temporal down- and up-sampling initializations.
- Initialization: Identity initialization lets the model better benefit from the pretrained text-to-image prior, whereas standard random initialization produces non-meaningful samples.
- MultiDiffusion: MultiDiffusion averages predictions across overlapping temporal windows to avoid boundary artifacts between video segments.
D.1. User study
The user study evaluated Lumiere and baseline methods on 109 prompts spanning varied objects, actions, complex scenes, and actions. Participants viewed the two videos in randomly ordered presentation.
- 109 prompts covered varied objects, actions, complex scenes, and actions in the text-to-video and image-to-video surveys.91 prompts came from recent text-to-video methods, while the remainder described complex scenes and actions.
- For each baseline method, results were generated for all prompts through the methods’ official APIs.
- The user-study display showed Lumiere and a baseline video in randomly ordered positions.
D.2. Zero-shot evaluation on UCF101
The zero-shot UCF101 evaluation generated 10,235 videos following the dataset’s class distribution and compared their video embeddings with those of UCF101 videos. The accompanying prompt set spans diverse objects, scenes, actions, and visual styles.
- 10,235 videos were generated according to the UCF101 class distribution for Fréchet Video Distance evaluation.Videos were resized to 244×244, and I3D embeddings were extracted from their first 16 frames before comparison with UCF101 embeddings.
- The evaluation used the same prompt set as Ge et al. (2023) for video generation.
- The prompt set included animals, vehicles, landscapes, weather, performances, food, and other varied visual scenarios.Examples include a bear presenting, a bicycle on a boat, a fire dragon breathing, and waves crashing against a lighthouse.