PaperlayerView exact revision on arXiv

Source-linked AI summary

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, Karsten Kreis

arXiv:2304.08818v2cs.CVcs.LG

TL;DR

High-resolution video generation is expensive, while prior video diffusion models generally produce shorter, lower-resolution videos. The paper reuses pretrained image LDMs by adding and training temporal alignment components, including for decoders and upsamplers. Video LDMs achieve state-of-the-art high-resolution driving-video synthesis, support multi-minute videos, and extend Stable Diffusion to expressive and personalized text-to-video generation.

  • Problem

    High-resolution video generation is computationally intensive, and existing video diffusion models generally produce relatively low-resolution, often short videos.

  • Method

    Video LDMs reuse pretrained image diffusion models and train temporal layers on encoded videos, while also temporally fine-tuning decoders, upsamplers, and long-video prediction mechanisms.

  • Results

    Video LDMs achieve state-of-the-art high-resolution driving-video synthesis, generate videos lasting multiple minutes, and transfer temporal layers across image-model checkpoints for text-to-video generation.

  • Takeaways & Limitations

    The approach enables efficient high-resolution, temporally coherent video generation and supports personalized text-to-video generation through transferable temporal layers.

  • Takeaways & Limitations

    The synthesized videos are not yet indistinguishable from real content, and the research data and internet-trained backbones make the model unsuitable for productization.

Abstract

from arXiv · show

Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion model in a compressed lower-dimensional latent space. Here, we apply the LDM paradigm to high-resolution video generation, a particularly resource-intensive task. We first pre-train an LDM on images only; then, we turn the image generator into a video generator by introducing a temporal dimension to the latent space diffusion model and fine-tuning on encoded image sequences, i.e., videos. Similarly, we temporally align diffusion model upsamplers, turning them into temporally consistent video super resolution models. We focus on two relevant real-world applications: Simulation of in-the-wild driving data and creative content creation with text-to-video modeling. In particular, we validate our Video LDM on real driving videos of resolution 512 x 1024, achieving state-of-the-art performance. Furthermore, our approach can easily leverage off-the-shelf pre-trained image LDMs, as we only need to train a temporal alignment model in that case. Doing so, we turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048. We show that the temporal layers trained in this way generalize to different fine-tuned text-to-image LDMs. Utilizing this property, we show the first results for personalized text-to-video generation, opening exciting directions for future content creation. Project page: https://research.nvidia.com/labs/toronto-ai/VideoLDM/

1. Introduction

Video LDMs address the computational and data challenges of high-resolution video generation by reusing image diffusion models and adding temporal alignment. The approach supports coherent long videos, temporally consistent upsampling, and text-to-video generation from Stable Diffusion.

  • Video modeling has lagged behind image synthesis because video training is computationally expensive and large, general public datasets are scarce.
  • Video LDMs reuse image-pretrained diffusion models, introducing temporal layers that align encoded image sequences while keeping pretrained spatial layers fixed.The method also temporally fine-tunes the decoder and diffusion-model upsamplers for coherent reconstructions and video super resolution.
  • Temporal video fine-tuning converts independently generated image samples into temporally aligned samples that form coherent videos.The alignment is performed in the LDM latent space before decoding into video frames.
  • The approach achieves state-of-the-art high-resolution synthesis on real driving videos, generates videos lasting multiple minutes, and supports Stable Diffusion text-to-video generation.Learned temporal layers can also be combined with different image-model checkpoints, including DreamBooth, enabling personalized text-to-video generation.
  • The framework combines temporal decoder fine-tuning with a frozen, frame-independent encoder and a video-aware discriminator to improve temporal reconstruction consistency.

2. Background

Diffusion models learn data distributions through iterative denoising from progressively corrupted inputs. Latent Diffusion Models improve efficiency by performing this process in a compressed latent space and decoding the resulting features into images.

  • Diffusion models train denoisers to model a data distribution by iteratively removing noise from corrupted inputs.Inputs are formed by combining data with Gaussian noise according to a diffusion-time-dependent schedule.
  • The denoiser can receive optional conditioning such as a text prompt and predict either random noise or the v-prediction target.The cited formulation uses a uniform distribution over diffusion time and supports a variance-preserving schedule.
  • Latent Diffusion Models reduce computational and memory demands by encoding images into a lower-dimensional latent space before diffusion modeling.A regularized autoencoder reconstructs images through an encoder E and decoder D, with the reconstruction approximating the input.

3. Latent Video Diffusion Models

Video LDMs reuse pretrained image LDMs by adding temporal layers that align frames, then extend the pipeline with temporal decoding, prediction, interpolation, and upsampling for high-resolution video.

  • 3.1. Turning Latent Image into Video Generators: A pretrained image LDM becomes a video generator by interleaving temporal layers with fixed spatial layers that process encoded frame sequences.The temporal layers operate across the video-time dimension while the spatial backbone retains image-generation capabilities.
  • 3.1. Turning Latent Image into Video Generators: The video-aware backbone combines independently processed spatial frames with temporal outputs through learned merge parameters and optional conditioning.Temporal mixing uses attention and 3D-convolutional residual blocks with sinusoidal time embeddings.
  • 3.3. Temporal Interpolation for High Frame Rates: The high-resolution stack generates sparse key frames, interpolates them in two stages for higher frame rates, decodes the latent video, and optionally applies a video upsampler.The LDM components share the same image backbone, while upsampling is temporally aligned for video.
  • 3.1. Turning Latent Image into Video Generators: Only the temporal layers are trained on videos, allowing large image datasets to pretrain spatial layers while limited video data focuses temporal alignment.Skipping the temporal layers recovers the original image model.
  • 3. Latent Video Diffusion Models: A temporally fine-tuned decoder reduces flickering by adding temporal decoder layers trained on videos with a patch-wise temporal discriminator, while the encoder remains unchanged.This preserves reuse of the image-trained latent diffusion model for encoded video frames.
  • 3.2. Prediction Models for Long-Term Generation: Long-video prediction masks future frames and iteratively reuses recent predictions as context, conditioning models on zero, one, or two frames.Context guidance is used during sampling to stabilize iterative generation.

4. Experiments

Experiments evaluate Video LDMs on real driving scenes and text-to-video generation, including temporal alignment, conditioning, upsampling, long-horizon synthesis, and personalized generation.

  • Datasets and evaluation: The RDS dataset contains 683,060 eight-second videos at 512 × 1024 and up to 30 fps, with day/night and crowdedness annotations.
  • Driving video synthesis: Video LDM generally outperforms LVG on driving videos, while conditioning on crowdedness and day/night further reduces FVD and improves human preference.
  • Upsampling: Temporal alignment of the video upsampler is crucial: independent frame-wise upsampling significantly degrades FVD while leaving FID essentially unaffected.
  • Long-horizon synthesis: The combined Video LDM and video upsampler generates high-quality, temporally coherent driving videos validated for up to 5 minutes.
  • Ablations: Ablations show that image pre-training, 3D convolutional temporal layers, and video-fine-tuned decoders improve results, while context guidance lowers FVD at the cost of higher FID.
  • Text-to-video and personalization: Temporal layers transferred from Stable Diffusion generalize to DreamBooth backbones, enabling personalized coherent videos that preserve training-image identities.

5. Conclusions

The paper presents Video LDMs as an efficient approach to high-resolution, temporally coherent video generation. It demonstrates driving-video synthesis, Stable Diffusion text-to-video generation, temporal transfer across checkpoints, and personalized video generation.

  • Video LDMs build video generators from pre-trained image diffusion models by adding temporal alignment layers and optionally temporally aligning super-resolution diffusion models.
  • The approach synthesizes high-resolution, temporally coherent driving videos lasting many minutes and supports efficient Stable Diffusion text-to-video generation.
  • Temporal layers transfer across model checkpoints, enabling personalized text-to-video generation with fine-tuned image backbones.

B. Broader Impact and Limitations

The paper discusses applications of Video LDMs to creative content creation and autonomous-driving simulation, alongside efficiency advantages and related diffusion-model limitations. It also notes unresolved realism, safety, data, and commercialization boundaries.

  • Potential impact: Video generative models may streamline digital-art workflows and support artistic expression, while driving-video models may serve as autonomous-driving research simulators.
  • Limitations: Synthesized videos are not yet indistinguishable from real content, and the internet-trained backbones and research-only data are unsuitable for productization.
  • Efficiency context: Latent diffusion and cascaded super-resolution models reduce computational complexity by moving resource-intensive modeling into lower-dimensional spaces and staged pipelines.
  • Related work: Video LDM differs from autoregressive transformer approaches by using continuous diffusion models with fewer parameters and non-autoregressive alignment layers.
  • Related work: The approach is more efficient and reproducible than concurrent pixel-space systems described as requiring substantially larger models and GPU resources.

D. Using Video LDM “Convolutional in Time” and “Convolutional in Space”

Video LDM extends convolutional sampling across time and space to generate sequences longer or larger than the training configuration. The method reaches longer high-resolution outputs, but long-video generation remains fragile and can degrade in quality.

  • Stable Diffusion’s convolutional image backbone preserves generalization to spatial resolutions larger than its training resolution by sampling larger noise tensors.
  • Relative temporal positional encodings and temporally constant mixing factors support convolutional-in-time sampling with more frames.
  • Convolutional-in-time generation generalizes beyond training sequences, producing high-resolution videos up to 30 seconds although training sequences lasted 4 seconds.
  • Convolutional-in-time generation can be fragile for long videos, and long text-to-video samples with prediction models remain future work.
  • The approach is among the first to combine long duration, high spatial resolution, and frame rates up to 30 fps while keeping training cost tolerable.

E. Datasets

The paper evaluates Video LDMs on real driving, text-described web videos, and first-person mountain biking clips. These datasets vary substantially in scale, resolution, frame rate, and annotation structure.

  • Real driving scenes: 683,060 real driving videos each contain 8 seconds at 512 × 1024 resolution, with day/night and crowdedness annotations.Most scenes are relatively empty highways with low crowdedness.
  • Real driving scenes: An additional 100k annotated driving frames provide car bounding boxes for initializing a bounding-box-conditioned image LDM.These frames are not used as video training data.
  • Text-to-video: WebVid-10M contains 10.7M video-caption pairs totaling 52k video hours from diverse stock footage.The dataset supports Stable Diffusion-based text-to-video training.

F. Architecture, Training and Sampling Details

The implementation builds on convolutional LDM and U-Net backbones, samples with DDIM, and exposes temporal-layer hyperparameters for architecture and conditioning. Tables 6–8 summarize the model and training configurations, while Table 7 covers ablations.

  • Backbones and diffusion: Image LDMs use convolutional encoders and decoders, while the latent diffusion and pixel-space upsampler models use U-Net-based backbones.The diffusion process uses discretized time steps and a linear noise schedule.
  • Backbones and diffusion: Sampling generally uses DDIM, with sampling steps, stochasticity η, and guidance scale varied across configurations.The detailed values are reported in Tables 6 and 7.
  • Temporal layers: The temporal-layer configuration includes a learnable skip-connection dimension, concatenated spatial-conditioning dimension, and context-channel count.These parameters control temporal-layer architecture and conditioning inputs.
  • Configuration tables: Tables 6–8 provide hyperparameters for diffusion models, ablation models, and autoencoders, including patch-based training for one configuration.Table 7 notes that one baseline uses twice as many attention layers while matching trainable-parameter counts.
  • Temporal layers: Temporal convolutional layers use a 3D kernel whose dimensions are specified as time, height, and width.This parameter directly controls the temporal convolutional receptive field.

G. Quantitative Evaluation

The evaluation combines automated image, video, action-recognition, and text-alignment metrics with human judgments, using standardized sampling protocols across driving, text-to-video, and mountain biking experiments.

  • Metrics: FID and FVD provide quantitative evaluation, while human evaluation is added because FVD can be unreliable.Text-to-video experiments additionally use Inception Score and CLIP Similarity.
  • Metrics: FVD compares real and generated videos using features from a pretrained I3D action-classification model.The protocol generally generates 2,048 videos of 16 frames at 30 fps, with 10k samples for one UCF-101 evaluation.
  • Metrics: FID is computed from pretrained Inception features extracted from randomly sampled generated and dataset frames.The protocol uses 10k frames, or 50k frames for mountain biking.
  • Human evaluation: Human evaluation compares 100 four-second videos from Video LDM and Long Video GAN in pairwise realism judgments with an equally-real option.Each pair is rated by four participants, producing 400 responses per dataset.
  • Text-to-video evaluation: The UCF-101 Inception Score uses 10k samples and a C3D model trained on UCF-101, whereas MSR-VTT CLIP Similarity averages scores over 47,840 generated frames.MSR-VTT uses 2,990 generated videos, one random prompt per test example.
  • Driving evaluation pipeline: The driving pipeline trains an image backbone on frames, then temporal layers on sparse key-frame videos, followed by a 4× pixel-space upsampler and temporal interpolation stages.The generation procedure starts from one image, predicts key frames, optionally interpolates from 1.875 to 7.5 to 30 fps, and optionally applies the upsampler.
  • Baselines: For driving comparisons, Long Video GAN is trained at 10 fps and 128 × 256 resolution as the principal baseline.The baseline uses the authors’ reported training settings and default hyperparameters.
  • Text-to-video results: The SD 2.1-based Video LDM performs better than the SD 1.4-based model and slightly outperforms Make-A-Video on UCF-101 Inception Score.The reported quantitative text-to-video evaluations use the SD 2.1-based model.

H.2.1 Number of Model Parameters

Video LDM’s Stable Diffusion-based variants retain large pretrained image backbones while training selected temporal and interpolation components. The resulting systems are smaller than cited large-scale alternatives, and decoder video fine-tuning improves reconstruction performance.

  • Stable Diffusion variants: The fine-tuned Stable Diffusion upsampler contains 977 million autoencoder and diffusion-model parameters, of which 449 million are trained.The trained parameters are concentrated in the temporal layers.
  • Parameter breakdown: For SD 1.4, the image backbone contains 860M parameters and the temporal layers 649M; SD 2.0/2.1 uses 865M and 656M, respectively.The text encoders are not trained in these parameter accounting figures.
  • Parameter breakdown: The shared interpolation latent diffusion model contains 1,509M trained parameters across the SD 1.4 and SD 2.0/2.1 variants.The same interpolation model is used for both model families.
  • Stable Diffusion variants: The SD 2.0/2.1 low-resolution text-to-video Video LDM contains around 3.1B autoencoder and diffusion-model parameters, with around 2.2B trained.The count excludes CLIP text embedders.
  • Model-size comparison: Imagen Video has 11.6B parameters, whereas the authors report their model as much smaller while still producing high-quality videos.CogVideo is also reported as larger, at around 9B parameters.
  • Decoder fine-tuning: Video fine-tuning of the decoder produces a significant performance boost for the text-to-video model and the mountain biking Video LDM.Table 11 evaluates reconstruction FVD and FID on 2,048 examples from the respective datasets.

I.1.2 More Samples

The paper presents additional samples demonstrating expressive text-to-video generation across resolutions, temporal extensions, personalized content, mountain biking, and driving scenarios. It also reports comparisons and decoder fine-tuning results for these applications.

  • Text-to-video samples: 1280 × 2048 videos are generated with Stable Diffusion-based Video LDMs and a video fine-tuned 4× upscaler.These samples include SD 2.0-based models and, on the project page, SD 2.1-based models.
  • Text-to-video samples: The models generate long, high-resolution, high-frame-rate, expressive, and artistic videos using spatial and temporal convolutional extensions.The reported extensions include “convolutional in space,” “convolutional in time,” or both.
  • Personalized generation: Personalized text-to-video samples successfully include learned objects while capturing their identities well.The samples use DreamBooth personalization and are shown at 320 × 512 resolution.
  • Mountain biking: The model outperforms Long Video GAN in FID and human evaluation but slightly underperforms it in FVD on mountain biking videos.The authors attribute the trade-off to more realistic single frames alongside weaker temporal consistency in rapidly changing background details.
  • Decoder and driving results: Video fine-tuning of the decoder provides a significant performance boost on mountain biking.Additional driving samples are upsampled to 512 × 1024 using a temporally aligned video upsampler.

I.3.2 Ablation on Image-level Quality Degradation after Temporal Video Fine-Tuning

The ablation evaluates whether temporal video fine-tuning degrades image-level quality. The reported change is small, indicating that image quality is affected only slightly in this experiment.

  • Evaluation setup: The evaluation compares image-level FID from independent image-backbone frames against frames generated by the full Video LDM.The full model includes learned decoder parameters and temporal alignment layers.
  • Image-level quality: 48.26 FID after temporal video fine-tuning compares with 47.00 FID from the independent image backbone, indicating only a tiny degradation.The experiment uses the smaller Video LDM for driving video generation.
Loading 2304.08818v2…