Source-linked AI summary
Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, Xi Yin
TL;DR
Text-to-video generation is difficult to scale because pixel-space pipelines are computationally complex and extending image U-Nets typically adds temporal modules. Latent-Shift instead uses parameter-free temporal shifting in a pretrained T2I latent diffusion model, achieving competitive or better performance with greater efficiency while retaining T2I generation.
Problem
Text-to-video generation faces limited high-quality text-video data and complex temporal modeling, while pixel-space pipelines require costly interpolation and super-resolution stages.
Method
Latent-Shift finetunes a pretrained T2I latent diffusion model with a parameter-free temporal shift module that moves feature channels across neighboring frames.
Results
Latent-Shift achieves better performance than latent video diffusion models with temporal attention while using fewer parameters and remaining competitive with prior methods.
Takeaways & Limitations
A video-finetuned Latent-Shift model can also generate images, preserving T2I capability through its parameter-free temporal-shift design.
Takeaways & Limitations
The method can produce artifacts, text mismatches, or limited motion, and these issues may relate to training-data scale and an image-trained VAE.
Abstract
from arXiv · showhide
We propose Latent-Shift -- an efficient text-to-video generation method based on a pretrained text-to-image generation model that consists of an autoencoder and a U-Net diffusion model. Learning a video diffusion model in the latent space is much more efficient than in the pixel space. The latter is often limited to first generating a low-resolution video followed by a sequence of frame interpolation and super-resolution models, which makes the entire pipeline very complex and computationally expensive. To extend a U-Net from image generation to video generation, prior work proposes to add additional modules like 1D temporal convolution and/or temporal attention layers. In contrast, we propose a parameter-free temporal shift module that can leverage the spatial U-Net as is for video generation. We achieve this by shifting two portions of the feature map channels forward and backward along the temporal dimension. The shifted features of the current frame thus receive the features from the previous and the subsequent frames, enabling motion learning without additional parameters. We show that Latent-Shift achieves comparable or better results while being significantly more efficient. Moreover, Latent-Shift can generate images despite being finetuned for T2V generation.
1. Introduction
Latent-Shift extends a pretrained T2I latent diffusion model to efficient T2V generation using parameter-free temporal shifting. It targets video-generation complexity while retaining image-generation capability and competitive performance.
- Results: Latent-Shift achieves better performance than latent video diffusion models with temporal attention while using fewer parameters.The comparison supports the paper’s efficiency claim for latent-space video generation.
- Method: Latent-Shift uses a parameter-free temporal shift module to extend a spatial T2I U-Net for T2V generation without adding new parameters.Channels are shifted forward and backward across frames so each frame receives features from neighboring frames.
- Capabilities: A video-finetuned Latent-Shift model can also perform T2I generation.The paper identifies this retained image-generation capability as a distinctive property of the parameter-free temporal shift design.
- Evaluation: The method is evaluated on MSRVTT, UCF-101, and through a user study.These evaluations are used to demonstrate effectiveness and efficiency.
2. Related Work
Related work spans Transformer–VAE and diffusion-based approaches for text-to-video generation. Latent-based methods seek to reduce the complexity of pixel-space video pipelines, which may require multiple models.
- Text-to-Video Generation: Transformer–VAE and diffusion U-Net models are established frameworks for text-to-video generation.CogVideo and Phenaki use Transformer–VAE models, while Make-A-Video and Imagen Video use diffusion models.
- Joint Image-and-Video Generation: Imagen Video and Phenaki support joint image-and-video generation by treating images as independent frames or frozen videos.These designs incorporate image generation into their respective video-generation frameworks.
- Latent Diffusion for Video Generation: Pixel-space video generation pipelines can require multiple models to produce high-resolution videos.Make-A-Video uses six models and Imagen Video uses eight models according to the cited table.
- Latent Diffusion for Video Generation: Latent-based video-generation models are explored to reduce pipeline complexity.The related-work discussion focuses specifically on latent diffusion models.
3. Method
Latent-Shift extends a pretrained latent image-diffusion U-Net to video generation by inserting parameter-free temporal shift modules, preserving the spatial architecture while incorporating neighboring-frame information.
- Temporal Shift: Temporal shift splits feature channels and moves portions forward and backward across frames before merging them, allowing a 2D spatial network to process temporal information.Zero padding handles boundary frames, and the operation is inserted in each 2D ResNet residual branch.
- Latent-Shift for T2V Generation: The framework adapts a pretrained autoencoder and latent-diffusion U-Net from image generation to video generation, encoding and decoding video frames independently.The U-Net is finetuned on encoded latent video representations for text-conditioned denoising.
- Temporal Shift: Unlike prior extensions that add temporal convolution or attention layers, Latent-Shift inserts a parameter-free module while retaining the pretrained spatial U-Net blocks.The U-Net combines 2D ResNet blocks for convolutional processing with transformer blocks for spatial attention and text cross-attention.
- Latent-Shift for T2V Generation: Video training adds noise to latent video representations at sampled diffusion steps, and the U-Net learns to estimate that noise under text conditioning.The text encoder remains fixed, while inference uses a DDPM sampler with 100 resampled steps.
4. Experiments
Latent-Shift is evaluated against prior video-generation methods using automatic metrics, user preferences, efficiency comparisons, and qualitative analyses. It generally performs competitively or better with fewer parameters, while retaining image-generation ability and exhibiting known failure modes.
- Main Results: On MSR-VTT, Latent-Shift is competitive with prior methods and often outperforms several baselines, although it does not surpass Make-A-Video.The authors attribute the gap to Latent-Shift's limited model size and report that its performance is much closer to Make-A-Video than that of other models.
- Main Results: Latent-Shift generates semantically richer videos with meaningful motion faithful to the input text in visual comparisons with CogVideo.These comparisons cover the user-study evaluation set and UCF-101.
- Ablation Study: MSR-VTT metrics do not account for motion information, motivating better automatic metrics for zero-shot text-to-video evaluation.This limitation arises because images can be treated as frozen videos in the comparison.
- Ablation Study: Temporal shift preserves reasonable T2I generation after T2V finetuning, whereas removing the shift module causes image generation to fail and disabling it during inference prevents meaningful videos.The authors explain this behavior through training with missing temporal context at sequence boundaries.
- Qualitative Results: Latent-Shift can distort complex objects, omit text-specified content, or produce limited motion when actions are subtle.The authors relate these failures to mixed concepts, limited text-video data, an image-trained VAE, and subtle actions.
5. Conclusion
Latent-Shift is a simple, efficient T2V framework that finetunes a pretrained T2I model with parameter-free temporal shift. It preserves T2I generation and demonstrates effectiveness and efficiency on MSR-VTT, UCF101, and user studies.
- Latent-Shift finetunes a pretrained T2I model with temporal shift to model temporal information without adding parameters.
- The finetuned model preserves T2I generation capability despite being trained on videos.
- Experiments on MSR-VTT, UCF101, and user studies demonstrate the approach’s effectiveness and efficiency.
A. Hyperparameter Settings
The experiments sample 16 frames from two-second clips, resize and crop them to 256 × 256, and use a 32 × 32 × 4 latent space.
- 16 frames are evenly sampled from each two-second video clip and processed at 256 × 256 resolution.
- The latent representation has dimensions 32 × 32 × 4.
- Temporal shift retains one-third of the channels from the previous, current, and subsequent frames.
B. Text-to-Video Generation
Figure 9 compares Latent-Shift with CogVideo and Video Diffusion Models for text-to-video generation. The figure is presented as a continued qualitative comparison across the displayed examples.
- Figure 9 compares Latent-Shift, CogVideo, and Video Diffusion Models in text-to-video generation examples.