Source-linked AI summary
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, Mike Zheng Shou
TL;DR
Existing text-to-video diffusion models typically trade precise text-video alignment against computational efficiency when choosing pixel or latent representations. Show-1 combines both: pixel VDMs generate aligned low-resolution video, while latent expert translation performs efficient, artifact-reducing super-resolution; the model achieves state-of-the-art benchmark performance and supports temporal-attention-based adaptation.
Problem
Previous text-to-video methods typically rely exclusively on pixel VDMs with high computational costs or latent VDMs that can struggle with precise text-video alignment.
Method
Show-1 generates low-resolution video with pixel-based VDMs and uses latent-based expert translation for efficient high-resolution super-resolution that reduces artifacts and corruptions.
Results
Show-1 achieves state-of-the-art performance on popular video-generation benchmarks including UCF101, MSR-VTT, and VBench.
Takeaways & Limitations
Show-1 produces high-resolution videos with precise text-video alignment at low computational cost and can be adapted for motion customization and video stylization by fine-tuning temporal attention layers.
Takeaways & Limitations
High-resolution synthetic-video upsampling remains challenging because temporal consistency and visual quality with minimal artifacts require high model complexity, while real-data training creates a domain gap.
Abstract
from arXiv · showhide
Significant advancements have been achieved in the realm of large-scale pre-trained text-to-video Diffusion Models (VDMs). However, previous methods either rely solely on pixel-based VDMs, which come with high computational costs, or on latent-based VDMs, which often struggle with precise text-video alignment. In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation. Our model first uses pixel-based VDMs to produce a low-resolution video of strong text-video correlation. After that, we propose a novel expert translation method that employs the latent-based VDMs to further upsample the low-resolution video to high resolution, which can also remove potential artifacts and corruptions from low-resolution videos. Compared to latent VDMs, Show-1 can produce high-quality videos of precise text-video alignment; Compared to pixel VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G vs 72G). Furthermore, our Show-1 model can be readily adapted for motion customization and video stylization applications through simple temporal attention layer finetuning. Our model achieves state-of-the-art performance on standard video generation benchmarks. Our code and model weights are publicly available at https://github.com/showlab/Show-1.
1 Introduction
Show-1 combines pixel-based and latent-based video diffusion models to balance precise text-video alignment with efficient high-resolution generation. It generates aligned low-resolution keyframes with pixel VDMs, then uses latent expert translation for artifact-aware super-resolution and broader adaptation.
- Motivation: Pixel-based VDMs align motion more accurately with text but require high computational costs, whereas latent-based VDMs are efficient but may compromise text-video alignment.Pixel models operate from very low resolutions, while latent models use compressed representations that may omit prompt-relevant visual details.
- Approach: Show-1 first generates low-resolution, low-frame-rate keyframes with pixel-based VDMs, then enhances temporal and spatial resolution through subsequent modules.This coarse-to-fine strategy uses pixel VDMs where they improve alignment and reserves latent processing for efficient upsampling.
- Approach: Expert translation uses latent-based VDMs for final super-resolution while addressing artifacts and corruptions that typical SDx4 upsampling leaves unresolved.Compared with temporal SDx4 upsampling, the approach improves temporal consistency and reduces noise in the X-T slice.
- Results: Show-1 achieves state-of-the-art performance on popular video-generation benchmarks including UCF101, MSR-VTT, and VBench.The reported benchmark result summarizes the model-level outcome of combining the two VDM types.
- Contributions: Show-1 integrates pixel and latent VDMs to produce high-resolution videos with precise text-video alignment at low computational cost.The model reports 15G GPU memory during inference and supports motion customization and video stylization through temporal-attention-layer fine-tuning.
2 Related Work
Related work spans text-to-image and text-to-video generation, including GANs, autoregressive models, implicit neural representations, and diffusion models. Show-1 differs by explicitly combining pixel-based and latent-based video diffusion models to leverage their complementary strengths.
- Text-to-image generation: Text-to-image research progressed from GAN-based methods toward approaches improving text-image alignment and diffusion-based photorealistic, compositional synthesis.Text-to-image diffusion models also provide initialization for text-to-video models.
- Text-to-video generation: Text-to-video research has used GANs, autoregressive models, implicit neural representations, and diffusion models for conditional generation.These model families form the broader methodological context for video generation.
- Show-1: Show-1 investigates combining pixel-based and latent-based VDMs to leverage their strengths while avoiding their weaknesses.This distinguishes the approach from methods that use only one representation type throughout the generation pipeline.
3 Show-1
Show-1 converts a text-to-image UNet into a video model by adding temporal processing while retaining spatial weights, then uses a coarse-to-fine cascade for interpolation and upsampling. Its final stages address the difficulty of 8× video super-resolution through separate pixel and latent modules.
- 3.2 Turn Image UNet to Video: Show-1 adds temporal convolution and attention layers to each UNet block while retaining spatial weights from a robust text-to-image model.The temporal layers model video dynamics, while the spatial layers remain inherited from the image model and are fixed during training.
- 3.3 Keyframe Generation: Pixel-based diffusion first generates low-resolution, low-frame-rate keyframes to improve text-to-video alignment by prioritizing text guidance over visual clarity and temporal consistency.The keyframe modules operate at very low spatial and temporal resolution, where appearance and consistency requirements are reduced.
- 3.4 Temporal Interpolation Model: A masked pixel-based interpolation model predicts intermediate frames between keyframes, using conditional frames and binary masks as input signals.The model concatenates noisy frames, keyframes, and masks, then predicts noise for the interpolated frames.
- 3.5 Super-resolution at Low Spatial Resolution: Because 8× upsampling is difficult for one module, Show-1 separates spatial enhancement into an initial 4× pixel super-resolution stage and a later high-resolution stage.The first module maps low-resolution video to a video with fourfold spatial dimensions and uses bilinearly resized low-resolution frames as conditioning.
- 3.6 Super-resolution at High Spatial Resolution: The final super-resolution module uses latent diffusion to refine high-resolution details and manage artifacts or temporal corruptions from previous stages, while reducing memory demands versus pixel processing.A latent VDM is selected for efficient final upsampling, whereas prior-stage artifacts motivate its artifact-management role.
4 Experiments
Experiments show that Show-1 combines strong text-video alignment, visual quality, motion fidelity, and efficiency across quantitative, qualitative, and ablation evaluations. It also supports motion customization and video stylization.
- Implementation Details: Show-1 produces videos at 29 × 576 × 320 after keyframe generation and two superresolution stages.The pipeline progresses from 8 × 64 × 40 × 3 keyframes through temporal interpolation and spatial upsampling.
- Quantitative Results: Show-1’s zero-shot capabilities outperform or match other methods on UCF-101, despite training its early modules only on WebVid-10M.The comparison contrasts Show-1’s publicly trained components with Make-A-Video’s additional internal text-video data.
- Quantitative Results: 12.97 FID-vid ranks second-best, 536 FVD ranks best, and 0.3104 CLIPSIM exceeds Make-A-Video on MSR-VTT.The CLIPSIM comparison favors Show-1 despite Make-A-Video using additional training data beyond WebVid-10M.
- Quantitative Results: 10 of 16 VBench metrics rank Show-1 first, surpassing methods trained on large-scale proprietary text-video datasets.The reported results were obtained using the publicly accessible WebVideo-10M dataset.
- Qualitative Results: Human evaluations prefer Show-1 across all evaluation parts, with superior text-video alignment and motion fidelity against several open-source methods.Visual quality matches or surpasses Imagen Video and Make-A-Video, while text-video alignment exceeds Gen-2 and Pika.
- Ablation Studies: Expert translation improves video quality and reduces artifacts compared with standard SDx4 temporal layers, while Show-1 is faster and more memory-efficient than Make-A-Video.The model also adapts motion effectively and supports video stylization and editing aligned with accompanying text.
5 Conclusion
Show-1 combines pixel- and latent-based Video Diffusion Models to generate high-quality text-to-video outputs while reducing computational costs.
- 5 Conclusion: Show-1 uses pixel-based VDMs for initial generation and latent-based VDMs for efficient high-resolution super-resolution.The combined strategy targets precise text-video alignment and motion portrayal while transitioning from lower to higher resolution.
Ackmowledgement
The research received Singapore Ministry of Education support and used National Supercomputing Centre resources.
- Ackmowledgement: The work was supported by Singapore’s Ministry of Education Academic Research Fund Tier 1 and National Supercomputing Centre resources.