Source-linked AI summary
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, Jie Tang
TL;DR
CogVideoX addresses the difficulty of generating long-term, temporally consistent videos with rich motion semantics. It combines a 3D causal VAE, expert Transformer, progressive training, and video-data processing, and reports state-of-the-art performance with CogVideoX-5B outperforming well-known video models while CogVideoX-2B remains competitive across most dimensions.
Problem
Previous video generation methods struggled to produce long-term consistent videos with dynamic plots and coherent narratives from text prompts.
Method
CogVideoX combines a 3D causal VAE, expert Transformer, progressive training, multi-resolution frame packing, and video captioning for text-to-video generation.
Results
CogVideoX-5B outperforms well-known video models in machine and human evaluations, while CogVideoX-2B is competitive across most dimensions.
Takeaways & Limitations
CogVideoX provides open 5B and 2B text-to-video and image-to-video models for generating coherent, long-duration videos with significant motion.
Takeaways & Limitations
The automated evaluation excludes metrics such as color because they tend to give higher scores to simple, static videos.
Abstract
from arXiv · showhide
We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos aligned with text prompt, with a frame rate of 16 fps and resolution of 768 * 1360 pixels. Previous video generation models often had limited movement and short durations, and is difficult to generate videos with coherent narratives based on text. We propose several designs to address these issues. First, we propose a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions, to improve both compression rate and video fidelity. Second, to improve the text-video alignment, we propose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. Third, by employing a progressive training and multi-resolution frame pack technique, CogVideoX is adept at producing coherent, long-duration, different shape videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method, greatly contributing to the generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weight of both 3D Causal VAE, Video caption model and CogVideoX are publicly available at https://github.com/THUDM/CogVideo.
1 Introduction
CogVideoX addresses the difficulty of generating long-term, temporally consistent videos with dynamic plots through a diffusion-transformer design combining specialized video compression, multimodal fusion, data processing, and progressive training. The resulting models support long, high-resolution videos and achieve strong automated and human-evaluated performance.
- Motivation: Previous text-to-video models struggled with long-term consistency and dynamic plots in generated videos.The paper gives prompts requiring multiple coherent events as an example of this difficulty.
- Approach: CogVideoX combines a 3D causal VAE, expert Transformer, progressive training, and video captioning pipeline for long-term video generation.The 3D VAE compresses videos spatially and temporally, while the expert Transformer targets video-text fusion.
- Evaluation: Both machine and human evaluations indicate that CogVideoX-5B outperforms well-known video models, while CogVideoX-2B remains competitive across most dimensions.The paper also reports that performance improves as model parameters, data volume, and training volume increase.
- Capabilities: CogVideoX generates videos with multiple aspect ratios, up to 768×1360 resolution, 10 seconds in length, at 16fps.The contribution describes the model as designed for coherent, long-duration, high-action videos.
- Evaluation: The authors evaluate CogVideoX against openly accessible top-performing text-to-video models using automated metrics and human assessment, reporting state-of-the-art performance.The paper publicly releases 5B and 2B text-to-video and image-to-video models.
2 The CogVideoX Architecture
CogVideoX combines a 3D causal VAE with transformer designs that jointly process spatial, temporal, and textual information. Its architecture compresses video efficiently, preserves temporal continuity, and addresses long-sequence training and cross-modal alignment.
- 3D Causal VAE: CogVideoX compresses videos spatially and temporally with a 3D VAE before patchifying the latents and combining them with T5 text embeddings.The resulting sequence is processed by expert transformer blocks.
- 3D Causal VAE: 8×8×4 compression maps pixels to latents through an encoder, decoder, and KL regularizer, reducing video representation size.The architecture uses interleaved spatial and spatiotemporal downsampling or upsampling stages.
- 3D Causal VAE: Temporally causal convolutions prevent future information from influencing present or past predictions, while context parallelism distributes long-video computation across devices.Each device communicates a segment of length k −1 to the next rank, limiting communication overhead.
- 3D Causal VAE: 3D VAE reconstruction shows almost no jitter, but overly aggressive 16×16×8 compression makes model convergence extremely difficult.Increasing latent channels improves restoration quality; larger compression ratios remain future work.
- Expert Transformer: The transformer uses 3D-RoPE, expert adaptive LayerNorm, and hybrid 3D attention to represent video coordinates and align text with visual features.Expert adaptive LayerNorm processes text and vision modalities independently, while 3D attention models spatial and temporal relationships together.
- Progressive Training: Mixed-duration training and Multi-Resolution Frame Pack place videos with different lengths and resolutions in consistent batches.This design addresses fixed-frame training limitations and supports broader data utilization and generalization.
3 Training CogVideoX
CogVideoX training combines mixed-duration, multi-resolution data handling with progressive resolution training, explicit timestep sampling, and extensive video filtering and captioning. These choices address variable video shapes, unstable diffusion training, noisy data, and sparse textual descriptions.
- Training strategy: Mixed-duration training packs videos with different lengths and resolutions into consistent batches through Multi-Resolution Frame Pack.This avoids discarding short videos or truncating long ones while enabling mixed-duration training.
- Training strategy: 3D RoPE models spatial and temporal positions across varying video shapes, using extrapolation to preserve relative pixel positions and detail.The paper contrasts extrapolation with interpolation and selects extrapolation for RoPE.
- Training strategy: Progressive training starts at 256px for semantic and low-frequency learning, then increases resolution while preserving aspect ratios.The final stage applies high-quality fine-tuning.
- Training strategy: Explicit Uniform Sampling divides diffusion timesteps across data-parallel ranks, producing a more uniform distribution and a more stable loss curve.Each rank samples within its assigned interval from 1 to T.
- Data processing: The captioning pipeline uses Panda70M or a video caption model, frame extraction, image recaptioning, GPT-4 summarization, and fine-tuned captioning models.CogVLM2-Caption is further fine-tuned to provide detailed descriptions of video content and changes.
4 Experiments
Experiments evaluate CogVideoX through ablations, automated metrics, and human judgments. The results support expert AdaLN, 3D full attention, explicit uniform sampling, the proposed VAE, and strong overall performance across dynamic video-generation criteria.
- Ablation Study: Expert AdaLN outperforms models without it and same-parameter MMDiT according to FVD, CLIP4Clip score, and loss.The authors infer that expert adaptive LayerNorm can alleviate cross-modal feature-space differences without two independent transformers.
- Ablation Study: Replacing 3D full attention with 2D+1D attention produces higher early-step FVD and unstable training prone to collapse.The paper reports that 2D+1D attention is unsuitable for video generation, especially as model size increases.
- Ablation Study: Explicit Uniform Sampling yields a more stable loss decrease, better performance, and lower loss across diffusion timesteps.The authors attribute this to reducing randomness caused by uneven timestep sampling.
- Ablation Study: The proposed 3DVAE achieves the best PSNR and least jitter on 256 × 256, 17-frame WebVid validation videos.Other compared VAE methods use fewer latent channels.
- Automated Metric Evaluation: CogVideoX-5B achieves the best performance on five of seven automated metrics and remains competitive on the other two.The evaluation covers video quality and complex dynamic scenes.
- Human Evaluation: CogVideoX-5B receives higher human preference than Kling across Sensory Quality, Instruction Following, Physics Simulation, and Cover Quality.Human evaluation uses scores of 0, 0.5, or 1 for each aspect.
5 Conclusion
The conclusion presents CogVideoX as a text-to-video diffusion model using 3D VAE and Expert Transformer components to generate coherent, long-duration videos with significant motion. It also identifies scaling toward longer and higher-quality videos as future work.
- Conclusion: CogVideoX combines a 3D VAE with an Expert Transformer to generate coherent long-duration videos with significant motion.The paper presents this as the model’s central architectural approach.
- Future work: Future work explores scaling video-generation models to produce longer and higher-quality videos.The authors aim to train larger and more powerful models.
- Supplementary material: The appendices provide training, loss-curve, example, image-to-video, related-work, captioning, filtering, and human-evaluation details.These materials cover implementation and additional evaluations beyond the main sections.
A Training Details
Additional training details describe quality-focused fine-tuning and RoPE adaptation for higher resolutions, alongside reported compute measurements and supplementary loss-curve materials. Fine-tuning improves visual quality but slightly reduces semantic ability.
- Training Details: High-quality fine-tuning on 20% of the dataset removes generated subtitles and watermarks and slightly improves visual quality.The authors also observe a slight degradation in semantic ability.
- Training Details: RoPE adaptation compares interpolation and extrapolation when increasing resolution, selecting extrapolation to preserve relative pixel positions and local detail.Interpolation preserves global information more effectively, whereas extrapolation retains local details better.
- Training Details: Tables report CogVideoX hyperparameters, inference time and memory consumption, attention-comparison timing, and validation loss by diffusion timestep.The appendix also includes additional examples and model-related materials.
D Image To Video Model
CogVideoX extends text-to-video generation to image-conditioned video by adding an image condition alongside text and training for robustness across image styles.
- Image-to-video conditioning: CogVideoX finetunes an image-to-video model from its text-to-video model by adding an image as an additional condition alongside text.The image is processed through the 3D VAE before conditioning generation.
- Image-to-video conditioning: The image condition is passed through the 3D VAE and concatenated with the noised input in the channel dimension.
- Robustness training: Large noise is added to the image condition during training to reduce the distribution gap between real-world images and video first frames.This improves robustness under image-to-video inference conditions.
- Results and examples: The model handles different styles of image input, with image-to-video examples shown in Figures 13 and 14.The figure captions identify these as image-to-video showcases.
- Prompt processing: A language model upsamples user prompts during inference so their distribution more closely matches the detailed prompts used during training.For image-to-video, a vision-language model such as GPT4V or CogVLM performs prompt upsampling.
G Dense Video Caption Data Generation
The dense caption pipeline samples video frames, captions them individually, and asks GPT-4 to synthesize a detailed chronological description within a fixed length.
- Frame sampling: The pipeline extracts one frame every two seconds for image captioning and collects 50,000 data points to fine-tune the summary model.
- Caption synthesis: GPT-4 receives frame-level captions and is prompted to describe objects, scenery, animals, characters, and camera movements in detail.
- Caption synthesis: The generated description must present video content and changes chronologically and remain within 100 English words.
H Video Caption Example
Examples compare captions from Panda-70M and CogVLM2-Caption, showing that CogVLM2-Caption produces substantially more detailed descriptions used for video-to-video generation.
- Caption comparisons: The examples compare the performance of Panda-70M and CogVLM2-Caption on video captioning.
- Caption comparisons: Panda-70M captions provide brief descriptions, such as a man reading on a cloud or a close-up of a white dragon with blue eyes.
- Caption comparisons: CogVLM2-Caption adds detailed attributes, settings, expressions, object changes, and motion to the corresponding descriptions.Examples include a crab’s illuminated bulb, a dragon’s changing horns, and a reader’s clothing and posture.
- Video-to-video generation: The video-to-video pipeline captions an original video with CogVLM2-Caption and feeds that caption into CogVideoX to generate a new video.
- Video-to-video generation: The authors report high fidelity to the original videos, attributing this to captions that capture almost all video details.
J Human Evaluation Details
Human evaluation measures video quality through broad prompts and four criteria: sensory quality, instruction following, physics simulation, and cover quality.
- Evaluation setup: Human evaluators score 100 broadly distributed, clearly articulated prompts with well-defined conceptual scope.
- Scoring: Evaluators assign each detail a score from 0 to 1 and an overall video-quality score from 0 to 5, with higher scores indicating better quality.
- Instruction following: Instruction-following scores cannot exceed 2 when a generated video fails to follow the prompt.
- Sensory quality: Sensory quality evaluates subject consistency, frame continuity, and stability.
- Instruction following: Instruction following evaluates alignment with the prompt, including subject, quantity, elements, and details.
- Physics and cover quality: Physics simulation evaluates physical realism in lighting, object interactions, and fluid dynamics, while cover quality evaluates single-frame aesthetics, clarity, and fidelity.
K Data Filtering Details
CogVideoX filters low-quality video data using negative labels and evaluates the filtering classifier on a test set. Figure 16 illustrates the labels and sample videos, while Table 14 summarizes classifier performance.
- Negative labels are designed to filter out low-quality video data.These labels are presented with sample videos in Figure 16.
- Figure 16 presents negative labels alongside sample videos for each label.
- Table 14 reports classifier accuracy and recall on a test set containing 10% randomly labeled data.The table uses TP, FP, TN, and FN terminology for classifier outcomes.