PaperlayerView exact revision on arXiv

Source-linked AI summary

CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, Jie Tang

arXiv:2205.15868v1cs.CVcs.CLcs.LG

TL;DR

Text-to-video generation is constrained by difficult temporal action alignment, scarce or weakly relevant text-video data, and the cost of training large models from scratch. CogVideo adapts pretrained CogView2 and introduces multi-frame-rate hierarchical training to align text with video semantics. The paper presents CogVideo as a large-scale open-source pretrained text-to-video transformer with improved generation accuracy and control over change intensity.

  • Problem

    Text-to-video models struggle with complex actions because text-video data are scarce or weakly relevant and fixed-frame training can disrupt temporal alignment.

  • Method

    CogVideo finetunes pretrained CogView2 using multi-frame-rate hierarchical training, with dual-channel attention transferring text-image knowledge to video generation.

  • Results

    CogVideo is reported as the largest and first open-source pretrained transformer for general-domain text-to-video generation, with hierarchical training improving generation accuracy, especially for complex movements.

  • Takeaways & Limitations

    Pretrained text-to-image models can be efficiently leveraged for text-to-video generation while preserving the ability to control the intensity of changes during generation.

Abstract

from arXiv · show

Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video generation is still facing many challenges: The potential huge computation cost makes the training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movement semantics. In this work, we present 9B-parameter transformer CogVideo, trained by inheriting a pretrained text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, CogVideo outperforms all publicly available models at a large margin in machine and human evaluations.

1 Introduction

Text-to-video generation remains difficult because autoregressive models can lose alignment with prompts during motion, while available datasets and fixed-length training clips inadequately represent temporal actions. CogVideo addresses these challenges by adapting pretrained text-to-image knowledge and using multi-frame-rate hierarchical training.

  • Challenges: Autoregressive video models can gradually deviate from prompts, failing to generate complex actions such as a lion drinking water.Regular or random patterns may be easier because the first frame already informs subsequent changes, whereas complex actions require precise temporal understanding.
  • Challenges: The main hypothesized causes of weak text-action understanding are limited datasets and ineffective data utilization.The paper contrasts strong text-image understanding with weaker text-action understanding in videos.
  • Challenges: Text-video data are scarcer than text-image data: VATEX contains only 41,250 videos, while retrieval-based pairs are often weakly relevant and lack temporal information.Many retrieved captions describe scenes without describing how actions unfold over time.
  • Challenges: Fixed-frame training clips can destroy alignment between text and temporal counterparts, confusing a single action label with separate sub-actions.A drinking video may be divided into holding, lifting, drinking, and putting-down clips that all retain the same text.
  • Present Work: CogVideo is a 9.4-billion-parameter model trained on 5.4 million text-video pairs by inheriting knowledge from pretrained CogView2.The model uses multi-frame-rate hierarchical training to align text with complete actions and control change intensity during generation.
  • Present Work: CogVideo combines sequential key-frame generation with recursive frame interpolation, varying frame rates to produce coherent videos.The sequential model generates key frames from text, while the interpolation model fills middle frames.

2 Related Work

Video generation progressed from deterministic next-frame prediction and GAN-based synthesis toward VQVAE and autoregressive-transformer frameworks. Open-domain text-to-video transformers emerged recently, but chronological frame generation can still suffer from poor text-video alignment.

  • Earlier Video Generation: Most earlier video-generation work focused on forecasting future frames from an initial video frame.This next-frame prediction framing differs from unconditional or text-conditioned video synthesis.
  • Earlier Video Generation: Deterministic CNN- and RNN-based methods could not capture stochastic temporal patterns or coherent complex scenes.CDNA and PredRNN directly predicted the next frame using deterministic methods.
  • Generative Models: GANs enabled unconditional or class-conditional video synthesis without requiring initial frames.Representative approaches decomposed videos into background and foreground or separated temporal and spatial factors.
  • Autoregressive Transformers: VQVAE combined with autoregressive transformers became a mainstream framework for video generation, following its success in token-based image modeling.Recent work also introduced video diffusion models for text-to-video generation.
  • Text-to-Video Generation: GODIVA and NÜWA target open-domain text-to-video generation but generate frames or frame blocks chronologically, which may weaken text-video alignment.The cited limitation follows the related-work discussion’s comparison with the paper’s alignment-focused approach.

3 Method

CogVideo combines multi-frame-rate hierarchical training, dual-channel attention, and autoregressive shifted-window attention to align text with video semantics, preserve pretrained image knowledge, and reduce generation overhead.

  • 3 Method: The method addresses large-scale video training challenges through hierarchical temporal alignment, pretrained image-model inheritance, and local autoregressive attention.These components target semantic alignment, knowledge transfer, and the time-memory costs of long video-token sequences.
  • 3.1 Multi-frame-rate Hierarchical Training: Fixed-rate clipping can mismatch text with incomplete actions, while adjacent-frame similarity encourages copying instead of learning long-range temporal correlations.These issues motivate selecting frame rates that preserve text-video alignment and expose broader temporal structure.
  • 3.1 Multi-frame-rate Hierarchical Training: Multi-frame-rate hierarchical training matches text and frames by sampling at the lowest feasible rate, then recursively interpolating frames at finer rates.The process sequentially generates key frames before inserting transition frames; frame-rate conditioning is used throughout.
  • 3.1 Multi-frame-rate Hierarchical Training: CogLM divides tokens into bidirectional and unidirectional attention regions so interpolation can use known-frame context without disrupting autoregressive prediction.Text, frame rate, and known frames are bidirectional, while predicted frames remain in the unidirectional region.
  • 3.2 Dual-channel Attention: Dual-channel attention adds a trainable spatial-temporal channel to frozen CogView2 attention, while shared FFNs transfer visual knowledge and reduce parameters.The design avoids directly finetuning CogView2, which the paper says can rapidly damage pretrained weights under mismatched temporal attention.
  • 3.3 Shifted Window Attention in Auto-regressive Generation: Autoregressive shifted-window attention applies masks within local 3D windows, allowing parts of later frames to generate in parallel and accelerating inference.The paper gives a maximum parallelism of ⌊XY/(AxY+Ay)⌋ tokens under its stated window dimensions.

4 Training

CogVideo uses a dual-channel Transformer backbone with frozen pretrained components and trains on millions of captioned videos using frame-rate-specific sampling and multi-rate clips.

  • Model: CogVideo has 9.4 billion parameters across 48 Transformer layers, with 6 billion parameters fixed to CogView2 components.The model uses hidden size 3,072 per attention channel and 48 attention heads.
  • Dataset: The pretraining dataset contains 5.4 million captioned videos at 160 × 160 resolution, with Stage 1 adapting frame rates to each full video.Stage 2 uses clips of different lengths to support 2, 4, and 8 fps prediction.
  • Pretraining: Both training stages use sequence length 2,065, comprising 64 text tokens, 5 × 400 image tokens, and 1 separator token.Optimization uses Adam with maximum learning rate 2×10^-4, β1 = 0.9, β2 = 0.95, and weight decay 1 × 10^-2.

5 Experiments

CogVideo is evaluated on standard video-generation benchmarks, human judgments, and ablations of hierarchical generation and CogView2 initialization. Across these evaluations, hierarchical generation and pretrained initialization improve quality, realism, consistency, and semantic relevance.

  • 5.1 Machine Evaluation: CogVideo is evaluated on UCF101 and Kinetics-600 using Fréchet Video Distance (FVD) and Inception score (IS).UCF101 evaluation uses 2,048 samples for FVD and 10,000 for IS; Kinetics-600 evaluation measures FVD on 64×64 frames.
  • 5.2 Human Evaluation: 90 evaluators rate CogVideo highest overall, with 49.53% choosing it as the best method versus 15.42% for VideoGPT and 5.6% for TGANv2.CogVideo also significantly outperforms the baselines in frame texture, motion realism, and semantic relevance.
  • 5.3.1 Quantitative Evaluation: The hierarchical method is clearly superior to 1-stage generation across different N_overlap settings, while CogView2 initialization yields lower FVD than random initialization.The ablations are trained for 11,000 iterations with batch size 160.
  • 5.3.1 Quantitative Evaluation: CogView2 initialization accelerates loss decrease, and fixing part of the parameters reduces training time and memory cost.The training-loss ablation compares finetuning CogVideo, random initialization, and CogView2 initialization with partial parameter fixing.
  • 5.3.2 Qualitative Evaluation: Random initialization tends to produce irrational deformation, whereas CogView2 produces realistic objects and hierarchical generation improves content consistency and motion realism.The comparison uses samples primed on a class label and the first five Kinetics-600 frames.
  • 5.3.2 Qualitative Evaluation: Human evaluation finds the hierarchical model superior to the 1-stage model in semantic relevance, motion realism, and texture quality.The authors attribute this difference to the 1-stage model’s difficulty estimating an appropriate change intensity from previous frames.

6 Conclusion

CogVideo is presented as a large, open-source pretrained transformer that efficiently transfers text-to-image knowledge to text-to-video generation. Its hierarchical training improves text-video relations and change-intensity control, but sequence length remains constrained by model scale and GPU memory.

  • 6 Conclusion: CogVideo is presented as the largest and first open-source pretrained transformer for general-domain text-to-video generation.The claim is explicitly qualified as being to the authors’ knowledge.
  • 6 Conclusion: CogVideo leverages a pretrained text-to-image model without hurting its image-generation capacity.The approach is described as an efficient transfer of pretrained knowledge to text-to-video generation.
  • 6 Conclusion: Multi-frame-rate hierarchical training improves text-video understanding and enables control over the intensity of changes during generation.The conclusion attributes both capabilities to the proposed training framework.
  • 6 Conclusion: The model’s large scale and GPU-memory limits still restrict the length of its input sequence.The authors leave this limitation for future work.
  • Broader Impact: The paper argues that transferring knowledge from text-to-image models can avoid training from scratch, reducing energy consumption and carbon emissions.The broader-impact discussion also identifies misinformation as a potential negative impact.

A Attention Analysis

The attention analysis shows that CogVideo distributes attention among text, individual frames, and multiple frames, with layer-dependent patterns. Its temporal channel emphasizes inter-frame relationships while the spatial channel retains image-derived feature analysis.

  • Attention Visualization: The analysis visualizes temporal-channel attention and the mixture factor α controlling spatial-versus-temporal attention.Only selected attention heads and layers are displayed in the heat maps.
  • Attention Patterns: Attention heads focus on text, particular frames, or several frames, and some heads combine multiple patterns.Examples include attention to the previous frame, first frame, current frame, and distributed frame context.
  • Layerwise Patterns: Lower layers tend to allocate attention according to position, whereas higher layers allocate more attention to text or multiple frames.The authors suggest this may reflect higher-level video-semantic features in deeper layers.
  • Channel Roles: Many higher-layer temporal heads attend little to the current frame, while the spatial channel handles within-frame analysis and the temporal channel explores relationships across frames.The authors interpret this as partial decoupling of temporal and spatial feature analysis.

B Training Details

CogVideo uses two large models for sequential generation and recursive interpolation, sharing extensive CogView2-derived parameters. Training spans millions of captioned videos and multiple frame rates, while attention analysis aggregates token-level frame-text scores.

  • Model Configuration: CogVideo has sequential-generation and recursive-interpolation models, each with 7.7 billion parameters and 6 billion fixed to CogView2.Together, the models contain 9.4 billion different parameters.
  • Training Setup: CogVideo is trained on 5.4 million captioned videos at 160×160 resolution, with separate pretraining schedules for its two stages.Stage 1 uses minimum frame rates of 0.25 and 1 fps, while Stage 2 uses 2, 4, and 8 fps.
  • Attention Analysis: Stage 1 sequential generation uses frame-text and frame-frame attention, with the displayed heat map summarizing attention across five frames and text.Only 24 of 48 attention heads in six layers are shown.
  • Attention Analysis: The attention statistic sums attention scores from tokens in frame F_i to tokens in the text set T.The notation defines attn_x,y as the attention score from token x to token y.

C Details about Human Evaluation

The human evaluation compares CogVideo with other video-generation models using randomly selected UCF101 classes and human ratings of visual, motion, semantic, and overall quality.

  • 30 UCF101 classes were randomly selected, and corresponding dataset videos served as ground-truth items for evaluation.
  • The evaluation compared TGANv2, VideoGPT, CogVideo, and a one-stage CogVideo variant fine-tuned on Kinetics-600.
  • Evaluators scored frame texture, motion realism, and semantic relevance from 1 to 5, then assigned overall quality scores from 1 to 10.
  • Figure 10 displays 16-frame original evaluation clips downsampled uniformly to 4 frames, while Figure 11 shows the evaluation website.
  • After rating all samples for a caption, evaluators selected the best video among the compared models.
  • Nearly 100 anonymous evaluators participated, with 90 completing the full evaluation and contributing to the final results.Each evaluator received 75 RMB, and there were no time limits for questions.
Loading 2205.15868v1…