PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoTetris: Towards Compositional Text-to-Video Generation

Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Jingmin Chen, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, Di Zhang, Bin Cui

arXiv:2406.04277v2cs.CV

TL;DR

Existing video-generation methods can struggle with complex multi-object prompts and progressive changes in object numbers. VideoTetris combines spatio-temporal attention composition with enhanced data preprocessing and reference-frame attention. The paper reports state-of-the-art compositional-video quality and high-quality long videos aligned with progressive prompts, while noting limits in long-video generation and autoregressive control.

  • Problem

    Existing video-generation methods struggle with complex multi-object scenes and long-video prompts involving dynamic changes in object number.

    Source: §1 · p. 1

  • Method

    VideoTetris manipulates denoising cross-attention spatially and temporally and pairs data preprocessing with Reference Frame Attention for long-video consistency.

    Source: §1 · pp. 1, 3, §5 · p. 11

  • Results

    Experiments report state-of-the-art compositional-video quality and high-quality long videos aligned with progressive prompts while maintaining consistency.

    Source: §1 · p. 3

  • Takeaways & Limitations

    VideoTetris extends video diffusion models’ generative capabilities to compositional video content.

    Source: §5 · p. 11

    Long-video generation remains constrained by base-model limitations, while the ControlNet autoregressive setup is computationally costly and can produce excessive transition frames that challenge object consistency and position control.

    Source: §5 · p. 11

Abstract

from arXiv · show

Diffusion models have demonstrated great success in text-to-video (T2V) generation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, a novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, we propose an enhanced video data preprocessing to enhance the training data regarding motion dynamics and prompt understanding, equipped with a new reference frame attention mechanism to improve the consistency of auto-regressive video generation. Extensive experiments demonstrate that our VideoTetris achieves impressive qualitative and quantitative results in compositional T2V generation. Code is available at: https://github.com/YangLing0818/VideoTetris

1 Introduction

VideoTetris targets compositional text-to-video generation, where existing models struggle with complex multi-object prompts and progressive changes in object number. It combines spatial and temporal attention manipulation with data preprocessing and reference-frame consistency, and reports stronger compositional generation results.

1 Introduction

  • Existing models struggle to follow complex multi-object relationships and dynamic object-count changes in long videos.The introduction gives the example of a man on the left walking a dog on the right and notes that prior multi-prompt methods are typically limited to simple single-object scene changes.Source: §1 · p. 1
  • Spatio-Temporal Compositional Diffusion manipulates denoising-network cross-attention spatially and temporally to follow complex or progressive instructions.The paper defines compositional video generation to include both integrating objects with varied attributes and relationships, and long-video prompts that progressively change object positions, quantities, and presence.Source: §1 · p. 1
  • Enhanced Video Data Preprocessing augments training data for motion dynamics and prompt semantics, while Reference Frame Attention regularizes consistency across frames.Reference Frame Attention aligns reference and noise information in a coherent latent representation and accepts arbitrary image inputs.Source: §1 · pp. 1, 3
  • VideoTetris reports superior compositional video generation, with accurate positions and attributes, consistent scene transitions, and high motion dynamics in long videos.The introduction contrasts it with systems that misplace objects, merge attributes, abruptly introduce or transform characters, or omit new characters.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

Prior work adapts image diffusion models for video and explores longer or spatially guided generation, but the cited approaches face quality, consistency, or spatial-guidance challenges. VideoTetris instead uses cross-attention maps to model object positions and support natural integration during denoising.

Text-to-Video Diffusion Models

  • Video generation work progressed from adapting 2D image diffusion architectures to video, to adding temporal attention and training video-image transformers jointly.The discussion names LVDM and ModelScope, AnimateDiff, and more recent transformer-based methods, respectively.Source: §2 · p. 3

Long Video Generation

  • Existing long-video approaches address frame limits through varied strategies, but masked-diffusion methods can suffer inconsistencies and quality degradation, while StreamingT2V outputs can show artifacts.The cited discussion attributes StreamingT2V’s artifacts to low-quality training data.Source: §2 · p. 3

VideoTetris

  • VideoTetris combines a Spatio-Temporal Compositional module with Reference Frame Attention; a ControlNet-like branch can be adopted for autoregressive long-video generation.The figure caption identifies the first module with compositional generation and Reference Frame Attention with consistency regularization.Source: Figure 2 · p. 4

Compositional Video Generation

  • Cross-attention maps explicitly model object positions, allowing objects to integrate and blend during denoising.The paper contrasts this compositional-region approach with layout-based methods that provide rudimentary guidance and struggle with overlapping objects.Source: §2 · pp. 3–4

3 Method

VideoTetris combines training-free spatio-temporal attention composition with an autoregressive long-video pipeline trained on motion-filtered, recaptioned data and reference-frame consistency regularization.

3.1 Spatio-Temporal Compositional Diffusion

  • Prompts are decomposed by frame and then into spatial sub-objects with region masks, providing the information used to compose each part of the video.An optional LLM-based decomposer can generate frame-wise prompts and spatial sub-prompts with region masks.Source: §3.1 · p. 5, Figure 3 · p. 5
  • Sub-object and original-prompt cross-attention are blended to balance global information with individual characteristics, then concatenated across frames.The approach is described as directly applicable to pretrained or autoregressive video models in a training-free manner.Source: §3.1 · pp. 5–6

3.2 Enhanced Video Data Preprocessing

  • The preprocessing pipeline filters Panda-70M videos by RAFT average optical-flow scores and recaptions selected samples to strengthen motion dynamics and prompt semantics.The filter excludes videos with optical-flow scores outside a specified range; recaptioning uses three multimodal LLMs followed by a local LLM.Source: §3.2 · p. 6

3.3 Consistency Regularization with Reference Frame Attention

  • Reference Frame Attention encodes reference frames in the T2V latent space and adds cross-attention between current and reference objects; new objects’ references are precomputed in relevant regions.The method is motivated by maintaining object-feature consistency across frames as objects are added or removed.Source: §3.3 · pp. 6–7

4 Experiments

Experiments evaluate VideoTetris on compositional prompts and progressive compositional long-video prompts, using quantitative metrics and qualitative comparisons. Results report strong compositional scores and more consistent, dynamic long-video generation, alongside ablations of preprocessing and Reference Frame Attention.

4.1 Experimental Setups

  • VideoTetris is evaluated with 16-frame VideoCrafter2 generations for compositional prompts and a StreamingT2V-like backbone trained on preprocessed Panda-70M for long videos.For each scenario, 100 prompts or prompt lists were generated, six videos per prompt were sampled, and one was randomly selected for comparison.Source: §4.1 · p. 7

4.2 Metrics

  • VBLIP-VQA and VUnidet average BLIP-VQA and Unidet scores across frames; CLIP-SIM measures CLIP similarity between adjacent frames.The first two metrics extend T2I-CompBench to video to assess compositional generation.Source: §4.2 · p. 8

4.3 Video Generation with Compositional Prompts

  • For compositional prompts, VideoTetris maintains distinct object characteristics and aligns objects with requested positions, unlike baselines that merge features or misplace characters.The qualitative comparisons also include prompts with more than two objects.Source: §4.3 · p. 9

4.3 Video Generation with Compositional Prompts; 4.4 Long Video Generation for Progressive Compositional Prompts

  • VideoTetris achieves the best VBLIP-VQA and VUnidet scores across all models for both compositional-prompt and progressive long-video generation.FreeNoise achieves a better CLIP-SIM score in long-video generation, but the paper reports that this empirically damages natural story transitions.Source: §4.3 · p. 9, Table 1 · p. 9, §4.4 · p. 10

4.4 Long Video Generation for Progressive Compositional Prompts

  • For progressive long-video prompts, VideoTetris adds characters naturally and maintains stable changes, while producing more dynamic motion than FreeNoise.FreeNoise’s new character appears abruptly and identities switch; StreamingT2V produces videos where the knight disappears and a merged character appears.Source: Figure 6 · p. 9, §4.4 · p. 10

4.5 Ablation Study

  • Enhanced Video Data Preprocessing better captures “early morning sunlight”; MAWE, CLIP, AE, and CLIP-SIM are also reported across StreamingT2V test prompts.The ablation compares auto-regressive generation with original StreamingT2V using its original prompts and comparison methods.Source: §4.5 · p. 10, Figure 7 · p. 10, Table 4 · p. 11
  • Reference Frame Attention produces more consistent outputs with fewer color artifacts and more uniform overall color.The authors attribute this benefit to aligning reference and noise information semantically in latent space.Source: Figure 7 · p. 10, §4.5 · p. 11, Table 4 · p. 11

5 Conclusion and Discussion

VideoTetris combines spatio-temporal compositional diffusion with data preprocessing and Reference Frame Attention to support high-quality compositional video generation. The authors note limits in long-video cases and identify more efficient methods and broader input conditions as future work.

5 Conclusion and Discussion

  • VideoTetris combines spatial and temporal cross-attention composition with preprocessing and Reference Frame Attention to support high-quality compositional video generation.The authors report that experiments confirmed the framework's superiority in extending video diffusion models' generative capabilities.Source: §5 · p. 11
  • Long-video generation still encounters bad cases because of current long-video model limitations; ControlNet adds high computation cost and excessive transition frames.Those transitions make object consistency and continuous position control challenging.Source: §5 · p. 11
  • Future work will explore more efficient long-video training than ControlNet, varied input conditions, stronger pretrained backbones, and larger high-quality video datasets.Source: §5 · p. 11
  • The authors flag potential misuse of video diffusion, particularly impersonation and deepfakes targeting public figures.They state that their algorithm is intended to enhance image-generation quality and does not support such malicious uses.Source: §5 · p. 11

A.1 LLM-based Automatic Spatio-Temporal Decomposer

The decomposer uses an LLM's in-context learning to derive spatial regions and temporal information from prompts, avoiding manual decomposition. The authors report better suggestions with chain-of-thought reasoning.

A.1 LLM-based Automatic Spatio-Temporal Decomposer

  • An LLM uses in-context learning to generate spatial regions and temporal information, eliminating manual spatio-temporal decomposition for each prompt.Prompt templates combine task instructions, in-context demonstrations, and the user's test prompt.Source: §A.1 · p. 15
  • The authors report that chain-of-thought prompting empirically produced more accurate suggestions than prompting without explicit reasoning.Source: §A.1 · p. 15
  • The appendix provides full prompt templates for temporal decomposition, recaptioning, and spatial decomposition.Source: Table 5 · p. 15, Table 6 · p. 16, Table 7 · p. 17

A.2 Dataset Prompt Recaptioning

The recaptioning pipeline combines captions from multiple video-captioning models with the original dataset caption, then consolidates them into a training caption.

A.2 Dataset Prompt Recaptioning

  • Three caption models—Video-LLaMA, Video-ChatGPT, and Video-llava—generate 40–50-word captions for each filtered video.Source: §A.2 · p. 17
  • The generated captions are appended to the original Panda-70M prompt caption and passed to a local LLM for consolidation.The LLM extracts common elements and adds details.Source: §A.2 · p. 17
  • The resulting unified caption is around 40–50 words and is used to train on each filtered video.Source: §A.2 · p. 17

A.3 Model Hyperparameters and Implementation Details

The implementation uses classifier-free guidance prompt dropout and reports the training batch size, learning rate, hardware, and total steps.

A.3 Model Hyperparameters and Implementation Details

  • Five percent of text prompts are randomly dropped for classifier-free guidance training.Source: §A.3 · p. 17
  • Training used batch size 2 and learning rate 1e-5.Table 8 is identified as the hyperparameter table for VideoTetris.Source: §A.3 · p. 17, Table 8 · p. 18
  • The model was trained on four A800 GPUs for 16k steps.Source: §A.3 · p. 17

Related papers

Loading 2406.04277v2…