PaperlayerView exact revision on arXiv

Source-linked AI summary

T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation

Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, Xihui Liu

arXiv:2407.14505v2cs.CV

TL;DR

Compositional text-to-video generation remains challenging, and common metrics do not fully capture it. T2V-CompBench evaluates seven compositional categories with tailored metrics, and its evaluation finds that current models struggle, especially with dynamic attribute binding.

  • Problem

    Generating videos that accurately depict multiple objects, attributes, and motions from fine-grained prompts remains challenging, while common metrics cannot fully reflect compositionality.

    Source: §1 · pp. 1–2

  • Method

    T2V-CompBench evaluates seven compositional categories across 1400 prompts using tailored metrics whose effectiveness is verified through correlation with human evaluations.

    Source: §1 · p. 2

  • Results

    Compositional text-to-video generation remains highly challenging for current models, with dynamic attribute binding the most challenging category.

    Source: §5.5 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The findings point to compositionality as an area for future improvement in text-to-video models.

    Source: §6 · p. 9

    The work lacks a unified evaluation metric across categories and evaluates videos within a 2-to-5-second scope.

    Source: G. Limitations and Future Work · p. 22

Abstract

from arXiv · show

Text-to-video (T2V) generative models have advanced significantly, yet their ability to compose different objects, attributes, actions, and motions into a video remains unexplored. Previous text-to-video benchmarks also neglect this important ability for evaluation. In this work, we conduct the first systematic study on compositional text-to-video generation. We propose T2V-CompBench, the first benchmark tailored for compositional text-to-video generation. T2V-CompBench encompasses diverse aspects of compositionality, including consistent attribute binding, dynamic attribute binding, spatial relationships, motion binding, action binding, object interactions, and generative numeracy. We further carefully design evaluation metrics of multimodal large language model (MLLM)-based, detection-based, and tracking-based metrics, which can better reflect the compositional text-to-video generation quality of seven proposed categories with 1400 text prompts. The effectiveness of the proposed metrics is verified by correlation with human evaluations. We also benchmark various text-to-video generative models and conduct in-depth analysis across different models and various compositional categories. We find that compositional text-to-video generation is highly challenging for current models, and we hope our attempt could shed light on future research in this direction.

1. Introduction

The introduction frames compositional text-to-video generation as a difficult, understudied problem and presents T2V-CompBench as a benchmark with category-specific metrics and systematic model evaluation.

1. Introduction

  • Existing T2V work often uses simple prompts, while benchmarks [21] [39] [40] emphasize video quality, motion, or alignment with single-object prompts, leaving compositional generation understudied.Generating videos that accurately depict multiple objects, attributes, and motions from fine-grained prompts remains challenging.Source: §1 · pp. 1–2
  • T2V-CompBench introduces the first compositional T2V benchmark, with seven categories and 200 prompts per category designed to include temporal dynamics and at least one active verb.The categories cover consistent and dynamic attribute binding, spatial relationships, motion binding, action binding, object interactions, and generative numeracy.Source: §1 · p. 2
  • Because common metrics cannot fully reflect compositionality, the benchmark uses MLLM-based, detection-based, and tracking-based metrics tailored to categories and validates them against human evaluations.MLLM-based metrics cover attribute binding, action binding, and object interactions; detection-based metrics cover spatial relationships and numeracy; tracking-based metrics cover motion binding.Source: §1 · p. 2
  • The authors evaluate various T2V models across compositional categories and provide systematic analysis intended to inspire future research.Source: §1 · p. 2

2. Related Work

The related work surveys text-to-video model families and prior image- and video-domain compositionality benchmarks, identifying the need to evaluate spatio-temporal composition in video.

2.1. Text-to-video Generation.

  • Text-to-video diffusion models are broadly categorized as diffusion U-Net-based or Diffusion Transformer-based, and the paper evaluates open-source and commercial models across these approaches.The evaluated set spans multiple architectures to examine model capabilities and limitations.Source: §2.1 · p. 2

2.2. Compositional Text-to-image Generation.

  • T2I-CompBench evaluates image compositionality through attribute binding, relationships, and complex compositions, but video benchmarking must also account for spatio-temporal dynamics.Prior compositionality evaluations are tailored exclusively to the image domain.Source: §2.2 · pp. 2–3

2.3. Benchmarks for Text-to-video Generation.

  • Existing quality and text-video alignment metrics are unsuitable for complex compositional prompts, while prior benchmarks largely lack dynamic multi-object prompts; T2V-CompBench introduces the first compositional T2V benchmark with tailored metrics validated through human-correlation studies.ChronoMagic-Bench [80] focuses on natural metamorphic timelapse videos and excludes unnatural attribute-change videos.Source: §2.3 · p. 3

3. Benchmark Construction

T2V-CompBench defines seven spatial and temporal compositional categories and builds a 1,400-prompt suite using real-user vocabulary, GPT-4 generation, and human filtering.

3.1. Problem Definition and Categorization

  • The benchmark separates spatial compositionality—consistent attributes, spatial relationships, and numeracy—from temporal compositionality—dynamic attributes, motion, actions, and interactions.Its categories test whether video frames align with prompt descriptions spatially and whether video dynamics follow them temporally.Source: §3.1 · p. 3

3.2. Prompt Categories

  • Prompts test consistent and changing object attributes, six spatial relationships, and four movement directions, including contrastive left/right pairs.Consistent binding covers color, shape, texture, and human-related attributes; 20% of its prompts use challenging, uncommon combinations. Dynamic binding covers color/light, shape/size, texture, and combined changes, with 80% common and 20% less common or artificial changes.Source: §3.2 · pp. 3–4
  • Action binding pairs two objects with activities, object interactions cover physical and social dynamics, and generative numeracy varies object counts; each category has 200 prompts.Action prompts are 80% common and 20% uncommon; numeracy prompts vary quantities for one or two object types.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Prompt Suite Generation

  • From 1.67 million VidProM prompts, the authors selected 260 object nouns, 200 active verbs, and 80 attributes, favoring frequent, labelable objects and dynamic actions.The vocabulary spans object metaclasses such as people, artifacts, animals, plants, and food; static verbs and poorly bounded “stuff” categories were avoided.Source: §3.3 · p. 4
  • GPT-4 generated 200 prompts per category from the selected vocabulary and category requirements; every prompt includes an active verb, and humans filtered improper prompts.The authors chose generated free-form prompts over existing captions or fixed templates to better accommodate the benchmark requirements and prompt diversity.Source: §3.3 · pp. 4–5

3.4. Prompt Suite Statistics

  • T2V-CompBench prompts average approximately 3.6 nouns and 1.4 verbs per prompt; all contain verbs, and average 10.4 words, ranging from 3 to 23, to avoid distracting T2V models.The authors use an LLM to generate free-form prompts, producing auxiliary words beyond those provided for selection and a rich variety of nouns, verbs, and adjectives.Source: §3.4 · p. 5

4. Evaluation Metrics

Section 4 adapts evaluation to video with MLLM-, detection-, and tracking-based metrics, using sampling choices suited to each evaluator. The methods assess attribute and action binding, object interactions, spatial relationships, numeracy, and motion.

4. Evaluation Metrics

  • The benchmark uses 6 evenly sampled frames for MLLM evaluation, 16 for detection, and 8 FPS sampling for tracking because video dynamics and frame counts hinder direct use of compositional T2I metrics.Figure 1(b) illustrates the three metric types.Source: §4 · p. 5

4.1. MLLM-based Evaluation Metrics

  • Grid-LLaVA uses Image Grid to feed 6 uniformly sampled frames to LLaVA, then applies chain-of-thought and disentangled questions to reduce hallucinations; Image Grid performed better than PLLaVA in these categories [23] [35] [70] [74].The metric evaluates consistent attribute binding, action binding, and object interactions.Source: §4.1 · p. 5
  • For consistent binding, action binding, and interactions, GPT-4 parses phrases or objects and actions for video-LLM grading, while interaction grading also considers dynamics, development, and outcome [47].Consistent-binding phrase grades are combined and averaged; action binding checks object presence and object-action alignment.Source: §4.1 · p. 5
  • D-LLaVA uses GPT-4 to extract initial and final states, then scores each frame with LLaVA to encourage initial-state alignment first, final-state alignment last, and intermediate states between them [35] [47].This frame-by-frame Image LLM metric addresses dynamic attribute binding, where current video LLMs perform poorly.Source: §4.1 · pp. 5–6

4.2. Detection-based Evaluation Metrics

  • GroundingDINO detections support rule-based scores for 2D and 3D spatial relations and numeracy; 3D relations additionally use Segment Anything masks and Depth Anything depth maps [25] [37] [75].Spatial scores average per-frame results across the video; numeracy scores whether detected object counts match prompt quantities.Source: §4.2 · p. 6

4.3. Tracking-based Evaluation Metrics

  • DOT estimates an object's true motion by subtracting tracked background motion from foreground motion, then scores whether that relative direction matches the prompt [46] [55].GroundingSAM supplies foreground and background masks, and DOT tracks points in both regions.Source: §4.3 · p. 6

5. Experiments

The experiments evaluate 23 text-to-video models, compare automatic metrics with human judgments, and find that compositional generation remains challenging across categories. They identify category-specific metric strengths and recurring model failures.

5.1. Evaluated Text-to-video Models

  • The benchmark evaluates 23 text-to-video models, including 17 open-source and 6 commercial models.MagicTime is specifically tested on dynamic attribute binding because it is designed to generate metamorphic time-lapse videos.Source: §5.1 · pp. 6–7

5.2. Evaluation Metrics

  • The study compares conventional alignment metrics with detection-, tracking-, and MLLM-based metrics designed for particular compositional categories.Proposed metrics include D-LLaVA for dynamic attribute binding, G-Dino for spatial relationships and numeracy, DOT for motion binding, and Grid-LLaVA and PLLaVA across categories; LLaVA is tested on a frame basis except for dynamic attributes and motion. Comparisons also include CLIPScore, BLIP-based metrics, ViCLIP, VPEval-S, and M-GDino.Source: §5.2 · p. 7

5.3. Human Evaluation Correlation Analysis

  • Human-correlation testing identifies different best metrics by category, with Grid-LLaVA leading three categories and specialized metrics leading elsewhere.In evaluations of 651 videos scored by three annotators, Grid-LLaVA leads consistent attribute binding, action binding, and object interactions; D-LLaVA leads dynamic attribute binding, G-Dino spatial relationships and generative numeracy, and DOT motion binding. CLIP, ViCLIP, B-CLIP, and B-BLEU show relatively low human correlations; Grid-LLaVA’s multiple-frame analysis helps it capture temporal changes, while LLaVA performs reasonably across the categories it evaluates.Source: §5.3 · pp. 7–8, Table 1 · p. 8

5.5. Qualitative Evaluation

  • Dynamic attribute binding is the hardest category; spatial relationships, motion binding, and numeracy are next-hardest, while other categories are easier but still error-prone.Models often miss attribute transitions, confuse left and right, produce incorrect motion directions or insufficient movement, and struggle to generate more than three objects. Interaction videos may remain static; action binding can assign the wrong actions or omit an object, and consistent attributes may be misbound or overlooked.Source: §5.5 · pp. 8–9

5.4. Quantitative Evaluation

  • PixVerse-V3 excels across all models; T2V-Turbo-V2 leads diffusion U-Net models, while CogVideoX-5B leads DiT models.VideoTetris, Vico, and T2V-Turbo-V2 improve in most categories over their VideoCrafter2 base; MagicTime improves dynamic attribute binding over AnimateDiff, and LVD improves over ModelScope. Open-Sora 1.2 excels in numeracy among open-source models, while VideoTetris is strong in numeracy and interaction among diffusion U-Net models.Source: §5.4 · p. 8, Table 2 · p. 9

6. Conclusion

The paper introduces T2V-CompBench for systematic evaluation of compositional text-to-video generation, with human-validated metrics and analysis of current models. It concludes that the task remains highly challenging.

6. Conclusion

  • T2V-CompBench provides a comprehensive benchmark with 1,400 prompts spanning seven categories of compositional text-to-video generation.The work presents this as the first systematic study of compositionality in text-to-video generation.Source: §6 · p. 9
  • Metrics designed for all seven categories are validated through correlations with human evaluations.Source: §6 · p. 9
  • Benchmarks across text-to-video models with different architectures find compositional generation highly challenging for current models.The authors hope the work will inspire future efforts to improve text-to-video model compositionality.Source: §6 · p. 9

Appendices · Appendix Contents · A. Prompt Suite

T2V-CompBench’s prompt suite is built from words grouped by WordNet meta classes, selected for frequency and compositional relevance, and used to generate 1,400 prompts. Its stability analysis finds that several metrics stabilize with fewer videos, while the benchmark uses 200 videos per category.

A.1. Vocabulary Construction

  • Using WordNet to group VidProM [67] prompt words, the authors select 260 object nouns, 66% of which occur over 900 times.They focus on the 2,000 most frequent nouns, then add words from the same meta classes to broaden noun coverage.Source: §A.1 · p. 14
  • The vocabulary includes 200 active verbs, with 60% occurring over 370 times in the real-user prompts.The top 1,200 verbs exceed that frequency threshold.Source: §A.1 · p. 14
  • Attributes are drawn from adjective definitions and noun meta classes, yielding 14 color, 26 shape, and 26 texture attributes.Human-related attributes are drawn from classes such as body covering and clothing.Source: §A.1 · p. 14

A.2. Prompt Generation with LLM

  • For LLM prompt generation, the authors provide high-frequency nouns, verbs, attributes, or other word lists alongside prompt templates.The templates are listed in Appendix H, Tables 5–11.Source: §A.2 · p. 15

A.3. Prompt Suite Statistics

  • The 1,400-prompt suite’s nouns cluster in artifact, object, and person classes, while verbs center on travel, move, and change.Verbs associated with express and think occur nearly negligibly.Source: §A.3 · p. 15, Figure 10 · p. 15

A.4. Stability of 1400 Prompts and Videos

  • G-Dino for spatial relationships and numeracy stabilizes at about 125 videos, while Grid-LLaVA for three categories stabilizes around 150.The analysis uses one generated video per prompt and reports that stability can vary with the T2V model.Source: §A.4 · p. 15, Figure 13 · p. 16, Figure 15 · p. 16
  • The benchmark limits each category to 200 videos, yielding 1,400 prompts and videos to constrain inference time and computational costs.Current T2V models typically take minutes to generate a video.Source: §A.4 · p. 15

B. Implementation Details · C. Using MLLMs as Evaluation Metrics · C.1. How to query MLLMs

The evaluation follows the T2V models’ official default implementations and uses structured MLLM queries to reduce hallucinations and assess compositional aspects separately. The MLLM protocol begins with an independent visual description, then uses tailored questions and grading options.

B. Implementation Details

C.1. How to query MLLMs

  • MLLM evaluation can hallucinate by misidentifying visual content or selecting unmatched grades or scores.Source: §C.1 · p. 16
  • The protocol uses chain-of-thought [70]: it first asks the MLLM to describe the image without revealing the evaluation question, avoiding influence from subsequent questions.Source: §C.1 · pp. 16–17
  • To balance sufficient grading options against overload, the evaluation disentangles aspects into parallel or sequential queries that address each aspect separately.Source: §C.1 · p. 17
  • For consistent attribute binding, queries check each object’s attribute; for dynamic attribute binding, they ask whether the image shows the initial or final state.Source: §C.1 · p. 17
  • For action binding, the protocol first checks object presence, then whether present objects’ actions match the prompt; for interactions, it checks object presence before interaction quality.Source: §C.1 · p. 17

C.2. How to obtain reliable and reproducible results … D.6. Object Interactions

The section recommends averaging three MLLM evaluations per video for more reliable scores and model rankings. Its category analyses show distinct challenges, including changing attributes, spatial layout, motion, uncommon actions, and physical interactions.

C.2. How to obtain reliable and reproducible results

  • Averaging three MLLM scores per video improves reliability and stability, while varying the evaluation seed can change model rankings, especially for close scores.Across consistent attribute binding, action binding, and object interactions, larger samples generally produced higher and more stable correlations with human evaluations; the authors chose three queries for practicality and reliability.Source: §C.2 · pp. 17–18, Figure 19 · p. 18

D. Evaluation Results Analysis

  • The evaluation analysis extracts important subgroup scores and interim-stage data to examine category-specific sub-dimensions.These results are documented in Table 4, with explanations and visualizations provided for each category.Source: D. Evaluation Results Analysis · p. 18

D.1. Consistent Attribute Binding

  • Color binding was easiest, texture next, and shape hardest; T2V-Turbo-V2 accurately bound color to objects and showed noticeable movement.The paper illustrates these examples alongside ModelScope videos.Source: §D.1 · p. 18, Figure 20 · p. 18

D.2. Dynamic Attribute Binding

  • 0.0687 was PixVerse-V3’s highest dynamic-attribute score; only 31 of 200 videos showed relevant elements or transitions, and just 9 showed meaningful transitions.Models often produced static content reflecting the initial state, final state, or both simultaneously; PixVerse-V3 showed better prompt understanding and temporal dynamics.Source: §D.2 · p. 18, Table 4 · p. 19

D.4. Motion Binding

  • LVD achieved the highest motion-direction accuracy at 47%, while Gen-3 reached 42%.PixVerse-V3 and CogVideoX-5B also generated significant object motion, as measured by Motion Level.Source: Table 4 · p. 19, §D.4 · p. 20

D.3. Spatial Relationships

  • Vico and Mochi performed best among open-source models at generating multiple objects, while LVD led in spatial accuracy and score despite low coexistence.Coexist measures whether both objects appear; Acc measures correct relationships among videos containing both objects, and Acc.Score averages scores for correctly related objects. High Acc. and Acc.Score are both needed for accurate spatial relationships.Source: Table 4 · p. 19, §D.3 · pp. 19–20

D.5. Action Binding

  • Uncommon action prompts, including anthropomorphic animal actions, were more challenging than common prompts.Source: §D.5 · p. 20

D.6. Object Interactions

  • Physical interactions were harder to depict than social interactions because they require understanding physical laws.Examples contrast a generated video that failed to represent the described interaction process with one that captured its progression and outcome more accurately.Source: §D.6 · pp. 20–21, Figure 26 · p. 21

D.7. Generative Numeracy

Generative-numeracy scores tend to decline as prompted object quantities increase, while model performance varies across quantity groups and architectures.

D.7. Generative Numeracy

  • As the prompted object quantity increases, the average generative-numeracy score tends to decrease.Figure 22 illustrates the score pattern across quantities.Source: §D.7 · p. 21
  • Commercial models generally outperform open-source models; PixVerse-V3 scores highest across almost all quantity groups, while Open-Sora 1.2 leads among DiT-based models.Diffusion U-Net-based models show comparable numeracy results.Source: §D.7 · p. 21
  • Open-Sora 1.2 represents the correct object quantity despite producing a video that is not realistic in style.The passage compares generated videos from Open-Sora 1.2 and ModelScope [65].Source: §D.7 · p. 21, Figure 27 · p. 21

E. Human Evaluation · F. Societal Impacts

The human evaluation uses structured AMT ratings across seven categories to assess metric alignment with human preferences. The societal-impacts discussion flags misuse and bias risks and proposes future evaluation of unbiased composition.

E. Human Evaluation

  • AMT human evaluations across all seven categories assess how well T2V-CompBench metric scores correlate with human preferences.The authors prepare a separate evaluation interface for each category.Source: E. Human Evaluation · p. 21
  • Interfaces explain each category’s evaluation dimension, provide rating criteria and examples with rationales, and present video-text pairs for scores from 5 to 1.The consistent-attribute-binding interface is shown in Figure 28.Source: E. Human Evaluation · p. 21, Figure 28 · p. 22
  • Quality controls include reviewing a random 20% of each worker’s tasks, rejecting evident instruction failures, and blocking workers rejected more than five times.Rejected workers’ tasks are reassigned; the authors state these measures enhance evaluation reliability and accuracy.Source: E. Human Evaluation · p. 21
  • Three annotators rate each task, and their average score is assigned to the corresponding video-text pair.Source: E. Human Evaluation · p. 21
  • AMT workers must have at least 90% historical task acceptance; participants receive an estimated hourly wage of 9.60 USD, with 195 USD spent in total.Source: E. Human Evaluation · p. 21

F. Societal Impacts

  • Video-generation misuse may produce misleading or harmful content that worsens misinformation and deepfakes, while training-data biases may perpetuate stereotypes or exclude underrepresented groups.Source: F. Societal Impacts · p. 22
  • The authors argue T2V evaluation should consider societal implications alongside technical performance and plan to add an unbiased-composition dimension.Source: F. Societal Impacts · p. 22

G. Limitations and Future Work · H. Templates for Generating Prompts and Metadata Using GPT-4

The authors identify limitations in unified evaluation and long-video coverage, then document GPT-4 prompt and metadata templates for T2V-CompBench. The templates span seven prompt categories and metadata for six categories.

G. Limitations and Future Work

  • The benchmark lacks a unified metric across categories, and fixed-frame evaluation may underserve videos longer than five seconds; longer motion-binding videos may improve performance.The authors leave evaluation of long videos for future work and call for larger multimodal LLMs or video-understanding models as potential evaluation metrics.Source: G. Limitations and Future Work · p. 22

H. Templates for Generating Prompts and Metadata Using GPT-4

  • The appendix provides GPT-4 [47] instructions for generating T2V-CompBench prompts and evaluation metadata, which are available in the code repository.Source: H. Templates for Generating Prompts… · p. 22
  • Prompt-generation templates cover consistent and dynamic attribute binding, spatial relationships, motion binding, and action binding, with specified replaceable attributes, relationships, directions, or nouns.Source: Table 5 · p. 23, Table 6 · p. 23, Table 7 · p. 24
  • Additional prompt templates address object interactions and generative numeracy, including interaction-type substitutions and quantities from two through eight.The numeracy template restricts the object-noun list to 30.Source: Table 10 · p. 25, Table 11 · p. 25
  • Metadata-generation templates are provided for consistent and dynamic attribute binding, spatial relationships, motion binding, and action binding.Source: Table 12 · p. 26, Table 13 · p. 26, Table 14 · p. 27
  • A separate template generates evaluation metadata for generative numeracy prompts.Source: Table 17 · p. 29

Related papers

Loading 2407.14505v2…