TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation

• Prior action or spatial prompts generally describe compositions invariant over time, potentially leaving human-noticeable flaws uncaptured by current benchmarks.Source: §1 · p. 2
• TC-Bench targets attribute transitions, object relations, and background shifts with prompts specifying initial and final scene states.Source: §1 · p. 2
• Most video generation models accomplish less than ∼20% of TC-Bench test cases.Source: §1 · p. 2
• TCR and TC-Score use frame-level compositionality assertions checked throughout videos to measure transition completion and text-video alignment.Source: §1 · p. 2
• The proposed metrics correlate better with human judgments than existing metrics for temporal compositionality.Source: §1 · p. 2
• Analysis of nine baselines identifies prompt understanding and temporal consistency as key weaknesses.Source: §1 · p. 3

2.1 Conditional Video Generation

• VideoCrafter2 uses a single latent diffusion model, while Modelscope, LaVie, Show-1, SEINE, and DynamiCrafter use distinct multi-component or image-injection designs.Most open-source models generate only 2–3 seconds per sampling sequence, motivating benchmark transitions that can occur within a few seconds.Source: §2.1 · p. 3

2.2 Video Generation Benchmarks

• Existing video evaluation uses measures such as FVD and CLIP similarity, while newer suites assess multiple dimensions with specialized evaluators.Source: §2.2 · p. 3
• Existing benchmarks under-address attribute and object-relation transitions, and their metrics have marginal correlations with human ratings.Source: §2.2 · p. 3
• Prior compositional video methods use entities, action graphs, spatial-temporal conditions, or LLM-generated layouts, yet still struggle with compositional transitions.Source: §2.2 · p. 3

3 TC-Bench

• TC-Bench consists of prompts following a defined scene-graph space and corresponding ground-truth videos.Source: §3 · p. 3

3.1 Temporal Compositionality Prompts

• The benchmark covers attribute transitions, changes in object relations, and background shifts, represented as scene-state changes over time.The examples include an attribute changing on one object, an object interacting with different objects, and a background attribute changing while another object distracts.Source: §3.1 · p. 4
• The prompt taxonomy focuses on single transition events that could occur within roughly one to ten seconds.Source: §3.1 · p. 4

3.2 Data Collection

• Human-in-the-loop prompt creation uses GPT-4 to expand crafted captions and definitions, with manual selection yielding 150 T2V prompts that specify scene start and end states.Source: §3.2 · p. 4
• Annotators match prompts to YouTube videos, revising text or replacing unsuitable prompts to create 120 I2V prompt-video pairs.The ground-truth videos provide image inputs for I2V models and references for metric computation.Source: §3.2 · pp. 4–5

4.1 Assertion-Based Evaluation

• GPT-4 generates indexed assertion sets covering transition completion, transition-object consistency, and other objects.Each assertion is paired with up to five frame indices for examination.Source: §4.1 · p. 5
• A transition counts as complete only when the VLM verifies every transition-completion and consistency assertion against selected frames.When multiple frames are selected, they are concatenated horizontally as one image for the VLM.Source: §4.1 · p. 5
• TCR reports the percentage of videos aligned with prompts, while averaged TC-Score measures assertion pass rates across all concepts.TC-Score is defined for individual text-video pairs and ranges from 0 to 1.Source: §4.1 · p. 6

4.2 Consistency Evaluation for Image-to-Video Generation

• For I2V, TC-Score penalizes adversarial intermediate frames using frame consistency; among four candidates, CLIP latent features are more reliable.Frame similarity is measured against either the next frame or a ground-truth frame.Source: §4.2 · p. 6

5 Method

• An LLM derives initial and final scene prompts; SDXL generates endpoint frames with shared noise and partially shared self-attention, then SEINE generates the transition.The generated frames guide video generation between the endpoints.Source: §5 · p. 6

6.1 Experiment Setup

• The setup evaluates six additional T2V models and two I2V models, alongside TCR, TC-Score, four alignment metrics, and human ratings.The I2V models receive the first and last ground-truth frames and generate intermediate frames; human ratings use a 5-point Likert scale.Source: §6.1 · pp. 6–7

6.2 Quantitative Results

• All direct T2V models complete transitions less than 10% of the time; SEINE and DynamiCrafter achieve much higher TCR but struggle with consistency when endpoints differ substantially.VideoCrafter2 achieves slightly better TCR and TC-Score than other direct T2V models.Source: Table 1 · p. 7, §6.2 · p. 7, Table 2 · p. 7
• Multi-stage T2V methods improve overall TCR, but explicit intermediate representations yield unbalanced results across transition types.Bounding boxes cannot represent attribute or background changes, while SDXL+SEINE underperforms on object relations because T2I models struggle to control object positions across two diffusion paths.Source: §6.2 · p. 7

6.3 Qualitative Results

• Direct T2V models often blend attributes into static patterns or show one dominant attribute; SDXL+SEINE can generate gradual color changes, while object-binding transitions remain difficult for both T2V and I2V models.VideoCrafter2 fails to depict gloves on snow or a robot arm picking objects up; DynamiCrafter produces more dynamics but changes the robot's shape across frames.Source: Figure 4 · p. 8, §6.3 · p. 8

6.4 Analysis

• Figure 5 tracks start- and end-attribute similarity over time and consecutive-frame similarity; T2V curves stay flat, while I2V models follow the attribute trend but have weaker frame consistency.Higher consistency scores do not help T2V models complete most transitions, so the authors argue consistency need only be computed for I2V models.Source: Figure 5 · p. 8, §6.4 · p. 9

6.5 Human Evaluation

• The proposed metrics correlate with human judgments more strongly than existing metrics for both transition completion and overall text-video alignment.Averaged text-frame CLIP similarity and advanced alignment models such as ViCLIP and UMTScore have low correlations with human ratings of temporal compositionality.Source: Table 3 · p. 9, §6.5 · p. 9

7 Conclusion

• The benchmark and experiments expose temporal-compositionality weaknesses in existing T2V and I2V models, motivating work on video mining, more efficient alignment evaluation, and better T2V models.The authors also report that simple transitions occurring within several seconds remain challenging for existing T2V methods.Source: §7 · p. 9

A Implementation Details

• The implementation generates 750 videos at 8 fps and 16 frames per video, except Show-1 videos, which contain 29 frames.GPT-4-turbo generates frame indices and assertions, and GPT-4V evaluates TCR and TC-Score.Source: A Implementation Details · p. 14

B TC-Bench Dataset Construction

• Prompt construction covers attribute changes, object-binding relation changes, and background shifts, with concise, realistic transitions intended to occur within seconds.Examples include changing an object's color, transferring an object between bindings, and changing a background while a foreground object remains relatively static.Source: B TC-Bench Dataset Construction · pp. 14–15

B.1 Ground Truth Video Collection and Statistics

• About 80% of collected videos last 6 seconds or less, and 95% last less than 15 seconds; object-binding-change videos have more motion than attribute-transition and background-shift videos.Figure 6 shows video-length and moving-object dynamics distributions; motion is estimated from optical-flow magnitude after static background areas are thresholded out.Source: Figure 6 · p. 15, §B.1 · p. 15
• Domain-aware student annotators search YouTube for videos matching prompts, revise prompts when a video only partly aligns, and discard prompts without suitable matches.They record each video's YouTube ID, start time, and end time; selected clips range from several seconds to less than 20 seconds.Source: B TC-Bench Dataset Construction · p. 15, §B.1 · p. 15

C.1 Assertion Generation

• GPT-4 generates prompt-specific assertions across transition completion, transition consistency, and other objects, checking start, end, intermediate, and sampled frames.The authors provide three in-context exemplars for each prompt, including one manually written exemplar per type.Source: §C.1 · p. 16

C.2 Frame Consistency for I2V Models

• For I2V frame-consistency evaluation, four methods extract optical flow, trajectories, or latent features after 16 equally spaced ground-truth frames are sampled.EPE compares optical-flow vectors; ATE compares estimated point positions with ground-truth positions; CLIP measures adjacent-frame or generated-to-reference similarity.Source: §C.2 · p. 16
• Frame consistency is evaluated only for I2V models because transition completion could not be disentangled from consistency for T2V videos.The human-rating filter retained 128 I2V videos, while over 97% of T2V videos averaged below 3.Source: §C.2 · pp. 16–17
• Consecutive Frame CLIP Similarity achieves the highest correlation with processed human ratings among the four I2V frame-consistency metrics.The authors conjecture that EPE and ATE are too strict because natural transitions can differ from the ground-truth video’s dynamics.Source: §C.2 · p. 17, Table 4 · p. 17

D Human Evaluation

• The study collected 2,451 ratings for 817 generated videos after discarding videos with divisive ratings.About 900 videos were randomly sampled, with three annotators assigned to each; the released ratings accompany the benchmark data and metrics.Source: D Human Evaluation · p. 17
• Annotators answered two questions: one about transition completion and another about overall text-video alignment.The authors selected five models and generated five videos per prompt per model to unify the prompt space for T2V and I2V annotation.Source: D Human Evaluation · p. 17
• Eight native English-speaking Amazon Mechanical Turk workers completed the annotation tasks using instructions and five interface examples.Each task paid $0.35 and took about 40 seconds; the reported hourly rate was $31.5.Source: D Human Evaluation · p. 17, Figure 7 · p. 18

E Additional Results

• I2V models achieve a much higher human-rated completion rate than T2V models, which achieve around 1%.The human threshold measurement is designed to reflect overall model ranking and is not statistically equivalent or directly comparable to TCR.Source: E Additional Results · p. 18
• Direct T2V models mix concepts across frames or produce trivial motions, whereas I2V models show more significant transitions but suffer consistency and coherence issues.The qualitative comparisons also show SDXL+SEINE with more significant transitions than direct or multi-stage T2V models, yet difficulties generating dynamics for relation changes.Source: E Additional Results · p. 18, Figure 8 · p. 19
• Existing metrics fail to capture temporal compositionality and assign higher scores to static scenes without compositional changes.Source: E Additional Results · p. 18

F Limitation and Potential Social Impacts

• TCR and TC-Score correlate better with human judgments, but the metrics remain imperfect and discrepant from human ratings.The authors cite VLMs’ difficulty with multi-image understanding and the reliance on image-based assertions because strong video foundation models are lacking.Source: F Limitation and Potential Social… · p. 19
• Future work could develop end-to-end video-based metrics when stronger video VLMs become available.Source: F Limitation and Potential Social… · p. 19
• The authors caution that generated videos may involve abuse, hallucinations, or biases, and advise care when using TC-Bench materials.Source: F Limitation and Potential Social… · p. 19