Evaluation of Text-to-Video Generation Models: A Dynamics Perspective

1 Introduction

• The benchmark combines prompts categorized by dynamics grades with an overall dynamics score formed from scores at different temporal granularities.The prompts are collected from existing datasets, graded with GPT-4, and manually refined.Source: §1 · p. 2
• Dynamics-based quality integrates quality metrics with dynamics scores to reduce bias from the negative correlation between video dynamics and quality.The protocol also introduces a naturalness metric based on Gemini-1.5 Pro because naturalness decreases as dynamics increase.Source: §1 · pp. 1–2
• Existing datasets have biased dynamics distributions, so current models, including GEN-2, often generate slow-motion videos to obtain high quality scores.Source: §1 · p. 2
• Biased dynamics in training prompts limit T2V controllability, while statistical analyses indicate limited real-world simulation ability in existing methods.Source: §1 · p. 2
• DEVIL integrates dynamics metrics into T2V evaluation to complement existing measures of generation quality, text-video alignment, and content continuity.Source: §1 · pp. 1–2
• The protocol benchmarks dynamics range, controllability, and dynamics-based quality using prompts graded across multiple dynamics levels.Source: §1 · pp. 1–2
• DEVIL’s evaluation of existing T2V models is intended to support more sophisticated generation methods.Source: §1 · p. 3

2.1 Text-to-Video Generation Model

• Diffusion-based T2V research has progressed from pixel-space and cascaded models to latent diffusion, higher-quality generation, and longer-duration video methods.Source: §2.1 · p. 3

2.2 Evaluation Protocol

• Despite this progress, existing protocols largely ignore temporal dynamics, so DEVIL adds dynamics to improve the completeness of T2V evaluation.Source: §2.1 · p. 3, §2.2 · p. 3
• Evaluation protocols evolved from class-label and category-based tests toward larger prompt sets and multi-dimensional suites covering alignment, quality, and diversity.Source: §2.2 · p. 3

3.1 Overview

• DEVIL evaluates T2V models with dynamics range, dynamics controllability, and dynamics-based quality across a shared prompt benchmark.The metrics use per-video dynamics scores and assess variation coverage, prompt-response ranking consistency, and quality across dynamics levels.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.2 Dynamics Metrics

• Dynamics controllability measures whether generated-video dynamics preserve the ranking of prompt dynamics grades, while dynamics range excludes the top and bottom 1% scores.Range therefore captures a realistic spread, and controllability uses ranking consistency rather than exact prompt-video correspondence.Source: §3.2 · pp. 4–5
• Naturalness, Motion Smoothness, Subject Consistency, and Background Consistency each show Pearson correlations above 50% with overall dynamics, while quality generally decreases as dynamics increase.The protocol therefore evaluates quality across dynamics intervals rather than rewarding low-dynamics videos alone.Source: §3.2 · p. 5

3.3 Text Prompt Benchmark

• Approximately 800 prompts are sampled evenly across five dynamics grades, from static to very high dynamics, after GPT-4 labeling and human refinement.The benchmark is more balanced across dynamics grades than existing benchmarks, which are skewed toward lower-dynamics content.Source: §3.3 · p. 6

3.4 Dynamics Scores for Generated Videos

• Dynamics scores combine inter-frame, inter-segment, and video-level measurements covering motion, structural change, aperiodicity, temporal entropy, and semantic diversity.Inter-frame scores include optical flow strength, structural dynamics, and perceptual dynamics; inter-segment scores include patch-level and global aperiodicity.Source: §3.4 · pp. 6–7

3.5 Overall Dynamics Score

• The overall dynamics score averages human-aligned scores from the three temporal granularities, improving consistency with human perception.Linear regression models are fitted at each granularity using human ratings before the aligned scores are averaged.Source: §3.5 · p. 8

4.1 Human Alignment Assessment

• Strong alignment with human evaluation validates the dynamics and naturalness metrics, with improved Sf, Ss, and Sv metrics further enhancing alignment.Source: §4.1 · p. 9

4.2 Dynamic-Quality Bi-variate Analysis

• Pearson correlations above 50% for naturalness, motion smoothness, subject consistency, and background consistency show dynamics substantially affect quality metrics.Source: §4.2 · p. 9
• Quality metrics decline as dynamics increase, while high-ranking models concentrate generated videos in low-dynamic regions, allowing low-dynamics outputs to improve benchmark quality scores.Source: §4.2 · p. 9

4.3 Evaluation of Dynamics Metrics

• GEN-2 and Pika achieve high dynamics alignment but low dynamics range, whereas FreeNoise-Lavie [30] and StreamingT2V [18] show the reverse pattern.Source: Table 5 · p. 9, §4.3 · p. 10
• The original quality metric aligns closely with low-dynamics quality, while T2V models typically receive lower high-dynamics quality scores.Source: §4.3 · p. 10, Figure 5 · p. 10

4.4 Insights from Video Dynamics Analysis

• WebVid-2M [8] contains mostly videos with dynamics scores ≤0.4, limiting training examples for dynamics-rich content.Source: §4.4 · p. 10, Figure 6 · p. 10
• WebVid2M [8] and MSR-VTT obtain dynamics controllability scores of 36.31% and 52.98%, indicating insufficient dynamics guidance in their prompts.Source: §4.4 · p. 10
• As dynamics increase, SOTA models’ visual quality, motion smoothness, and naturalness scores significantly decrease, reflecting limited real-world simulation ability.Source: §4.4 · pp. 10–11

• DEVIL combines multi-temporal dynamics metrics with a benchmark of prompts spanning multiple dynamics levels to assess T2V generation capacity and alignment.Source: §5 · p. 11
• Experiments show 90% consistency between DEVIL and human evaluation results, supporting its potential as a tool for advancing T2V generation models.Source: §5 · p. 11
• The protocol currently uses limited dynamics grades and evaluates only a limited number of T2V models, requiring finer grades and broader future evaluation.Source: §5 · p. 11
• The authors identify potential misuse of advanced T2V models to create realistic but misleading video content, including deepfakes.Source: §5 · p. 11

• StreamingT2V performs exceptionally well across most dynamics metrics and generates complex dynamic content at the video level.ModelScope instead excels at rapid inter-frame motions, while GEN-2 and VideoCrafter2 perform poorly on several metrics.Source: A Dynamics Scores · p. 15, Figure 7 · p. 15

• Table 6 reports Pearson correlations between dynamics scores and aesthetic, technical, visual-quality, consistency, smoothness, and naturalness metrics.Source: B Correlation Between Existing Metrics… · p. 15
• As dynamic complexity increases, motion smoothness, subject consistency, background consistency, and naturalness tend to decline.Source: B Correlation Between Existing Metrics… · p. 15
• Aesthetic, technical, and visual-quality metrics show relatively low correlation with dynamics because they evaluate frames independently.Source: B Correlation Between Existing Metrics… · p. 15

C Detail of Dynamics-based Quality

• Existing metrics average scores across generated videos, but negative correlations with dynamics can reward models that produce low-dynamic videos.Source: C Detail of Dynamics-based Quality · p. 15
• The improved metric divides human-aligned dynamics scores into L = 12 intervals, averages quality within each interval, and then averages the interval means.Source: C Detail of Dynamics-based Quality · p. 16
• FreeNoise and StreamingT2V achieve high overall quality scores across a wide dynamic range, whereas Gen-2 and Pika excel mainly at low dynamics.Source: C Detail of Dynamics-based Quality · p. 16

D Assigning Dynamics Grades to Text Prompts

• The DEVIL benchmark samples about 800 uniformly distributed prompts from approximately 50,000 prompts classified by GPT-4 and refined by six human annotators.The source prompts cover 19 object categories and 4 scene categories, with refinement conducted over three months.Source: D Assigning Dynamics Grades to Text… · p. 16

• Gemini-1.5 Pro classifies video naturalness across five levels ranging from “Completely Fantastical” to “Almost Realistic.”Source: E Details of Naturalness · p. 16
• The naturalness assessment process uses video understanding to assign ratings based on alignment with the real world.Source: E Details of Naturalness · p. 16
• An almost-realistic dog video contrasts with a slightly unrealistic example containing an implausible number of legs.These examples illustrate how distortions affect naturalness ratings.Source: E Details of Naturalness · p. 16

F Human Annotation

• Videos generated by six advanced T2V models were annotated to align human evaluations with automated metrics.The models were GEN-2, Pika, VideoCrafter2, OpenSora, StreamingT2V, and FreeNoise-Lavie.Source: F Human Annotation · p. 16
• Table 7 integrates dynamics scores with Motion Smoothness, Naturalness, Subject Consistency, and Background Consistency across overall, low, mid, and high categories.The categories use modified dynamic intervals for comprehensive evaluation.Source: Table 7 · p. 17
• Annotators assessed dynamics across five grades and naturalness from almost real to completely unreal using a dedicated toolbox.Six trained annotators with undergraduate degrees conducted the evaluations.Source: F Human Annotation · p. 17

G Visual comparison

• The dynamics classifier assigns each text segment a grade from Almost Static through Very High based on scene and background changes.The criteria progress from minimal changes to extremely rapid and frequent content changes.Source: G Visual comparison · p. 18
• GPT-4 is used to illustrate coarse categorization of prompts.The figure presents an example of prompt coarse categorization.Source: Figure 8 · p. 18
• Gemini-1.5 Pro is used to illustrate naturalness calculation for generated videos.The figure presents an example of the naturalness-calculation process.Source: Figure 9 · p. 19
• The toolbox provides a combined visual framework for dynamics and naturalness annotation.Figure 10 identifies the toolbox as covering both annotation dimensions.Source: Figure 10 · p. 20
• The examples span static, medium, and very high dynamic grades for prompts ranging from portraits to yachting and volcanic eruptions.The displayed examples associate a business-people portrait with Static, yachting with Medium, and a volcanic eruption with Very High.Source: G Visual comparison · pp. 21–22
• The visual results show outputs from models including GEN2, OpenSora, VideoCrafter2, FreeNoise, Pika, and StreamingT2V.These model names accompany the displayed prompt examples.Source: G Visual comparison · pp. 21–22