PaperlayerView exact revision on arXiv

Source-linked AI summary

FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation

Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, Lu Hou

arXiv:2311.01813v3cs.CV

TL;DR

Open-domain T2V evaluation lacks fine-grained prompt coverage and reliable agreement between automatic metrics and human judgment. FETV addresses these gaps with multi-aspect, temporal-aware categorization, manual model evaluation, and metric diagnosis; its UMT-based metrics correlate better with humans than existing metrics, while still having room for improvement.

  • Problem

    T2V evaluation is limited by coarse prompt measurements and under-examined reliability of automatic metrics relative to human judgment.

    Source: §1 · pp. 1–2

  • Method

    FETV categorizes 619 prompts across major content, attribute control, prompt complexity, and temporal categories, then supports manual model evaluation and automatic-metric diagnosis.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    FETV reveals weaknesses in four representative T2V models and shows that UMT-based FVD-UMT and UMTScore correlate more strongly with humans than existing metrics.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    FETV provides specific performance information across prompt categories and a testbed for assessing automatic T2V metric reliability.

    Source: §6 · p. 10

    The UMT-based metrics still have room for improvement, and evaluation is limited by the small number of open-sourced T2V models.

    Source: §7 · p. 10

Abstract

from arXiv · show

Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation of T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories of text prompts. Although some benchmarks have categorized the prompts, their categorization either only focuses on a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose FETV, a benchmark for Fine-grained Evaluation of Text-to-Video generation. FETV is multi-aspect, categorizing the prompts based on three orthogonal aspects: the major content, the attributes to control and the prompt complexity. FETV is also temporal-aware, which introduces several temporal categories tailored for video generation. Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as a testbed to evaluate the reliability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://github.com/llyx97/FETV.

1 Introduction

Open-domain T2V has advanced, but evaluation remains largely qualitative and lacks both fine-grained prompt analysis and reliable automatic metrics. FETV addresses these gaps with multi-aspect, temporal-aware evaluation, manual model analysis, and improved metrics.

Motivation

  • Existing quantitative evaluations provide coarse overall scores and often omit temporal aspects, limiting understanding of model capabilities.Prior benchmarks may categorize prompts by content or challenge type but do not fully capture video-specific temporal information.Source: §1 · pp. 1–2

FETV Benchmark

  • FETV categorizes prompts by major content, attribute control, and prompt complexity, while adding temporal categories tailored to video generation.The benchmark collects 619 categorized prompts using automatic labeling followed by manual review.Source: §1 · p. 2

Manual Evaluation

  • Manual evaluations of four representative T2V models reveal poor quality for action and kinetic-motion videos and weak control of quantity, motion direction, and event order.The evaluation identifies strengths and weaknesses across prompt categories rather than only reporting aggregate performance.Source: §1 · p. 2

Automatic Metrics

  • Existing FID, FVD, and CLIPScore metrics correlate poorly with human evaluation, whereas FVD-UMT and UMTScore achieve much higher human correlation.The proposed metrics target video quality and video-text alignment using UMT.Source: §1 · p. 2

Contributions

  • FETV provides a comprehensive benchmark and testbed for comparing T2V models and diagnosing automatic metric reliability.Its multi-aspect categorization supports analysis across different prompt scenarios.Source: §1 · p. 2

2 Related Work

Prior T2V benchmarks often reuse datasets designed for other tasks or categorize prompts along only one dimension. Existing automatic metrics separately assess video quality or text-video alignment, but their reliability for open-domain T2V remains under-explored.

Benchmarks for Text-to-Video Generation

  • MSR-VTT categorizes videos by topic but omits temporal content and controllable attributes important for open-domain T2V.Its categories include topics such as music, sports, and news.Source: §2 · p. 2
  • DrawBench and PartiPrompts categorize text-to-image challenges but lack temporal descriptions needed for T2V evaluation.Make-a-Video-Eval is designed for T2V but categorizes prompts only by spatial content.Source: §2 · p. 3

Automatic Metrics for Text-to-Video Generation

  • Open T2V metrics evaluate either video quality or video-text alignment, commonly using IS, FID, FVD, or CLIPScore.FID and FVD compare real and generated distributions in neural-network latent spaces, while CLIPScore measures text-image similarity through CLIP.Source: §2 · p. 3

Models for Text-to-Video Generation

  • Existing open T2V models include Transformer-based and diffusion-based systems, with this work evaluating CogVideo, Text2Video-zero, ModelScopeT2V, and ZeroScope.The paper focuses on text-only conditioning rather than text-driven video editing.Source: §2 · p. 3

3 FETV Benchmark

FETV is an open-domain benchmark whose prompts receive multi-aspect, temporal-aware labels and can support fine-grained manual or automatic evaluation. It combines diverse existing prompts with unusual manually written cases and uses automatic categorization followed by manual correction.

3 FETV Benchmark

  • FETV is a prompt dataset organized for fine-grained T2V evaluation across multiple aspects.The benchmark introduces its categorization, collection process, statistics, data format, and application scope.Source: §3 · p. 3

3.1 Multi-Aspect Categorization

  • Each prompt is categorized by major content, attribute control, and complexity, with spatial and temporal subcategories for content and attributes.Complexity has simple, medium, and complex levels based on the number of non-stop words, and the resulting subsets enable fine-grained evaluation.Source: §3.1 · p. 4

3.2 Data Collection

  • Prompts come from MSR-VTT, WebVid, and manually written unusual scenarios covering object-action, object-quantity, object-direction, object-speed, and event-order cases.The unusual prompts are intended to test model generalization beyond scenarios represented in existing datasets.Source: §3.2 · p. 4
  • Category assignment combines keyphrases, WordNet synsets, and hand-crafted matching patterns with manual selection and label revision.The process selects roughly 20 prompts for combinations of attribute-control and temporal-content categories, then revises incorrect labels and mismatches with reference videos.Source: §3.2 · pp. 4–5

3.3 Dataset Statistics and Data Format

  • 619 prompts are included: 541 from existing datasets and 78 unusual prompts written by the authors, with distributions covering diverse categories.The benchmark’s scale is comparable to existing manual-evaluation benchmarks such as DrawBench, PartiPrompts, and Make-a-Video-Eval.Source: §3.3 · p. 5
  • Each sample contains a prompt, an optional reference video, and categorization labels, with multiple labels allowed within an aspect except complexity.Reference videos are unavailable for unusual prompts.Source: §3.3 · p. 5

3.4 Application Scope

  • FETV supports manual and automatic evaluation, but the paper primarily uses manual evaluation because existing automatic metrics correlate poorly with humans.Manual results can also serve as a reference for diagnosing automatic metrics.Source: §3.4 · p. 5

4 Manual Evaluation of Open T2V Models

The study manually evaluates four open-sourced T2V models across static quality, temporal quality, overall alignment, and fine-grained alignment using FETV’s prompt categories. Results expose category-specific quality and controllability weaknesses, including difficult motion and attribute-control cases.

4.1 Models

  • Four open-sourced models—CogVideo, Text2Video-zero, ModelScopeT2V, and ZeroScope—are evaluated with FETV’s multi-perspective manual protocol.The protocol scores static quality, temporal quality, and overall alignment on 1–5 scales, and fine-grained alignment on a 1–3 scale.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Evaluation Setups

  • 28,116 human ratings cover generated and reference videos, with three graduate-student ratings collected for each evaluated perspective.The evaluation averages ratings across videos and reports category-specific performance by aggregating prompts belonging to each category.Source: §4.2 · p. 6

4.3 Results and Analysis

  • Manual evaluation finds the weakest video quality for people, animals, actions, and kinetic motions, while plants and scenery or natural objects perform better.The paper attributes these differences to the higher-frequency details or temporal content involved in the poorer-quality categories.Source: §4.3 · pp. 6–7
  • Text2Video-zero leads static quality but lags in temporal quality, whereas ZeroScope matches ModelScopeT2V temporally but performs worse in static quality.The paper links Text2Video-zero’s temporal weakness to lacking video training and attributes ZeroScope’s trade-off to optimization for 16:9, watermark-free videos.Source: §4.3 · pp. 7–8
  • Models control color and camera view relatively well but struggle with quantity, motion direction, and event order.ModelScopeT2V shows preliminary ability on speed and motion direction; alignment is only slightly lower for complex prompts, without a clear complexity correlation.Source: §4.3 · p. 8
  • Fine-grained alignment more accurately measures specific attribute controllability than overall alignment, although overall alignment remains an acceptable approximation.The relative rankings across attributes and models are generally similar, with exceptions because overall alignment can reflect other attributes.Source: §4.3 · p. 8
  • The full FETV benchmark also provides a manual-evaluation leaderboard for comparing the evaluated T2V models.The leaderboard aggregates performance across the benchmark rather than isolating a single prompt category.Source: §4.3 · p. 8, Figure 7 · p. 8

5 Diagnosis of Automatic Evaluation Metrics

The paper tests automatic alignment and video-quality metrics against human evaluations on FETV. Standard CLIPScore, FID, and FVD correlate weakly with human judgments, while UMTScore and FVD-UMT produce more human-consistent rankings.

5.2 Evaluation Setups

  • Automatic metrics are evaluated by correlating their scores with manual judgments, using Kendall’s τc and Spearman’s ρ for alignment and system-wise rankings for group-level quality metrics.Alignment correlations are computed overall and within prompt categories; FID and FVD are evaluated through model rankings because they are not sample-wise metrics.Source: §5.2 · p. 9

5.3 Results and Analysis

  • UMTScore has the strongest human correlation and consistently matches human T2V model rankings for video-text alignment.Its stronger performance supports using video-trained vision-language models for evaluating open-domain T2V systems.Source: §5.3 · pp. 9–10, Figure 8 · p. 9
  • CLIPScore is positively but weakly correlated with manual alignment judgments, while CLIPScore-ft and BLIPScore improve correlation yet remain inconsistent with human model rankings.The results therefore do not support relying on CLIPScore or BLIPScore alone for accurately evaluating open T2V models.Source: §5.3 · p. 9
  • UMTScore correlates more strongly in challenging quantity and motion-direction categories than in simpler color and camera-view categories.The paper conjectures that simpler categories yield more similar model performance, making models harder to distinguish.Source: §5.3 · pp. 9–10
  • A case study shows CLIPScore and Otter-VQA rankings diverging strongly from humans, whereas UMTScore’s ranking is most consistent with human evaluation.CLIPScore-ft and BLIPScore correlate better but still assign a lower score to the ground-truth video in the presented example.Source: §5.3 · p. 10, Figure 9 · p. 10
  • FID and original FVD with I3D features are loosely coupled with human video-quality perception, while FVD-UMT produces rankings consistent with humans.FVD-UMT replaces the original video feature extractor with the UMT vision encoder.Source: §5.1 · p. 8, §5.3 · p. 10, Figure 10 · p. 10

6 Conclusions

FETV provides multi-aspect and temporal-aware evaluation of open-domain text-to-video generation, while diagnosing weaknesses in models and automatic metrics.

6 Conclusions

  • FETV is multi-aspect and temporal-aware, providing more specific information about text-to-video model performance.It categorizes prompts across different aspects and temporal content.Source: §6 · p. 10
  • Manual evaluation with FETV reveals representative text-to-video models’ strengths and weaknesses across prompt categories.Source: §6 · p. 10
  • Existing automatic metrics correlate poorly with human judgments, whereas two UMT-based metrics produce model rankings consistent with humans.Source: §6 · p. 10

7 Limitations and Future Work

The authors identify two limitations: UMT-based metrics still need better human alignment, and the evaluated model set is small because few open T2V models are open-sourced.

7 Limitations and Future Work

  • UMT-based metrics are more reliable than existing automatic metrics but still have room to improve their alignment with human judgments.Source: §7 · p. 10
  • The evaluation includes few models because open-sourced open-domain text-to-video models are limited.The authors invite future studies to evaluate additional models and encourage more projects to open-source them.Source: §7 · p. 10

A Overview of Appendix

  • The appendix includes further details on FETV categories, model setups, evaluation setups, experimental results, categorization rules, and maintenance plans.Source: A Overview of Appendix · p. 14

B.2 Data Distribution

FETV is more uniformly distributed across categories than MSR-VTT and WebVid, whose category biases leave several attribute-control abilities poorly represented.

B.2 Data Distribution

  • FETV has a more uniform category distribution than MSR-VTT and WebVid, which are biased toward particular categories.Source: §B.2 · p. 14
  • More than 60% of MSR-VTT and WebVid prompts omit the six FETV attributes, limiting evaluation of control over especially challenging categories.Source: §B.2 · p. 14

B.3 Data Examples

  • FETV examples cover all 22 categories across the three categorization aspects.Source: §B.3 · p. 14

C.1 CogVideo

The appendix documents the architectures, training or inference settings, category materials, and automatic alignment metrics used for the evaluated text-to-video models.

C.1 CogVideo

  • CogVideo hierarchically generates discrete visual tokens with two Transformer stages after VQVAE encoding.Its two stages generate keyframes and recursively interpolate remaining frames.Source: §C.1 · p. 14
  • CogVideo is trained on 5.4 million captioned videos and evaluated with 33-frame, 480×480 videos.The first stage uses 1 fps and the second uses 2, 4, and 8 fps during training.Source: §C.1 · p. 14
  • The appendix provides spatial, temporal, prompt-complexity, and example-category materials for FETV.Source: Figure 11 · p. 15, Figure 12 · p. 16, Table 3 · p. 16

C.2 Text2Video-Zero

  • Text2Video-Zero adapts text-to-image generation without video training by injecting motion, cross-attending frames to the first frame, and smoothing backgrounds.Source: §C.2 · p. 16

C.3 ModelScopeT2V

  • ModelScopeT2V adds temporal convolution and attention blocks to Stable Diffusion and trains on both video-text and image-text pairs.Source: §C.3 · pp. 17–18

C.4 ZeroScope

  • ZeroScope is trained from ModelScopeT2V weights and optimized for watermark-free 16:9 video compositions.Source: §C.4 · p. 18

D.1 Video-Text Alignment Metrics

  • Automatic alignment evaluation uses CLIPScore, BLIPScore, fine-tuned CLIPScore, UMTScore, and Otter-VQA-based question answering.Source: §D.1 · p. 18

D.2 Video Quality Metrics

FETV computes FID and FVD from sampled generated and reference videos, with category-level evaluation based on repeated random sampling. The appendix also documents the associated metric implementation details.

D.2 Video Quality Metrics

  • 2,055 generated and 2,328 reference videos were collected, with at least 300 sampled prompts per major content category.Category-specific FID and FVD use 300 generated and reference videos, averaged over four runs with different random seeds.Source: §D.2 · p. 18
  • FID and FVD were computed using clean-fid and stylegan-v implementations, respectively.Source: §D.2 · p. 18
  • Table 6 reports inter-human correlation statistics using Kendall’s τc and Spearman’s ρ, averaged across every pair of human evaluators.The reported p-values are less than 0.05.Source: Table 6 · p. 19

E More Details of Manual Evaluation Setups

The manual evaluation setup standardizes rating instructions and presentation conditions, then compares automatic metrics with human rankings across categories and sample sizes. These analyses identify stronger metric alignment while documenting the dependence of FID and FVD on video count.

E.1 Evaluation Instructions

  • Rating instructions define evaluation levels and examples, while the interface crops videos to 256×256 and controls duration and presentation order.Reference videos are cropped to four seconds for quality evaluation, retained at original duration for alignment evaluation, and interleaved across models to reduce order effects.Source: §E.1 · p. 19, §E.2 · p. 19, Figure 16 · p. 23

E.3 Correlation between Human Evaluators

  • Table 6 demonstrates strong inter-human agreement across the four evaluation perspectives.Source: §E.3 · p. 19

F.1 Video-Text Alignment Ranking Results

  • UMTScore shows the strongest correlation with human video-text alignment rankings in every attribute-control category.Source: §F.1 · p. 20, Figure 13 · p. 20

F.2 FID and FVD Ranking Results

  • FVD-UMT is more consistent with human video-quality rankings than FID and FVD in almost all spatial and temporal categories.All three metrics remain poorly aligned with humans for categories such as Illustration, Food & Beverage, and Plants.Source: §F.2 · p. 20, Figure 14 · p. 21, Figure 15 · p. 22

F.3 The Effect of Video Number on FID and FVD

  • FID and FVD values decrease as the number of videos increases, while more than 300 videos enables FVD-UMT to rank models more accurately than FID and FVD.Because generating thousands of videos is time-consuming, the study uses 300 videos per category; more videos can yield more reliable results.Source: §F.3 · pp. 20, 23

F.4 Results on Real-World and Unusual Prompts

  • Unusual prompts make video generation more challenging than real-world prompts, although video quality is less affected.Source: §F.4 · p. 23

F.5 Qualitative Analysis of Generated Videos

The qualitative appendix illustrates metric behavior, category-specific generation strengths and failures, and the benchmark’s operational scope. It also documents automatic categorization, distribution, maintenance, and known dataset limitations.

F.6 Case study of Video-Text Alignment Metrics

  • UMTScore produces the most human-consistent video-text alignment ranking, whereas other metrics often assign low scores to more complex ground-truth videos.The authors conjecture that such cases require deeper vision-language understanding because described entities may not appear in every frame.Source: §F.6 · p. 23, Figure 32 · p. 33

G Automatic Categorization Rules

  • FETV classifies prompts using WordNet synsets, key phrases, and matching patterns, followed by manual selection and label revision.Prompt complexity is based on the number of non-stop words, and prompts may receive multiple labels within an aspect except for complexity.Source: G Automatic Categorization Rules · p. 23

F.5 Qualitative Analysis of Generated Videos

For what purpose was the dataset created?

H Licensing, Hosting and Maintenance Plan

What (other) tasks could the dataset be used for?

Related papers

Loading 2311.01813v3…