PaperlayerView exact revision on arXiv

Source-linked AI summary

ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answering

Kaisi Guan, Zhengfeng Lai, Yuchong Sun, Peng Zhang, Wei Liu, Kieran Liu, Meng Cao, Ruihua Song

arXiv:2503.16867v2cs.CV

TL;DR

Text-to-video alignment metrics often provide coarse scores without fine-grained detail, limiting their alignment with human preference. ETVA generates atomic questions and answers them with knowledge-augmented, multi-stage reasoning. The paper reports a higher correlation with human judgment than existing metrics.

  • Problem

    Existing text-to-video alignment metrics provide coarse-grained scores that miss fine-grained alignment details and do not align well with human preference.

    Source: §1 · p. 1

  • Method

    ETVA uses a multi-agent system to generate atomic questions and knowledge-augmented, multi-stage reasoning to answer them.

    Source: §1 · p. 2

  • Results

    A Spearman’s correlation of 57.8, outperforming VideoScore by 27.5, is reported for ETVA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ETVA-Bench provides 2K diverse prompts and 12K questions across 10 categories for evaluating text-to-video alignment.

    Source: §1 · p. 2

Abstract

from arXiv · show

Precisely evaluating semantic alignment between text prompts and generated videos remains a challenge in Text-to-Video (T2V) Generation. Existing text-to-video alignment metrics like CLIPScore only generate coarse-grained scores without fine-grained alignment details, failing to align with human preference. To address this limitation, we propose ETVA, a novel Evaluation method of Text-to-Video Alignment via fine-grained question generation and answering. First, a multi-agent system parses prompts into semantic scene graphs to generate atomic questions. Then we design a knowledge-augmented multi-stage reasoning framework for question answering, where an auxiliary LLM first retrieves relevant common-sense knowledge (e.g., physical laws), and then video LLM answers the generated questions through a multi-stage reasoning mechanism. Extensive experiments demonstrate that ETVA achieves a Spearman's correlation coefficient of 58.47, showing a much higher correlation with human judgment than existing metrics which attain only 31.0. We also construct a comprehensive benchmark specifically designed for text-to-video alignment evaluation, featuring 2k diverse prompts and 12k atomic questions spanning 10 categories. Through a systematic evaluation of 15 existing text-to-video models, we identify their key capabilities and limitations, paving the way for next-generation T2V generation.

1. Introduction

Existing text-to-video alignment metrics provide coarse scores that can miss fine-grained details and diverge from human preferences. ETVA uses atomic questions and knowledge-augmented, multi-stage reasoning; experiments report stronger human alignment, and ETVA-Bench evaluates 15 models.

1. Introduction

  • Existing caption-, CLIP-, and MLLM-based metrics produce coarse-grained scores that can miss fine-grained alignment and diverge from human preference.For the microgravity example, a human annotator prefers Video 2 while existing metrics favor Video 1.Source: §1 · pp. 1–2
  • Video prompts can combine multiple elements, making intricate questions difficult for video LLMs to answer correctly.The example question about water pouring from a glass in a space station requires understanding actions, objects, and environment together.Source: §1 · p. 2
  • Video LLMs may hallucinate when questions require commonsense knowledge or deeper observation and reasoning.The passage contrasts human recognition of microgravity with video LLMs’ lack of that innate understanding.Source: §1 · p. 2
  • ETVA generates atomic questions with a multi-agent framework and answers them using knowledge-augmented, multi-stage reasoning.Its question-generation agents extract prompt elements, build a scene graph, and traverse it to generate questions.Source: §1 · p. 2
  • A Spearman’s correlation of 57.8, outperforming VideoScore by 27.5, is reported for ETVA.The authors also report that the question-generation and question-answering components contribute performance improvements of 14.67% and 26.2%, respectively.Source: §1 · p. 2
  • ETVA-Bench contains 2K diverse prompts and 12K questions, and the paper evaluates 10 open-source and 5 closed-source text-to-video models.The evaluation identifies challenges including camera movements and physics processes.Source: §1 · p. 2

2. Related Work

The paper situates ETVA among text-to-video models, alignment metrics, and benchmarks, then describes its question-generation and question-answering pipeline. The pipeline maps prompts to binary questions and an aggregate alignment score using scene graphs, commonsense knowledge, and staged video reasoning.

2. Related Work

  • Text-to-video models include GAN/VQ-VAE, UNet-based diffusion, and DiT architectures; this paper focuses on DiT-based models.The related-work discussion describes DiT as the state-of-the-art backbone for text-to-video models.Source: §2 · p. 2
  • Existing alignment metrics use caption similarity, CLIP embeddings, or MLLM reward models, while prior question-generation and answering approaches for text-to-image do not transfer directly to text-to-video.General benchmarks often overlook fine-grained details, whereas specific benchmarks target aspects such as human actions, composition, physics, or time-lapse video.Source: §2 · p. 3

3.1. Problem Formulation

  • ETVA maps a prompt and video to an alignment score using binary questions derived from the prompt, commonsense knowledge, and video-based answers aggregated across questions.The formulation identifies the prompt, video, generated questions, extracted knowledge, binary question scores, and final score as the pipeline inputs and outputs.Source: §3.1 · p. 4

3.2. Multi-Agent Question Generation

  • ETVA generates questions by extracting entities, attributes, and relations, structuring them in an entity-centered scene graph, and traversing the graph systematically.Attribute nodes connect outward from entities, while relation nodes connect entities; traversal handles entities, attributes, then relations after both connected objects are processed.Source: §3.2 · p. 4, §3.2.1 · p. 4, §3.2.2 · p. 4

3.2.1. Element Extractor

  • The Element Extractor identifies entities, descriptive attributes, and relationships in a text prompt.For the example prompt, it identifies water, cup, and space station as entities, alongside material attributes and spatial relationships.Source: §3.2.1 · p. 4

3.2.2. Graph Builder

  • The Graph Builder uses entity nodes as anchors: attribute nodes emit outgoing edges, while relation nodes connect entities bidirectionally.This entity-centered topology is described as enforcing category exclusivity between neighboring nodes and preventing redundant connections.Source: §3.2.2 · p. 4

3.2.3. Graph Traverser

  • The Graph Traverser asks about entities, then attributes, and only then relations after both connected objects have been processed.This ordering is intended to make questions build on previously confirmed information and maintain clear context.Source: §3.2.3 · p. 4

3.3. Multi-modal Question Answering

  • ETVA’s question-answering workflow augments video questions with commonsense knowledge and uses progressive video understanding, reflection, and conclusion stages.The auxiliary LLM generates knowledge-informed video descriptions; the video LLM observes frames, checks evidence against the question and knowledge, then gives a supported Yes/No answer.Source: §3.3 · pp. 4–5, §3.3.1 · p. 5, §3.3.2 · p. 5

4. Our Proposed ETVABench

ETVABench is an alignment benchmark built from prompts collected across open-source benchmarks and categorized using atomic questions. It provides a 2K-prompt set and a smaller subset for more cost-effective testing.

4. Our Proposed ETVABench

  • ETVABench collects prompts from open-source benchmarks and classifies them into 10 categories using a question-driven method.The taxonomy is based on atomic questions generated from scene-graph node traversals.Source: §4 · p. 5, §4.2 · p. 5

4.1. Our Prompts Collection and Sources

  • ETVABench-2k contains 2K prompts sampled across general, compositional, human-motion, physics-aware, temporal-coherence, and GPT-4-o-generated sources.The listed source counts are 880 general, 300 compositional, 510 human-motion, 160 physics-aware, 100 temporal-coherence, and 50 generated prompts.Source: §4.1 · p. 5

4.2. Prompt Classification based on Question Labels

  • ETVABench-2k contains 12K questions across existence, action, material, spatial, number, shape, color, camera, physics, and other categories.ETVABench-105 contains 0.6K questions and is sampled to preserve the full benchmark’s question-type distribution for cost-effective closed-source model testing.Source: §4.2 · p. 5, §4.1 · p. 5

5. Experiment

Experiments compare ETVA with existing alignment metrics, test its question-generation and answering components, and evaluate T2V models on two benchmark scales and in case studies.

5.1. Experiment Setup

  • ETVA uses Qwen2.5-72B-Instruct for question generation and knowledge augmentation, and Qwen2-VL-72B for video-question answering; evaluation includes five human annotators rating 1,575 videos.Source: §5.1 · pp. 5–6, §5.2.2 · p. 6

5.2.3. Comparison of T2V Alignment Metrics

  • 47.16 Kendall’s τ and 58.47 Spearman’s ρ: ETVA achieves the strongest reported alignment with human judgment across semantic categories.Source: Table 1 · p. 6, §5.2.3 · p. 6

5.2.1. Baselines metrics

  • The comparison includes caption-, CLIP-, and MLLM-based metrics, with VideoScore among the MLLM baselines.Source: §5.2.1 · p. 6

5.3.1. Evaluation on ETVABench-105

  • On ETVABench-105, Vidu-1.5 leads overall at 0.761 average, while all models reach at most 0.600 in physics and 0.474 in camera control.Source: Table 4 · p. 7, §5.3.1 · p. 7

5.3.2. Evaluation on ETVABench-2k

  • On ETVABench-2k, performance trends are consistent with ETVABench-105, and Hunyuan-Video leads among the 10 open-source models.Source: Figure 3 · p. 7, §5.3.2 · p. 8

5.2.4. Ablation Study of ETVA

  • ETVA improves Kendall’s τ by 34.6% and Spearman’s ρ by 34.1% over vanilla question generation, with scene-graph traversal supporting broader question coverage.Source: §5.2.4 · p. 7
  • 89.27% accuracy: ETVA exceeds direct Video LLM answers by 26.20%, while knowledge augmentation, video understanding, and critical reflection contribute to answering performance.Source: §5.2.4 · p. 7

5.4. Case Study

  • Case studies cover physics, color transition, number accuracy, and gestural semantics, with examples showing both accurate generations and specific alignment errors.Source: Figure 4 · p. 8, §5.4 · p. 8

6. Conclusion

ETVA evaluates text-to-video alignment through fine-grained question generation and answering, and ETVABench applies it to current T2V models. Experiments report higher human-preference correlation than existing metrics and identify model limitations.

  • ETVA pairs multi-agent generation of atomic, semantically comprehensive questions with knowledge-augmented, multi-stage question answering designed to address question complexity and video-LLM hallucinations.The paper describes this as its evaluation method for text-to-video alignment.Source: §6 · p. 8
  • ETVABench contains 2K diverse prompts and 12K generated questions categorized into ten types for evaluating current T2V models.Source: §6 · p. 8
  • Experiments report that ETVA correlates much more highly with human preferences than existing metrics and helps identify T2V model limitations.Source: §6 · p. 8

7. Details on Evaluation Categories

The benchmark groups prompts into ten categories, and ETVABench-2k and ETVABench-105 have similar question-category distributions.

  • The ten categories are existence, action, material, spatial, number, shape, color, camera, physics, and other.Source: §7 · p. 12
  • ETVABench-2k and ETVABench-105 generate questions with similar distributions across the categories, indicating consistent coverage.Source: §7 · p. 12
  • Table 5 provides a prompt example and a question example for each category.Source: §7 · p. 12

8. Detailed results on ETVABench-2k

This section points to detailed ETVABench-2k results and describes the settings used to generate videos with several evaluated models.

8. Detailed results on ETVABench-2k

  • Table 6 contains further details about the ten open-source models evaluated on ETVABench-2k.Source: §8 · p. 12

9. Details of Text-to-Video Models

  • Sora samples were generated at 480p in 16:9 for five seconds, while Vidu 1.5 used 720p in 16:9 for four seconds and Pika 1.5 used 16:9 for five seconds.Source: §9 · p. 12
  • Kling-1.0 and Kling-1.5 samples used a 16:9 aspect ratio and five-second duration; Hunyuan-Video samples used 544×966 resolution, 24 fps, and five seconds.Source: §9 · p. 12
  • Open-source model settings varied: OpenSora 1.2 used 720p at 24 fps for four seconds, while CogVideoX versions used differing resolutions, frame rates, and durations.CogVideoX-2B and CogVideoX-5B used 720×480 at 8 fps for six seconds; CogVideoX-1.5-5B used 1360×768 at 16 fps for five seconds.Source: §9 · p. 12
  • Figure 5 is a prompt-category distribution figure for ETVABench-2k and ETVABench-105, and Table 5 presents question and prompt examples by category.Source: Figure 5 · p. 13, Table 5 · p. 13
  • OpenSoraPlan v1.1 used 65 frames at 512×512 and 16 fps, whereas v1.2 used 93 frames at 720p and 16 fps.Source: §9 · p. 13

10. Details of Human annotation

Figure 6 presents detailed instructions for human annotation.

  • Figure 6 presents detailed instructions for human annotation.Source: §10 · p. 13
  • The figure's stated subject is the human-annotation instructions.Source: §10 · p. 13
  • The annotation guidance is presented as a figure rather than described further in this passage.Source: §10 · p. 13

11. Prompts of ETVA

Figures 7–11 present ETVA’s detailed prompts.

  • Figures 7–11 present the detailed prompts used in ETVA.Source: §11 · p. 13

12. More Cases about ETVA

The appendix reports ETVABench-2k evaluation results and illustrates comparisons between ETVA and conventional metrics on two prompts. It also shows prompts for human annotation, graph construction, knowledge augmentation, and multi-stage reasoning.

Related papers

Loading 2503.16867v2…