Source-linked AI summary
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, Ying Shan
TL;DR
Existing T2V evaluation often relies on limited metrics that miss text alignment, motion quality, and temporal consistency. EvalCrafter builds a 700-prompt benchmark and combines multi-aspect objective metrics with human alignment; its experiments show that the resulting evaluation effectively aligns with user opinions. The benchmark provides a broader assessment of T2V methods, while its current prompt coverage and annotator pool remain limited.
Problem
Existing T2V evaluations use limited metrics that do not adequately assess text-video alignment, motion quality, and temporal consistency.
Method
EvalCrafter constructs a diverse prompt benchmark and evaluates video quality, text-video alignment, motion quality, and temporal consistency using objective metrics aligned with human scores.
Results
EvalCrafter's experiments demonstrate that its evaluation methods effectively align with user opinions and provide reliable assessment of T2V approaches.
Takeaways & Limitations
The benchmark supports comprehensive assessment of large T2V models across video quality, alignment, temporal consistency, and motion quality.
Takeaways & Limitations
The benchmark currently contains only 700 prompts, and its alignment labels come from relatively few human annotators, which may introduce bias.
Abstract
from arXiv · showhide
The vision and language generative models have been overgrown in recent years. For video generation, various open-sourced models and public-available services have been developed to generate high-quality videos. However, these methods often use a few metrics, e.g., FVD or IS, to evaluate the performance. We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Thus, we propose a novel framework and pipeline for exhaustively evaluating the performance of the generated videos. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation, which is based on an analysis of real-world user data and generated with the assistance of a large language model. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics. To obtain the final leaderboard of the models, we further fit a series of coefficients to align the objective metrics to the users' opinions. Based on the proposed human alignment method, our final score shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.
1. Introduction
EvalCrafter addresses the difficulty of evaluating large text-to-video models with metrics that capture video-specific and multi-aspect abilities. It builds a diverse benchmark and aligns objective evaluation with human preferences.
- Existing T2V evaluations rely on limited metrics such as FVD, overlooking motion quality and temporal consistency.
- EvalCrafter builds a comprehensive prompt list from real-world prompts, T2I prompts, and LLM-expanded real-world knowledge.The prompts include everyday objects, attributes, and motions with metadata for evaluation.
- The framework evaluates video quality, text-video alignment, motion quality, and temporal consistency with objective metrics and human scores.
- Human-aligned regression coefficients convert objective metrics into final model scores and support evaluation of new videos.
- The evaluation also reports conclusions and findings intended to contribute to further T2V model development.
2. Related Work
Prior T2V methods have advanced through diffusion models and large-scale text-image pretraining, but detailed and fair evaluation remains underdeveloped. Existing benchmarks and metrics often target image or language tasks and miss key video properties.
- Recent T2V systems use diffusion models and large-scale text-image pretraining to improve video generation quality.Examples include Make-A-Video, Imagen-Video, LVDM, Align Your Latent, and MagicVideo.
- A fair and detailed T2V benchmark is still lacking.
- FVD, IS, and CLIP similarity do not adequately assess text alignment, motion quality, and temporal consistency in T2V.
- Large generative-model evaluation uses metrics tailored to different abilities, question types, and user platforms, especially for LLMs.
- Existing evaluation methods such as DALL-Eval, HRS-Bench, and TIFA focus on T2I or language-model evaluation rather than T2V.
3. Benchmark Construction
EvalCrafter constructs a trustworthy T2V benchmark by combining analyzed user prompts with LLM- and human-assisted generation, metadata checks, and diverse prompt categories. The initial benchmark contains over 700 annotated prompts.
- An automatic pipeline targets a diverse prompt list for computer-vision and user evaluation, with 700 prompts selected initially because video generation is time-consuming.
- The benchmark starts by analyzing large-scale real-world T2V prompts to guide prompt construction.Over 600k prompts are collected and filtered to 200k after removing repeated and meaningless prompts.
- GPT-4 generates recognizable scenes from metaclasses and sampled attributes, while self-checking identifies inconsistencies between descriptions and attributes.
- User and T2I prompts are integrated after GPT-4 metadata generation, prompt selection, and consistency checking.
- The benchmark covers human, animal, object, and landscape metaclasses, each with natural, stylized, and explicit-camera-motion subtypes.It contains 50 styles and 20 camera-motion prompts, randomly added to 250 prompts.
4. Evaluation Metrics
EvalCrafter evaluates text-to-video outputs across visual quality, text-video alignment, content correctness, motion quality, temporal consistency, and human opinions rather than relying on a single metric.
- The benchmark combines visual-quality, alignment, content, motion, and temporal-consistency metrics for text-to-video evaluation.This expands beyond FID-based evaluation to assess multiple properties of generated videos.
- Overall Video Quality Assessment: Dover provides aesthetic and technical video-quality scores, while Inception Score measures generated-content diversity.The technical rating targets distortions such as noise and artifacts.
- Text-Video Alignment: Text-video alignment includes CLIP similarity, SD-Score, BLIP-BLEU, object detection, object counts, colors, celebrity identity, and text recognition.These measures cover global alignment, content attributes, human faces, and generated text.
- Motion Quality: Motion quality is assessed with action recognition, average optical flow, and motion-amplitude classification.The framework uses MMAction2 with VideoMAE V2 for actions and RAFT for dense optical flow.
- Temporal Consistency: Temporal consistency uses warping error, consecutive-frame embedding similarity, and face consistency based on similarity to the first frame.Face consistency averages similarities between the first frame and later frames.
- User Opinion Alignments: Human studies evaluate video quality, text-video alignment, motion quality, and other aspects using user ratings and filtered professional feedback.The study asks 7 users to rate alignment from 1 to 5 and retains 1024 filtered scores from 8647 feedback scores.
5. Results
The benchmark evaluates available T2V models across multiple aspects and meta types, using human-aligned correlations to compare objective metrics with user judgments. Results show that multi-aspect and meta-type analyses reveal substantial performance differences, while the benchmark remains limited by prompt coverage and annotator diversity.
- Evaluation Setup: The evaluation covers available high-resolution and service-based T2V models using benchmark prompts and default hyper-parameters.Evaluated models include ModelScope, Floor33 Pictures, ZeroScope, Show-1, Hotshot-XL, VideoCrafter1, Gen2, and PikaLab.
- Human Preference Alignment: The proposed weighting method shows better correlation with human scores on unseen 20% samples than directly averaging metrics.Spearman’s and Kendall’s rank correlations are used to measure alignment between objective results and human judgments.
- Findings: Models’ rankings vary significantly across aspects, supporting multi-aspect evaluation for comprehensive performance analysis.The benchmark examines different aspects rather than relying on a single dimension.
- Findings: Models also perform differently across animal, human, landscape, object, and style categories, making meta-type evaluation necessary.For example, Gen2 performs better than Floor33 Pictures on VQAA for human, animal, and style videos.
- Limitations: The benchmark remains constrained by its 700 prompts and by alignment labels collected from relatively few human annotators.The authors identify both limited prompt coverage and potential annotator bias as challenges for broader evaluation.
6. Conclusion
The paper introduces a comprehensive benchmark for evaluating large T2V models across video quality, text-video alignment, temporal consistency, and motion quality. Human alignment improves the evaluation metrics’ correspondence with user opinions and supports more reliable assessment of T2V methods.
- Conclusion: The benchmark provides a comprehensive prompt set and objective metrics covering video quality, text-video alignment, temporal consistency, and motion quality.It is presented as a first step toward evaluating large, high-quality T2V models.
- Conclusion: Human alignment correlates objective metrics with user scores to produce more accurate evaluation metrics for T2V methods.The authors report that their experiments demonstrate effective alignment with user opinions.
- Conclusion: The authors position the benchmark as a foundation for future research on large video generation models.The stated purpose is to foster development of future T2V research.
A. Detailed Analysis of Real-World User Data
The benchmark’s prompt design is grounded in real-world T2V user data, whose prompt lengths and noun categories inform its structure. The detailed results section also organizes 17 metrics by model and direction of better performance.
- Real-World Prompt Analysis: 90% of real-world T2V prompts contain between 3 and 40 words, guiding the benchmark’s prompt-length design.The analysis uses collected user prompts to determine an appropriate length range.
- Real-World Prompt Analysis: The user-data word cloud identifies frequent words and key concepts requested in T2V generation.These frequencies provide insight into users’ prompt interests.
- Real-World Prompt Analysis: WordNet analysis identifies artifacts, humans, animals, and locations as important noun classes, with style added as a meta class.Communication, attribute, and cognition words are excluded from this class analysis.
- Benchmark Construction: The benchmark divides T2V prompts into human, animal, object, and landscape meta-subject classes to create a diverse and representative evaluation set.The classification is derived from the analysis of real-world user prompts.
- Metric Results: Table 4 reports 17 metrics for each evaluated model, with percentages used except for Warping Error and Flow-Score.The table lists raw results for the evaluated T2V systems.
- Metric Results: Metric arrows indicate whether higher or lower values are better, while Flow-Score uses a neutral rightwards arrow.This notation clarifies how to interpret each metric’s direction.
B.2. Correlations Between Metrics and Human Labels
The benchmark compares objective metrics with human judgments across visual quality, motion, temporal consistency, and text-video alignment. The proposed human-aligned method generally correlates better with users’ opinions than averaging metrics, while individual metrics vary in reliability.
- Correlation analysis: The correlation analysis covers visual quality, motion amplitude, temporal consistency, and text-video alignment using Spearman’s ρ and Kendall’s ϕ.Table 5 compares objective metrics and the proposed evaluation method against human judgment.
- Human alignment: 55.4 Spearman’s ρ and 41.1 Kendall’s ϕ for visual quality exceed average values of 55.0 and 41.0.The paper reports similar improvements for the other evaluation aspects.
- Human alignment: The proposed method consistently achieves higher correlations with human judgment than averaging the other methods.This result is presented as evidence that aligning objective metrics to users’ opinions improves evaluation effectiveness.
- Metric reliability: Color-Score and OCR-Score show negative correlations with human judgment in the TV Alignment aspect, limiting their reliability there.Detection-Score and Count-Score show relatively higher correlations and may be more useful for T2V alignment evaluation.
- Qualitative evaluation: Model performance differs across content types, motion, style, camera control, and generation tasks, motivating qualitative and multi-aspect comparisons.The qualitative analysis covers camera motion control, content generation, motion generation, style generation, and task-specific generation.
C.5. Task-Specific Generation
Task-specific evaluation reveals severe failures in some models, including noisy or distorted outputs, while user preferences can diverge from motion-only metrics. The benchmark uses a compact prompt set because sampling is slow and aims to track evolving expectations over time.
- Task-specific generation: Many models sometimes generate completely wrong task-specific videos with severe noise and distortions, especially ZeroScope, ModelScope, and Floor33 Pictures.The evaluated tasks include face generation, colored-object generation, counted-object generation, text generation, and activity generation.
- Benchmark design: The initial benchmark uses 700 prompts selected for diversity and balance because T2V sampling is typically slow.The authors report that objective metrics plateau as the number of prompts increases and that comparable benchmarks use similar prompt counts.
- Human evaluation: Human annotators perform 100 pre-labeling tasks, and those exceeding 90% accuracy are qualified to label the benchmark.Annotators below the qualification threshold are replaced by another supplier.
- Benchmark scope: The benchmark captures a snapshot of current user expectations and model abilities, with periodic updates intended to keep it dynamic.This defines the benchmark’s temporal scope rather than a fixed, permanent representation of user preferences.
- Motion preferences: Gen2 ranks 1st in user-rated motion quality despite ranking 7th on Motion AC-Score, reflecting preferences for subtle motion.The paper attributes this discrepancy mainly to users favoring subtle movements.
- Future evaluation: The authors describe pretrained models as an initial reference and plan to explore end-to-end evaluation models trained with more user opinions.This is presented as a direction for developing more straightforward metrics.
D.6. Costs of Evaluation
The benchmark is designed for offline, exhaustive evaluation rather than online monitoring. Its results are stable under added prompts and prompt noise, but the pipeline is computationally demanding and uses temporal-consistency measures such as warping error.
- Evaluation cost: Each generated video requires more than 2 minutes, making continuous online monitoring of T2V models impractical with this evaluation setup.The method is designed for offline evaluation rather than monitoring models during generation.
- Evaluation cost: The full benchmark requires around 2 hours on an A100 GPU and at least 16 GB of memory without code optimization.The authors characterize this as more demanding than traditional FVD evaluation.
- Benchmark stability: Objective scores remain stable as prompts increase, and prompt noise changes scores by less than 0.2 points in most cases.The tested noise includes adding, removing, or swapping words and symbols in 100 randomly selected prompts.
- Benchmark stability: Model rankings remain consistent across the tested prompt-noise conditions.This supports the reported robustness of the benchmark’s comparative results under those perturbations.
- Temporal consistency: Warping error measures the discrepancy between an actual subsequent frame and a frame predicted by warping the current frame with optical flow.Larger warping error indicates more dramatic frame changes, which are typically undesirable for real-world video consistency.
- Prompt-set expansion: The benchmark expanded from 512 to 700 prompts without significant changes in results, so the initial user-study labels were retained.The authors retained the earlier study to avoid wasting resources after observing stability following the prompt increase.