Source-linked AI summary
Video-Bench: Human-Aligned Video Generation Benchmark
Hui Han, Siyuan Li, Jiaqi Chen, Yiwen Yuan, Yuling Wu, Chak Tou Leong, Hanwen Du, Junchen Fu, Youhua Li, Jie Zhang, Chi Zhang, Li-jia Li, Yongxin Ni
TL;DR
Video-Bench addresses the gap between automated video-generation evaluation and human preferences, especially for cross-modal alignment and video-quality judgments. It combines a comprehensive benchmark with MLLM-based chain-of-query and few-shot scoring, achieving the highest reported correlation with human preferences and agreement on par with human raters. The framework is efficient and promising, but its accuracy remains constrained by MLLMs’ ability to perceive dynamic elements and fine-grained details.
Problem
Existing metrics, embeddings, and LLM-based benchmarks do not comprehensively align with human preferences because they struggle with cross-modal consistency or video-quality assessment.
Method
Video-Bench evaluates generated videos with an MLLM framework using iterative chain-of-query analysis and multimodal few-shot scoring across comprehensive dimensions.
Results
Video-Bench achieves the highest correlation with human preferences among compared evaluation methods, with human-score agreement averaging 0.52 and an average absolute mean difference of 0.18.
Takeaways & Limitations
Video-Bench provides an automatic, human-aligned benchmark for evaluating video-generation models across multiple dimensions.
Takeaways & Limitations
The MLLM-based framework is constrained in perceiving dynamic elements and capturing fine-grained details.
Abstract
from arXiv · showhide
Video generation assessment is essential for ensuring that generative models produce visually realistic, high-quality videos while aligning with human expectations. Current video generation benchmarks fall into two main categories: traditional benchmarks, which use metrics and embeddings to evaluate generated video quality across multiple dimensions but often lack alignment with human judgments; and large language model (LLM)-based benchmarks, though capable of human-like reasoning, are constrained by a limited understanding of video quality metrics and cross-modal consistency. To address these challenges and establish a benchmark that better aligns with human preferences, this paper introduces Video-Bench, a comprehensive benchmark featuring a rich prompt suite and extensive evaluation dimensions. This benchmark represents the first attempt to systematically leverage MLLMs across all dimensions relevant to video generation assessment in generative models. By incorporating few-shot scoring and chain-of-query techniques, Video-Bench provides a structured, scalable approach to generated video evaluation. Experiments on advanced models including Sora demonstrate that Video-Bench achieves superior alignment with human preferences across all dimensions. Moreover, in instances where our framework's assessments diverge from human evaluations, it consistently offers more objective and accurate insights, suggesting an even greater potential advantage over traditional human judgment.
1. Introduction
Video generation advances create an urgent need for benchmarks that assess both video quality and alignment with human instructions. Existing metrics, embeddings, and LLM-based methods each face limitations in comprehensively matching human preferences.
- Motivation: Humans prefer videos that align with input conditions and exhibit strong aesthetic quality and temporal consistency.The paper identifies video-condition alignment and video quality as central dimensions of human video preferences.
- Existing benchmarks: Existing benchmarks combine metrics such as IS, FID, and FVD with CLIP or BLIP embeddings to evaluate video quality and video-text alignment.These approaches represent one major category of automated video-generation benchmarks.
- Existing benchmarks: Metrics- and embedding-based benchmarks provide quantitative measurements but often misalign significantly with human preferences.This limitation motivates alternatives that better reflect human judgments.
- Existing benchmarks: LLM-based benchmarks can imitate human evaluation logic but struggle with cross-modal video-condition comparisons and video-quality assessment.The paper highlights these as two critical limitations of current LLM-based approaches.
- Proposed benchmark: Video-Bench uses an MLLM-based framework with chain-of-query and few-shot scoring to evaluate videos automatically across comprehensive dimensions.The framework is designed to improve direct video comprehension and alignment with human preferences.
2. Related work
Prior video-generation benchmarks primarily use metrics, model-based measures, or LLM-based evaluation to score multiple quality and alignment dimensions. LLM-based methods increasingly use descriptions and question-answering, but Video-Bench is positioned against these existing strategies.
- LLM-based evaluation: LLM-based evaluation methods include single-score grading, pairwise comparison, description-based consistency reasoning, and aspect-level scoring.Examples include GPT-4v Eval, LLMScore, and VIEScore.
- LLM-based evaluation: Many LLM-based approaches formulate evaluation as question generation and answering over visual content and specified viewpoints or temporal coverage.This reflects a broader trend toward decomposing evaluation into targeted questions.
- Traditional benchmarks: Traditional text-to-video benchmarks use metric-based or model-based evaluation across video quality, motion, temporal consistency, and text-video alignment.VBench and EvalCrafter are examples of benchmarks that score generated videos across multiple dimensions.
- Traditional benchmarks: EvalCrafter fits linear regression models between computed metrics and user scores, whereas VBench calculates Pearson and Spearman correlations.These methods differ in how they relate automated measurements to human ratings.
3. Benchmark
Video-Bench defines video-generation quality through video quality and video-condition alignment, covering object, action, color, scene, text, imaging, aesthetic, temporal, and motion-related dimensions. Its prompt suite contains 419 prompts targeting both static and dynamic properties.
- Benchmark dimensions: Video-generation quality is divided into video quality and video-condition alignment.Video quality concerns the video itself, while condition alignment concerns compliance with human instructions.
- Video-condition alignment: Video-condition alignment covers object, action, color, scene, and overall video-text consistency.Most consistency dimensions use three-point scales, while overall video-text consistency uses a five-point scale.
- Video quality: Video quality covers imaging quality, aesthetic quality, temporal consistency, and motion-related properties.These dimensions assess frame artifacts, visual appeal, cross-frame stability, and realistic movement.
- Video quality: Temporal consistency evaluates smooth visual transitions and stable semantic features across consecutive frames.The assessment considers color, brightness, texture, objects, subjects, and scenes.
- Prompt suite: The prompt suite contains approximately 70-90 prompts per dimension and 419 prompts overall.Dynamic dimensions combine Kinetics-400 action data with rigid-body and animal-motion data from VBench.
4. Evaluation Framework with MLLMs
Video-Bench addresses MLLM evaluation challenges by converting video content into richer textual representations and calibrating scores with multimodal examples. Its chain-of-query process supports iterative, dimension-specific assessment rather than single-turn judgment.
- Challenges: Direct MLLM scoring struggles with cross-modal alignment and ambiguous video-quality criteria.The model may miss details, hallucinate, or assign similar average scores across videos.
- Chain-of-query: Chain-of-query first converts video content into text, enabling more consistent comparison with textual prompts.This avoids direct cross-modal comparison as the initial alignment operation.
- Few-shot scoring: Few-shot scoring uses multiple multimodal examples to calibrate evaluation direction and scale.Compared with assigning one direct score, the strategy supports more nuanced distinctions from poor to good.
- Chain-of-query: The chain iteratively generates a video description, dimension-specific queries, refined answers, and a final score using conversation history and guidelines.The process probes overlooked details and inconsistencies before scoring.
5. Experiment
The experiments evaluate multiple video-generation models against human ratings and define the benchmark’s leaderboard and comparison protocol.
- Models and evaluation setup: Seven video-generation models spanning open-source and commercial systems are evaluated across different development stages.The passage lists LaVie, Show-1, VideoCrafter2, CogVideoX-5B, Pika-Beta, Kling, and Gen3.
- Human evaluation: 35,196 human evaluations are collected from 10 annotators, with four scores per generated video and expert quality control.Human and MLLM evaluations use the same scale and guidelines.
- Evaluation models: GPT-4o processes multimodal text and video-frame inputs, whereas GPT-4o-mini receives text prompts only.The comparison distinguishes multimodal MLLM evaluation from text-only LLM evaluation.
- Leaderboard: The Video-Bench leaderboard reports higher-is-better dimension scores and a lower-is-better average rank across multiple dimensions.The best score in each dimension is highlighted.
6. Main results
Video-Bench achieves the strongest reported alignment with human ratings and reaches human-level agreement on average across evaluation dimensions.
- Comparison with existing methods: Video-Bench achieves the highest Spearman correlation with human ratings among the compared evaluation methods.The comparison includes EvalCrafter, VBench, and ComBench on the Video-Bench prompt suite.
- Comparison with existing methods: Video-Bench’s human-alignment results show stronger correlations with metric-based evaluation than other LLM-based methods across all dimensions.This comparison is reported across the benchmark’s evaluation dimensions.
- Human preference alignment: 0.52 average agreement with human scores is on par with inter-rater agreement among humans.The paper also reports an average absolute mean difference of 0.18 across dimensions after bootstrapping 1000 iterations over 100k score pairs.
- Human preference alignment: Human evaluations show low agreement on semantic-consistency dimensions.The paper discusses this pattern in relation to the five Video-Condition Alignment dimensions.
Video-Condition Alignment
The benchmark studies component effectiveness, evaluation stability, robustness, model-version behavior, and performance across prompt complexity within Video-Condition Alignment.
- Evaluation stability: 67% of videos receive the same rating across three repeated runs, with Krippendorff’s α reaching 0.867.The results indicate high agreement across identical runs despite imperfect exact-rating agreement.
- Robustness: Gaussian blur produces less than 5% relative percentage error under the video-text consistency dimension.This tests robustness to small input variations that are hardly visible to humans.
- Comparing versus rating paradigms: Few-shot scoring raises average human-rating correlation by 10.33% across dimensions without increasing runtime complexity.It provides relative quality information across models while avoiding the position bias and higher cost associated with pairwise comparison.
- Ablation study: Adding each proposed component significantly increases human-alignment scores across all dimensions, while more reference videos further improve performance.The ablation results support the effectiveness of the benchmark’s components.
- Base-model comparison: GPT-4o-0806 reaches 0.807 Imaging Quality and 0.750 Video-text Consistency, while GPT-4o-1120’s Motion Effects falls to 0.309 from 0.469.The results indicate that newer GPT-4o versions do not consistently improve performance.
- Prompt complexity: Gen3, Kling, and Pika show consistent performance on both simple and complex prompts, including MovieGenBench prompts.The comparison is presented as evidence of robustness across prompt lengths and complexities.
7. Conclusion
Video-Bench is presented as a human-aligned benchmark using MLLM evaluations to assess generated videos. The conclusion emphasizes efficiency, strong human alignment, and the value of few-shot and chain-of-query designs for future video-generation research.
- Video-Bench uses Multi-Modal Large Language Models to provide a human-aligned benchmark for video generation.
- The benchmark’s experiments and human alignment study demonstrate advantages in evaluation efficiency and agreement with human preferences.
- Few-shot and chain-of-query technologies are identified as important component-design choices for improving automatic evaluations.
- Video-Bench is intended to support future research on video generation model development through MLLM-based visual evaluation.
Supplementary Material
The supplementary material expands the evaluation dimensions, experimental analysis, societal implications, and limitations of Video-Bench.
- Evaluation dimensions: Section A provides detailed explanations and examples of the evaluation dimensions.
- Experimental analysis: Section B presents extended experimental results and analysis.
- Implications and limitations: Sections C and D discuss broader societal implications, current limitations, and directions for future research.
A. More Details on Evaluation Dimension
The supplementary evaluation dimensions define prompt-consistency criteria for objects, actions, colors, and scenes, using three-point scales and explicit boundary cases. The criteria emphasize faithful, stable prompt realization while distinguishing consistency from visual quality.
- Object class consistency: Object class consistency measures whether prompted objects are correctly generated, identifiable, and faithful in appearance and structure throughout the video.
- Object class consistency: Object scores range from poor when objects fail to match the prompt, through moderate partial or unstable generation, to good when classes remain correct throughout.
- Action consistency: Action consistency evaluates whether prompted actions are correctly identifiable and realistically represented across their progression and temporal coverage.
- Color consistency: Color consistency assesses prompt matching, stability, placement, distribution, and distinction, with good scores requiring faithful colors without sudden changes or inconsistencies.
- Scene consistency: Scene consistency measures whether prompted scenes are identifiable and faithfully structured, with ambiguous terms judged using the most comprehensive generated interpretation.
- Important notes: “Most” generally means N −1 of up to five key prompt elements, while superior visual quality is not required for a high consistency score.
B.1. Statistical Analysis of Evaluation Discrepancy
The discrepancy analysis finds statistically significant differences between MLLM and human evaluations across all dimensions. The framework is stricter and more stable in many areas, while motion effects reveal a complementary role for human perception.
- Video quality metrics: 0.31 is the highest positive mean difference among video quality metrics, occurring for temporal consistency.
- Video quality metrics: Motion effects show the only negative contrast among the reported video quality metrics, highlighting complementary machine and human evaluation capabilities.
- Alignment metrics: Video-text consistency (0.19) and action consistency (0.22) have the largest positive differences among alignment metrics, while all 99% confidence intervals exclude zero.
- Statistical significance: Object-class consistency [0.03, 0.05] and color consistency [0.04, 0.06] have narrow confidence intervals, indicating stable evaluations in these aspects.
B.2. Mini-split for quick performance evaluation
The mini-split evaluation finds Sora highly competitive across nine metrics, while case studies identify persistent weaknesses in video-text consistency, temporal coherence, and motion generation. The evaluation also highlights specialized semantic distinctions that MLLM assessment can detect beyond typical human judgments.
- Overall results: Sora ranked first in 5 of 9 evaluated metrics and remained competitive across the others.The test used 25 representative prompts and generated 5-second, 720p videos.
- Evaluation setup: The evaluation used a mini-split scheme to provide a quick performance assessment of video generation models.The study selected about one-third of the dataset as 25 representative prompts.
- Overall results: Sora matched or exceeded Gen-3 and CogVideoX on all Video Quality metrics and Video-Text Consistency.It also performed strongly on Motion Effects and most Video-Condition Alignment metrics, except Video-Text Consistency.
- Limitations: Case studies exposed limitations in video-text consistency, temporal frame coherence, and motion generation.Reported issues include content mismatch, unnatural frame transitions, and inconsistent motion trajectories.
- MLLM assessment: MLLM assessment identified semantic distinctions that human evaluators often overlooked, including skiing versus snowboarding and air drumming versus drum playing.The framework is described as offering stronger discrimination in specialized content assessment.
D. Limitations and Future Work
Video-Bench’s MLLM-based evaluation remains limited in perceiving dynamic elements, fine-grained visual details, complex temporal relationships, and subtle visual nuances. Future work targets differentiable metrics, optimization mapping, stronger temporal understanding, and more capable efficient architectures.
- Limitations: MLLM-based evaluation is constrained in perceiving dynamic elements and capturing fine-grained details.These constraints limit evaluation accuracy when videos contain complex temporal relationships or subtle visual nuances.
- Limitations: Evaluation accuracy may inherit biases from the underlying MLLMs’ pre-training data and architectural design.
- Future Work: Future work proposes differentiable metrics, robust mappings from evaluations to optimization strategies, and adaptive feedback mechanisms.
- Future Work: Improving MLLMs’ temporal understanding is identified as a direction for assessing dynamic elements and temporal coherence.
- Future Work: More sophisticated architectures should capture global temporal dynamics and local visual details while maintaining computational efficiency.The paper anticipates improved evaluation accuracy and reliability as MLLM technology evolves.