Source-linked AI summary
VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation
Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Duc Do, Yuansheng Ni, Bohan Lyu, Yaswanth Narsupalli, Rongqi Fan, Zhiheng Lyu, Yuchen Lin, Wenhu Chen
TL;DR
Automatic video metrics lack reliable, fine-grained human-aligned evaluation, partly because large-scale human annotations are costly and limited. The paper constructs VideoFeedback and trains VideoScore on it; VideoScore consistently outperforms existing metrics across held-out benchmarks, supporting its use as a proxy for human feedback.
Problem
Existing video metrics do not reliably score generated videos across multiple aspects, while large-scale human feedback is costly to obtain.
Method
The paper curates VideoFeedback from 37.6K videos generated by 11 text-to-video models and fine-tunes Mantis to train VideoScore for automatic multi-aspect assessment.
Results
VideoScore consistently achieves the best performance across VideoFeedback-test, EvalCrafter, GenAI-Bench, and VBench, including 77.1 Spearman correlation on VideoFeedback-test.
Takeaways & Limitations
VideoScore can serve as a proxy for human raters to evaluate video models and simulate fine-grained feedback for RLHF.
Takeaways & Limitations
Feature-based automatic metrics can perform well on one aspect while failing on others.
Abstract
from arXiv · showhide
The recent years have witnessed great advances in video generation. However, the development of automatic video metrics is lagging significantly behind. None of the existing metric is able to provide reliable scores over generated videos. The main barrier is the lack of large-scale human-annotated dataset. In this paper, we release VideoFeedback, the first large-scale dataset containing human-provided multi-aspect score over 37.6K synthesized videos from 11 existing video generative models. We train VideoScore (initialized from Mantis) based on VideoFeedback to enable automatic video quality assessment. Experiments show that the Spearman correlation between VideoScore and humans can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about 50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and VBench show that VideoScore has consistently much higher correlation with human judges than other metrics. Due to these results, we believe VideoScore can serve as a great proxy for human raters to (1) rate different video models to track progress (2) simulate fine-grained human feedback in Reinforcement Learning with Human Feedback (RLHF) to improve current video generation models.
1 Introduction
Existing video metrics do not reliably assess single generated videos across multiple fine-grained dimensions, motivating human-feedback-based evaluation. The paper introduces VideoFeedback and VideoScore, achieving substantially stronger agreement with human judgments across several benchmarks.
- Motivation: Current metrics miss important evaluation needs, including single-output assessment, multiple quality dimensions, and fine-grained aspect subscores.Examples include distribution-based FVD and IS, single-aspect metrics such as CLIP and DINO, and mean-opinion-score metrics such as T2VQA and DOVER.
- Contribution: VideoFeedback contains human-provided scores for 37.6K videos generated by 11 text-to-video models.Videos are annotated across five key aspects on a 1-to-4 scale.
- Contribution: VideoScore is trained by fine-tuning the Mantis-Idefics2-8B backbone on VideoFeedback-train to simulate human feedback for video evaluation.Mantis supports multi-image and video content, up to 128 frames, and native resolution.
- Results: 77.1 Spearman correlation is achieved on VideoFeedback-test, while 59.5 is achieved on EvalCrafter for text-to-video alignment.These results surpass the best baselines by 54.1 and 4.4, respectively.
- Results: 78.5 pairwise comparison accuracy is achieved on GenAI-Bench, and 72.1 average accuracy is achieved across five VBench aspects.These results exceed previous best baselines by 11.4 and 9.6, respectively.
2 Related Work
Video-generation evaluation has progressed from feature-based and human-rater approaches to multimodal-language-model assessment, but existing automatic metrics remain incomplete or weakly aligned with human judgment. VideoScore is positioned as a human-feedback approximation for reinforcement learning.
- Existing evaluation metrics: Text-to-video evaluation commonly uses FVD for video quality and CLIP for text-frame alignment, but these metrics do not capture subject, temporal, or factual consistency.VBench addresses some additional aspects, yet its correlation with human judgment is reported as relatively low.
- Existing evaluation metrics: VBench can substantially overestimate current text-to-video capabilities, with most models receiving subject/background consistency scores above 97%.EvalCrafter instead uses human raters for comprehensive evaluation.
- Learned and multimodal evaluators: VideoPhy and VIEScore prompt multimodal language models such as Gemini and GPT-4o for video quality assessment, but later study reports low agreement with human raters.T2VQA also trains on human-annotated ratings, while VideoFeedback differs through multiple aspects and larger scale.
- RLHF: VideoScore aims to approximate human feedback for use with video-generation RLHF methods such as PPO or DPO.The stated motivation is to support enhancement of video generative models.
3 VIDEOFEEDBACK
VIDEOFEEDBACK combines generated and real-world videos with multi-aspect expert annotations to support video-evaluator training. Its construction uses diverse prompts, 11 text-to-video models, annotation guidance, quality checks, and a held-out test set.
- Data preparation: The dataset construction covers prompt filtering, generation with 11 text-to-video models, multi-aspect annotation, supplementary data, and a 760-example test set.The pipeline begins with diverse text prompts and ends with summarized dataset statistics.
- Data preparation: VidProM supplies diverse user-derived prompts, filtered by length and NSFW probability before random down-sampling.The source contains 1.04 million unique prompts; prompts shorter than 5 or longer than 100 words are removed.
- Data preparation: The dataset includes videos from 11 models, normalized into a unified format, totaling 33.6K generated videos plus 4K real-world supplementary videos.Real-world videos come from DiDeMo and Panda70M.
- Annotation Pipeline: The five evaluation dimensions combine low-level visual quality with higher-level text-to-video alignment and factual consistency.Checklists and definitions are provided to improve rating accuracy and consistency.
- Annotation Pipeline: Twenty expert raters score videos across aspects using ratings from 1 (Bad) to 3 (Good), with 4 (Perfect) added through post-annotation and agreement checks.Rater training, pilot rounds, inter-annotator agreement analysis, and periodic reviews support annotation consistency.
- Annotation Pipeline: 37.6K final examples are assembled after post-augmentation, real-video supplementation, rating-distribution inspection, and reserving 760 validation examples as test data.The rating distribution is described as balanced except for Dynamic Degree, which the authors judged expected after case-study inspection.
4 Experiments
The experiments compare feature-based metrics, MLLM prompting baselines, and VideoScore across human-aligned video quality and preference benchmarks. VideoScore achieves stronger reported performance across VIDEOFEEDBACK-test, EvalCrafter, GenAI-Bench, and VBench, while ablations favor Mantis and regression scoring.
- Baselines: The study compares statistical or neural feature-based metrics with MLLM-based evaluators across multiple video dimensions and benchmarks.Feature-based baselines typically assess one dimension, whereas MLLM methods evaluate videos across multiple dimensions.
- Evaluation Benchmarks: VIDEOFEEDBACK-test evaluates label prediction accuracy and Spearman’s ρ using 760 entries, including 680 annotated and 80 augmented videos.Predictions for each aspect are compared with human labels for that same aspect.
- Evaluation Results: 54.1 average improvement over GPT-4o on VIDEOFEEDBACK-test and 4.4 improvement on EvalCrafter text-to-video alignment establish VideoScore’s strongest reported gains.The reported VIDEOFEEDBACK-test comparison covers all five aspects.
- Baselines: Feature-based metrics can perform well on one aspect yet correlate negatively on others; SSIM-dyn and MSE-dyn score 31.5 and 38.0 for dynamic degree but negative correlations elsewhere.PIQE, BRISQUE, CLIP-Score, and X-CLIP-Score also show nearly all negative correlations across five aspects.
- Best-of-K Sampling with VIDEOSCORE: Selecting the highest-average VideoScore video among five candidates improves most EvalCrafter scores compared with random sampling.The best-of-5 procedure averages VideoScore across five dimensions before selection.
- Ablation Study: Changing the backbone to Mantis-Idefics2-8B raises average benchmark scores from 47.5 to 55.6, and regression scoring adds 12.1 points over the Idefics2-based version.The Mantis-based version is selected as the final model.
- Ablation Study: Regression scoring consistently improves benchmark correlation or pairwise accuracy, reaching 78.5 on GenAI-Bench versus 59.0 for generative scoring.Regression scoring produces more fine-grained scores than four-label generative scoring.
5 Conclusion
VideoScore is trained on the curated VideoFeedback dataset to evaluate generated videos and simulate human feedback. The authors report testing it across multiple benchmarks and evaluation dimensions.
- VideoScore is trained on VideoFeedback for automatic evaluation of generated videos and simulation of human feedback.VideoFeedback contains annotations for 37.6K videos generated from 11 text-to-video models across five aspects.
- The dataset uses 20 expert raters to score videos on Visual Quality, Temporal Consistency, Dynamic Degree, Text-to-Video Alignment, and Factual Consistency.The reported inter-annotator agreement match ratio exceeds 60%.
- VideoScore is evaluated with Spearman correlation on VideoFeedback-test and EvalCrafter, and pairwise comparison accuracy on GenAI-Bench and VBench.These benchmarks assess agreement with human judgments using complementary evaluation protocols.
- VideoScore consistently achieves the best reported performance, surpassing GPT-4o and Gemini 1.5 Flash/Pro by a large margin.The authors attribute the results to MLLM-based evaluation, rich world knowledge, and a fine-grained multi-dimensional rating dataset.
A Ethical Statement
The paper states that it complies with the ACL Ethics Policy and reports no ethical issues to the authors’ knowledge.
- The authors declare full compliance with the ACL Ethics Policy and report no known ethical issues.
B Risks and Limitation
The paper identifies annotation quality and occasional scoring errors as limitations of VideoFeedback and VideoScore. It also cautions that the reported inter-annotator agreement may not represent the full annotation set.
- Some expert annotations contain errors that may reduce the overall quality of VideoFeedback.
- The inter-annotator agreement score is based only on a small set of trial examples and may not represent the complete annotation set.
- VideoScore can sometimes produce scores that do not match the authors’ expectations.The authors list improving these scoring errors as future work.
C Dataset Licence
The paper releases VideoFeedback under the MIT license and documents the data sources, preprocessing, annotation, and scoring procedures used in the experiments.
- Dataset licence: Prompts are collected from VidProM under CC BY-NC 4.0, while GenAI-Bench and VBench use MIT and Apache 2.0 licenses, respectively.The authors report finding no license for EvalCrafter’s human annotations.
- Dataset licence: VideoFeedback is released under the MIT license.The release is intended to contribute to video evaluation datasets.
- Annotation and scoring: Twenty expert raters annotate the videos using interfaces, definitions, error checklists, and quality examples across five evaluation dimensions.The scoring prompts instruct the regression model to output a float from 1.0 to 4.0.
- Data processing: All generated videos are normalized to 8 fps through down-sampling or frame interpolation, depending on the source model.Pika and AnimateDiffusion are down-sampled, while Text2Video-Zero is interpolated from 4 fps to 8 fps.
- Data processing: Pika videos are cropped from 1088×640 to 768×480 to remove their watermark and match the format of videos from other models.
H Feature-based Baselines Discretization
The paper converts continuous baseline-metric outputs into four discrete labels to match the annotation format, while presenting the annotation interface used for human scoring.
- H Feature-based Baselines Discretization: Baseline metrics produce continuous float outputs that are discretized into labels [1, 2, 3, 4] to align with the annotation format.Table 12 specifies the discretization rules for feature-based baselines.
- H Feature-based Baselines Discretization: The annotation website includes definitions, error-point checklists, and diverse video examples for raters.
- H Feature-based Baselines Discretization: The website provides a dedicated working page for video annotation.
I Case study of VIDEOFEEDBACK
The case studies illustrate how videos receive scores across five aspects and reveal differing relationships among those aspects. Clear and distorted videos receive broadly consistent ratings, while dynamic degree behaves differently from the other dimensions.
- I Case study of VIDEOFEEDBACK: A clear video scores 3 across all five aspects, whereas a distorted video scores 1 on every aspect except dynamic degree.
- I Case study of VIDEOFEEDBACK: Visual quality correlates 0.6 with temporal consistency, while dynamic degree has very low correlation with the other aspects.
- I Case study of VIDEOFEEDBACK: Each video is paired with a text description and rated across five aspects.
J Leaderboard
The leaderboard evaluates existing text-to-video models using VIDEOSCORE on curated examples, while related materials document annotation error cases, prompting templates, and baseline discretization.
- J Leaderboard: The evaluation generates 200 videos from various text-to-video models and ranks models by their average score across five VIDEOSCORE dimensions.
- J Leaderboard: The supplementary materials include expected and error cases for annotation aspects, prompting templates, and discretization rules for feature-based baselines.
- J Leaderboard: The leaderboard ranks existing text-to-video models using VIDEOSCORE on 200 curated examples.