VideoScore2: Think before You Score in Generative Video Evaluation

Xuan He; Dongfu Jiang; Ping Nie; Minghao Liu; Zhengxuan Jiang; Mingyi Su; Wentao Ma; Junru Lin; Chun Ye; Yi Lu; Keming Wu; Benjamin Schneider; Quy Duc Do; Zhuofeng Li; Yiming Jia; Yuxuan Zhang; Guo Cheng; Haozhe Wang; Wangchunshu Zhou; Qunshu Lin; Yuanxing Zhang; Ge Zhang; Wenhao Huang; Wenhu Chen

Source-linked AI summary

VideoScore2: Think before You Score in Generative Video Evaluation

Xuan He, Dongfu Jiang, Ping Nie, Minghao Liu, Zhengxuan Jiang, Mingyi Su, Wentao Ma, Junru Lin, Chun Ye, Yi Lu, Keming Wu, Benjamin Schneider, Quy Duc Do, Zhuofeng Li, Yiming Jia, Yuxuan Zhang, Guo Cheng, Haozhe Wang, Wangchunshu Zhou, Qunshu Lin, Yuanxing Zhang, Ge Zhang, Wenhao Huang, Wenhu Chen

arXiv:2509.22799v1Sep 26, 2025cs.CVcs.AIcs.CL

TL;DR

Video evaluation must jointly assess visual fidelity, prompt alignment, and physical plausibility, but existing evaluators often provide opaque scores and generalize poorly beyond their training data. VideoScore2 addresses this gap with interpretable multidimensional scoring and reasoning, achieving stronger in-domain and out-of-domain benchmark performance while supporting Best-of-N generation.

Problem
Video evaluation requires jointly assessing visual fidelity, semantic prompt alignment, and physical plausibility, while prior evaluators often collapse judgments into opaque scores and rely on supervised training that limits out-of-domain generalization.
Method
VideoScore2 predicts visual quality, text alignment, and physical/common-sense consistency with detailed rationales, using the VideoFeedback2 dataset and supervised fine-tuning followed by GRPO reinforcement learning.
Results
VideoScore2 outperforms existing evaluators across in-domain and out-of-domain benchmarks, achieving 44.35 (+5.94) point-score accuracy in-domain and 50.37 (+4.32) average performance across four out-of-domain benchmarks.
Takeaways & Limitations
VideoScore2 provides interpretable evaluation and shows potential as a reward model for controllable text-to-video generation through Best-of-N sampling.
Takeaways & Limitations
Checkpoint selection assumes that benchmark performance must be balanced with training-loss stability when initializing reinforcement learning.

Abstract

from arXiv · show

Recent advances in text-to-video generation have produced increasingly realistic and diverse content, yet evaluating such videos remains a fundamental challenge due to their multi-faceted nature encompassing visual quality, semantic alignment, and physical consistency. Existing evaluators and reward models are limited to single opaque scores, lack interpretability, or provide only coarse analysis, making them insufficient for capturing the comprehensive nature of video quality assessment. We present VideoScore2, a multi-dimensional, interpretable, and human-aligned framework that explicitly evaluates visual quality, text-to-video alignment, and physical/common-sense consistency while producing detailed chain-of-thought rationales. Our model is trained on a large-scale dataset VideoFeedback2 containing 27,168 human-annotated videos with both scores and reasoning traces across three dimensions, using a two-stage pipeline of supervised fine-tuning followed by reinforcement learning with Group Relative Policy Optimization (GRPO) to enhance analytical robustness. Extensive experiments demonstrate that VideoScore2 achieves superior performance with 44.35 (+5.94) accuracy on our in-domain benchmark VideoScore-Bench-v2 and 50.37 (+4.32) average performance across four out-of-domain benchmarks (VideoGenReward-Bench, VideoPhy2, etc), while providing interpretable assessments that bridge the gap between evaluation and controllable generation through effective reward modeling for Best-of-N sampling. Project Page: https://tiger-ai-lab.github.io/VideoScore2/

1 INTRODUCTION

VideoScore2 addresses the difficulty of evaluating generated videos across visual, semantic, and physical dimensions while overcoming opaque scoring and limited out-of-domain generalization. It combines interpretable multi-dimensional reasoning with a two-stage training pipeline and achieves strong in-domain and out-of-domain results.

1 INTRODUCTION: Existing video evaluators can score accurately but commonly collapse judgments into a single opaque score without rationale, while supervised-only training limits out-of-domain generalization.These limitations motivate interpretable evaluation and reinforcement learning beyond direct supervised fine-tuning.
1 INTRODUCTION: VideoScore2 evaluates visual quality, text alignment, and physical/common-sense consistency with structured scores and detailed chain-of-thought-style analyses.The design is intended to make judgments interpretable and human-aligned.
1 INTRODUCTION: VIDEOFEEDBACK2 combines multi-dimensional scores and reasoning traces from diverse prompts and videos generated by more than twenty T2V models.Annotations include scores and diagnostic comments that are expanded into detailed rationales.
1 INTRODUCTION: The two-stage pipeline uses cold-start supervised fine-tuning for structured outputs and basic reasoning, followed by GRPO reinforcement learning for analytical robustness and human-preference alignment.This ordering is presented as the training strategy for VideoScore2.
1 INTRODUCTION: 44.35 (+5.94) point-score accuracy and 50.37 (+4.32) average performance across four out-of-domain benchmarks establish VideoScore2’s reported gains.It also reaches 90.78 (+4.01) relaxed accuracy and 60.37 (+8.32) PLCC on VideoScore-Bench-v2.

2 RELATED WORKS

Prior work advances text-to-video generation, reward modeling, and video-language reasoning, but existing reward models remain limited in dimensionality, explanatory reasoning, or dataset recency. VideoScore2 is positioned against these limitations through multi-dimensional evaluation with rationale support.

2 RELATED WORKS: Text-to-video generation progressed from open-source diffusion pipelines to larger architectures with improved temporal fidelity, resolution, and narrative coherence.The cited progression includes ModelScope, VideoCrafter2, CogVideoX, and industrial-scale systems.
2 RELATED WORKS: Existing reward models range from single scalar scores to multi-dimensional ratings, but the latter still generally lack explanatory reasoning.The comparison motivates VideoScore2’s combination of multiple evaluation dimensions and rationales.
2 RELATED WORKS: Video understanding research has established transformer-based models and video-language systems as foundations for multimodal temporal reasoning.The related work includes Video Swin Transformer, InternVideo, Video-LLaMA, Video-LLaVA, and mPLUG-Owl-V.
2 RELATED WORKS: The paper compares reward models by multi-dimensional scoring, rationale support, and dataset recency.These are the comparison dimensions identified in Table 1.

3 DATASET CURATION

The dataset combines filtered and manually designed prompts with videos from diverse T2V models, then collects three-dimensional human annotations and expands comments into aligned rationales. The resulting 27,168-sample resource supports SFT and RL training.

3 DATASET CURATION: Prompts combine existing datasets with manually designed multi-action, OCR-text, and camera-motion cases targeting known T2V weaknesses.Rule-based and LLM semantic filtering remove unsuitable prompts before collection.
3 DATASET CURATION: Videos come from over 20 T2V models grouped into four quality tiers, with ten models sampled per prompt to balance quality levels and enable same-content comparisons.This design improves scoring consistency across videos sharing semantic content.
3 DATASET CURATION: Annotations score visual quality, text alignment, and physical/common-sense consistency on a 1–5 scale, accompanied by comments and supported by dimension-specific checklists.Fifteen trained annotators completed pilot rounds and received reviewer feedback.
3 DATASET CURATION: Claude-4-Sonnet generates rationales from instructions, sampled frames, comments, and examples, while score reconciliation and GPT-5-mini alignment preserve consistency.Fewer than 10% of entries were discarded after resampling.
3 DATASET CURATION: 27,168 processed samples form VIDEOFEEDBACK2, with 500 videos held out for VIDEOSCORE-BENCH-V2 and the remainder used for training.The four quality tiers contain 10.36%, 33.53%, 41.77%, and 12.54% of videos, respectively.

4 VIDEOSCORE2

VIDEOSCORE2 is trained and evaluated as a multi-dimensional video evaluator using SFT initialization followed by GRPO-based reinforcement learning. It is tested across in-domain and out-of-domain preference and point-score benchmarks, with ablations examining rationales, initialization, and score format.

Training and Inference Setup: The training pipeline uses SFT for structured output and task familiarity, followed by GRPO reinforcement learning to strengthen analytical robustness and human alignment.The final reward combines accuracy and format terms, with the format coefficient adjusted depending on whether training starts from SFT or the base model.
Evaluation Setup: The evaluation spans pairwise preference and point-score benchmarks, using more than ten baselines that include prompted multimodal LLMs and vision reward or scoring models.Pairwise tasks compare video quality, whereas point-score tasks assess numerical agreement with ground-truth scores.
Evaluation Results: VIDEOSCORE2 surpasses the best baseline across all VIDEOSCORE-BENCH-V2 dimensions and metrics, while achieving the highest overall average across four out-of-domain benchmarks.The evaluation covers accuracy, relaxed accuracy, PLCC, pairwise preference, and point-score tasks.
Ablation Study: SFT initialization produces higher average performance across in-domain and out-of-domain benchmarks than starting reinforcement learning from the base model.The advantage is not universal on every benchmark, but the SFT checkpoint provides a stronger average starting point.
Ablation Study: Rationale-based SFT is slightly weaker on preference benchmarks but significantly better on point-score benchmarks, improving average generalization.The ablation links rationales to both interpretability and overall robustness.
Ablation Study: Normalized float scores balance performance across tasks by preserving point-score accuracy while capturing finer quality differences in preference settings.Integer scores have a slight advantage on out-of-domain point-score benchmarks but perform notably worse on preference tasks.

5 CONCLUSION

The conclusion presents VIDEOSCORE2 as a multi-dimensional, interpretable, human-aligned evaluator trained through a two-stage paradigm. It reports gains across in-domain and out-of-domain benchmarks while identifying physics and common-sense evaluation as an ongoing challenge.

Conclusion: VIDEOSCORE2 combines multi-dimensional video evaluation with interpretable rationales and human-aligned training.The framework is supported by an annotation pipeline containing diverse prompts, generated videos, scores, and rationales.
Conclusion: The model outperforms existing evaluators across in-domain and out-of-domain benchmarks, supporting its use for trustworthy evaluation and human-aligned generative-video training.The conclusion describes this as a path toward more trustworthy evaluation and training.
Conclusion: VIDEOSCORE2 still struggles with physics and common-sense evaluation, motivating future work on a world model for video evaluators.This limitation is explicitly identified as a future research direction.

ETHICS STATEMENT

The study reports compliance-oriented data handling, including privacy protections, safe prompt filtering, and trained annotation procedures. It states that no human or animal experimentation and no personally identifiable information were involved.

Ethics Statement: The study used no human subjects or animal experimentation and processed datasets under relevant usage guidelines.The statement specifically includes the curated VIDEOFEEDBACK2 dataset.
Ethics Statement: Prompt filtering excluded NSFW, harmful, inappropriate, personally identifiable, and sensitive content from the retained data.The retained prompts were restricted to non-sensitive and safe content.
Ethics Statement: The authors report that no personally identifiable information was collected or used and that annotators were trained to avoid biased or discriminatory judgments.The statement frames transparency, fairness, and integrity as annotation commitments.

A DATA COLLECTION AND PROCESSING

The dataset is assembled by filtering prompts, adding targeted scenarios for OCR text, multi-action sequences, and camera motion, and sampling videos from diverse T2V models. Annotation guidelines then organize quality judgments around visual quality, text alignment, and physical/common-sense consistency.

Prompt Processing: Prompts are filtered for safety, clarity, length, semantic suitability, and video-oriented motion before inclusion.Filters remove NSFW or harmful content, vague prompts, named people, image-like descriptions, excessive actions, and unsuitable caption lengths.
Prompt Sources: The collection adds challenging OCR-text, multi-action, and camera-motion scenarios through manually designed or LLM-expanded prompts.The OCR-text and multi-action categories each include 200 prompts, while camera-motion prompts append explicit movement instructions.
Video Generation: Videos are generated by more than twenty T2V models and sampled across four quality tiers to provide a broad quality spectrum.Each prompt receives ten videos from different models with a balanced distribution across tiers.
Dataset Documentation: The dataset documentation records video sources and properties such as frame rate, resolution, and duration.These details are summarized in the dataset information table.
Annotation Guidelines: Annotators assign dimension-specific scores and comments, with guidelines defining severe visual flaws, missing prompt elements, and physical or common-sense violations.Videos rated five may omit comments, while entirely black or unrecognizable videos are skipped.

B EVALUATION SUITE

The evaluation suite aligns heterogeneous benchmarks and baseline outputs with VideoScore2’s three dimensions and a common scoring range. It covers preference and point-score settings across visual quality, text alignment, and physical consistency.

B.1 DIMENSION MATCHING AND MODIFICATION IN OUT-OF-DOMAIN BENCHMARKS: VideoGenReward Bench evaluates pairwise preferences using visual quality, text alignment, and overall preference, with results reported both with and without ties.Its motion-quality dimension is not directly used because the first two dimensions align most closely with VideoScore2.
B.1 DIMENSION MATCHING AND MODIFICATION IN OUT-OF-DOMAIN BENCHMARKS: VideoScore2 aligns benchmark dimensions and baseline outputs to visual quality, text alignment, and physical consistency within a unified integer range of 1–5.The framework uses dimension matching and score rescaling to make heterogeneous evaluators comparable.
B.2 DIMENSION MATCHING AND SCORE RESCALING FOR BASELINES: Baseline compatibility uses broadcasting for single scores, direct reuse for good matches, and customized mappings for partially overlapping dimensions.Customized mappings can average fine-grained outputs, skip unavailable dimensions, or assign matched dimensions selectively.
B.2 DIMENSION MATCHING AND SCORE RESCALING FOR BASELINES: Scores without fixed bounds are converted into five ordinal categories using 20%, 40%, 60%, and 80% Gaussian quantile thresholds.The procedure defines z as the raw model score and Φ^-1 as the inverse standard-Gaussian CDF.

C.2 FULL RESULTS ON MJ-BENCH-VIDEO

MJ-Bench-Video is evaluated through three selected aspects aligned with VideoScore2’s dimensions, while Video-Phy2-Test directly covers semantic adherence and physical consistency. The section reports prediction accuracy and PLCC against human ground truths.

C.2 FULL RESULTS ON MJ-BENCH-VIDEO: MJ-Bench-Video selects fineness, alignment, and coherence & consistency because they correspond to visual quality, text alignment, and physical/common-sense consistency.Overall scores are also evaluated after the selected dimensions are aligned.
C.3 FULL RESULTS ON VIDEOPHY2-TEST: Video-Phy2-Test contains semantic adherence and physical consistency, which directly correspond to VideoScore2’s latter two evaluation dimensions.The reported metrics are prediction accuracy and PLCC between model outputs and ground truths.

D.2 ABLATION ON SAMPLING FPS IN SFT TRAINING

The sampling-rate ablation compares 2, 4, and 8 fps for SFT training. Increasing the rate does not significantly improve performance but increases computational cost and training time, motivating 2 fps as the default.

D.2 ABLATION ON SAMPLING FPS IN SFT TRAINING: 2 fps is adopted for SFT because global visual properties, semantic adherence, and most physical abnormalities remain capturable at that rate.The authors state that most relevant physical or abnormal events last longer than half a second.
D.2 ABLATION ON SAMPLING FPS IN SFT TRAINING: Higher sampling rates do not yield significant performance gains while noticeably increasing computational cost and training time.The comparison uses a 17k-subset ablation across 2, 4, and 8 fps.

D.3 ABLATION ON LEARNING RATE AND EPOCHS IN SFT TRAINING

The SFT ablation studies learning rate and epoch count using in-domain accuracy and loss behavior. The selected configuration is 5e-5 for two epochs, balancing accuracy, loss smoothness, and resistance to overfitting concerns.

D.3 ABLATION ON LEARNING RATE AND EPOCHS IN SFT TRAINING: 5e-5 is selected as the default learning rate because 1 × 10^-4 is slightly more accurate but less stable, while 2e-5 indicates underfitting.The less-stable 1 × 10^-4 loss curve suggests potential overfitting that could harm out-of-domain performance.
D.3 ABLATION ON LEARNING RATE AND EPOCHS IN SFT TRAINING: The 2-epoch setting outperforms both 1 and 3 epochs and becomes the main SFT version.This checkpoint initializes subsequent RL cold-start training.

D.4 ABLATION ON INFERENCE SETTINGS

The ablations identify inference settings that improve evaluation reliability, with performance favoring sparse frame sampling and a specific RL checkpoint.

Inference sampling rates: 2 fps achieves the best VIDEOSCORE-BENCH-V2 performance among the tested 2, 4, and 8 fps sampling rates.The authors attribute this to sufficient issue coverage with less redundant or noisy information.
RL training steps: 300 RL steps produce the best joint performance across three metrics, while later checkpoints show a clear decline.Main experiments therefore use the 300-step checkpoint.