Source-linked AI summary
VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models
Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, Ziwei Liu
TL;DR
Existing video-generation metrics do not fully align with human perception, and evaluations should reveal model strengths and weaknesses. VBench++ evaluates video generation through disentangled dimensions, tailored prompts, and human preference annotations, extending the benchmark to image-to-video and trustworthiness. Its evaluations are reported to align with human preferences and provide detailed feedback across model abilities.
Problem
Existing video-generation metrics are inconsistent with human judgment, and evaluation should reflect perception while revealing models’ strengths and weaknesses.
Source: §1 · p. 1Method
VBench++ evaluates video generation with 16 disentangled dimensions, tailored prompts and methods, human preference annotations, and extensions for image-to-video and trustworthiness.
Source: §1 · pp. 2–3, §3.1 · p. 4Results
VBench evaluations correlate highly with human preferences, and their multi-dimensional approach provides detailed feedback on model strengths and weaknesses.
Source: §1 · p. 2Takeaways & Limitations
Dimension-level feedback is intended to inform future training, data, and architectural choices for video generation.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Video generation has witnessed significant advancements, yet evaluating these models remains a challenge. A comprehensive evaluation benchmark for video generation is indispensable for two reasons: 1) Existing metrics do not fully align with human perceptions; 2) An ideal evaluation system should provide insights to inform future developments of video generation. To this end, we present VBench, a comprehensive benchmark suite that dissects "video generation quality" into specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. VBench has several appealing properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in video generation (e.g., subject identity inconsistency, motion smoothness, temporal flickering, and spatial relationship, etc). The evaluation metrics with fine-grained levels reveal individual models' strengths and weaknesses. 2) Human Alignment: We also provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception, for each evaluation dimension respectively. 3) Valuable Insights: We look into current models' ability across various evaluation dimensions, and various content types. We also investigate the gaps between video and image generation models. 4) Versatile Benchmarking: VBench++ supports evaluating text-to-video and image-to-video. We introduce a high-quality Image Suite with an adaptive aspect ratio to enable fair evaluations across different image-to-video generation settings. Beyond assessing technical quality, VBench++ evaluates the trustworthiness of video generative models, providing a more holistic view of model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and continually add new video generation models to our leaderboard to drive forward the field of video generation.
1 INTRODUCTION
Existing video-generation metrics can disagree with human judgment, while evaluation should also reveal models’ strengths and weaknesses. VBench++ addresses this with fine-grained, human-aligned evaluation spanning text-to-video, image-to-video, and trustworthiness.
1 INTRODUCTION
- VBench++ decomposes video-generation quality into hierarchical evaluation dimensions, with tailored prompts and methods for fine-grained assessment.The framework also samples generated videos and compares evaluations with human preference annotations.Source: §1 · p. 1, Figure 1 · p. 2
- Dimension- and category-level evaluations provide feedback on model strengths and weaknesses, including variation across eight content types.The authors say these insights can inform future training, data, and architectural choices.Source: §1 · p. 2
- Human preference annotations show that VBench evaluations correlate highly with preferences across fine-grained dimensions.Annotators compare videos generated from the same prompt according to each dimension.Source: §1 · p. 2
- For image-to-video evaluation, VBench++ adds an adaptive-resolution Image Suite and methods for video-image and video-text consistency.The new methods are described as aligned with human perception through annotation and experiments.Source: §1 · p. 2
- VBench++ adds four trustworthiness dimensions covering cultural and demographic fairness and avoidance of harmful or offensive content.The evaluations are designed to align with human perception.Source: §1 · p. 3
- Compared with the earlier VBench, VBench++ supports image-to-video and trustworthiness evaluation, adds 32 models to the initial four, and releases VBench-Long.The authors also maintain open access to evaluation videos and data.Source: §1 · p. 3
2 RELATED WORKS
Video generation spans multiple conditioning approaches, but its evaluation benchmarks have lacked the detailed, human-aligned feedback available in some image-generation assessments. VBench++ addresses this gap while extending evaluation to image-to-video and trustworthiness.
2.1 Video Generative Models
- Video generation works include text-to-video, image-to-video, video-to-video, and control-map-conditioned approaches such as pose, depth, and sketch.The passage presents these as directions associated with diffusion-based video generation.Source: §2.1 · p. 3
2.2 Evaluation of Visual Generative Models
- Common video-generation metrics and limited prompt sets provide less specific evaluation than the comprehensive, human-aligned feedback sought for video models.The passage notes that UCF-101 labels and MSR-VTT captions are used with common metrics, while video benchmarks remain less comprehensive than text-to-image benchmarks.Source: §2.2 · p. 3
- VBench evaluates video generation with 16 dimensions, specialized prompts, human-alignment validation, and multi-dimensional, multi-category analysis.The authors distinguish their work from concurrent research, on these grounds.Source: §2.2 · pp. 3–4
- Prior image-to-video evaluations are described as lacking comprehensive human-perception validation and accounting for models’ differing default resolutions.VBench++ uses adaptive resolution and evaluation dimensions aligned with human perception through experiments and annotation.Source: §2.2 · p. 4
- VBench++ introduces human-aligned benchmarks and evaluation pipelines for trustworthiness in video generation.The passage situates this work alongside prior research on cultural representation, human attributes, and harmful-content filtering,,, [133], [134], [135],,,,,, [139],,,.Source: §2.2 · p. 4
3 VBENCH++ SUITE
The VBench++ suite comprises evaluation dimensions, prompt suites, an adaptive Image Suite, and human preference annotation. The paper presents these components as the basis for evaluating dimensions and validating alignment with human perception.
3 VBENCH++ SUITE
- The paper introduces VBench++’s main components; Section 3.1 presents the rationale for designing 16 evaluation dimensions and provides each dimension’s definition and evaluation method.Source: §3 · p. 4
- VBench++ uses prompt suites and a newly proposed Image Suite as test inputs for evaluation.The prompt suites are discussed in Section 3.2.1 and the Image Suite in Section 3.3.Source: §3 · p. 4
- Human preference annotation for each dimension is used to validate VBench++’s alignment with human perception.The paper places experiments and insights drawn from the suite in Sections 4 and 5.Source: §3 · p. 4
3.1 Evaluation Dimension Suite
The suite replaces single-number assessment with 16 disentangled dimensions spanning video quality and consistency with user conditions. It also defines criteria for image-to-video and trustworthiness evaluation.
3.1 Evaluation Dimension Suite
- Sixteen top-down dimensions divide evaluation into Video Quality and Video-Condition Consistency, separating perceptual quality from alignment with the user’s condition.The first asks whether the video looks good apart from the prompt; the second asks whether it matches the guiding condition.Source: §3.1 · p. 4
3.1.1 Video Quality
- Video Quality separates temporal consistency and dynamics from frame-wise quality, with distinct measures for subject and background consistency, flickering, smoothness, dynamics, aesthetics, and distortion.The suite uses DINO and CLIP similarities, frame differences, motion priors and RAFT, plus aesthetic and MUSIQ predictors.Source: §3.1.1 · pp. 4–5
3.1.2 Text-to-Video: Video-Condition Consistency
- T2V condition consistency covers semantic entities and attributes plus appearance and temporal style, using specialized checks and ViCLIP for overall consistency.Semantic checks include object classes, multiple objects, human actions, colors, spatial relations, and scenes.Source: §3.1.2 · p. 5
3.1.3 Image-to-Video Dimensions
- I2V adds subject and background consistency with the input image, alongside consistency between prompted camera motion and generated motion.It combines image-to-frame and consecutive-frame similarities for subject and background scores, and uses tracked edge points to infer camera motion.Source: §3.1.3 · p. 5
3.1.4 Trustworthiness
- Trustworthiness covers cultural fairness, gender and skin-tone bias, and safety, including checks for cross-cultural scenes, distribution skew, and unsafe content.The methods use ViCLIP for cultural prompts, face and attribute classifiers for bias, and three safety detectors with a video-level threshold.Source: §3.1.4 · pp. 5–6
3.2 Prompt Suite
The prompt suites balance efficient evaluation with targeted coverage across dimensions, content categories, and trustworthiness scenarios.
3.2.1 Prompt Suite for Text-to-Video
- For each VBench evaluation dimension, the authors carefully curated around 100 test prompts to probe its specific ability, using movable subjects for Subject Consistency and requiring a specific object class for Object Class.Source: §3.2.1 · p. 6
- The Image Suite’s resolution distribution is reported using pixel area and the lengths of both sides; the suite primarily contains images of 4K resolution or higher.Source: Table 1 · p. 7
- Eight content-category suites contain 100 manually cleaned prompts each, covering animals, architecture, food, humans, lifestyle, plants, scenery, and vehicles.The prompts were human-curated, labeled with LLM assistance, and manually cleaned.Source: §3.2.1 · p. 7
3.2.2 Prompt Suite for Trustworthiness
- Culture Fairness crosses nine cultural classes with 14 scenarios, producing 126 prompts designed around visually distinctive settings.Scenarios include weddings, dances, architecture, holidays, and dining.Source: §3.2.2 · p. 7
- A shared set of 90 semantically gender- and skin-tone-agnostic prompts tests bias across six human-description aspects.The aspects are occupation, location, emotion, appearance, behavior, and clothing.Source: §3.2.2 · p. 7
- The culture classes make slight modifications to Huntington’s original classification to support a more robust culture classification.Source: §3.2.2 · p. 7
- Safety uses 90 manually screened prompts across seven harmful-content classes, including generalized historical scenes and events.GPT-4 assists with prompt generation; prompts avoid harmful language and specific sensitive individuals or events.Source: §3.2.2 · p. 7
3.3 Image Suite
The Image Suite is designed for I2V evaluation across model-specific resolutions and aspect ratios while preserving image content and balancing foreground, background, and prompt coverage.
3.3 Image Suite
- The Image Suite uses high-resolution images and adapts crops to I2V models’ default resolutions and aspect ratios to avoid quality degradation from resizing.Images are selected when the main subject can remain intact through cropping; intermediate aspect ratios are derived from square and widescreen crops.Source: §3.3 · p. 7, Figure 5 · p. 8
- For portrait images (width less than height), cropping proceeds from 1:1 to 16:9; for landscape images (width greater than height), it proceeds from 16:9 to 1:1, preserving key content.Additional crops interpolate between these ratios; common ratios include 1:1, 7:4, 8:5, and 16:9.Source: Figure 5 · p. 8
- Foreground coverage spans humans, animals, plants, food, transportation, and other content; backgrounds include architecture, scenery, indoor, and abstract scenes.The suite also maintains diversity within foreground categories, including human age, race, and gender.Source: §3.3 · p. 8, Figure 6 · p. 8
- Each image receives a refined caption, and camera-motion evaluation prompts append controls such as panning or zooming.Captions are generated with models including CoCa and BLIP2, then reviewed and edited to reflect image content and add motion details.Source: §3.3 · p. 8
- The suite combines adaptive aspect ratios, diverse content, primarily 4K-or-higher images, and image-specific prompts for I2V evaluation.Source: §3.3 · p. 8
3.4 Human Preference Annotation
Human preference annotations are collected to validate VBench’s alignment with human judgment across T2V, I2V, and trustworthiness dimensions.
3.4 Human Preference Annotation
- The annotation program targets human-alignment validation across all 16 dimensions, including newly proposed I2V and trustworthiness evaluation methods.Source: §3.4 · p. 8
3.4.1 Human Annotations for Text-to-Video Evaluations
- For T2V, annotators compare randomized video pairs generated from the same prompt and judge only the specified dimension; instructions and quality checks support consistent labeling.Four models generate five video groups per prompt, yielding six pairwise comparisons per group; annotators receive dimension-specific instructions and undergo trial and post-labeling checks.Source: §3.4.1 · pp. 8–9
3.4.2 Human Annotations for Image-to-Video Evaluations
- For I2V, annotators compare two generated videos while viewing the input images and prompts, with documentation and multiple quality checks.Source: §3.4.2 · p. 9
3.4.3 Human Annotations for Trustworthiness Evaluations
- Trustworthiness annotations compare groups of ten videos per prompt for fairness, bias mitigation, or safety, with quality checks and relabeling.Source: §3.4.3 · p. 9
4 EXPERIMENTS
The experiments evaluate text-to-video, image-to-video, and trustworthiness dimensions across model groups, then assess human alignment, content-category performance, and video-versus-image generation. The supplied passages report high human alignment across dimensions but do not provide numerical model-comparison results.
4.1.1 Text-to-Video Evaluation
- The T2V evaluation initially covered LaVie, ModelScope, VideoCrafter, and CogVideo, with 32 more models subsequently added to the leaderboard.Empirical Max, Empirical Min, and WebVid-Avg provide reference baselines for per-dimension scores.Source: §4.1.1 · pp. 9–10
4.1.2 Image-to-Video Evaluation
- The I2V evaluation scores seven models across VBench++ dimensions, with results reported in Table 3.The table compares the models across I2V dimensions; higher scores indicate relatively better performance for a dimension.Source: Table 3 · p. 10, §4.1.2 · p. 10
4.1.3 Trustworthiness Evaluation
- Trustworthiness scores were calculated for seven video-generation models and reported in Table 4 and Figure 7.Table 4 compares trustworthiness scores for image and video generative models; higher scores indicate relatively better performance for a dimension.Source: §4.1.3 · p. 10, Table 4 · p. 10, Figure 7 · p. 10
4.2 Validating Human Alignment of VBench++
- VBench++ evaluations closely correlate with human preferences across T2V, I2V, and trustworthiness dimensions.The T2V, I2V, and trustworthiness alignment results are presented in Figures 8, 9, and 10, respectively.Source: §4.2 · p. 11, Figure 8 · p. 11, Figure 9 · p. 11
4.3 Per-Category Evaluation
- The evaluation tests T2V models across eight content categories by generating videos from each category’s Prompt Suite and calculating model performance across different evaluation dimensions.Source: §4.3 · p. 11
4.4 Video Generation V.S. Image Generation
- The comparison evaluates T2V models against T2I models on ten dimensions representing frame-wise generation capability.The analysis also compares trustworthiness between T2V and T2I models.Source: §4.4 · p. 11
5 INSIGHTS AND DISCUSSIONS
The discussion identifies trade-offs in temporal consistency, dynamic motion, and input-image alignment, alongside variation across content categories. It also highlights compositionality and camera-motion control as challenges, and compares trustworthiness across model origins.
Trade-off across Ability Dimensions
- Models with stronger temporal consistency often have lower Dynamic Degree, while models with high Dynamic Degree can perform poorly on temporal consistency.LaVie exemplifies the first pattern and VideoCrafter the second; the authors identify achieving both with large motions as a challenge.Source: §5 · p. 12
Bottleneck in Temporally Complex Categories Affecting Spatial and Temporal Performance
- For complex categories such as Human and Vehicle, intense motion is associated with poor performance across dimensions and can affect both spatial and temporal results.The authors suggest poor temporal modeling may produce distorted and blurred imagery.Source: §5 · p. 12, Figure 12 · p. 13
Challenges of Data Quantity in Handling Complex Categories like Human
- Human is among the weakest content categories despite accounting for 26% of WebVid-10M, suggesting data volume alone may not suffice for complex categories.The authors suggest human-related priors or controls, such as skeletons, as a potential approach.Source: §5 · p. 12
Prioritizing Data Quality Over Quantity in Large-Scale Datasets
- Food tends to score highest in Aesthetic Quality despite comprising 11% of WebVid-10M, which the authors interpret as evidence that data quality may matter more than quantity at million scale.They also suggest VBench dimensions could help clean datasets for specified quality dimensions.Source: §5 · p. 12
Compositionality: T2I versus T2V
- T2V models underperform T2I models in Multiple Objects and Spatial Relationship, highlighting a compositionality gap.Possible responses include training data with explicit multi-object captions, spatial-control modules, or more representative text embeddings.Source: §5 · p. 12
Video-Image Consistency versus Between-Frame Consistency
- Among I2V models with comparable Dynamic Degree, stronger alignment with the input image is associated with weaker between-frame consistency.The ranking in the I2V Background dimension is inversely related to the ranking in Background Consistency for I2VGen-XL, ConsistI2V, and VideoCrafter-I2V.Source: §5 · p. 12
Limited Performance in Camera Motion Control
- 35.44% was the highest classification accuracy for I2V Camera Motion, indicating limited performance in following camera-motion instructions.The authors suggest more annotated camera-motion training videos or explicit camera-control mechanisms as potential approaches.Source: §5 · pp. 12–13
Trustworthiness in Video Generative Models
- Evaluated industry-developed models showed relatively stronger trustworthiness performance than models originating from academic research.The authors offer more diverse in-house data and internal trustworthiness reviews as possible explanations, and urge caution in model use.Source: §5 · p. 13
6 CONCLUSION
The paper presents VBench and extends it as VBench++ for image-to-video and trustworthiness evaluation. It describes the framework as multidimensional, human-aligned, insight-rich, and versatile, and says the leaderboard will continue to include newly released models.
6 CONCLUSION
- VBench++ extends VBench to support image-to-video and trustworthiness evaluation, while the authors continue adding newly released models to its leaderboard.The conclusion describes the benchmark as multidimensional, human-aligned, insight-rich, and versatile.Source: §6 · p. 13