Source-linked AI summary
VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Lulu Gu, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, Ziwei Liu
TL;DR
Existing benchmarks capture superficial faithfulness but provide limited coverage of intrinsic realism involving physical laws, commonsense, anatomy, and composition. VBench-2.0 introduces a five-dimensional benchmark combining generalist and specialist evaluators with human validation, and its evaluation identifies strengths in human-related generation and creativity alongside weaknesses in complex plots, dynamics, and commonsense reasoning.
Problem
Existing evaluations emphasize visual convincingness, while advanced video-generation applications require adherence to physical laws, commonsense reasoning, anatomical correctness, and compositional integrity.
Method
VBench-2.0 evaluates Human Fidelity, Controllability, Creativity, Physics, and Commonsense using tailored VLM, LLM, specialist, and human-annotation evaluation procedures.
Results
Recent models show emerging abilities in human anatomy, consistency, and novel creativity but struggle with complex plots, simple dynamic changes, and commonsense reasoning.
Takeaways & Limitations
VBench-2.0 complements VBench by broadening standardized evaluation from superficial quality toward intrinsic faithfulness in video generation.
Takeaways & Limitations
Current foundation video models typically generate single-shot videos under 10 seconds, limiting coherent narrative expression in complex plots.
Abstract
from arXiv · showhide
Video generation has advanced significantly, evolving from producing unrealistic outputs to generating videos that appear visually convincing and temporally coherent. To evaluate these video generative models, benchmarks such as VBench have been developed to assess their faithfulness, measuring factors like per-frame aesthetics, temporal consistency, and basic prompt adherence. However, these aspects mainly represent superficial faithfulness, which focus on whether the video appears visually convincing rather than whether it adheres to real-world principles. While recent models perform increasingly well on these metrics, they still struggle to generate videos that are not just visually plausible but fundamentally realistic. To achieve real "world models" through video generation, the next frontier lies in intrinsic faithfulness to ensure that generated videos adhere to physical laws, commonsense reasoning, anatomical correctness, and compositional integrity. Achieving this level of realism is essential for applications such as AI-assisted filmmaking and simulated world modeling. To bridge this gap, we introduce VBench-2.0, a next-generation benchmark designed to automatically evaluate video generative models for their intrinsic faithfulness. VBench-2.0 assesses five key dimensions: Human Fidelity, Controllability, Creativity, Physics, and Commonsense, each further broken down into fine-grained capabilities. Tailored to individual dimensions, our evaluation framework integrates generalists such as SOTA VLMs and LLMs, and specialists, including anomaly detection methods proposed for video generation. We conduct extensive human annotations to ensure evaluation alignment with human judgment. By pushing beyond superficial faithfulness toward intrinsic faithfulness, VBench-2.0 aims to set a new standard for the next generation of video generative models in pursuit of intrinsic faithfulness.
I. INTRODUCTION
Existing benchmarks emphasize superficial faithfulness—visual quality, temporal smoothness, and simple prompt adherence—while advanced applications require intrinsic faithfulness to physical, commonsense, anatomical, and compositional principles. VBench-2.0 addresses this gap with five dimensions, specialized evaluation methods, and human-validation annotations, revealing persistent weaknesses in complex plots, dynamic changes, and commonsense reasoning.
- Motivation: Existing benchmarks assess per-frame aesthetics, temporal smoothness, and simple prompt adherence, defining superficial faithfulness as visual convincingness.Recent state-of-the-art models perform strongly on these metrics, with many aspects approaching saturation.
- Motivation: Intrinsic faithfulness requires generated videos to follow physical laws, commonsense reasoning, anatomical correctness, and compositional integrity.These requirements matter for applications involving real-world simulation, AI-driven storytelling, filmmaking, and embodied intelligence.
- Approach: VBench-2.0 evaluates Human Fidelity, Controllability, Creativity, Physics, and Commonsense through fine-grained capability dimensions.The suite combines VLM and LLM generalists, video-based question answering, text-description alignment, specialists, safeguards, and human preference annotations.
- Findings: Recent models show emerging abilities in human anatomy, consistency, and novel creativity but struggle with complex plots, simple dynamic changes, and commonsense reasoning.These results identify open challenges in generating videos that faithfully synthesize the world.
- Contribution: VBench-2.0 extends VBench with a broader framework for evaluating intrinsic faithfulness and standardizing assessment of future video-generation advances.The benchmark is intended to complement VBench and remain open-sourced while incorporating newly released models.
II. RELATED WORKS
Video generation progressed from short clips optimized for visual fidelity and temporal consistency toward foundation models targeting physical principles, commonsense reasoning, human-centric generation, and creativity. Existing benchmarks do not systematically cover these emerging capabilities, motivating VBench-2.0’s comprehensive intrinsic-faithfulness evaluation.
- Video Generation Benchmarks: VBench-2.0 is presented as the first comprehensive benchmark assessing intrinsic faithfulness while complementing VBench.Its comparison scope includes frame quality, temporal consistency, simple and complex prompts, creativity, commonsense, physics, and human anatomy.
- Video Generative Models: Recent video foundation models target deeper principles such as physical laws and commonsense reasoning alongside realistic action continuity and human-centric creativity.These directions extend beyond earlier emphasis on visual quality and temporal consistency.
- Video Generation Benchmarks: Earlier metrics including FID, IS, and FVD offered limited insight into the diverse and complex capabilities of modern video generation.VBench and related frameworks introduced structured evaluation by disentangling performance into multiple capability dimensions.
III. VBENCH-2.0 SUITE FOR INTRINSIC FAITHFULNESS
VBench-2.0 evaluates intrinsic faithfulness through five dimensions and fine-grained capabilities, combining generalist reasoning, specialist detectors, and human-alignment validation. Its suite covers human fidelity, creativity, controllability, physics, and commonsense through tailored evaluation procedures.
- Evaluation framework: VBench-2.0 evaluates Human Fidelity, Creativity, Controllability, Physics, and Commonsense through fine-grained sub-dimensions.The framework combines generalist reasoning models with specialist detectors for dimension-specific assessment.
- Generalist evaluation: Text description alignment uses a VLM to describe generated videos and an LLM to judge caption-reference matching with binary scores.This decoupling targets semantic dimensions such as Complex Plot and Human Interaction, where high-level reasoning is important.
- Generalist evaluation: Video-based multi-question answering queries salient concepts with complementary questions, averaging or requiring all binary responses depending on the scoring scheme.The method is used for targeted visual understanding, including dynamic attribute changes.
- Human Fidelity: Human Anatomy uses three ViT-base anomaly detectors for bodies, hands, and faces, with detected regions evaluated across video frames.The final score is the percentage of frames not flagged as abnormal; training uses real and generated samples plus manually annotated negatives.
- Creativity: Creativity measures diversity from 20 sampled videos and composition through species combinations, single-entity actions, and multi-entity interactions.Composition evaluation omits videos containing more than one creature after a preliminary question to reduce discrete VQA errors.
- Controllability: Controllability tests whether models follow complex prompts and render specified entities, events, content, camera movements, and dynamic attribute changes.Dynamic attributes are tested with questions about initial and final states and whether the attribute changes.
- Controllability: Complex Plot evaluates sequential matching of five summarized plot elements against five video captions, stopping when an element is absent.Long prompts describe multi-stage events, and LLM post-processing verifies caption numbering before scoring.
- Physics: Multi-View Consistency combines feature-matching stability with camera-motion speed to assess geometric consistency without explicit 3D ground truth.The procedure extracts SIFT keypoints, matches them with FLANN and RANSAC, and estimates motion using RAFT.
B. Prompt Suite
The VBench-2.0 Prompt Suite uses compact, representative prompts designed for diverse evaluation while reducing sampling costs and isolating the capability under test.
- B. Prompt Suite: The prompt suite limits test cases to reduce sampling costs while maintaining coverage across diverse evaluation scenarios.This design addresses the computational cost of sampling longer and higher-resolution videos.
- B. Prompt Suite: Prompts eliminate confounding factors by restricting changes to the entity or attribute relevant to the evaluated dimension.For example, Dynamic Spatial Relationship and Dynamic Attribute allow only one entity to move or change.
- B. Prompt Suite: These structured principles produce a compact, diverse, and reliable benchmark spanning real-world and abstract scenarios.
C. Human Preference Annotation
VBench-2.0 validates its evaluations through large-scale human preference annotations using pairwise comparisons, structured interfaces, win-ratio scoring, and quality control.
- C. Human Preference Annotation: Human preference labeling compares generated videos across all VBench-2.0 dimensions to assess alignment with human perception.The annotations also provide a resource for fine-tuning generation and evaluation models toward human judgments.
- C. Human Preference Annotation: For each prompt, four-model video groups generate six randomized pairwise comparisons for human preference judgments.The comparisons are (A,B), (A,C), (A,D), (B,C), (B,D), and (C,D).
- C. Human Preference Annotation: A second annotation format compares two groups whose content distributions are evaluated, with each video group containing 20 videos from one prompt.
- C. Human Preference Annotation: The interface sequentially presents annotation instructions to improve readability and efficiency for detailed evaluations and prompts exceeding 150 words.
- C. Human Preference Annotation: Win ratio divides each model’s total pairwise score by its number of comparisons, assigning 1 for a preference, 0 for the alternative, and 0.5 for ties.
- C. Human Preference Annotation: Verification samples 20% of annotated pairs and requires a 95% success rate for quality control.
- C. Human Preference Annotation: Annotation across 18 evaluation dimensions required 15.75 hours of individual effort and 284 hours across 18 annotators.
A. Video Generation Models in Evaluation
The evaluation compares four recent video generation models using standardized model information and generation settings, while VBench-2.0 evaluation alignment is visualized against human preferences.
- A. Video Generation Models in Evaluation: Four video generation models are used for comparison, with additional models planned as they become open-sourced.Their basic information is summarized in Table III.
- A. Video Generation Models in Evaluation: Figure 11 plots human preference win ratios against VBench-2.0 evaluation win ratios for each dimension and computes Spearman’s correlation coefficient.
- A. Video Generation Models in Evaluation: Table III presents information on the evaluated models.
- A. Video Generation Models in Evaluation: Each prompt uses 161 continuous frames at 768×1360 resolution and 16 FPS, with the initial random seed fixed at 42.
B. VBench-2.0 Evaluation Results
VBench-2.0 evaluates four recent video generation models across 18 dimensions, producing scores from 0 to 1 in which higher values indicate stronger performance.
- B. VBench-2.0 Evaluation Results: Each sub-ability receives a numerical score between 0 and 1, with higher values indicating relatively stronger performance in that dimension.Results for four video generation models are summarized in Table II and visualized in Figure 2.
C. Human Alignment of VBench-2.0
VBench-2.0 evaluations are compared with human preference annotations to assess alignment across evaluation dimensions.
- Human preference labeling and correlation analysis were used to evaluate how closely VBench-2.0 results align with human judgment.Figure 11 and Table IV present the correlation plot and numerical win ratios.
V. INSIGHTS AND DISCUSSIONS
This section summarizes evaluation insights about model characteristics, trade-offs, and the relationship between superficial and intrinsic faithfulness.
- The evaluation highlights model trade-offs and characteristics while examining superficial versus intrinsic faithfulness in video generation.
A. Characteristics of Recent SOTA Models
Recent models exhibit distinct strengths across human fidelity, creativity, controllability, physics, and commonsense, while revealing persistent challenges in complex plots and dynamic relationships. Prompt refinement can improve controllability or physical plausibility, but may trade off diversity and does not consistently improve knowledge-driven capabilities.
- A. Characteristics of Recent SOTA Models: Sora excels in Human Fidelity and Creativity but performs comparatively poorly in Controllability, Physics, and Commonsense.Its human figures show reasonable anatomical consistency, while some outputs violate real-world principles.
- A. Characteristics of Recent SOTA Models: Kling is relatively strong in Commonsense, Controllability, Multi-View Consistency, and Camera Motion without a significantly weak dimension.
- A. Characteristics of Recent SOTA Models: VBench-2.0 and human win ratios are reported together for each evaluation dimension and model, demonstrating close alignment with human perception.
- A. Characteristics of Recent SOTA Models: HunyuanVideo shows strengths in Human Fidelity and Motion Rationality, possibly benefiting from high-quality human-related training data.
- B. Key Limitations of Recent SOTA Models: Complex Plot generation remains difficult because current foundation models typically produce single-shot videos under 10 seconds, limiting coherent narratives.
- B. Key Limitations of Recent SOTA Models: Models fail about 80% of Dynamic Spatial Relationships and Dynamic Attributes cases, likely reflecting insufficiently granular temporal captioning.
- C. The Role of Prompt Engineering: Sora’s creativity-controllability profile contrasts with other models, which show the reverse pattern, indicating a trade-off between novelty and strict prompt following.
- C. The Role of Prompt Engineering: External Prompt Refiners can support physically plausible outcomes and controllability, but may sacrifice diversity and show no consistent gains on Human Fidelity, Camera Motion, Geometry, or Commonsense.
D. Superficial Faithfulness vs. Intrinsic Faithfulness: Do Not Miss Out on Any Pillar
Visual quality can bias judgments of video generators, while intrinsic faithfulness captures deeper requirements for real-world applications. VBench-2.0 addresses this gap with broader evaluation and highlights challenges in reliably assessing complex dimensions.
- Visually pleasing and smooth videos can be perceived as better even when structure-driven capabilities remain weaker.CogVideoX performs relatively well across many VBench-2.0 dimensions, while HunyuanVideo’s visual strengths coexist with room for growth in structure-driven dimensions.
- Using VBench and VBench-2.0 together enables evaluation across both superficial and intrinsic faithfulness.
- Intrinsic-faithfulness evaluation is difficult because it requires complex scene understanding, commonsense reasoning, and physical-world perception.Current large models are unreliable for some areas, including Human Anatomy and Motion Rationality, partly because training data lacks generated-video anomalies and models struggle with object counting and 3D understanding.
- A unified clip-level anomaly detector trained with abnormal data can capture differences and commonalities across anomaly types.
- VBench-2.0 complements VBench through a multidimensional, human-aligned framework using VLM generalists and specialists for challenging dimensions.