VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness

• VBench-2.0 extends evaluation beyond superficial faithfulness to intrinsic realism grounded in physical laws, commonsense reasoning, anatomical correctness, and compositional integrity.The paper identifies this broader realism as important for applications including AI-assisted filmmaking and virtual environments for embodied intelligence.Source: §I · p. 1
• VBench-2.0 assesses Human Fidelity, Controllability, Creativity, Physics, and Commonsense through fine-grained capabilities, comprising 18 dimensions.Figure 1 presents the suite’s five broad categories and 18 fine-grained capability dimensions.Source: §I · p. 1, Figure 1 · p. 2
• The evaluation combines VLM- and LLM-based text-description alignment and video multi-question answering with specialist anomaly-detection pipelines and safeguards.The safeguards include pre-filtering and redundant questioning; extensive human preference annotations validate and align automated results with human judgment.Source: §I · pp. 1–2
• Recent models show emerging abilities in human anatomy, consistency, and some novel creativity, but struggle with complex plots, simple object dynamics, and stable commonsense reasoning.These limitations highlight open challenges in generating videos with intrinsic faithfulness.Source: §I · p. 2
• VBench assesses superficial faithfulness, while VBench-2.0 complements it with intrinsic-faithfulness evaluation and aims to provide an open-source standardized framework.The introduction notes that superficial-faithfulness metrics are approaching saturation as recent state-of-the-art models perform strongly on them.Source: §I · pp. 1–2

A. Video Generative Models

• Earlier text-to-video models mainly generated 2–3-second clips through incremental improvements in visual fidelity and temporal consistency, before Sora pioneered scaling through large-scale training.The passage describes Sora’s demonstration of unprecedented model capacity, then notes that next-generation foundation models achieved visual quality and spatiotemporal coherence.Source: A. Video Generative Models · pp. 2–3
• Next-generation video foundation models pursue deeper principles such as physical laws, commonsense reasoning, action continuity, realistic physical-world perception, and creative human-centric generation.Existing benchmarks cannot systematically evaluate these emerging explorations, motivating VBench-2.0.Source: A. Video Generative Models · p. 3

B. Evaluation of Video Generative Models

• VBench-2.0 provides a comprehensive framework for evaluating emerging video-generation capabilities and intrinsic faithfulness, beyond benchmarks limited to fundamental or specific domains.It complements VBench and covers aspects including compositional creativity, commonsense reasoning, physics, human anatomy, and complex-prompt adherence.Source: B. Evaluation of Video Generative Models · p. 3, Table I · p. 3, A. Video Generative Models · p. 3
• Early evaluation relied on FID, IS, and FVD, which offered limited insight into modern video-generation capabilities.Later frameworks –,, [71] structured evaluation across capability dimensions.Source: B. Evaluation of Video Generative Models · p. 3
• VBench, [26] assesses per-frame quality, temporal consistency, and basic prompt adherence, but some of its dimensions are beginning to saturate as models improve.This motivates a broader scope that evaluates deeper intrinsic faithfulness.Source: B. Evaluation of Video Generative Models · p. 3
• Specialized benchmarks target physical-law understanding with VLMs in PhyGenBench [72], compositionality in T2V-CompBench [73], and storytelling in StoryEval [74].The passage contrasts these domain-specific efforts with benchmarks focused on fundamental capabilities and VBench-2.0’s comprehensive framework.Source: B. Evaluation of Video Generative Models · p. 3

III. VBENCH-2.0 SUITE FOR INTRINSIC FAITHFULNESS

• The suite assesses Human Fidelity, Creativity, Controllability, Physics, and Commonsense through fine-grained capabilities, using generalist reasoning models and specialist detectors.For semantic alignment, a VLM describes video content and an LLM compares it with a prompt, reference, or metadata; targeted VQA is used when a salient concept can be queried directly.Source: §III · p. 3, A. Evaluation Dimension Suite · pp. 3–4

A. Evaluation Dimension Suite

• Human Fidelity checks structural anomalies and temporal consistency in human appearance, clothing, and identity.Frame-level body, hand, and face detectors flag anomalies; clothing consistency uses video QA, while identity consistency compares facial features across frames with ArcFace and RetinaFace.Source: A. Evaluation Dimension Suite · pp. 4–5
• Creativity measures inter-sample diversity and uncommon compositions, using VGG-19 features for diversity and video QA for species combinations, actions, and interactions.Composition evaluation pre-filters clips to exclude cases containing more than one creature, addressing unreliable VQA results in those cases.Source: A. Evaluation Dimension Suite · p. 5
• Controllability tests dynamic attributes and spatial relationships, ordered motions, specified content, and camera movements in response to detailed prompts.The suite uses video QA or text-description alignment for these tasks and assesses camera motion with point tracking and heuristics, extending VBench++ [26]’s taxonomy to nine types.Source: A. Evaluation Dimension Suite · pp. 5–6
• Complex Plot and Complex Landscape test adherence to long prompts through ordered alignment of video captions with plot elements or landscape descriptions.For complex plots, prompts are split into five ground-truth descriptions, matched sequentially to video captions, and scored by the correctly matched elements.Source: A. Evaluation Dimension Suite · pp. 5–6
• Physics evaluates mechanical, thermal, and material state changes alongside multi-view geometric consistency.State-change evaluation uses video QA; mechanics prompts describe expected visual outcomes and filter videos whose initial state conflicts with the prompt. Geometry combines feature-matching stability with camera-motion speed, using SIFT, FLANN, RANSAC, and RAFT.Source: A. Evaluation Dimension Suite · p. 6
• Commonsense evaluates whether motions produce expected real-world consequences, using redundant video questions to distinguish effective actions from visually plausible but ineffectual motions.Examples include checking whether food decreases after eating, forward progress follows walking, or an object splits after cutting.Source: A. Evaluation Dimension Suite · p. 6
• Instance Preservation detects unnatural entity merging, duplication, splitting, appearance, or disappearance with a clip-level VLM detector.A clip is scored normal only when every clip in the video is judged normal; training uses real and generated normal samples and manually annotated anomalies.Source: A. Evaluation Dimension Suite · pp. 6–7

B. Prompt Suite

• Approximately 70 tailored prompts per dimension cover varied capabilities and scenarios while limiting video-sampling costs.Examples include object- and scene-level Multi-View Consistency, Composition tasks spanning species combinations and entity actions, and Physics prompts covering mechanics, thermotics, and material properties.Source: B. Prompt Suite · p. 7
• Prompts reduce confounds by allowing only one entity to move or change in Dynamic Spatial Relationship and Dynamic Attribute tests.This focuses evaluation on positioning or attributes rather than multi-object interactions.Source: B. Prompt Suite · p. 7
• Explicitly recognizable actions and observable counterexamples strengthen evaluation robustness, such as handshakes and fake cutting that leaves objects unchanged.The suite avoids ambiguous social scenarios and visually inconclusive examples such as lip-syncing.Source: B. Prompt Suite · p. 7
• The structured design yields a compact, diverse, and reliable benchmark across real-world and abstract scenarios.Source: B. Prompt Suite · p. 7

Data Preparation

• For each prompt, annotators compare all six model pairs across five groups, with video order randomized to reduce bias.The four models generate videos for each group; annotators select the preferred video in each pairwise comparison.Source: C. Human Preference Annotation · pp. 7–8
• For content-distribution dimensions, pairwise comparisons use two video groups, each containing 20 videos generated from a single prompt.Source: C. Human Preference Annotation · p. 8

Labeling Instructions

• The interface presents annotation instructions sequentially to improve readability and efficiency for detailed multi-question evaluations and prompts exceeding 150 words.Source: C. Human Preference Annotation · p. 8

Win Ratio

• Win ratios award 1 to a preferred video, 0 to the other, and 0.5 to each in a tie, then divide each model’s total score by its comparisons.The results are reported in Table IV, which is not included in the supplied passages.Source: C. Human Preference Annotation · p. 8

Quality Assurance

• Quality control verifies a random 20% of annotated pairs against a 95% required success rate, while the full effort totaled 284 hours across 18 annotators.The process also required 15.75 hours of individual effort; most dimensions underwent two trial-labeling rounds, official labeling, and post-labeling verification.Source: C. Human Preference Annotation · p. 8

A. Video Generation Models in Evaluation

• The evaluation compares Kling 1.6, Sora-480p, HunyuanVideo, and CogVideoX-1.5 [59], with more models planned as they become open-sourced.A modified CogVideoX Prompt Refiner processes prompts for the other three models; it improved video quality and text alignment for those models, while Sora was excluded because it may have an embedded refiner.Source: A. Video Generation Models in Evaluation · p. 8
• Kling 1.6 is sampled through its standard API at 720×1280 for 241 continuous frames at 24 FPS, using the official default cfg scale of 0.5.The paper uses the standard API version.Source: A. Video Generation Models in Evaluation · p. 8
• Sora-480p is sampled on the official website at 480×854 for 150 continuous frames at 30 FPS.This is identified as the low-resolution version of Sora.Source: A. Video Generation Models in Evaluation · p. 8
• HunyuanVideo uses an MLLM [91] and a dual-branch diffusion transformer; evaluation samples 129 frames at 720×1280 and 24 FPS with default inference hyperparameters and seed 42.The best version is used without classifier-free guidance.Source: A. Video Generation Models in Evaluation · p. 8
• VBench-2.0 evaluations closely align with human judgment in all dimensions, as assessed by human-preference and benchmark-evaluation win ratios.Figure 11 plots the win ratios and shows a linear fit with Spearman’s correlation coefficient for each dimension.Source: Figure 11 · p. 9
• CogVideoX-1.5 [59] uses a VAE to compress spatial and temporal dimensions and an expert transformer generative architecture for better text-video alignment; its best 5B version is sampled as 161 continuous 768×1360 frames at 16 FPS, with seed 42.Source: A. Video Generation Models in Evaluation · pp. 8–9

B. VBench-2.0 Evaluation Results

• Scores from 0 to 1, with higher values indicating stronger relative performance, summarize four video generative models in Table II and Figure 2.Videos for each sub-ability dimension were generated from its corresponding prompt suite and evaluated using the method introduced in Section III-A.Source: B. VBench-2.0 Evaluation Results · p. 9
• Figures 4–8 visualize one model pair for Human Anatomy, Diversity, Complex Landscape, Mechanics, and Instance Preservation across the five major dimensions.The figures correspond respectively to Human Fidelity (Figure 4), Creativity (Figure 5), Controllability (Figure 6), Physics (Figure 7), and Commonsense (Figure 8).Source: B. VBench-2.0 Evaluation Results · p. 9

C. Human Alignment of VBench-2.0

• Human preference labels, following VBench, [26], were used to correlate VBench-2.0 evaluations with annotations; Figure 11 and Table IV report model-level alignment by dimension.Figure 11 presents the correlation plot, while Table IV gives numerical win ratios.Source: C. Human Alignment of VBench-2.0 · p. 9

• The discussion highlights trade-offs identified through VBench-2.0’s evaluation.Source: §V · p. 9
• The section discusses characteristics of the evaluated models.Source: §V · p. 9
• The section examines superficial versus intrinsic faithfulness in video generation.Source: §V · p. 9

A. Characteristics of Recent SOTA Models

• Sora excels in Human Fidelity and Creativity but is weaker in Controllability, Physics, and Commonsense, whereas Kling is strong in Commonsense, Controllability, and camera-related dimensions.Kling shows no significantly weaker area; the paper describes it as a useful reference for future model development.Source: A. Characteristics of Recent SOTA Models · p. 9
• CogVideoX-1.5 [59] is relatively strong in complex prompt adherence and Physics but weaker in Human Fidelity and Motion Rationality; HunyuanVideo shows the opposite human-related strengths.The paper suggests CogVideoX has limited high-quality human-related training data, while HunyuanVideo likely benefits from data rich in such content.Source: A. Characteristics of Recent SOTA Models · pp. 9–10

B. Key Limitations of Recent SOTA Models

• Models fail in about 80% of simple dynamic-attribute or spatial-relationship cases, while current models struggle to generate coherent complex plots.The paper links these limitations to videos typically being under 10 seconds and to dataset captions that insufficiently describe temporal changes.Source: B. Key Limitations of Recent SOTA Models · p. 10

C. The Role of Prompt Engineering

• Sora performs well on Creativity but struggles with Controllability, while other models show the reverse pattern, suggesting a creativity-control trade-off.The Prompt Refiner used with the other three models may improve controllability through finer-grained prompts at the expense of diversity; Sora’s internal prompt optimization appears different.Source: C. The Role of Prompt Engineering · p. 10
• Except for Sora-480p, models guided by the external Prompt Refiner show reasonably strong Physics performance, suggesting prompts can steer videos toward plausible outcomes.The authors suggest the core difficulty may lie more in precise video-text alignment than in physical reasoning itself.Source: C. The Role of Prompt Engineering · p. 10
• Prompt Refiners show no consistent performance trend on Human Fidelity, Camera Motion, Geometry, or Commonsense, dimensions tied to visual understanding and prior knowledge.The paper suggests performance in these areas may depend more on data quality and model architecture than prompt engineering.Source: C. The Role of Prompt Engineering · p. 10

D. Superficial Faithfulness vs. Intrinsic Faithfulness: Do Not Miss Out on Any Pillar

• Visual quality alone can mislead model judgments: CogVideoX performs relatively well across VBench-2.0 dimensions despite room to improve on VBench’s Quality Score, while HunyuanVideo looks impressive but has room to grow on structure-driven dimensions.Superficial faithfulness shapes first impressions, but intrinsic faithfulness—including storytelling, logical progression, and world simulation—is also important for real-world applications.Source: D. Superficial Faithfulness vs.… · p. 10
• The authors recommend using VBench and VBench-2.0 together to assess both superficial and intrinsic faithfulness.Visual quality can bias judgments: aesthetically pleasing, smooth outputs may be perceived as better even when structure-driven performance differs.Source: D. Superficial Faithfulness vs.… · p. 11

E. Challenge, Current Solution and Future Work of Video Generation Evaluation

• Current VLM evaluators struggle with human anatomy and motion rationality because training data lacks generated-video anomalies and models perform poorly at object counting and 3D understanding.Intrinsic faithfulness evaluation requires complex scene understanding, commonsense reasoning, and physical-world perception.Source: E. Challenge, Current Solution and… · p. 11
• VBench-2.0 trains task-specific anomaly specialists on abnormal human-body, motion, and entity splitting or merging data mixed with real datasets to balance accuracy and generalizability.Source: E. Challenge, Current Solution and… · p. 11
• VBench-2.0 avoids fine-tuning, which risks overfitting and poor generalization to longer videos, by decomposing tasks into VLM-understandable phenomena, asking redundant questions, and adding LLMs for high-level understanding.Source: E. Challenge, Current Solution and… · p. 11
• Future work proposes a unified clip-level anomaly detector trained on abnormal data, since anomaly types overlap, such as limb anomalies during entity splitting.Source: E. Challenge, Current Solution and… · p. 11
• Future benchmarks should address sustainability and accuracy by evaluating models’ understanding of physical laws and 3D scenes; fine-tuning would need length-independent clips and a virtual database matching VLM training-data scale.Suggested evaluation areas include 3D space reasoning, cinematography understanding [93], and physics understanding,.Source: E. Challenge, Current Solution and… · p. 11

VI. CONCLUSION

• VBench-2.0 complements VBench with a human-aligned framework assessing five dimensions of intrinsic faithfulness, using VLMs and LLMs where capable and specialists where they are not.The conclusion frames intrinsic faithfulness around physical laws, commonsense reasoning, and structured coherence.Source: §VI · p. 11
• The authors plan to add video-generation models as they become available and emphasize responsible use and safety as generated scenes become more realistic.Although VBench-2.0 does not directly generate videos, evaluation involves generated content and may carry societal risks.Source: §VI · p. 11