Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains

• In science-domain videos, visual plausibility can conceal violations of conservation laws, reversed causal relations, or impossible state transitions, so reliable generation requires mechanistic fidelity as well as realism (Bansal et al., 2025a; Meng et al., 2025).Source: §1 · p. 2
• Existing benchmarks emphasize perceptual quality, alignment, compositionality, or generic physical commonsense, while a multidisciplinary benchmark for scientific mechanisms with portable expert-defined criteria was still missing.Source: §1 · p. 2
• Sci-VBench comprises 1,253 expert-authored, independently reviewed tasks across 60 subjects and four disciplines, pairing minimal prompts with reference guides and detailed rubrics anchored to observable 1–5 scores.The disciplines are Natural Science, Healthcare, Humanities & Social Sciences, and Engineering.Source: §1 · p. 2
• The evaluation protocol combines VBench-based Vision Tools for Low-level Perceptual Fidelity with rubric-conditioned MLLM judging of grounding, scientific and causal correctness, and spatiotemporal consistency.Evaluation specifications improve non-expert agreement with experts; expert rerating is stable (Cohen’s κ = 0.842), and rubric-conditioned MLLM scores align more closely with experts than prior video scoring methods (He et al., 2024; 2025; Guan et al., 2025).Source: §1 · p. 2
• Across 16 evaluated models, a pronounced proprietary–open-source gap centers on Prompt Grounding and Scientific and Causal Correctness, while automatic perceptual-quality scores are nearly flat.Qualitative analysis also finds systematic mechanistic errors in leading models despite visually convincing outputs.Source: §1 · p. 2
• Prompt rewriting improves Prompt Grounding and Scientific and Causal Correctness but has limited effect on Spatiotemporal Consistency, suggesting temporal failures often stem from generators rather than underspecified instructions.Source: §1 · p. 2

2.1 Video Generative Models

• Video-generation systems have progressed from short, low-fidelity clips to high-fidelity, temporally coherent outputs, alongside work on physical-law adherence and commonsense reasoning (Li et al., 2025b; Henschel et al., 2024; The Sora Team, 2025; Google DeepMind, 2025; Xue et al., 2025; Gillman et al., 2025; Hu et al., 2025c).This progress is attributed to diffusion models combined with large-scale transformer architectures (Ma et al., 2025).Source: §2.1 · pp. 2–3
• Related work also studies scientific multimodal reasoning and static scientific visual generation (Wang et al., 2025a; Chen et al., 2026; Zhao et al., 2025b; Wang et al., 2025b).Source: §2.1 · p. 3

2.2 Evaluation of Video Generative Models

• Existing video-generation benchmarks mainly assess perceptual fidelity and motion, with some testing compositionality and prompt following, while reasoning-focused benchmarks test generic physical commonsense rather than science-domain knowledge (Huang et al., 2024; He et al., 2024; Li et al., 2025a; Huang et al., 2026; Zheng et al., 2025; Sun et al., 2025; Bansal et al., 2025a;b).Prior work also reports reliability gaps in model-based evaluation of generated videos and complex scientific tasks, motivating structured specifications such as Sci-VBench's per-example rubrics (Song et al., 2025; Zhao et al., 2025a;d).Source: §2.2 · p. 3
• Compared with VideoScience-Bench's 200 expert-annotated physics and chemistry prompts, Sci-VBench spans 60 subjects, releases reusable per-example guides and rubrics, and evaluates portability across expert, non-expert, and MLLM raters (Hu et al., 2025b).Source: §2.2 · p. 3

3 Sci-VBench Benchmark

• Sci-VBench contains 1,253 expert-authored examples across 60 subjects and four disciplines, designed around breadth, expert reasoning, spatiotemporal reasoning, and reproducible rubric-based evaluation.Each prompt requires grounded scientific understanding and multi-step causal reasoning, and its target phenomenon must unfold over time rather than be captured by a single frame.Source: §3 · p. 3

3.1 Evaluation Dimensions

• The benchmark scores videos on Low-level Perceptual Fidelity, Prompt Grounding, Scientific and Causal Correctness, and Spatiotemporal Consistency.The latter two dimensions target mechanism-level failures that generic quality or prompt-alignment criteria do not capture.Source: §3.1 · p. 4

3.2 Preliminary Setup for Benchmark Construction

• Subject selection drew on a study of 133 students, while 61 trained expert annotators authored examples and independent experts reviewed them.The selected 60 subjects span four disciplines and have core concepts that are expert-level yet verifiable from video evidence.Source: §3.2 · p. 4

3.3 Video Prompt Annotation

• Prompts derive from canonical textbooks and focus on concepts with mechanism-governed visual realizations that can be verified from video evidence.Each minimal prompt specifies the observable initial setup and any intervention or objective, but omits the expected mechanistic trajectory so models must infer it.Source: §3.3 · pp. 4–5

3.4 Evaluation Specification Curation

• Each prompt includes a high-level reference guide and a 1–5 rubric that externalize expert knowledge for non-expert or MLLM evaluation.Guides specify target concepts, mechanistic assumptions, phase-based expected phenomena, causal transitions, and verification constraints; rubrics tie observable evidence to scoring anchors, with per-example anchors for reasoning dimensions and a shared rubric for perceptual fidelity.Source: §3.4 · p. 5

3.5 Data Quality Control

• An audit of 200 examples found high agreement across reasoning dimensions: quadratic weighted κ = 0.79 for Prompt Grounding, 0.75 for Scientific and Causal Correctness, and 0.73 for Spatiotemporal Consistency.Independent expert review checks prompt clarity, visual testability, and consistency across the prompt and specification; Sci-VBench also provides a 150-example testmini split alongside the 1,253-example full split.Source: §3.5 · pp. 5–6

4.1 Human Evaluation Protocols

• Discipline-matched experts score each video from 1–5 across four dimensions using the prompt, video, and per-example specification, with calibration to observable rubric cues.The cues include state transitions, causal consistency, and verifiable evidence in the video.Source: §4.1 · p. 6

4.2 Automated Evaluation Protocols

• For benchmarking, Vision Tools scores Low-level Perceptual Fidelity, while a rubric-conditioned judge scores Prompt Grounding, Scientific and Causal Correctness, and Spatiotemporal Consistency.Vision Tools averages six VBench Video Quality metrics after normalization and maps the composite to the rubric dimensions’ 1–5 range.Source: §4.2 · p. 6
• The Qwen3.5-397B-A17B judge receives the prompt, video, reference guide, and dimension-specific 1–5 rubric, gives a video-grounded justification, and averages three independent scores.Restricting the rubric to one dimension is intended to prevent evidence from bleeding across dimensions.Source: §4.2 · p. 6

4.3 Measuring Reliability of Evaluation Protocol

• Across 300 videos rated twice by independent discipline-matched experts, Cohen’s κ was 0.842, indicating strong inter-expert agreement.The reliability analysis uses expert reference ratings for 1,500 videos generated by ten systems released on or before January 2026.Source: §4.3 · p. 7
• With evaluation specifications, non-expert Pearson correlations were LPF 84.7 vs. 83.2, PG 83.3 vs. 74.1, SCC 82.5 vs. 68.2, and SC 79.6 vs. 70.3, exceeding every tested MLLM judge on every dimension.The specifications modestly improved Low-level Perceptual Fidelity and more substantially improved the other three dimensions.Source: Table 3 · p. 7, §4.3 · p. 7
• Qwen3.5-397B-A17B had the highest MLLM-judge correlations overall, agreement generally increased with evaluator scale, and the strongest rubric-conditioned judge beat VideoScore, VideoScore2, VideoReward, and ETVA on every dimension.MLLM judges aligned more strongly on Prompt Grounding and Scientific and Causal Correctness than on Low-level Perceptual Fidelity and Spatiotemporal Consistency.Source: §4.3 · p. 7

5.1 Evaluated Models

• The benchmark compares 16 frontier text-to-video models—eight proprietary and eight open-source—using verbatim prompts and each model’s default configuration.The reported version, resolution, frame rate, and clip duration for each model are listed in Appendix A.3.Source: §5.1 · pp. 7–8

5.2 Main Results

• VT is nearly flat (3.79–4.12), but automatic SCC spans 1.24–3.34 and human SCC 1.12–3.06; Gemini-Omni-Flash leads automatic (3.38) and human (3.18) averages.VT is the VBench-based proxy for low-level perceptual fidelity; the reported averages are over the testmini results.Source: §5.2 · p. 8, Table 4 · p. 8
• Gemini-Omni-Flash leads three disciplines, but Sora-2 leads Healthcare; open-source MiniMax-H3 reaches 1.63 SCC while Wan2.2-5B has the highest automatic Spatiotemporal Consistency score, 2.79.Gemini-Omni-Flash scores 3.33 on Engineering, 3.13 on Natural Science, and 2.95 on Humanities & Social Sciences; Sora-2 scores 2.97 on Healthcare.Source: Figure 3 · p. 8, §5.2 · p. 9

5.3 Error Analysis

• Observed errors fall into three categories: poor instruction adherence, inaccurate simulation of scientific principles, and deficiencies in temporal coherence and visual quality.These include missed prompt details, visually appealing but physically incorrect dynamics, and illogical changes over time.Source: §5.3 · p. 9

5.4 Effect of Prompt Rewriting

• +23.3%/+51.7% SCC, +12.4%/+26.7% Prompt Grounding, and +7.5%/+16.5% Spatiotemporal Consistency gains occurred for Wan2.2-5B/HunyuanVideo-1.5, respectively.The gap narrows but remains: HunyuanVideo-1.5 reaches SCC 2.20, above the best verbatim open-source score of 1.63 but below the strongest proprietary score of 3.34.Source: §5.4 · p. 9

6 Conclusion

• Across 16 frontier models, Sci-VBench reveals a persistent gap between perceptual realism and scientific validity, alongside a substantial proprietary/open-source gap on reasoning-centric dimensions.The benchmark contains 1,253 expert-authored prompts spanning 60 subjects across four disciplines.Source: §6 · p. 9
• Observed failure modes point to opportunities to improve instruction adherence, mechanistic consistency, and spatiotemporal stability in expert-domain video generation.Source: §6 · p. 9

A.1 Subject Selection

• Table 5 lists Sci-VBench’s subjects by major discipline, with subfields shown under each of the four disciplines.The supplied passage identifies the table’s scope and organization but does not provide the subject names or subfield values.Source: Table 5 · p. 15

A.2 Data Quality Control

• Across 200 audited examples, independently authored specifications yielded κ=0.79 for Prompt Grounding, κ=0.75 for Scientific and Causal Correctness, and κ=0.73 for Spatiotemporal Consistency.Two independent scorers rated the same video under both specifications in counterbalanced order, using a crossed scorer–rubric design.Source: §A.2 · p. 15

A.3 Video Generation Models

• Evaluated models used verbatim benchmark prompts under default configurations, with durations closest to the 15-second target; several models had shorter duration caps, and HappyHorse-1.1 clips carried a watermark.The table gives exact model identifiers or repositories, and reports measured resolution and frame rate; duration caps include 8 seconds for Veo-3.1 and 5 seconds for CogVideoX1.5-5B.Source: Table 6 · p. 16, §A.3 · p. 16

A.4 Annotator Information

• Biographies for 61 Sci-VBench annotators are listed, with author-annotator biographies withheld to protect anonymity.Source: §A.4 · p. 17, Table 7 · p. 17, Table 8 · p. 18

B.1 Low-level Video Quality: Definitions and Implementation Details (Adapted from VBench)

• Subject Consistency tracks whether subject identity and attributes remain stable across frames, while Background Consistency assesses continuity in scene layout, lighting, and spatial structure.Source: §B.1 · p. 19
• Motion Smoothness evaluates temporal continuity and physical plausibility, whereas Dynamic Degree measures motion intensity and richness without judging correctness.Source: §B.1 · p. 19
• Aesthetic Quality assesses visual appeal and artistic composition, while Imaging Quality measures frame-level fidelity, including sharpness, noise, compression artifacts, and clarity.Source: §B.1 · p. 19

B.2 Comparing with Prior Video Scoring Methods

• ETVA (Guan et al., 2025) scores prompt alignment by parsing prompts into scene graphs, generating atomic questions, and aggregating knowledge-augmented, multi-stage question-answering scores.Its score follows the official protocol.Source: §B.2 · p. 19
• VideoScore (He et al., 2024) averages five aspect scores, while VideoScore2 (He et al., 2025) averages three reported dimensions for each video.VideoScore is trained on VideoFeedback; VideoScore2 uses VideoFeedback2 human scores and reasoning traces with a two-stage SFT+RL pipeline.Source: §B.2 · p. 19
• VideoReward (Liu et al., 2025) averages its per-video Visual Quality, Motion Quality, and Text Alignment scores into an overall score.The VLM-based reward model was trained on a large-scale human preference dataset.Source: §B.2 · p. 19
• The rubric-conditioned MLLM-as-Judge has the highest instance-level Pearson correlation among compared automatic methods on Prompt Grounding, Scientific and Causal Correctness, and Spatiotemporal Consistency.ETVA is comparatively competitive on Prompt Grounding and Scientific and Causal Correctness but degrades sharply on Low-level Perceptual Fidelity and Spatiotemporal Consistency.Source: §B.2 · pp. 19–20

B.3 Evaluation Prompt Templates

• Figure 5’s single-dimension prompt template fills angle-bracketed fields per example and includes only that dimension’s rubric anchors.The template is reproduced from the scoring code.Source: §B.3 · p. 21, Figure 5 · p. 21
• Figure 6’s released evaluation specification presents the verbatim generation prompt followed by the judged dimension’s anchored 1–5 rubric.The illustrated specification is for the knee-jerk reflex example and is reproduced verbatim from the released dataset.Source: §B.3 · p. 21, Figure 6 · p. 22

C.1 Full-Benchmark Results

• The corresponding overall averages appear in Table 4’s Full Avg. column.Source: §C.1 · p. 23
• Table 9 covers per-dimension automatic evaluation for open-source models across all 1,253 Sci-VBench examples.Its results are sorted by the rightmost Avg. column.Source: Table 9 · p. 23, §C.1 · p. 23

Poor Adherence to Instructions

• Models often ignore fine-grained prompt details, such as a specified push-button switch, rendering a fingertip touching the board instead.This illustrates poor adherence to instructions even in leading models.Source: §C.2 · p. 23, Figure 7 · p. 24

Inaccurate Simulation of Scientific Principles

• Models fail to respect scientific principles and preconditions, producing errors such as an unstruck leg kicking during a knee-jerk reflex.In another example, squeezing a bag beside the eye does not trigger a blink; the analysis says models prioritize visual aesthetics over scientific realism.Source: §C.2 · p. 23, Figure 8 · p. 24

Deficiencies in Temporal Coherence and Visual Quality

• Temporal and visual defects include merging objects, discontinuous style shifts, coarse textures, and events with weak causal links, disrupting believability.Examples include two balls merging mid-motion, realistic prism imagery switching to animation, and an LED changing state independently of a hand waving before a motion sensor.Source: §C.2 · p. 23