PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench

Lanxiang Hu, Abhilash Shankarampeta, Yixin Huang, Zilin Dai, Haoyang Yu, Yujie Zhao, Haoqiang Kang, Daniel Zhao, Tajana Rosing, Hao Zhang

arXiv:2512.02942v1cs.CVcs.AI

TL;DR

Existing video benchmarks primarily test physical commonsense, leaving limited evidence about scientific reasoning across concepts. VideoScience-Bench evaluates generated scientific phenomena, and VideoScience-Judge supports automated assessment; results show that strong visual quality and temporal coherence do not ensure instruction-following or adherence to physical laws.

  • Problem

    Existing video benchmarks largely focus on everyday physical commonsense, offering limited evaluation of models’ ability to reason across interacting scientific concepts.

    Source: §1 · p. 2

  • Method

    VideoScience-Bench evaluates expert-rated generations across five dimensions, while VideoScience-Judge uses checklists, key frames, and computer-vision evidence to support automated scoring.

    Source: §1 · p. 2

  • Results

    Across evaluated models, videos often had high visual quality and temporal coherence but struggled with instructions and physical laws; Sora-2 and Veo3 scored approximately 64% and 58.7% on Phenomenon Congruency, respectively.

    Source: §4.1 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    Scientific world modeling requires assessing whether generated phenomena reflect interacting scientific concepts, not only visual quality or physical commonsense.

    Source: §1 · p. 2

    The evaluation incurs prohibitively high API cost.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

The next frontier for video generation lies in developing models capable of zero-shot reasoning, where understanding real-world scientific laws is crucial for accurate physical outcome modeling under diverse conditions. However, existing video benchmarks are physical commonsense-based, offering limited insight into video models' scientific reasoning capability. We introduce VideoScience-Bench, a benchmark designed to evaluate undergraduate-level scientific understanding in video models. Each prompt encodes a composite scientific scenario that requires understanding and reasoning across multiple scientific concepts to generate the correct phenomenon. The benchmark comprises 200 carefully curated prompts spanning 14 topics and 103 concepts in physics and chemistry. We conduct expert-annotated evaluations across seven state-of-the-art video models in T2V and I2V settings along five dimensions: Prompt Consistency, Phenomenon Congruency, Correct Dynamism, Immutability, and Spatio-Temporal Continuity. Using a VLM-as-a-Judge to assess video generations, we observe strong correlation with human assessments. To the best of our knowledge, VideoScience-Bench is the first benchmark to evaluate video models not only as generators but also as reasoners, requiring their generations to demonstrate scientific understanding consistent with expected physical and chemical phenomena. Our data and evaluation code are available at: \href{https://github.com/hao-ai-lab/VideoScience}{github.com/hao-ai-lab/VideoScience}.

1. Introduction

VideoScience-Bench asks whether video models can reason across interacting scientific concepts and portray the resulting phenomena, beyond visual fidelity and physical commonsense. The paper introduces a benchmark and an automated VLM-based judge to evaluate that capability.

1. Introduction

  • VideoScience-Bench targets undergraduate-level scientific reasoning across 103 physics and chemistry concepts, evaluating five dimensions from prompt alignment to spatio-temporal coherence.Its prompts require reasoning across multiple scientific concepts to generate expected phenomena.Source: §1 · p. 2
  • Existing video benchmarks largely test everyday physical commonsense, leaving limited evaluation of scientific reasoning across interacting principles.Scientific world modeling matters for predicting outcomes of real-world processes; for example, specific-heat and heat-transfer principles predict that water absorbs heat and prevents immediate balloon rupture.Source: §1 · p. 2
  • VideoScience-Judge combines prompt-specific checklists, key-frame selection, and evidence aggregation to produce weighted scores, with judgments reported to align better with domain-expert annotations than other physical-commonsense and visual-fidelity benchmarks.The judge uses salient video frames to ground its evaluations.Source: §1 · p. 2
  • The study reports that current models can produce photorealistic, temporally coherent videos yet often fail to model complex scientific phenomena or follow basic physical laws.Sora-2 and Veo-3 perform best among the evaluated models and show early signs of scientific reasoning.Source: §1 · p. 2
  • The data-creation pipeline has researchers select multiple scientific concepts and reference educational materials or videos before prompts undergo peer, model, and quality review.Prompts are finalized for dataset inclusion after these checks.Source: Figure 2 · p. 3

2. VideoScience-Bench

VideoScience-Bench evaluates videos depicting complex scientific events that require multiple concepts. It defines the benchmark’s scope and scoring dimensions, then describes expert-authored prompts and review procedures.

2.1. Benchmark Overview

  • The benchmark contains 160 questions spanning 14 undergraduate topics and 103 concepts, with 9 physics topics and 5 chemistry topics.The concepts comprise 79 physics and 24 chemistry concepts.Source: §2.1 · p. 3
  • Each challenge requires at least two undergraduate-level principles, unlike commonsense benchmarks built around everyday scenes explainable with basic or high-school science.The sugar-solution laser example requires both Snell’s Law and diffusion physics to explain beam bending across a concentration gradient.Source: §2.1 · p. 3
  • The benchmark has 160 T2V prompts and 40 paraphrased I2V prompts, with initial experimental-setup images provided for I2V.Source: §2.1 · p. 3
  • Videos are scored on five dimensions using a four-point ordinal scale, separating perceptual qualities from physical fidelity and requiring prompt-aligned setups and procedures.Physical fidelity covers phenomenon congruency and correct dynamism; perceptual ratings cover immutability and spatio-temporal coherence.Source: §2.1 · pp. 3–4

2.2. Benchmark Building

  • Eight graduate-student experts each authored 20 prompts designed to require at least two concepts, drawing on public educational experiments and targeting phenomena feasible within 10 seconds.A panel checked prompt specificity, procedural accuracy, and clarity, then video models synthesized opening seconds to verify that key setups could be captured.Source: §2.2 · p. 4

3. VideoScience-Judge

VideoScience-Judge uses checklists, salient frames, and computer-vision evidence to support video ratings. Its design responds to under-justified judgments and false positives from direct VLM evaluation.

3. VideoScience-Judge

  • Directly prompting VLMs to evaluate generated videos can yield under-justified assessments and false-positive ratings, motivating an evidence-grounded evaluation harness.Source: §3 · p. 4
  • The judge’s ratings can be accompanied by salient frames, checklist items, and CV-derived evidence to make evaluation traceable.The scoring prompt can selectively enable the checklist and CV-based evidence table.Source: Figure 3 · p. 4
  • An LLM agent generates prompt-specific checklists, and the fraction of satisfied items in each category determines an ordinal rating on a four-point scale.Deductions are made only when supported by concrete evidence, with reference phenomena and I2V videos guiding identification of violation frames.Source: §3 · p. 4
  • Computer-vision modules supply evidence on entity presence, tracking and coherence, motion, text-video alignment, and—when available—similarity to a reference video.The modules include Grounding DINO, ByteTrack, RAFT optical flow, CLIP4Clip, and LPIPS with a VGG backbone.Source: §3 · p. 4
  • The examples contrast Sora-2’s expected aluminum-iodine ignition with Hailuo-2.3’s failure, while even top-ranked Sora-2 and Veo-3 miss the rotating-cup phenomenon and procedure.Source: Figure 4 · p. 5, §3 · p. 5

4. Experiments

Experiments assess seven video models with expert ratings and evaluate how well VideoScience-Judge aligns with human judgments. The results show persistent difficulty generating scientifically correct phenomena, alongside gains from evidence-based judging.

4.1. Model Evaluation.

  • Even the strongest models scored only about 64% and 58.7% on Phenomenon Congruency, while most performed well on coherence and immutability.Sora-2 and Veo3 reached approximately 64% and 58.7%, respectively; Wan-2.5, the strongest open-source model, reached around 46%.Source: §4.1 · p. 6, Table 1 · p. 6
  • VideoScience-Judge achieved the highest alignment with human evaluations among the compared baselines.The study measures ranking correlations with human annotations.Source: §4.1 · p. 6, Table 2 · p. 7

4.2.1. VideoScience-Judge Design Choice

  • Replacing GPT-4o with GPT-5 pro mitigated prompt-consistency score inflation by up to 57%, but ranking misalignments with human judgments remained.The discrepancy remained noticeable despite using a reasoning model.Source: §4.2.1 · p. 6
  • Adding checklist hints, key-frame evidence, and CV tools reduced judge overestimation across dimensions, narrowing coherence and immutability gaps by 0.07 and 0.09.The reductions are relative to vanilla VideoScience-Judge without the checklist.Source: §4.2.1 · p. 6, Table 3 · p. 7

4.2.2. Comparison with Baselines

  • PhyGenEval aligned less with human judgments than VSci-Judge, and its rigid binary checks miss holistic motion and cascading multi-concept dynamics.VSci-Judge had ρ = 0.89 versus ρ = 0.61 for PhyGenEval.Source: §4.2.2 · pp. 6–7
  • VideoScore2 and T2V-CompBench's VLM judge aligned less with human assessments than VSci-Judge on this benchmark.The passages attribute this gap to their focus on broad commonsense or compositional evaluation rather than nuanced scientific reasoning.Source: §4.2.2 · p. 7

4.3. Qualitative Analysis

  • Sora-2 produced a coherent reaction video in one example, whereas Hailuo-2.3 maintained other dimensions but failed to show the expected phenomenon.The contrast underscores the importance of matching the experimental setup and prompt.Source: §4.3 · pp. 7–8
  • Even top-ranked models struggled with a simpler setup: Sora-2 made significant errors, while Veo-3 depicted it poorly.Both retained some positive dynamism, immutability, or spatio-temporal coherence despite failing to capture the setup or expected phenomenon.Source: §4.3 · p. 8

5. Related Work

Related work covers video-generation benchmarks, VLM-based evaluation, and benchmarks for physical commonsense. VideoScience addresses a gap in evaluating undergraduate-level reasoning across multiple scientific concepts and domains.

5.1. Video Generation Models

  • Video-generation evaluation suites span perceptual dimensions, compositionality, physical realism, and dynamic coherence.VBench and VBench++ decompose quality across dimensions, while WorldModelBench, VideoPhy, and Physion emphasize dynamics, physics, and material interactions.Source: §5.1 · p. 8

5.2. VLM-as-a Judge for Video Model Evaluation

  • Automated VLM judging is increasingly common, and T2V-CompBench combines frame reasoning with detection and tracking modules in a structured rubric.Its pipeline uses Grid-LLaVA alongside tools including GroundingDINO, SAM, Depth-Anything, and DOT.Source: §5.2 · p. 8

5.3. Physical Commonsense Benchmarks for Video Models

  • Prior physical-commonsense benchmarks assess basic laws, while VideoScience targets undergraduate-level synthesis across multiple physics domains and provides automated evaluation.The paper identifies a remaining need for SOTA models to understand physical phenomena.Source: §5.3 · p. 8

6. Conclusion

VideoScience-Bench evaluates scientific reasoning through generated phenomena, while VideoScience-Judge aligns most strongly with expert-rated rankings among the compared benchmarks. The appendix maps scientific subcategories to subjects and fine-grained concepts to subcategories.

6. Conclusion

  • VideoScience-Bench targets undergraduate-level physics and chemistry reasoning, and VideoScience-Judge aligns most strongly with expert rankings among the compared benchmarks.The conclusion reports that current models still struggle to follow instructions and adhere to physical laws despite high visual quality and temporal coherence.Source: §6 · p. 8

A.1. Subcategory-to-Subject Mapping

  • The subcategory-to-subject mapping groups nine physics and five chemistry subcategories, assigning each a two-letter code and subject.Examples include Classical Mechanics and Optics for physics, and Liquid Chemistry and Reaction Kinetics for chemistry.Source: §A.1 · p. 11

A.2. Concept-to-Subcategory Mapping

  • The concept-to-subcategory mapping assigns each fine-grained concept to exactly one subcategory, including optics concepts such as dispersion, reflection, and refraction.The appendix provides the complete mapping used.Source: §A.2 · p. 11

B.1. T2V Examples

The examples show that generated videos can depict scientific setups and phenomena correctly yet remain static or fail to show expected outcomes. The appendix also specifies checklist generation, evidence-grounded scoring, rubrics, and output fields.

B.1. T2V Examples

  • Sora-2 depicts the curved-refraction setup and expected phenomenon, but the output is largely static.Its human ratings are PCS 4/4, PCG 3/4, CDN 4/4, IMB 4/4, and STC 3/4.Source: Figure 5 · p. 14, §B.1 · p. 14
  • For water separation, Sora-2 fails to show the expected minimal mixing of the two colored liquids.The human ratings are PCS 3/4, PCG 2/4, CDN 3/4, IMB 4/4, and STC 3/4.Source: Figure 6 · p. 14, §B.1 · p. 14

B.2. I2V Examples

  • The I2V example conditions a Newton’s Cradle video on its first-frame image, but the expected phenomenon fails after the first strike.The expected behavior is the opposite-end ball swinging outward while the others remain nearly stationary.Source: §B.2 · p. 14, Figure 9 · p. 15, §C.1 · p. 15

C.1. Checklist Generation

  • Checklist generation uses the prompt and expected phenomenon to produce items that guide the VLM judge’s scoring and justification.Examples include dry ice in soapy water, gas passed into limewater, and Newton’s Cradle; the limewater example lacks the expected milky color and bubbles.Source: §C.1 · pp. 14–16, Figure 8 · p. 15

Prompt for Checklist Generation

  • The checklist-generation prompt requires the reference phenomenon and a comprehensive checklist tailored to applicable categories, including phenomenon congruency.It asks the judge to evaluate a generated video from the prompt and expected phenomenon.Source: Prompt for Checklist Generation · p. 16

C.2. Scoring

  • Scoring can selectively use checklist items and CV-based evidence, with the VLM judge justifying each rating from salient video frames.When available, justifications also draw on the corresponding checklist items and CV-derived evidence.Source: §C.2 · p. 16

RUBRIC TEXT

  • The rubric scores prompt consistency, expected phenomenon, immutability, dynamism, and coherence from 1 to 4, requiring clear justification for each rating.Its scale runs from absent or contradictory (1) to clearly correct (4).Source: RUBRIC TEXT · p. 17

OUTPUT SCHEMA FORMAT

  • The output schema returns dimension scores, explanations, and timestamped candidate and reference evidence observations.The scoring prompt combines the question, ground-truth phenomenon, optional checklist, optional CV evidence table, rubric, and output schema.Source: OUTPUT SCHEMA FORMAT · p. 17

D. Baseline Evaluation Results

The baseline comparisons show differing results across evaluation frameworks: T2VBench aligns moderately to strongly with human annotations, while VideoScore2 aligns weakly. PhyGenEval reports a low overall score for its top model on VideoScience.

PhyGenEval

  • Veo-3 scores 0.47 overall under PhyGenEval, and models score better on physics than chemistry-related topics.The passage characterizes PhyGenEval as a physical-commonsense framework and contrasts its approach with VideoScience-Judge’s evidence-centric evaluation.Source: D. Baseline Evaluation Results · p. 17

Correlation weighting

  • Equal weighting of all five dimensions changes τ and ρ very little compared with Table 2.The passage reports this as a sensitivity check on the reported correlations.Source: D. Baseline Evaluation Results · p. 17

T2VBench

  • T2VBench normalized scores range from 0.75 to 0.8, with Veo-3.1 highest and Ray2 lowest, and its automatic metric shows moderate-to-strong rank correlation with human annotations.The evaluation concerns consistent attribute binding; the passage notes that models perform somewhat similarly on this controlled task.Source: D. Baseline Evaluation Results · p. 18, Table 5 · p. 18

VideoScore2

  • VideoScore2 normalized aggregate scores range from 0.67 to 0.79, with Wan-2.5-T2V-Preview highest and Ray2 lowest, but alignment with human annotations is weak.The passage attributes weak alignment to VideoScore2’s emphasis on everyday commonsense plausibility and simple physical rules rather than deeper scientific reasoning and non-trivial dynamics.Source: D. Baseline Evaluation Results · p. 18

E. VideoScience-Judge Model Choice

Replacing GPT-4o with GPT-5-Pro reduces VideoScience-Judge’s discrepancy from human annotations, although the comparison reports that the size of this reduction varies by dimension.

E. VideoScience-Judge Model Choice

  • Up to 57%, GPT-5-Pro reduces VideoScience-Judge’s discrepancy from human annotations relative to GPT-4o.GPT-4o’s absolute deviations reach 0.384 for Coherence and 0.417 for Immutability; the comparison reports that its ratings are positively skewed and overestimate video quality.Source: E. VideoScience-Judge Model Choice · p. 18, Table 7 · p. 19

F. Details of Expert Annotation Results on VideoScience-Bench

Expert annotations are reported as raw and normalized model-by-dimension scores, with normalized ratings averaged after instance-level min-max normalization.

F. Details of Expert Annotation Results on VideoScience-Bench

Related papers

Loading 2512.02942v1…