PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation

Atin Pothiraj, Jaemin Cho, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal

arXiv:2606.25306v1cs.CVcs.AI

TL;DR

Video generators can look realistic yet violate physical laws, and evaluating physical plausibility at specific failure points remains challenging. This paper introduces PQSG, a dependency-structured question framework, and FinePhyEval; on the benchmark, PQSG correlates better with overall human ratings than prior metrics and ranks proprietary models above open-source models.

  • Problem

    Video generators still struggle with physical realism, while evaluating physical plausibility at specific failure points remains challenging.

    Source: §1 · p. 2, §3 · p. 5

  • Method

    PQSG represents video evaluation as a dependency-structured graph of atomic questions, and the authors create FinePhyEval with human-annotated videos for evaluating its scores.

    Source: §3 · p. 5, §6 · p. 15

  • Results

    PQSG correlated better with overall human ratings than prior metrics, ranked proprietary models above open-source models, and found all models struggled more with action and physics than object generation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Human overall ratings correlated most strongly with physics ratings (r = 0.85), indicating annotators' overall assessments were more predicted by physics errors than by errors in other categories.

    Source: §5.1 · p. 10

    PQSG's performance is bounded by the question-answering capability of its underlying VLM.

    Source: §6 · p. 15

Abstract

from arXiv · show

Video generation models are increasingly capable of producing realistic videos, but they still struggle to generate videos that follow basic physical laws. Compounding this is a lack of reliable granular evaluation methods for localizing and specifying physical law violations in videos. We address this by introducing Physics Question Scene Graph (PQSG), a hierarchical question-based evaluation pipeline. PQSG evaluates generated videos by checking their faithfulness to a prompt across objects, actions, and adherence to physical laws using a graph-based hierarchy of questions generated by a vision-language model (VLM), guided by high-quality in-context examples. By representing questions as a graph, PQSG introduces logical dependencies within questions, ensuring that each query is contextually valid. Moreover, PQSG provides granular assessments of which qualities of the video violate physical plausibility constraints. We validate PQSG by creating FinePhyEval, a dataset with physics-based prompts and corresponding generated videos from diverse state-of-the-art video generation models (Sora 2, Veo 3, and Wan 2.1), with each video annotated across multiple categories by humans. Using FinePhyEval, we measure the correlation between PQSG's fine-grained scores and human judgments, showing higher overall correlations than prior work. We also find that PQSG ranks closed-source models higher than Wan 2.1 on physical realism. Lastly, we show that the annotations we provide in FinePhyEval can also be used for subtask evaluation: we benchmark two strong VLMs on generating and answering questions, finding that while models can create human-like questions, they still fall short of human performance in answering them.

1 Introduction

Video generators can produce photorealistic videos yet violate physical laws, while existing evaluation methods provide coarse scores that obscure why a video fails. PQSG addresses this with dependency-aware questions and fine-grained scores; FinePhyEval supports its evaluation and comparison with human judgments.

Motivation

  • Prior metrics collapse object, action, and physics into aggregate scores, obscuring failure causes and sometimes rewarding visually realistic but physically implausible videos; finer feedback is needed for evaluation and repair.A plausible physics assessment first requires checking that the relevant object is present and performing the relevant action.Source: §1 · pp. 2–3

PQSG

  • PQSG generates prompt-conditioned object, action, and physics questions in a dependency graph, then uses a VLM to answer them and produce interpretable category scores.The dependencies prevent asking about an object's actions or physical interactions when that object is absent.Source: §1 · p. 3

FinePhyEval

  • FinePhyEval contains 195 human-annotated prompt-video pairs from Physics-IQ prompts and videos generated by Sora 2, Veo 3, and Wan 2.1, with four quality ratings and annotations for PQSG's question-generation and question-answering subtasks.The quality categories are object, action, physics, and overall quality; the annotations show strong inter-annotator agreement.Source: §1 · p. 3

Results

  • On FinePhyEval, PQSG correlates better with overall human ratings than prior metrics, ranks Sora 2 and Veo 3 above Wan 2.1 and Cosmos-14B, and finds action/physics harder than objects.Unlike prior metrics, PQSG also provides fine-grained feedback at the level of individual attribute categories.Source: §1 · p. 3
  • GPT-5.5 reaches ∼65% QA accuracy, with action and physics posing challenges; tested VLMs perform adequately at question generation but fall short of human QA, while human QA yields high upper-bound correlation.The videos average 4.39 seconds in length.Source: §1 · pp. 3–4

2 Related Work

Prior video-generation methods increasingly address physical realism, but common evaluation approaches miss fine-grained physical constraints. PQSG responds with structured, detailed questions that localize video shortcomings.

Video Generation Models

  • Physics-aware generation methods use simulator-guided motion, post-training and reward optimization, physical priors, or explicit law encoding, motivating actionable fine-grained evaluation.Recent studies still find physical-law violations in state-of-the-art models.Source: §2 · p. 4

Evaluation of Generated Videos

  • General video metrics overlook physical constraints; VideoScore emphasizes semantic and factual alignment, while VideoPhy-2 and PhyGenEval assess physical realism.Earlier metrics rely on feature embeddings, classifier predictions, semantic matching, or evaluation toolkits, but do not pinpoint physical-law violations.Source: §2 · p. 4
  • PQSG generates hierarchical detailed questions to localize needed improvements, whereas image-focused DSG cannot test temporal properties such as actions or physics-based interactions.The paper characterizes PQSG's granular QA format as interpretable, consistent across answerers, human-aligned, and generalizable.Source: §2 · pp. 4–5

3 Physics Question Scene Graph (PQSG)

PQSG evaluates generated videos with a graph of dependent questions spanning objects, actions, and physical plausibility. Its prerequisites make judgments contextually valid and its fine-grained scores help locate failure modes.

3 Physics Question Scene Graph (PQSG)

  • PQSG models a physical scene as a directed acyclic graph of Object, Action, and Physics questions, separating prompt-specified actions from implicit physical-commonsense interactions.Object questions check presence, Action questions check whether objects perform the prompt’s actions, and Physics questions assess whether those actions are physically plausible.Source: §3 · p. 5
  • Binary judgments with reasoning provide per-question failure diagnoses, category scores for Object, Action, and Physics, and an overall score.The question-level results help pinpoint detailed video failure modes.Source: Figure 2 · p. 6, §3 · p. 7
  • Object-to-action-to-physics and same-category edges enforce affirmative prerequisites, preventing unsupported judgments and yielding stronger human correlation than unstructured approaches.A child is queried only when all parent questions are answered affirmatively; otherwise, its dependent questions are automatically marked “no.”Source: §3 · pp. 6–7
  • The pipeline generates questions from a prompt using an in-context example, then answers each question with an open-ended response followed by yes/no categorization.The two-step answering strategy improves QA quality while adding negligible latency; invalid nodes are marked “no,” and scores can be separated by category.Source: §3 · p. 7

4 FinePhyEval: Human Annotation of Fine-grained Video Scores

FinePhyEval provides human annotations to evaluate PQSG’s fine-grained video assessments, covering verification questions and video scores across multiple categories.

4 FinePhyEval: Human Annotation of Fine-grained Video Scores

  • FinePhyEval is a dataset of human judgments across fine-grained categories for videos generated by recent state-of-the-art video models.It was collected to analyze the utility of PQSG’s fine-grained video evaluation.Source: §4 · p. 7

4.1 Prompt and Video Generation

  • Using all 65 Physics-IQ prompts, the study generates 260 prompt-video pairs (65 each) from Sora 2, Veo 3, Wan 2.1, and Cosmos-Predict2.5-14B [1].The videos use each model’s default configuration.Source: §4.1 · pp. 7–8

4.2 Human Annotation of Question Generation and Answering

  • Human annotations cover 20 prompts for atomic, non-overlapping verification questions and 444 yes/no answers across 30 prompt-video pairs.The question annotations support measuring automated question-generation precision and recall; the human answers establish an upper bound for question-answering performance.Source: §4.2 · p. 8

4.3 Likert-scale Video Judgments

  • For 195 videos, eight non-author annotators provide 780 Likert-scale (1–5) scores across object, action, physics, and overall categories.The overall score rates video-text alignment without focusing on the other three criteria; inter-annotator agreement is 0.84 across categories on a subset.Source: §4.3 · pp. 8–9

5 Experiments and Discussion

Experiments evaluate PQSG against human judgments, compare video generators, test question generation and answering, and examine generalization, iterative refinement, and design choices.

5.1 Correlation to Human Judgments: Generated Video Evaluation

  • PQSG correlates similarly or better with human judgments than VideoScore, VideoPhy-2-Autoeval, PhyGenEval, DSG, and direct VQA across three coefficients.Human physics ratings correlate most strongly with overall ratings (r = 0.85); automated QA nearly matches human scores for objects but has room to improve for actions and physics.Source: §5.1 · pp. 9–10, Table 2 · p. 10, Table 3 · p. 11

5.2 Comparing Video Generation Models via PQSG

  • Sora 2 and Veo 3 outperform Wan 2.1 and Cosmos-14B by a large margin, while all models score lower on action and physics than on objects.Narrow confidence intervals show that rankings remain consistent across repeated question-generation runs.Source: §5.2 · pp. 10–11, Table 4 · p. 11

Question Generation.

  • Gemini-2.5-Pro and GPT-5.5 generate PQSG questions with precision and recall above 90%, though mismatches often omit possible future object interactions.The evaluation compares generated questions with human annotations on 20 graphs, assessing accuracy and coverage of prompt aspects.Source: Question Generation · pp. 11–12, §5.3 · p. 11, Table 6 · p. 12

Question Answering.

  • VLMs answer object and high-level action questions accurately but struggle with physics, rapid actions, and complex interactions, often exhibiting “yes-bias”.A qualitative example shows GPT-5.5 mischaracterizing smoke direction and confidently asserting actions absent from the video.Source: Table 5 · p. 11, Question Answering · pp. 12–13, Figure 3 · p. 12

5.4 Generalization Study: External Dataset and Open-source VLM

  • On 100 VideoPhy2 test prompt-video pairs, PQSG increases both semantic adherence and physical commonsense, demonstrating generalization to another dataset and an open-source VLM.Source: Table 7 · p. 12, §5.4 · p. 13

5.5 Iterative Refinement with PQSG

  • One PQSG-guided prompt-refinement iteration raises the average PQSG score by nearly 15% for videos generated with Wan 2.2 TI2V-5B.The loop evaluates the initial video, uses GPT-5.5 to refine the prompt, and repeats generation and evaluation.Source: §5.5 · p. 13, Figure 4 · p. 14

5.6 Ablation Study

  • Removing fine-grained questions or the dependency graph degrades correlation with human ratings, showing that both contribute to PQSG’s alignment.The ablations also test adding a reference video during question generation; each tested design change reduces performance.Source: §5.6 · p. 14, Table 8 · p. 14

6 Conclusion … A.1 Annotation Guidelines

PQSG evaluates physical realism through a dependency graph of atomic questions, and FinePhyEval provides fine-grained human annotations for generated videos. The conclusion highlights limits from VLM question-answering ability and prompt-scoped assessment, while the annotation appendix specifies objective, physically grounded guidelines.

6 Conclusion

  • PQSG addresses the lack of fine-grained automated physical-realism metrics with a dependency graph of atomic questions and FinePhyEval’s human-annotated videos.The experiments analyze human-score correlations, compare generation methods, evaluate question generation and answering, and ablate design choices.Source: §6 · p. 15
  • 64.6% accuracy on physics questions limits PQSG’s performance, while human QA reaches Pearson’s correlation of 0.80, indicating a high ceiling.PQSG is model-agnostic, and human correlation scales with VLM capability; it improves reliability over direct VQA, but backbone performance remains important.Source: §6 · p. 15
  • Closed-source VLMs in the main experiments may limit reproducibility; the authors release code, prompts, and annotations and show open-source VLM generalization.The open-source VLM generalization result is reported in Table 7.Source: §6 · p. 15
  • PQSG assesses only prompt-entailed interactions, leaving unconstrained regions unevaluated; prompt-free assessment is proposed as future work.Humans can assess physics without a prompt.Source: §6 · p. 15

A Human Annotation Details

  • The annotation framework combines strict guidelines and quality-control measures to maximize objectivity and data consistency.The guidelines are intended to minimize variance and ensure consistent, objective, physically grounded evaluations.Source: A Human Annotation Details · p. 19, §A.1 · p. 19

A.1 Annotation Guidelines

  • Annotators judge videos by real-world standards, verify literal prompt-object and action correctness, and assess physical plausibility independently of text alignment.Physical plausibility is judged solely by adherence to real-world laws.Source: §A.1 · p. 19

A.2 Quality Control and Reliability

The study reports consistent, reliable human annotations overall, while physics-adherence judgments show the lowest agreement and greater variability. Reliability was assessed through a pilot involving six annotators and 50 videos.

A.2 Quality Control and Reliability

  • Six non-author undergraduate and graduate students annotated 50 videos, providing three ratings per video across object, action, physics, and overall judgment.Each rated a subset of eight or nine videos on 5-point Likert scales, yielding 150 annotations; annotators received a detailed instruction guide.Source: §A.2 · p. 19
  • ICC averaged 0.840 and Krippendorff’s Alpha was 0.592; physics adherence had the lowest agreement, but the authors conclude annotations are reliable overall.The paper characterizes Alpha values from 0.4 to 0.6 as moderate and ICC values above 0.75 as excellent; it notes that judgments become noisier when videos break physics laws.Source: §A.2 · pp. 19–20, Table 9 · p. 20
  • After validating annotator agreement, the researchers collected annotations for the remaining generated videos from a broader selection of students using the same Likert scales.The annotation page displayed category guidelines.Source: §A.2 · p. 20

B Additional Experimental Results · B.1 QA with GPT 5.1

Stronger video question-answering capabilities are associated with closer alignment between PQSG and human judgments. GPT 5.5 achieves the highest reported Pearson correlation, 0.478.

B.1 QA with GPT 5.1

  • 0.478 Pearson correlation is the highest reported, with GPT 5.5, supporting the finding that stronger VLMs align PQSG more closely with human judgments.The paper suggests that improving VLM performance on questions about rapid actions and intricate physical properties could further increase correlation with humans.Source: §B.1 · p. 20

B.2 Ablation Study

Adding a reference video or imposing stricter action-versus-physics prompt constraints significantly impeded question-generation performance.

B.2 Ablation Study

  • Both adding reference video and imposing action–physics prompt constraints significantly impeded performance.Reference video led to questions about irrelevant ground-truth occurrences rather than the prompt, while added constraints made coherent question generation difficult.Source: §B.2 · pp. 20–21

C Additional FinePhyEval Details and Video Examples · D Prompts

The appendix illustrates FinePhyEval videos with concrete physical-plausibility failures and provides additional dataset, annotation, evaluation, ablation, and PQSG prompt materials.

C Additional FinePhyEval Details and Video Examples

  • Four FinePhyEval examples show water flowing despite a low dispenser level and changing color, incorrect Newton’s cradle motion and ball counts, an implausible teapot reflection, and erratic tennis-ball collisions.The examples illustrate limited physics-rendering capabilities in video diffusion models.Source: C Additional FinePhyEval Details and… · p. 21
  • Figure 6 presents annotation guidelines and examples.Source: Figure 6 · p. 22
  • Table 10 extends the reported correlations between human overall Likert scores and metrics on FinePhyEval.Source: Table 10 · p. 23
  • Table 11 presents additional ablations of PQSG design choices.Source: Table 11 · p. 23

D Prompts

C Additional FinePhyEval Details and Video Examples

Related papers

Loading 2606.25306v1…