Source-linked AI summary
Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation
Atin Pothiraj, Jaemin Cho, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal
TL;DR
Video generators still struggle with physical realism, while existing metrics lack fine-grained localization of violations. PQSG evaluates videos through dependency-structured questions and, on FinePhyEval, correlates better with human ratings than prior metrics, though VLM answering remains a bottleneck.
Problem
Video generation models often violate physical laws, while existing evaluation methods do not reliably localize specific physical-realism failures.
Method
PQSG represents video evaluation as a dependency-structured graph of atomic questions across objects, actions, and physical laws, supported by FinePhyEval human annotations.
Results
PQSG correlates better with overall human ratings than prior metrics, ranks proprietary models above open-source models, and yields human-like questions while VLMs achieve only ~65% QA accuracy.
Takeaways & Limitations
PQSG provides fine-grained feedback on physical-realism failures, while current VLMs remain substantially weaker at answering physics questions than generating them.
Takeaways & Limitations
PQSG’s performance is bounded by the question-answering capability of its underlying vision-language model.
Abstract
from arXiv · showhide
Video generation models are increasingly capable of producing realistic videos, but they still struggle to generate videos that follow basic physical laws. Compounding this is a lack of reliable granular evaluation methods for localizing and specifying physical law violations in videos. We address this by introducing Physics Question Scene Graph (PQSG), a hierarchical question-based evaluation pipeline. PQSG evaluates generated videos by checking their faithfulness to a prompt across objects, actions, and adherence to physical laws using a graph-based hierarchy of questions generated by a vision-language model (VLM), guided by high-quality in-context examples. By representing questions as a graph, PQSG introduces logical dependencies within questions, ensuring that each query is contextually valid. Moreover, PQSG provides granular assessments of which qualities of the video violate physical plausibility constraints. We validate PQSG by creating FinePhyEval, a dataset with physics-based prompts and corresponding generated videos from diverse state-of-the-art video generation models (Sora 2, Veo 3, and Wan 2.1), with each video annotated across multiple categories by humans. Using FinePhyEval, we measure the correlation between PQSG's fine-grained scores and human judgments, showing higher overall correlations than prior work. We also find that PQSG ranks closed-source models higher than Wan 2.1 on physical realism. Lastly, we show that the annotations we provide in FinePhyEval can also be used for subtask evaluation: we benchmark two strong VLMs on generating and answering questions, finding that while models can create human-like questions, they still fall short of human performance in answering them.
1 Introduction
Current video generators can produce photorealistic content but still struggle with physical plausibility, while existing evaluation methods lack reliable fine-grained analysis. PQSG addresses this gap through structured sub-queries, and FinePhyEval supports validation against human judgments and model subtasks.
- Motivation: Current video generation models produce increasingly photorealistic videos but still lack reliable adherence to basic physical laws.The paper frames physical understanding as part of world modeling and notes that current models remain limited despite photorealistic generation advances.
- Motivation: Existing physics-evaluation methods provide only coarse-grained scores and fail to identify specific physical inaccuracies.The paper argues that increasingly subtle model errors require fine-grained evaluation and feedback for assessment and video repair.
- PQSG: PQSG decomposes physical-law adherence into multiple logically structured sub-queries that localize which video attributes violate physical plausibility.Its design emphasizes granular feedback rather than a single overarching physics score.
- FinePhyEval: FinePhyEval contains 195 human-annotated prompt-video pairs generated by Sora 2, Veo 3, and Wan 2.1 for evaluating PQSG and video-generation models.The dataset uses prompts sourced from Physics-IQ and provides human annotations across object, action, physics, and additional video-quality categories.
- Results: PQSG correlates better with overall human ratings than prior metrics while also providing feedback at individual attribute-category levels.The paper additionally evaluates question generation and answering with Gemini-2.5-Pro and GPT-5.5, finding adequate question generation but substantially weaker question answering than humans.
- Results: ∼65% QA accuracy was achieved by GPT-5.5, the best model, with action and physics categories posing particular challenges.Human-performed question answering yields a high upper-bound correlation with human judgments, indicating room for improvement as VLMs advance.
2 Related Work
Recent text-to-video generation models can render increasingly realistic, complex scenes, yet they frequently violate basic physical principles. Existing graph-based question evaluation, such as DSG, targets images and does not test temporal actions or physics-based interactions.
- Video Generation Models: Video diffusion models have rapidly advanced and can realistically render complex, high-resolution scenes approaching human-shot video realism.
- Video Generation Models: Even state-of-the-art video generation models frequently violate basic physical principles.
- Video Generation Models: DSG provides graph-based, question-based evaluation for images but fails to test temporal properties such as actions and physics-based interaction.Its limitation is demonstrated by the evaluation in Sec. 5.1.
3 Physics Question Scene Graph (PQSG)
PQSG evaluates generated videos through a directed acyclic graph of prompt-derived questions spanning objects, actions, and physical plausibility. Its dependency-aware, per-question judgments produce category and overall scores while localizing detailed failure modes.
- Graph structure: PQSG represents the physical scene as a directed acyclic graph of atomic verification questions connected by explicit logical dependencies.All nodes are generated from the text prompt describing the scene.
- Question categories: The hierarchy separates Object presence, prompt-tied Action execution, and implicit Physics plausibility grounded in physical commonsense.Physics questions cover interactions such as deformation and absorption, distinguishing physical failures from omitted prompted actions.
- Dependency enforcement: PQSG permits object-to-action, action-to-physics, and same-category dependencies, while rejecting other connections and invalidating child chains when a parent answer is “no.”This structure captures prerequisites such as earlier actions or physical states required by later questions.
- Evaluation pipeline: The pipeline uses a VLM for question-graph generation from an in-context example and answers video-based questions one at a time during QA.The QA model receives the generated video and an individual PQSG question for each answer.
- Outputs: Per-question binary judgments and reasoning yield per-category scores and an overall score while pinpointing detailed video failure modes.The method is designed to identify whether a video omits an action or depicts an action with implausible physical behavior.
4 FinePhyEval: Human Annotation of Fine-grained Video Scores
FinePhyEval is a human-judgment dataset for evaluating generated videos across fine-grained physical and text-video alignment categories. It uses Physics-IQ prompts and includes both manually authored verification questions and Likert-scale annotations.
- Dataset construction: FinePhyEval collects human judgments across fine-grained categories for videos generated by recent state-of-the-art video generation models.The dataset is introduced to analyze the utility of PQSG’s fine-grained video evaluation.
- Evaluation prompts and reference videos: 65 Physics-IQ prompts provide complex, compositional tests of physical principles, with a reference ground-truth video for each prompt.The prompts include multiple objects and object interactions.
- Human annotation of verification questions: 20 prompts receive manually annotated verification questions that are comprehensive, atomic, and non-overlapping.These annotations establish a ground-truth set for question generation.
- Human annotation of video scores: 780 Likert-scale scores cover 195 generated videos, four alignment categories, and eight non-author annotators, with each score ranging from 1–5.The categories are object, action, physics, and overall alignment.
- Human annotation of video scores: The overall score captures video-text alignment generally, while object, action, and physics scores assess specific alignment criteria.These four-category judgments are compared with video evaluation metrics and category-specific human scores.
5 Experiments and Discussion
Experiments show that PQSG correlates well with human judgments, distinguishes physical-realism weaknesses across video models, and generalizes beyond FinePhyEval. Its question-generation component aligns closely with human annotations, while question answering remains difficult for physical-plausibility judgments.
- Overall score correlation: PQSG is evaluated against aggregate human Likert scores alongside VideoScore, VideoPhy-2-Autoeval, PhyGenEval, DSG, and direct VQA baselines using Pearson’s r, Kendall’s τ, and Spearman’s ρ.PQSG can aggregate its fine-grained scores into an overall score when an aggregate evaluation is desired.
- Model comparison: Video-generation models score lower on action and physics than on object prediction, indicating greater difficulty producing correct actions and physical interactions.The comparison covers Sora 2, Veo 3, Wan 2.1, and Cosmos-14B on FinePhyEval prompts.
- Model comparison: Sora 2 and Veo 3 substantially outperform Wan 2.1 and Cosmos-14B, while narrow confidence intervals indicate stable model rankings across repeated question-generation runs.PQSG scores are averaged over three question-generation runs in the model comparison.
- Question generation: Gemini-2.5-Pro and GPT-5.5 generate questions highly aligned with human annotations in precision and recall, with most mismatches involving failure to anticipate future prompt states.The evaluation manually annotates 20 graphs and compares generated questions with human-annotated questions.
- Question answering: VLMs answer object and high-level action questions accurately but achieve low accuracy on physics questions, reflecting persistent difficulty with physical and spatio-temporal reasoning.Question-answering performance is measured by comparing model yes/no responses with human-derived ground-truth answers.
- Generalization: On 100 VideoPhy2 test prompt-video pairs, PQSG improves both semantic adherence and physical commonsense, demonstrating generalization with an open-source VLM.The evaluation uses the same in-context examples and prompts as the other experiments.
6 Conclusion … A.1 Annotation Guidelines
The paper introduces PQSG and FinePhyEval to enable fine-grained evaluation of physical realism in generated videos. Its annotation protocol emphasizes objective, consistent, and physically grounded judgments through strict guidelines and quality controls.
- 6 Conclusion: PQSG addresses the lack of fine-grained, automated metrics for physical realism in video generation.The framework is presented as an evaluation approach for a critical gap in assessing generated videos.
- 6 Conclusion: PQSG decomposes evaluation into a structured dependency graph of atomic questions.The graph structure supports systematic evaluation by organizing questions according to their dependencies.
- 6 Conclusion: FinePhyEval provides fine-grained human annotations of videos from strong video generation models.The dataset is intended to facilitate research on reliable evaluation metrics.
- A Human Annotation Details: The annotation framework combines strict guidelines with quality-control measures to maximize objectivity and data consistency.These measures are designed to minimize annotation variance across generated videos.
- A.1 Annotation Guidelines: Annotators follow a human-realism assumption, judging videos against real-world standards rather than adapting judgments to generative model limitations.This principle is one of three protocol requirements intended to produce physically grounded evaluations.
- A.1 Annotation Guidelines: The protocol requires absolute semantic correctness checks so that objects and actions match the literal prompt definition.This guideline is part of the framework for consistent and objective evaluation across generated videos.
A.2 Quality Control and Reliability
A pilot study established annotation reliability for FinePhyEval using three ratings per video across object, action, physics, and overall judgment dimensions. Annotators showed strong overall agreement, although physics adherence was the noisiest dimension, after which broader student annotations were collected with the same guidelines and scales.
- Pilot annotation study: 150 annotations from 6 non-author students covered 50 videos, with 3 annotations per video across four dimensions on 5-point Likert scales.Each student rated a subset of 8 or 9 videos using a detailed instruction guide.
- Inter-annotator agreement: 0.840 average ICC indicated excellent agreement, while Krippendorff’s Alpha of 0.592 indicated moderate agreement.ICC above 0.75 is considered excellent, whereas Alpha values between 0.4 and 0.6 are considered moderate.
- Inter-annotator agreement: Physics adherence had the lowest agreement because judging very poor-physics videos produced more opinionated and noisy annotations.The study attributes this variability to annotators’ differing judgments when generated videos break physics laws.
- Full annotation collection: The remaining generated videos were annotated by a broader student group using the same Likert scales and category guidelines.Guidelines were displayed at the top of the annotation page to remind annotators of category meanings.
B Additional Experimental Results · B.1 QA with GPT 5.1
The results indicate that stronger video question-answering can improve PQSG’s alignment with human judgments. GPT 5.5 achieves the highest reported Pearson’s correlation of 0.478.
- B.1 QA with GPT 5.1: GPT 5.5 achieves the highest reported Pearson’s correlation of 0.478 for video question-answering.The result supports stronger alignment between PQSG and human judgments as video question-answering capabilities improve.
- B.1 QA with GPT 5.1: Improving VLM performance on rapid actions and intricate physical properties could yield higher correlation with human judgments.The passage identifies these question types as areas where better answering performance may improve PQSG’s correlation.
- B.1 QA with GPT 5.1: The findings further support that stronger models make PQSG more aligned with human judgments.This conclusion is drawn from the reported correlation result and the improved video question-answering capabilities of models such as GPT 5.5.
B.2 Ablation Study
The ablation study finds that adding reference videos or imposing stricter prompt variations on question generation significantly harms performance. Reference videos induce irrelevant questions, while added constraints reduce question coherence.
- Question generation ablations: Adding the reference video caused the model to generate questions unrelated to the prompt and focus on irrelevant ground-truth occurrences.This experiment targeted the question generation phase.
- Question generation ablations: Additional prompt constraints to distinguish action from physics questions made question generation struggle to produce coherent questions.The attempted constraints required explicit prompt actions for action nodes and physics content for physics nodes.
- Question generation ablations: Both question-generation experiments significantly impeded performance.The comparison is reported in Table 11.
C Additional FinePhyEval Details and Video Examples · D Prompts
The supplementary sections provide FinePhyEval video examples, annotation materials, dataset statistics, ablations, and the prompts and examples used for PQSG question generation and answering. They illustrate concrete physics-rendering failures and document the evaluation and prompt-construction procedures.
- C Additional FinePhyEval Details and Video Examples: FinePhyEval examples show failures in water dispensing, liquid color, and Newton’s-cradle motion, including an incorrect change in the number of metal balls.One video depicts water below the dispenser spout while the liquid turns orange in the glass; another misrepresents edge-ball motion in a Newton’s cradle.
- C Additional FinePhyEval Details and Video Examples: Human annotators received strict guidelines designed to make evaluations consistent, objective, and physically grounded across generated videos.The supplementary material labels these materials as annotation guidelines and examples.
- C Additional FinePhyEval Details and Video Examples: The supplementary material includes annotation-guideline examples and partial screenshots of the annotation interface provided to annotators.These materials are presented in Fig. 6 and Figs. 7–8.
- C Additional FinePhyEval Details and Video Examples: Extended FinePhyEval results report correlations between human overall Likert scores and different metrics, alongside an ablation of PQSG design choices.These results appear in Tables 10 and 11 as supplementary extensions and additional ablations.
- D Prompts: FinePhyEval statistics are provided in Table 12 as additional dataset documentation.The supplementary material also includes figures showing the annotation interface.
- D Prompts: The question-generation prompt instructs a graph-construction assistant to return a valid JSON object with exactly two top-level keys: “nodes” and “edges”.An example uses the video prompt “A water dispenser is dispensing water into a glass which has orange juice in it.”