PaperlayerView exact revision on arXiv

Source-linked AI summary

PhyGround: Benchmarking Physical Reasoning in Generative World Models

Juyi Lin, Arash Akbari, Yumei He, Lin Zhao, Haichao Zhang, Arman Akbari, Xingchen Xu, Zoe Y. Lu, Enfu Nan, Hokin Deng, Edmund Yeh, Sarah Ostadabbas, Yun Fu, Jennifer Dy, Pu Zhao, Yanzhi Wang

arXiv:2605.10806v1cs.CVcs.AIcs.LG

TL;DR

PhyGround addresses gaps in evaluating whether generated videos follow physical laws by combining per-law diagnostics with quality-controlled human evaluation and an open judge. Models with similar overall scores can have different physical-domain strengths, and PhyJudge-9B has lower aggregate relative bias than Gemini-3.1-Pro.

  • Problem

    Existing physics-focused video benchmarks can obscure law-specific failures, while human evaluation faces response biases and automated judges may lack physics specialization or auditability.

    Source: §1 · p. 2

  • Method

    PhyGround evaluates generated videos with a 13-law taxonomy across three domains, per-law sub-questions, quality-controlled human judgments, and an open physics-specialized VLM judge.

    Source: §1 · p. 2

  • Results

    Models with nearly identical Overall scores show sharply different physical-domain strengths; PhyJudge-9B’s aggregate relative bias is 3.3% versus Gemini-3.1-Pro’s 16.6%.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    PhyGround provides a diagnostic foundation for understanding current video-generation models as world models and supporting future improvements in physical reasoning.

    Source: §5 · p. 9

Abstract

from arXiv · show

Generative world models are increasingly used for video generation, where learned simulators are expected to capture the physical rules that govern real-world dynamics. However, evaluating whether generated videos actually follow these rules remains challenging. Existing physics-focused video benchmarks have made important progress, but they still face three key challenges, including the coarse evaluation frameworks that hide law-specific failures, response biases and fatigue that undermine the validity of annotation judgments, and automated evaluators that are insufficiently physics-aware or difficult to audit. To address those challenges, we introduce PhyGround, a criteria-grounded benchmark for evaluating physical reasoning in video generation. The benchmark contains 250 curated prompts, each augmented with an expected physical outcome, and a taxonomy of 13 physical laws across solid-body mechanics, fluid dynamics, and optics. Each law is operationalized through observable sub-questions to enable per-law diagnostics. We evaluate eight modern video generation models through a large-scale, quality-controlled human study, grounded on social science lab experiment design. A total of 459 annotators provided 5,796 complete annotations and over 37.4K fine-grained labels; after quality control, the retained annotations exhibited high split-half model-ranking correlations (Spearman's rho > 0.90). To support reproducible automated evaluation, we release PhyJudge-9B, an open physics-specialized VLM judge. PhyJudge-9B achieves substantially lower aggregate relative bias than Gemini-3.1-Pro (3.3% vs. 16.6%). We release prompts, human annotations, model checkpoints, and evaluation code on the project page https://phyground.github.io/.

1 Introduction

PhyGround targets gaps in evaluating whether generated videos follow physical laws, introducing law-specific diagnostics, quality-controlled human evaluation, and an open physics-specialized judge.

1 Introduction

  • Existing benchmarks can obscure law-specific failures through broad or binary ratings, while human judgments face fatigue and response biases and automated judges lack physics specialization or auditability.These gaps complicate reliable comparison and diagnosis of physical plausibility.Source: §1 · p. 2
  • PhyGround’s taxonomy covers 13 laws across solid-body mechanics, fluid dynamics, and optics, using sub-questions and 1–5 ratings for per-law and per-domain diagnostics.The scale distinguishes violation severity rather than collapsing deviations into binary judgments.Source: §1 · p. 2
  • 459 annotators produced 5,796 complete annotations and over 37.4K fine-grained labels; quality controls yielded split-half model-ranking correlations above Spearman ρ > 0.90.The protocol randomized annotator arrival and video order, capped workloads, and filtered low-quality annotations.Source: §1 · p. 2
  • 3.3% vs. 16.6% aggregate relative bias: PhyJudge-9B scored lower than closed-source Gemini-3.1-Pro.PhyJudge-9B is an open VLM judge fine-tuned on PhyGround human annotations for per-law scoring.Source: §1 · pp. 2–3
  • PhyGround is fully open-source, with prompts, human annotations, model checkpoints, and evaluation code publicly released for reproducible and auditable physical video evaluation.The project presents these resources as a foundation for evaluation.Source: §1 · p. 3

2 Related Work

Prior physics-focused video benchmarks vary in diagnostic resolution, human-evaluation scale, and evaluator openness; PhyGround combines explicit outcomes, per-law decomposition, a larger quality-controlled annotator pool, and an open judge.

2 Related Work

  • VideoPhy and VideoPhy-2 use binary judgments, PhyGenBench has 160 prompts, and Physics-IQ relies on reference-based pixel metrics rather than per-law diagnosis.Other benchmarks use binary scoring or target counterfactual rule-breaking and science scenarios rather than per-law physical-correctness diagnosis.Source: §2 · p. 3
  • Open video-physics judges trained on holistic labels cannot provide per-law diagnostics or severity calibration, while closed-source VLM judges raise reproducibility concerns.General video metrics and learned video-quality metrics are not designed or trained to detect physical-law violations.Source: §2 · p. 3
  • PhyGround’s positioning combines explicit expected outcomes and per-law decomposition with a larger quality-controlled annotator pool and an open-source judge.The comparison covers six physics-focused video-generation benchmarks.Source: §2 · p. 3

3 Benchmark Design

PhyGround combines curated prompts with explicit outcomes, a per-law taxonomy, criterion-focused VLM scoring, structured human annotation, and a judge trained on human score labels.

3.1 Prompt Suite Construction

  • 250 curated prompts specify expected physical outcomes and span varied actions and objects, giving evaluators clear scoring references for the intended physical events.Prompts were selected for unambiguous outcomes, visually verifiable events, and concise everyday scenarios.Source: §3.1 · p. 4

3.2 Physics Criteria Taxonomy

  • The taxonomy separates 13 laws across three domains, scores 2–4 applicable laws per video, and uses a 1–5 scale to preserve violation severity.Fluid dynamics is decomposed into distinct constraints such as flow dynamics, boundary interaction, displacement, and continuity.Source: §3.2 · p. 4

3.3 Automatic VLM Judge

  • The automatic judge evaluates one criterion per query, using law-specific observation sub-questions; each video receives 5–7 VLM queries.The focused queries are intended to reduce interference among heterogeneous criteria and produce diagnostic scores.Source: §3.3 · p. 5

3.4 Scoring

  • Scoring combines three general dimensions with applicable per-law scores; human annotators rate randomized video assignments on desktop and undergo two rounds of quality control.General dimensions measure semantic alignment, plausible event order, and object persistence; each applicable law receives an independent score.Source: §3.4 · p. 5

3.5 Judge Training

  • PhyJudge-9B is trained with LoRA supervised fine-tuning on video, single-criterion prompts, and structured human score labels, holding out Veo-3.1 annotations for evaluation.Training uses annotations from the other seven models.Source: §3.5 · p. 5

4 Experiments

Experiments assess eight video generators with 250 prompts and a quality-controlled human study, then compare model performance across physical domains and evaluate automated judges. The results show that holistic scores can conceal different domain strengths, while annotator aggregates yield stable rankings and PhyJudge-9B has lower bias than closed-source judges.

4.1 Setup

  • The study evaluates eight video generators on 250 prompts and gathers annotations from 459 annotators, retaining 352 after quality control.The retained pool provides 4,576 complete annotations and 29.5K individual labels; four mutually independent signals inform quality control.Source: §4.1 · p. 6

4.2 Main Results

  • No model exceeds 3.3/5 overall, and Wan2.2-27B-A14B and Veo-3.1 tie at 3.28 despite sharply different domain strengths.Wan2.2 leads on solid-body, temporal coherence, and persistence; Veo-3.1 leads on fluid and optical scores.Source: §4.2 · pp. 6–7
  • Domain and per-law decomposition reveals uneven model capabilities and a benchmark difficulty profile, with solid-body scoring lowest across models at 2.79.Cosmos-Predict2.5-2B has the widest within-model domain range, from 2.53 on solid-body to 3.35 on optical.Source: §4.2 · p. 7

4.3 Annotation Reliability

  • Split-half rankings remain strong across 100 annotator partitions, with Spearman ρ of 0.93/0.91/0.93 for General/Physics/Overall.Dropping 30% of annotators shifts model scores by 0.033 RMS, compared with 0.204 RMS when sampling down to 20 annotators.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Judge Results

  • On the strictly held-out Veo-3.1 evaluation, PhyJudge-9B has lower per-dimension aggregate relative bias than closed-source Gemini-3.1-Pro and Claude (Opus 4.7), with lower values better.Source: §4.4 · p. 7, Table 5 · p. 8
  • Adding sub-questions moves Gemini-3.1-Pro closer to human means on Fluid and Optical, while adding chain-of-thought on top is not consistently beneficial.The reported means shift from 1.86 to 3.29 on Fluid and from 2.64 to 3.28 on Optical.Source: §4.4 · p. 8, Table 6 · p. 9
  • On a rotating-mirror teapot case, PhyJudge-9B scores reflection and shadow near the human rating, while Gemini-3.1-Pro rates both at 1.Humans rate the clip 4.4 and PhyJudge-9B 4.6; Gemini misreads the physically plausible scene as a near-total optical violation.Source: §4.4 · p. 8, Figure 3 · p. 8
  • PhyJudge-9B’s fixed weights and full scoring pipeline support independent reproduction and auditing, and its results remain closer to human labels across tested frame rates than base judges.The frame-rate sweep tests 1/2/4/8 fps; main results use 4 fps.Source: §4.4 · p. 9

5 Conclusion

PhyGround evaluates physical reasoning in generative world models at the level of individual physical laws. It combines a quality-controlled human annotation set with an open physics-specialized judge to support scalable and reproducible evaluation.

5 Conclusion

  • PhyGround provides a criteria-grounded benchmark for evaluating physical reasoning in generative world models at the level of individual physical laws.The benchmark reveals distinct strengths and weaknesses across physical constraints.Source: §5 · p. 9
  • The work provides a large-scale, quality-controlled human annotation set and an open physics-specialized automatic judge for evaluation.These resources are intended to support scalable and reproducible evaluation.Source: §5 · p. 9
  • The benchmark offers a diagnostic foundation for future improvements in physical reasoning in video generation models.The paper frames this as advancing understanding of current models as world models.Source: §5 · p. 9

A Prompt Suite and Physics Law Selection Rationale

This section explains how PhyGround builds an evaluable prompt suite around clear physical outcomes and a ti2v-compatible taxonomy of 13 laws, enabling law-specific judgments while excluding phenomena or metrics unreliable in short videos. It also describes quality-control rules that use converging indicators to remove annotations with uninformative scoring patterns or signs of non-deliberate ratings.

A.1.1 Source Complementarity

  • The suite draws from four complementary prompt sources rather than adopting one benchmark wholesale, targeting evaluable scenarios for text+image-to-video.The 250-prompt suite redistributes prompts and first-frame images from sources under permissive research-use licenses with attribution preserved.Source: §A.1.1 · p. 12, Table 7 · p. 12, §A.1.3 · p. 13

A.1.4 Filtering Criteria

  • Prompts are retained only when outcomes are unique, visually verifiable, concise and familiar, aligned with the target domain, and driven by observable physical processes.This excludes ambiguous outcomes, specialized or rare objects, superficial keyword matches, machine operations, and active human or animal motion with unobservable initial conditions.Source: §A.1.4 · p. 13

A.1.5 Expected-Outcome Augmentation

  • Expected outcomes are made explicit so models must attempt scorable physical events and evaluators have an unambiguous reference.Gemini 2.5 Flash appends a concise outcome description to implicit prompts, which are manually checked; lengthy OpenVid captions are rewritten as concise physics-focused prompts, while prompts with explicit outcomes remain unchanged.Source: §A.1.5 · p. 13

A.2.1 Selection Criteria

  • The 13-law taxonomy is selected for visual observability, VLM discriminability, and compatibility with ti2v evaluation without a ground-truth video.It excludes invisible quantities and short-duration phenomena, and avoids reference-dependent pixel metrics that penalize physically plausible but visually different generations.Source: §A.2.1 · p. 14

A.2.2 Included Laws

  • The taxonomy covers six solid-body, five fluid, and two optical laws, with reflection and shadow treated separately because their evaluation criteria differ.Reflection tests whether mirrored content matches the scene; shadow tests directional consistency with the light source. The shadow/reflection split has not been adopted by prior benchmarks.Source: §A.2.2 · pp. 14–15
  • Each law is operationalized through binary violation probes for the VLM judge, while human annotators do not see the checklist.The probes structure judge reasoning and reduce ambiguity without making human scoring overly mechanical.Source: §A.2.2 · p. 15, Figure 4 · p. 16

B.2 Evaluation Design

  • Human evaluation uses a controlled university-based pool, randomized assignment and ordering, desktop viewing, training, hidden model identities, and capped workloads.The pool includes 35 physics-coursework graduate students and more than 400 non-experts; annotators rate observable plausibility rather than solve equations or estimate physical parameters.Source: §B.1 · p. 17, §B.2 · p. 18

B.3 Annotation Setup

  • The study collected 5,796 complete annotations and 37,408 score labels from 459 annotators before filtering, retaining annotations from 352 annotators after quality control.Two-round filtering uses score consistency, dimension discriminability, peer agreement, and behavioral engagement; score-constancy and per-video copy-paste checks are complementary signals.Source: §B.2 · p. 18, §B.3 · p. 19, §B.4 · p. 21

B.4 Quality Control Pipeline

  • Peer MAE measures annotators’ average absolute score deviation from peers on the same video-dimension cells, with values above 1.8 flagged as extreme.On the 1–5 scale, the threshold corresponds to nearly two scale points of average disagreement and roughly twice typical population disagreement.Source: §B.4 · p. 21
  • Peer MAE alone did not trigger removal because honest annotators may disagree; it had to coincide with a behavioral anomaly or high copy-paste rate.Source: §B.4 · p. 21
  • Behavioral flags included median page stay below 30 seconds, stay below 10 seconds, or zero video plays across all submissions.The study treated these as suspicious because rating the dimensions and laws required deliberate video viewing, but either signal alone could be noisy.Source: §B.4 · p. 21
  • 23.3% of annotators and their ratings were removed, reducing the pool from 459 raw annotators to 352 retained annotators.Source: §B.4 · p. 21

B.5 Removed Annotators

  • Constant or near-constant scoring was the largest removal source, defined as zero score variation or standard deviation below 0.3.Source: §B.5 · p. 21
  • A distinct per-video copy-paste pattern assigned identical scores across dimensions within each video while varying scores across videos, making per-law diagnostics uninformative.This pattern could have nonzero overall variance, so the constancy rule alone would miss it.Source: §B.5 · p. 21
  • Removal also followed combinations of weak signals, including copy-paste rates of at least 75% with short stays or unplayed videos, and extreme peer MAE paired with behavioral abnormalities.Extremely low annotation volume or stay time was also grounds for removal.Source: §B.5 · p. 22

B.6 Difficult-Item Analysis

Annotation difficulty did not produce a systematic skipping pattern: annotators spent longer on harder items, while copy-paste responses rose only slightly on the hardest items. The analysis also found no physical dimension markedly harder to annotate than the others.

B.6 Difficult-Item Analysis

  • Very hard items received the longest average stay time, while total skip rate was low and showed no clear relationship with difficulty.Roughly 30% of videos were classified as very hard; the evidence did not indicate systematic skipping.Source: §B.6 · p. 22
  • Copy-paste rates rose slightly on very hard items, but few annotators contributed few hard-item labels, limiting their effect on aggregated means.The cleaning procedure removed annotators with clear copy-paste patterns; remaining low-effort cases were described as sparse and isolated.Source: §B.6 · p. 22
  • No physical dimension was markedly harder to annotate than the others.Source: §B.6 · p. 22

B.7 Ranking Robustness

Model rankings remained stable under alternate aggregation and annotator sampling, with split-half ratings recovering similar orderings. The annotator pool’s diversity index was 2.01× that of a hypothetical 12-CS-PhD panel.

B.7 Ranking Robustness

  • Table 11 shows that median aggregation changed none of the eight model ranks and shifted Overall scores by at most ≈0.02 on the 1–5 scale.Both bootstrap confidence intervals were tight and mostly non-overlapping between adjacent models.Source: Table 11 · p. 22, §B.7 · p. 22
  • Split-half rank correlations were high across score views: Spearman ρ was 0.930 for General, 0.907 for Physics, and 0.928 for Overall.Across 100 random splits, the lower empirical interval bounds were Spearman ρ ≥0.773 and Kendall τ-b ≥0.605.Source: §B.7 · p. 23
  • Dropping 30% of annotators perturbed scores by 0.033 RMS, while sampling only 20 annotators increased the deviation to 0.204 RMS.Table 12 compares these perturbations with the full-pool baseline.Source: §B.7 · p. 23, Table 12 · p. 23

B.8 Annotator Diversity

  • The Annotator Diversity Index was 2.01× that of a hypothetical 12-CS-PhD panel, with the largest gaps in major and cohort.ADI is the geometric mean of effective category counts across gender, education, major, age, and cohort.Source: §B.8 · p. 23

C Aggregation Sensitivity

Alternative Physics weighting schemes left model rankings nearly unchanged, although equal-law-global weighting produced some score shifts. The analysis also reports per-domain weighting gaps and compares the annotator pool’s diversity with a hypothetical expert panel.

C Aggregation Sensitivity

  • Rankings remained nearly unchanged under equal-law aggregation schemes, suggesting high-frequency laws did not drive the headline leaderboard.The Physics score weights Solid-Body, Fluid, and Optical by 693, 136, and 47 scored prompt-video-law instances, respectively.Source: C Aggregation Sensitivity · p. 23
  • The annotator diversity index was 2.01× that of a hypothetical 12-CS-PhD panel, with the largest gaps on major and cohort.ADI is the geometric mean of effective category counts across five demographic facets.Source: C Aggregation Sensitivity · p. 24, Table 13 · p. 24

C.1 Schemes

  • The tested alternatives are equal-law-within-domain weighting and equal-law-global weighting, contrasted with the current sample-weighted scheme.Equal-law-global assigns each of 13 laws 1/13 of the Physics aggregate; Overall remains 0.5×General + 0.5×Physics.Source: §C.1 · p. 24

C.2 Per-Domain ∆from Removing Within-Domain Law-Count Imbalance

  • Table 14 reports each domain’s absolute gap between equal-law-within-domain and sample-weighted means across the eight comparison models.It gives the mean and maximum gap and identifies the model attaining the maximum.Source: Table 14 · p. 24, §C.2 · p. 24

C.4 Takeaway

  • Within-domain law imbalance changed Overall by a mean absolute ∆≈0.007; equal-law-global left rankings stable up to a one-position shift.The only persistent swap was Wan2.2-27B-A14B ↔ Veo-3.1, already a near-tie with ∆Overall = 0.002 under sample weighting.Source: §C.4 · p. 24

C.3 Overall Sensitivity Under Each Alternative Scheme

  • Table 15 compares each alternative scheme’s absolute Overall shift against the sample-weighted baseline and reports induced rank changes.Source: §C.3 · p. 24, Table 15 · p. 25

D Per-Law Score Breakdown

Per-law results reveal model strengths and weaknesses that domain averages conceal, while qualitative cases show PhyJudge-9B often tracks human judgments more closely than Gemini-3.1-Pro. The section also describes the no-sub-question judge ablation and points to training configuration details.

D Per-Law Score Breakdown

  • Within solid-body mechanics, domain leadership reverses across laws, and the law each model should target first differs, making per-law scores useful for targeted post-training.Wan2.2-27B-A14B leads the domain aggregate, but other models win two of its six laws; Material is the only law where neither leading open-source nor closed-source model is best.Source: D Per-Law Score Breakdown · pp. 25–26
  • Momentum and Collision are the hardest laws across models, with dynamic contact resolution weaker than free motion or static non-overlap.The cited bottlenecks include rebound direction, restitution, and fracture patterns.Source: D Per-Law Score Breakdown · p. 25
  • Veo-3.1’s fluid lead is concentrated in flow and liquid–solid interaction, while conservation is near-tied; model profiles also vary from balanced to spiky across laws.Wan2.2-27B-A14B is the most balanced, Cosmos-Predict2.5-2B the most spiky, and LTX-2-19B flips rank between Impenetrability and fluid laws.Source: D Per-Law Score Breakdown · p. 25

E Per-Law Qualitative Examples of Judge–Human Alignment

  • The qualitative cases select one example per law where PhyJudge-9B tracks humans and Gemini-3.1-Pro differs by at least two rating levels, across six generators and three domains.They show frames, prompts, score comparisons, and Gemini’s conclusion sentence, using the judges’ main-results sub-question schemas.Source: E Per-Law Qualitative Examples of… · p. 26
  • Other cases show opposite-direction errors: Gemini-3.1-Pro misses broken gravity and momentum, while also over-penalizing plausible liquid–ice interaction.PhyJudge-9B matches human scores in the cited gravity and momentum cases and stays close on boundary interaction.Source: Figure 8 · p. 26, Figure 12 · p. 28, Figure 13 · p. 28
  • Inertia, Collision, and Material examples show Gemini-3.1-Pro assigning 1.0 to human-plausible motion or contact, while PhyJudge-9B scores near the human ratings.Examples include plausible frisbee inertia, colliding bearings coming to rest, and an axe cutting a pumpkin.Source: Figure 7 · p. 26, Figure 10 · p. 27, Figure 11 · p. 27

F Judge Bias without Per-Law Sub-Questions

  • The no-SubQ ablation trains a 9B checkpoint on the same data with one score per video instead of per-law sub-questions.The supplied passage describes the ablation setup but gives no result.Source: F Judge Bias without Per-Law… · p. 28

G VLM Judge Training Details

  • The judge fine-tune’s LoRA, optimization, and compute settings are reported in Table 18, including hardware, peak per-GPU memory, and training time.The passage identifies the table’s contents without giving its values.Source: G VLM Judge Training Details · p. 28

G.1 Training Data

The training and evaluator configuration specify structured human-score supervision, video-frame inputs, and independent scoring calls. The appendix also illustrates judge–human agreements and disagreements across physical laws.

G.1 Training Data

  • Each training sample combines video frames, the evaluation-time prompt augmented with an expected outcome, and a JSON target containing the evaluation key and human score.The prompt is kept identical between training and evaluation; frames are sampled at 4 fps and capped at 12 during training.Source: §G.1 · pp. 28, 32
  • Examples across buoyancy, displacement, flow, continuity, reflection, and shadow show PhyJudge-9B tracking human ratings, while Gemini-3.1-Pro errs in either direction.The supplied cases include Gemini under-scoring plausible filling and flow, and over-scoring a continuity violation and a changing reflected object.Source: Figure 14 · p. 29, Figure 15 · p. 29, Figure 16 · p. 29
  • The no-SubQ checkpoint is trained with a no-sub-question template, and Table 17 defines its bias measures and comparison scope.The table reports grouped mean-bias and signed relative bias for judges under baseline prompts without per-law sub-questions.Source: Table 17 · p. 30

H Evaluator Configuration Details

  • The evaluator appendix describes its prompt structure and compares schema choices and inference-time frame-sampling rates.The supplied system-prompt excerpt says only that the evaluator is a strict video evaluation model.Source: H Evaluator Configuration Details · p. 32, §H.1 · p. 32

H.2 Scoring Dimensions and Prompt Structure

  • Each general or physical-law dimension is scored in a separate VLM call that returns a 1–5 score, with sub-questions serving as a checklist rather than required individual responses.The three general dimensions are Semantic Alignment, Physical Temporal Validity, and Object Persistence; their prompts include a video description, reference sub-questions, and scale anchors.Source: §H.2 · p. 32, §H.2.1 · p. 32

H.3 Schema Ablation

The schema ablation favors per-law sub-questions without chain-of-thought as the default, while frame-sampling results indicate modest sensitivity across tested rates and closer human agreement from PhyJudge-9B than base judges.

H.3 Schema Ablation

  • Gemini-3.1-Pro’s mean-bias drops from 39.2%/47.9%/27.1% on Persist./Fluid/Optical under the bare rubric to 17.3%/7.8%/9.4% under +SUBQ.The authors adopt +SUBQ as the default; the same direction holds on the Persist. row for the open 9B reference.Source: §H.3 · p. 32
  • Adding chain-of-thought to +SUBQ is non-monotone: Optical bias falls to 1.8%, but Fluid bias rises from 7.8% to 20.6%, so the authors retain +SUBQ without CoT by default.The ablation compares bare rubric, +SUBQ, and +SUBQ +COT schemas at physics-domain granularity.Source: §H.3 · p. 32
  • The schema ablation reports per-law and per-domain errors against per-law human labels and directly compares prompt schemas, making rubric and reasoning-style effects separately measurable.Table 19 distinguishes general-dimension and physics-domain columns, which score independent sub-rubrics.Source: §H.3 · pp. 32–33, Table 19 · p. 33

H.4 Frame Sampling Rate

  • Across 1, 2, 4, and 8 fps, sampling rate shifts aggregate bias only slightly without a consistent trend; 4 fps is adopted near the per-judge optimum and halves visual-token input versus 8 fps.PhyJudge-9B stays closer to human labels than either base judge at every tested rate, while the two base judges are roughly comparable.Source: §H.4 · p. 33
  • At 4 fps, high-speed transient events may still lose key details for all VLM judges.This is the stated limitation of the frame-sampling setting.Source: §H.4 · p. 33

H.5 Evaluation Cost

The appendix describes evaluation cost, reproducibility, and video-generation settings. LTX-2 results are bounded by a deliberately weaker single-stage configuration rather than the recommended refinement pipeline.

H.5 Evaluation Cost

  • The Claude Opus 4.7 chain-of-thought judge dominates estimated automatic-evaluation cost because of image-token pricing.Table 21 summarizes approximate API and compute costs; the supplied passage gives no cost values.Source: §H.5 · p. 33, Table 21 · p. 34

H.6 Reproducibility.

  • The released pipeline includes model and LoRA weights, code, sub-question definitions, training configuration, and score records, runs on one GPU, and needs no closed-source API or reasoning traces.Version-pinned judge weights prevent evaluation results from drifting with provider updates.Source: §H.6 · p. 34

I Video Generation Inference Settings

  • All generators receive the same Gemini 2.5 Flash rewritten captions to hold text-conditioning quality constant, while released defaults supply most model-specific inference settings.Veo-3.1 exposes none of its inference knobs through its closed API.Source: I Video Generation Inference Settings · p. 34, Table 22 · p. 35
  • The eight generators differ by scheduler family and output sizing, with four using FlowMatchEulerDiscrete and two using FlowUniPCMultistep.Three models use fixed output resolution, while five resize to track input aspect ratio.Source: I Video Generation Inference Settings · p. 34
  • LTX-2-19B and LTX-2.3-22B use a single Stage-1 pass without the recommended Stage-2 refinement because the full benchmark exceeded the compute budget, so their scores are lower bounds for the family.The two-stage configuration is described as Stage-1 base plus Stage-2 distilled-LoRA refiner.Source: I Video Generation Inference Settings · p. 35

J Limitations

PhyGround trades benchmark scale for curated, evaluable prompts and restricts its taxonomy to visually verifiable laws. Its qualitative examples illustrate physical-law violations, including cases unanimously labeled by annotators.

J.1 Prompt Suite Size

  • The 250-prompt suite prioritizes unambiguous expected outcomes and practical generation and multi-annotator scoring over raw benchmark scale.Prompt and conditioning-frame curation supports law-level diagnostic scoring.Source: §J.1 · p. 35

J.2 Physics Law Coverage

  • The taxonomy is finite, excluding categories that are not visually verifiable in short videos or decomposable into two or three observable sub-questions.Excluded areas include thermodynamic phase transitions, combustion, electromagnetism, chemistry, waves, energy, and modern physics.Source: §J.2 · p. 35

K Qualitative Examples of Physical Law Violations

L Qualitative Examples of Physical Law Violations by Model

Loading 2605.10806v1…