PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysicsLENS: Diagnosing Physical Property Blindness in Video Generation Models

Isaiah Milkey, Som Sagar, Aditya Taparia, Xinyuan Liu, Jiqing Wen, Ransalu Senanayake

arXiv:2610.01162v1cs.CVcs.RO

TL;DR

Existing video benchmarks do not determine whether a model grounds physical properties that are hidden from the conditioning image, despite the use of generated videos as predictive environments for robotics. PhysicsLENS tests this with matched prompts and human and automated evaluation. Human ratings show that plausible clips often fail to follow the stated property, and the study is limited to 80 initial frames and four models.

  • Problem

    Existing benchmarks only partially assess whether generated videos ground prompt-specified properties such as weight, friction, or viscosity that may not be visible in the initial frame.

    Source: §1 · p. 1

  • Method

    PhysicsLENS uses matched robot-video scenarios that hold the frame and task fixed while varying text-specified properties, evaluated through human annotations and an automated motion-measurement and VLM pipeline.

    Source: §1 · pp. 1–2, §3.4 · p. 5

  • Results

    34 of 47 clips rated plausible under unobservable conditions did not follow the stated property, while 15 of 28 property-adherent clips were still physically implausible.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Physical plausibility and adherence to a stated property are distinct: a video can satisfy either while failing the other.

    Source: §5 · p. 9

    The study focuses on 80 initial frames and four models; multiple generations per prompt remain a next step.

    Source: §5 · p. 9

Abstract

from arXiv · show

Reliable video world models could provide scalable predictive environments for robot learning, planning, and evaluation. However, generated robot videos can violate physical principles and complete tasks through physically implausible behavior, limiting their reliability for robot learning and planning. Current video-generation benchmarks exclude physics that are inherently hidden by visuals (e.g., weight, viscosity, friction). Due to this, video models are evaluated on the fidelity of physics, not the underlying accuracy of physics. We introduce PhysicsLENS, a dataset and benchmark for evaluating plausibility of physical properties grounded in robotics. PhysicsLENS uses matched scenario pairs that hold the same conditioning frame and task, while varying underlying physics in the scene description. Scenarios are curated from public robot video sources and annotated across seven physical domains: collision, gravity, momentum, friction, deformation, fluid, and causality. We evaluate across four video generation models, producing over 400 human-annotated labels. Results show that plausible-looking videos often ignore the stated property (34 of 47), and that stating the property lowers plausibility only slightly and not significantly.

1 INTRODUCTION

Generated robot videos could support learning, planning, and evaluation, but visual realism and task completion do not establish that they obey the stated physics. PhysicsLENS benchmarks physical plausibility and property grounding using matched scenarios, and finds that plausible-looking videos often ignore the specified property.

1 INTRODUCTION

  • Visual quality and task success can miss violations of physical constraints, limiting generated videos as predictive environments for robotics.Properties such as weight, friction, and viscosity may be hidden by the initial image.Source: §1 · p. 1
  • PhysicsLENS curates robot-video prompts and evaluates more than 400 generated clips from four models with structured human annotations.Its scenarios cover seven physical domains, including collision, gravity, momentum, friction, deformation, fluid, and causality.Source: §1 · pp. 1–2
  • Matched scenarios hold visual context and task fixed while varying the text-specified physical property, including properties not inferable from the frame.This design tests whether generated interactions reflect the specified property’s physical consequences.Source: §1 · p. 2
  • Across four video-generation models and 439 clips, plausible-looking videos often ignore the stated physical property.The benchmark evaluates physical plausibility and property grounding as distinct qualities.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates visual and physical plausibility, robot-video quality, and inference of hidden physical properties, while PhysicsLENS tests whether video generators enact properties stated only in text. It also combines tracked measurements with targeted VLM questions in its automated evaluation.

Evaluating physics in generated video

Hidden physical properties

  • PhysicsLENS tests whether general-purpose image-to-video models follow plain-language hidden properties with the frame and task held fixed.Earlier work infers properties from observed interactions [Chen et al., 2022], while T2V-CompBench tests visible prompt attributes [Sun et al., 2025].Source: §2 · p. 3

VLM judges of physical plausibility

  • PhysicsLENS combines tracked measurements with targeted VLM questions, addressing reported limits in VLM physical understanding and fine-grained motion grounding.Prior studies report near-chance intuitive-physics performance and limited temporal grounding [Bordes et al., 2025; Hong et al., 2025; Wang et al., 2025].Source: §2 · p. 3

3 PHYSICSLENS BENCHMARK

PhysicsLENS holds the initial frame and task fixed while varying a text-specified latent property, enabling separate assessments of property grounding and physical plausibility. It combines human ratings with an automated evaluator that uses motion measurements and targeted VLM questions.

3.1 OBSERVABLE AND UNOBSERVABLE CONDITIONS

  • Matched observable and unobservable prompts keep the initial frame and task fixed while changing whether the text matches or conflicts with the frame’s implied physical properties.The paired design tests property grounding and whether specifying a latent property affects plausibility; individual pair differences may reflect generation noise, so pairs are analyzed in aggregate.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 SCENARIOS AND GENERATED VIDEOS

  • PhysicsLENS contains 110 prompts drawn from 80 initial frames across 11 public robot datasets, with 80 observable scenarios and 30 matched unobservable variants.Four models generated one video per prompt; the realized benchmark has 439 videos, including 320 observable and 119 unobservable clips, and does not estimate variation across generation seeds.Source: §3.2 · p. 4

3.3 HUMAN ANNOTATION

  • Human annotators rate visible physical plausibility, task completion, and—only for unobservable videos—adherence to the hidden property.P and H use 1–4 scales, while task completion uses a binary scale; violations can be assigned one or more of eight labels, with contact merged into collision in analyses.Source: §3.3 · p. 5, Table 2 · p. 6

3.4 AUTOMATED DIAGNOSTIC PIPELINE

  • The automated evaluator pairs motion measurements with targeted VLM questions and combines evidence into per-video reports.Its four stages screen motion and VLM responses, localize potential violations, verify task completion and physical errors, and report findings; the Figure 4 caption identifies mean plausibility and adherence by model as plotted quantities.Source: §3.4 · pp. 5–6, Figure 3 · p. 5, Figure 4 · p. 6

4 EXPERIMENTAL RESULTS

Human annotations show that physical plausibility and adherence to a text-specified property can diverge. Stating a latent property produces only a small, statistically nonsignificant plausibility decrease, while automated evaluation agrees with human judgments on some outcomes but struggles with hidden-property adherence and violation attribution.

4.1 RQ1: DOES HIGH PLAUSIBILITY IMPLY PROPERTY GROUNDING?

  • 34 of 47 plausible videos (72%) ignored the specified property, while 15 of 28 property-adherent videos were rated implausible.The pattern appeared across all four generators; without MAGI, 13 of the remaining 26 plausible videos still ignored the property.Source: §4.1 · p. 6

4.2 RQ2: DOES STATING A LATENT PROPERTY IN TEXT CHANGE PLAUSIBILITY?

  • Across 29 scenarios, unobservable-property prompts lowered mean plausibility by 0.28 points; the decrease was not statistically significant (p = 0.085).Mean plausibility was lower for all four generators, by 0.13 to 0.38 points on the 1–4 scale.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3 RQ3: WHAT DO FAILURE CASES REVEAL ABOUT GROUNDING AND PLAUSIBILITY?

  • A plausible molasses-like tea pour could ignore viscosity, while property-adherent videos could remain implausible through other violations, including motion without contact.MAGI also produced nearly static videos rated plausible without demonstrating how the stated property affected the task.Source: §4.3 · p. 7, Figure 6 · p. 7

4.4 RQ4: HOW WELL DO AUTOMATED EVALUATORS MATCH HUMAN JUDGMENTS ON PHYSICSLENS?

  • The full evaluator achieved AUCs of 0.72 for overall plausibility, 0.81 for task completion, 0.73 for violation detection, and 0.71 for hidden-property adherence.It combines ten VLMs with Stage-1/2 signals; evaluation used human annotations for 439 videos, with scenario versions kept together in five-fold cross-validation.Source: §4.4 · pp. 7–8, Table 4 · p. 8
  • Detection AUC spans 0.56–0.84 across violation families, whereas attribution is 0.53–0.58 for most (fluids: 0.82); 66% of violation videos contain multiple types.Adding signals improves detection for five families, leaves momentum unchanged, and lowers performance for fluids.Source: Figure 8 · p. 8, Figure 7 · p. 8, §4.4 · p. 9
  • Targeted physics-error questions improved plausibility AUC for 8 of 10 VLMs; Qwen3-VL-32B rose from 0.55 to 0.75 on observable videos, while its hidden-property adherence AUC was 0.57.Across VLMs, adding signals raised mean adherence AUC from 0.57 to 0.70, but this agreement did not establish whether a particular hidden property was realized.Source: §4.4 · pp. 8–9, Table 5 · p. 9

5 CONCLUSION

Across 439 clips from four generators, plausible-looking videos often failed to follow the stated hidden property, while property-following clips could still be physically implausible. The study argues that plausibility and property grounding should be evaluated separately, while noting its deliberately narrow study scope.

5 CONCLUSION

  • 34 of 47 clips rated plausible under unobservable conditions did not follow the stated property, while 15 of 28 clips that followed it were still physically implausible.These results distinguish visible physical plausibility from adherence to a hidden property.Source: §5 · p. 9
  • Pairing targeted VLM questions with measured video signals reached an AUC of 0.72 for plausibility and 0.81 for task completion against human judgments.Judging adherence to a specific hidden property and assigning a violation to a physics family remained difficult.Source: §5 · p. 9
  • The study focused on 80 initial frames and four video-generation models, with further work proposed on more models, hidden-property types, and generations per prompt.Multiple generations per prompt could help separate variation from the generation process from variation tied to the property being tested.Source: §5 · p. 9

A.1 SOURCE DATASETS AND LICENSES

The benchmark draws starting frames from public robot demonstrations, creates prompts that preserve the frame and task while specifying hidden properties, and generates one clip per prompt and model.

A.1 SOURCE DATASETS AND LICENSES

  • The 80 starting frames come from demonstrations in 11 public robot datasets, with each frame taken before manipulation begins.The dataset manifest records each frame’s source, episode, and frame index; released files retain their source licenses.Source: §A.1 · p. 15

A.2 IMAGE AND PROMPT PREPARATION

  • For hidden-property variants, prompt preparation retained the same frame and action while rewriting the scene description to state the property.Expected outcomes and failure signatures were excluded from the generation prompt.Source: §A.2 · p. 15

A.3 VIDEO GENERATION

  • Each model received the same 1280 × 720 conditioning frame and text prompt, and one video was generated per prompt and model.The generated clips lasted about 5 seconds.Source: §A.3 · p. 15

A.4 TAXONOMY AND CURATION NOTES

The appendix describes how scenarios were sourced and assigned to physical domains, and clarifies that observability is a separate scenario dimension. Its sourcing observations are limited to the curation process, not a systematic estimate of public robotics video.

A.4 TAXONOMY AND CURATION NOTES

  • Collision and friction scenarios were comparatively easy to source, while fluid, deformation, and free-fall gravity scenarios were harder to find.The authors caution that these curation observations do not establish the distribution of public robotics video.Source: §A.4 · p. 15
  • The video-generation settings describe image-to-video clips conditioned on a source start frame and text prompt, lasting about 5 seconds.MAGI-1 generates autoregressively in 24-frame chunks with 16 denoising steps per chunk; only Wan uses a negative prompt.Source: Table 7 · p. 16, §A.4 · p. 16
  • Scenarios were curated from public robotics videos and manually assigned to the domain their physics most directly tests, using seven domains fixed in advance.The annotation was independent of taxonomy construction to avoid circularity.Source: §A.4 · p. 16
  • Observability is orthogonal to domain: matched observable and unobservable scenarios retain the same domain label.The domain identifies the physical law under test, while observability indicates whether its governing property is visually apparent or text-specified.Source: §A.4 · p. 16

B.1 ANNOTATION INSTRUMENT

The annotation instrument separates visible physical plausibility, task completion, and adherence to hidden properties, and reports these dimensions separately. It also requires annotators to describe and categorize issues for low-plausibility clips.

B.1 ANNOTATION INSTRUMENT

  • When plausibility is below 4, annotators must describe the issue and select at least one violation category.Physical plausibility and action completion are collected for every clip; hidden-property adherence is collected only for unobservable scenarios.Source: §B.1 · p. 16
  • Physical plausibility rates visible consistency of contact, motion, deformation, and object persistence, while action completion scores the final task outcome rather than process correctness.A completed task can still contain severe physical violations, and an incomplete task can have otherwise plausible motion.Source: §B.1 · p. 16
  • Hidden-property adherence is collected only for unobservable scenarios, and low scores can reflect either contradiction of the property or a lack of an interaction that reveals it.Because there is no separate insufficient-evidence response, the authors retain this ambiguity when interpreting H ≤2.Source: §B.1 · p. 16
  • Thresholded summaries treat P ≥3 as mostly plausible and H ≥3 as mostly or fully following the stated property, while reporting the three dimensions separately.The reported summaries include rating distributions, descriptive ordinal-score means, action-completion proportions, and the joint distribution of P and H.Source: §B.1 · pp. 16–17

B.2 VIOLATION-LABEL CODEBOOK

The codebook records eight human violation labels, distinct from the benchmark’s seven physical domains, and analyses merge Contact into Collision. Labels can overlap, and click order is not treated as a calibrated severity ranking.

B.2 VIOLATION-LABEL CODEBOOK

  • When P<4, annotators must select a violation category and are instructed to select the most obvious violation first.Frequency analyses treat selected labels as an unordered set, not a severity ranking.Source: §B.2 · pp. 17–18
  • 66% of videos with a violation have multiple labels, so the violation shares sum to more than 100%.Contact is merged into Collision in the analyses.Source: Figure 10 · p. 18
  • The eight human labels differ from the seven-domain taxonomy: Contact and Collision remain distinct in the codebook, but Contact is merged into Collision for analysis.A blank category field means no label was recorded, not that the clip is confirmed physically correct.Source: §B.2 · p. 18, Table 9 · p. 18
  • The multi-select interface preserves overlapping violations rather than forcing each failure into one category.For example, passing through a supporting surface may involve Contact and Collision, while unsupported floating may involve Contact and Gravity.Source: §B.2 · p. 18

B.3 ANNOTATOR ALLOCATION AND RELIABILITY LIMITS

Research-team members, mainly computer-science graduate students, annotated the clips, but the annotation records do not support estimating inter-annotator reliability.

B.3 ANNOTATOR ALLOCATION AND RELIABILITY LIMITS

  • Annotators were research-team members, mainly computer-science graduate students, and clips were assigned randomly.Observable clips were distributed in batches of 16–17; unobservable clips in batches of six.Source: §B.3 · p. 18
  • Combined annotator labels in some observable Cosmos records are treated as a single annotation source, not overlapping independent ratings.Those records cannot estimate agreement between the named annotators.Source: §B.3 · pp. 18–19
  • Each clip received one final annotation rather than independent ratings from multiple annotators, so the dataset cannot estimate inter-annotator reliability.Source: §B.3 · p. 19

C PHYSICSLENS PIPELINE DETAILS

The pipeline combines motion measurements with targeted VLM questions, then reports findings for each video. Its appendix evaluation also tests a physics-error question against the standard question on VideoPhy-2.

Screening

  • Stage 1 screens each clip with 22 motion and DINOv2-embedding signals, alongside a VLM physics-error question; signals are not individually thresholded into flags.The screening covers the first five seconds at eight frames per second.Source: C PHYSICSLENS PIPELINE DETAILS · p. 19

Localization

  • Stage 2 tracks points after removing camera motion and computes 16 motion signals; the interactive tool adds object tracking and ranks violation families with confidence and time windows.Section 4.4 uses only the 16 point-track signals.Source: C PHYSICSLENS PIPELINE DETAILS · p. 19

VLM verification

  • Stage 3 asks VLMs about task completion and violation type, and asks about stated-property adherence only for unobservable videos.The property question supplies the expected outcome; the violation choices are shuffled, and answer probabilities are converted to scores.Source: C PHYSICSLENS PIPELINE DETAILS · p. 19

Combining evidence

  • The evaluator combines averaged VLM ranks with five selected motion-signal ranks, choosing signals and their mixing weight within scenario-grouped five-fold cross-validation.The final score mixes the VLM and signal ranks; adherence and family attribution are fitted only on their relevant video subsets.Source: C PHYSICSLENS PIPELINE DETAILS · p. 20

Reporting and routing

  • Stage 4 produces a per-video report with a timeline and findings but adds no score; Section 4.4 applies no routing, so all signals and questions are computed for all 439 videos.The appendix describes the pipeline as run for Section 4.4, which omits the interactive tool’s specialist modules.Source: C PHYSICSLENS PIPELINE DETAILS · pp. 19–21

VideoPhy-2 check

  • VideoPhy-2 AUC rose from 0.645 to 0.720 for Qwen3-VL-32B (p < 10^-6), 0.666 to 0.708 for Qwen3-VL-8B (p = 0.006), and 0.581 to 0.655 for Qwen2.5-VL-7B (p < 0.001).The paired comparison used 926 test videos after dropping 274 videos rated 3 on human physical-commonsense scores.Source: C PHYSICSLENS PIPELINE DETAILS · p. 21
Loading 2610.01162v1…