PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysTrace: Interpretable Temporal Evaluation of Physical Consistency in Video Generative Models

Taekoan Yoo, Wonkyung Jung, Kyunghun Kim, Kyeongbo Kong

arXiv:2610.31429v1cs.AIcs.RO

TL;DR

Pretrained video models are increasingly used for robotic prediction and planning, but existing assessments rarely identify which history-conditioned physical expectations conflict with observations. PhysTrace aligns MLLM-generated expectations with video windows to localize and describe violations, and reports stronger possible–impossible discrimination on IntPhys2 alongside temporal-localization results on IntPhys2-Timing.

  • Problem

    Existing evaluations assess plausibility, probe representations, or measure predictive surprise, but rarely identify which history-conditioned physical expectation conflicts with an observation.

    Source: §1 · pp. 1–2

  • Method

    PhysTrace uses MLLM-generated, history-conditioned expected states and aligns them with observed video windows to localize and describe physical violations without task-specific probes or classifiers.

    Source: §1 · p. 2

  • Results

    PhysTrace achieves 59.9% overall accuracy, the highest among compared methods, and outperforms them on the Medium and Hard subsets of IntPhys2.

    Source: §5.3 · p. 7

  • Takeaways & Limitations

    Applied to generative-model reconstructions, PhysTrace reveals that violation localization varies across timesteps and physical attributes.

    Source: §1 · p. 2

    PhysTrace depends on the MLLM's contextual reasoning capability and is influenced by model capability and prompt design.

    Source: §7 · p. 8

Abstract

from arXiv · show

World action models increasingly rely on pretrained video models assumed to capture physical dynamics for prediction, planning, and action generation. Existing evaluations assess plausibility, probe representations, or measure predictive surprise, but rarely identify which history-conditioned physical expectations conflict with observations. We introduce PhysTrace, a training-free framework aligning MLLM-generated, history-conditioned expectations with observed video windows to temporally localize physical violations and provide interpretable descriptions without task-specific probes or classifiers. We also introduce IntPhys2-Timing, adding violation intervals to IntPhys2. PhysTrace achieves a Hit Rate of up to 70.7% across physical principles on IntPhys2-Timing and outperforms the compared methods in overall possible-impossible discrimination accuracy on IntPhys2. Applied to reconstructed videos along the trajectories of pretrained diffusion/flow-based video models, PhysTrace reveals attribute- and noise-dependent patterns in physical violation localization, enabling interpretable physical consistency analysis across timesteps.

1. Introduction

As pretrained video models become backbones for robotic prediction, planning, and action generation, assessing whether they capture physical dynamics matters. PhysTrace traces history-conditioned expectations against observations to localize and describe violations, alongside a temporal-annotation benchmark and analysis of generative-model reconstructions.

1. Introduction

  • Existing evaluations assess plausibility, probe encoded physics, or measure predictive surprise, but rarely identify which history-conditioned expectation conflicts with an observation.Physical consistency is temporal: sparse observations may miss brief violations, and a single plausibility score cannot show where expected and observed dynamics diverge.Source: §1 · pp. 1–2
  • PhysTrace aligns MLLM-generated, history-conditioned expected states with video observations to localize physical violations and describe which expectations conflict with observations.The framework is training-free and does not rely on task-specific probes or classifiers.Source: §1 · p. 2
  • IntPhys2-Timing augments IntPhys2 with violation intervals to evaluate temporal localization of physical violations.It enables evaluation of physical violation localization in video reconstructions across generative timesteps.Source: §1 · p. 2
  • PhysTrace is applied to reconstructions along pretrained diffusion/flow-based video-model trajectories to examine how violation localization varies across timesteps and physical attributes.The framework uses MLLMs to formulate expected states from temporal context, while assessing consistency against video evidence rather than relying on direct MLLM judgments of physical validity.Source: §1 · p. 2

2. Related Work

Prior work uses video representations and predictive dynamics as priors for robot policies, while physics evaluations and probes generally do not identify which history-conditioned expectation conflicts with an observation. PhysTrace focuses on interpretable, temporally localized diagnostics for that gap.

2.1. VLA and World-Model-Based Policies

  • World-model-based policies couple video-derived future prediction with action generation or planning, using predictive dynamics as priors for robot control.Examples span generated future videos, latent future states, and visual rollouts paired with inverse dynamics.Source: §2.1 · p. 2

2.2. Pretrained Video World Models as Policy Priors

  • Pretrained video world models include self-supervised representation learners, robot-specific action-conditioned models, and general diffusion/flow-based generators.The paper examines Wan2.1 and Cosmos-Predict2, pretrained generative models also used as backbones for robot control and planning.Source: §2.2 · p. 3

2.3. Physical Understanding and Probing in Video Models

  • IntPhys2 evaluates four intuitive-physics principles at video level but lacks violation timing; IntPhys2-Timing adds temporal annotations to impossible videos.Prior work also probes physics-related representations and evaluates video-level plausibility or time-resolved predictive surprise, without explicitly identifying the conflicting expectation.Source: §2.3 · p. 3

3. PhysTrace: Temporal Physical Consistency Tracing

PhysTrace repeatedly compares history-conditioned expected states with observations in local video windows, selectively retaining salient observations to build temporal context. IntPhys2-Timing supplies violation intervals for evaluating temporal localization and video-level inconsistency.

3.1. Temporal Prediction–Observation Alignment

  • Each window yields an observed State description and a history-conditioned Prediction of what should appear in that same window, which are compared through video–text alignment.This differs from future-directed predictive-surprise comparisons that can shift a discrepancy according to prediction horizon.Source: §3.1 · pp. 3–4
  • In the Guided setting, generated State and Prediction descriptions are closer to ground-truth captions than unrelated-prompt and unrelated-video baselines.This supports MLLMs' role in generating descriptions under controlled conditions, not their ability to directly judge physical validity.Source: §3.1 · pp. 3–4, Table 1 · p. 4
  • The alignment score is cosine similarity between video and text embeddings, and its trace across windows is used to locate physical-consistency breakdowns.Alignment tends to decrease when observed events differ from history-conditioned expectations; videos with score range below τ count as misses in localization evaluation.Source: §3.1 · p. 4

3.2. History-Conditioned Prediction and Selective State Retention

  • Selective history retention preserves salient observations while maintaining local windows, balancing brief-event localization with the broader context needed for violations spanning time.The MLLM retains a state when a new object, motion, or meaningful state change occurs; selective retention yields lower Prediction cosine distances than retaining all observations.Source: §3.2 · p. 5
  • After a violation, retaining the newly observed state updates subsequent predictions, allowing the trace to emphasize the first deviation rather than remain low afterward.IntPhys2-Timing annotations use the corresponding possible video as a reference, then manually refine the interval in which the impossible video's violation occurs.Source: §3.2 · p. 5, §3.3 · p. 5

3.3. Temporal Evaluation with IntPhys2-Timing

  • IntPhys2-Timing uses Minimum Alignment Score for video-level inconsistency and Hit Rate for localization, counting a hit when the minimum-score interval overlaps the annotated violation interval.The minimum alignment score represents the largest measured mismatch; localization associates each score with an interval of length Δt.Source: §3.3 · pp. 5–6

4. PhysTrace along Generative Trajectories

PhysTrace is applied without additional training to reconstructed videos along diffusion/flow-model trajectories, using clean-video estimates as output-space readouts. Across noise levels, alignment becomes more informative as reconstructions approach the clean state; model-specific input and sampling choices support the evaluation.

4. PhysTrace along Generative Trajectories

  • PhysTrace evaluates Wan2.1 and Cosmos-Predict2 without additional training by tracing reconstructed videos at intermediate generative timesteps.The analysis targets pretrained diffusion/flow-based models used as action-model backbones.Source: §4 · p. 6

4.1. Output-Space Readout via Video Inversion

  • At each noise level, deterministic inversion provides a trajectory checkpoint whose clean-latent estimate is decoded into a video for PhysTrace.Wan2.1 and Cosmos-Predict2 use different native readouts, but both supply decoded clean-video estimates.Source: §4.1 · p. 6
  • Alignment is generally low and its minimum location can be unstable at high noise, while low-noise reconstructions reveal a decrease at the annotated violation.This lets PhysTrace track when physical behavior becomes observable along a generative trajectory, not only in the final output.Source: §4.1 · p. 6

4.2. Model-Specific Evaluation Setup

  • Wan2.1 and Cosmos-Predict2 inputs are formed by uniformly sampling 81 and 93 frames, respectively, across each 636-frame source video.The sequences are reconstructed in one pass and saved at 7.5 and approximately 8.8 fps, respectively, to preserve the approximately 10.6-second duration; prompts are empty.Source: §4.2 · pp. 6–7

5. Experiments

Experiments assess temporal localization, possible–impossible discrimination, and tracing across reconstructed generative trajectories. PhysTrace records the highest overall IntPhys2 accuracy among compared methods, while localization and reconstruction results vary by physical principle and temporal-window configuration.

5.1. Temporal Localization of Physical Violations on IntPhys2-Timing

  • PhysTrace localizes violations more accurately than the VoE baseline on three of four physical principles, and provides interpretable State and Prediction descriptions.Table 2 reports Hit Rates across physical attributes and difficulty categories; a hit requires overlap with the annotated violation interval.Source: §5.1 · p. 7, Table 2 · p. 7

5.2. Ablation Study on Temporal Windowing

  • The 2-second window preserves broader context for interpreting motion, while the 1-second hop offers a balance between localization resolution and alignment stability.Shorter windows can capture brief events but may omit context; smaller hops offer finer resolution but can produce fluctuations that destabilize minimum-alignment locations. Figure 5 reports a sharper drop within the solidity ground-truth interval with broader temporal coverage.Source: §5.2 · p. 7, Figure 5 · p. 8

5.4. Physical Tracing in Diffusion Reconstruction

  • Hit Rate tends to increase as noise decreases, but attribute trends differ: continuity remains strongest, immutability degrades sharply at higher noise, and permanence and solidity remain weaker.For both models, continuity has the smallest drop at σ = 0.9; immutability loses more than 10% for σ ≥ 0.8, while permanence and solidity are already weaker on clean reconstructions.Source: §5.4 · pp. 7–8

5.3. Possible-vs-Impossible Classification Accuracy on IntPhys2

  • 59.9% overall accuracy is the highest on IntPhys2, with PhysTrace outperforming compared methods on the Medium and Hard subsets.For each possible–impossible pair, PhysTrace compares MAS and counts a prediction as correct when the impossible video has lower MAS.Source: §5.3 · p. 7, Table 3 · p. 8

5.4. Physical Tracing in Diffusion Reconstruction

  • Reconstruction visibility helps explain these differences: continuity cues persist at high noise, while appearance, occlusion, re-emergence, and box-bottom cues can be insufficient or ambiguous.For solidity, a falling box passing through an obstacle can appear plausible if its bottom is assumed open, and resolving the ambiguity requires inspecting that bottom.Source: §5.4 · p. 8

6. Conclusion

PhysTrace aligns history-conditioned expected states with observed video windows to localize physical violations and describe expectation–observation conflicts. IntPhys2-Timing adds violation intervals, and experiments report improved localization over a VoE baseline for three of four principles plus attribute-dependent patterns across generative timesteps.

6. Conclusion

  • PhysTrace is training-free and aligns MLLM-generated, history-conditioned expected states with observed windows in a shared multimodal embedding space.It provides Prediction and State descriptions without direct MLLM judgments of physical validity or task-specific probes and classifiers.Source: §6 · p. 8
  • Experiments show improved localization over a VoE baseline for three of four physical principles, while IntPhys2-Timing adds annotated violation intervals for quantitative temporal localization.Source: §6 · p. 8
  • Across output-space reconstructions of pretrained diffusion/flow video models, localization patterns vary by attribute and generative timestep, indicating when inconsistencies become observable.Source: §6 · p. 8

7. Limitation

PhysTrace depends on the MLLM's contextual reasoning, making results sensitive to model capability and prompt design. Property-specific prompt constraints are used to improve robustness to weak temporal grounding and confusion from incidental visual changes.

7. Limitation

  • PhysTrace relies on MLLM contextual reasoning, so its behavior is influenced by model capability and prompt design.Source: §7 · p. 8
  • Property-specific prompt constraints target weak temporal grounding and confusion caused by incidental visual changes.Source: §7 · p. 8
  • The corresponding prompt-design criteria are provided in Appendix C.Source: §7 · p. 8

A. Alignment Model Selection and Temporal Window Configuration

PE-AV-Large is selected as PhysTrace’s alignment model because it combines sensitivity to caption errors with denser temporal coverage. The section also explains how window length and frame sampling trade off temporal context against preservation of brief events.

A. Alignment Model Selection and Temporal Window Configuration

  • Alignment-model selection considers frame-sampling strategy as well as retrieval performance because sampling determines how much temporal information each window retains.Source: A. Alignment Model Selection and… · p. 13
  • PE-AV-Large can use all frames within a window, whereas InternVideo2-CLIP represents each clip with eight sampled frames.Longer windows offer broader context, but dense sampling is needed to preserve short or transient events.Source: A. Alignment Model Selection and… · p. 13
  • For IntPhys2’s roughly 636-frame, 10.6-second videos, eight-frame sampling can leave sparse temporal coverage as window length increases, motivating PE-AV-Large’s selection.Source: A. Alignment Model Selection and… · p. 13
  • The WebVid sensitivity evaluation uses 24 clips and 86 captions perturbed by replacing a subject, location, action, or attribute; ∆Det. is normalized against unrelated-sentence replacements.Table 4 defines Det. Rate as the fraction of substitutions lowering the original alignment score and ∆Det. as the normalized score change.Source: A. Alignment Model Selection and… · p. 13, Table 4 · p. 13
  • PE-AV-Large achieves the largest-magnitude ∆Det. (−18.9%), versus −12.1% for InternVideo2-Stage2-6B, while Det. Rate is comparable across evaluated models.The authors interpret this as similar frequency of detecting caption errors but stronger penalties for detected mismatches.Source: A. Alignment Model Selection and… · p. 13, Table 4 · p. 13

B. Qualitative Physical Consistency Traces

Qualitative traces compare history-conditioned expected states with observed states across four physical principles, exposing where expectations conflict with video. The examples suggest that visibility and duration of violations, as well as the consistency of spatial tracking, affect localization difficulty.

B. Qualitative Physical Consistency Traces

  • PhysTrace uses contextual reasoning about object attributes, position, motion, and interactions to form expected states and align them with observations over time.Prediction and State descriptions expose which expectations conflict with observations.Source: B. Qualitative Physical Consistency… · p. 13
  • The four principles cover persistence through occlusion, preservation of intrinsic properties, continuous movement, and solid objects occupying distinct spaces.Their violations include unexpected appearance or disappearance, abrupt transformations, discontinuous motion, and passing through obstacles or reacting to nonexistent obstacles.Source: B. Qualitative Physical Consistency… · pp. 13–14
  • Immutability violations are easiest for PhysTrace to identify when object identity or visible properties such as color and shape change across multiple frames.Object-centric descriptions make these persistent discrepancies between Prediction and State easier to identify.Source: B. Qualitative Physical Consistency… · p. 14
  • Continuity violations may persist over longer intervals, but inconsistent tracking of motion direction and relative position can make some harder to localize.The authors suggest these spatial relationships are captured less consistently than intrinsic object properties.Source: B. Qualitative Physical Consistency… · p. 14

C. Prompt Design and Selection

The prompts separate what is currently observed, what history implies should be present, and what information should be retained for later predictions. The accompanying figures show caption-substitution sensitivity and qualitative localization examples.

C. Prompt Design and Selection

  • State and Trusted describe the current clip without letting history alter observed content, while Prediction uses trusted history to describe what should be visible in the current window.Prediction may use the current state to identify scene phase, motion, and position, but should not overwrite history-established object properties or ignore the current state.Source: C. Prompt Design and Selection · p. 14
  • Trusted retains only novel, task-relevant information grounded in the current clip; information already represented in history is marked False, while needed new information is marked True.The prompts focus on foreground objects, persistence, intrinsic properties, position, motion, and interactions rather than incidental background changes or camera motion.Source: C. Prompt Design and Selection · p. 14
  • Prompt inputs specify the demonstration count, current window index and times, overlap with the next window, and the next window’s duration.Source: C. Prompt Design and Selection · p. 14
  • Figure 6 rows replace a caption’s subject, place, action, attribute, or entire sentence; columns are alignment models, and colors encode normalized signed ∆Det. bands.Green denotes ∆Det. ≤−25%, yellow −25% < ∆Det. < 0, and red ∆Det. ≥0.Source: Figure 6 · p. 15
  • Figure 7 presents qualitative localization examples across physical attributes, with time in seconds on the x-axis.Source: Figure 7 · pp. 16–17
Loading 2610.31429v1…