PaperlayerView exact revision on arXiv

Source-linked AI summary

VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction

Jiarong Liang, Max Ku, Ka-Hei Hui, Ping Nie, Wenhu Chen

arXiv:2602.13294v3cs.CVcs.AI

TL;DR

MLLM physical reasoning is difficult to assess because recognition-based benchmarks do not require explicit, testable physical hypotheses. VisPhyWorld addresses this by reconstructing scenes as executable simulator code and evaluates them with VisPhyBench; results show strong semantic understanding but weak physical parameter inference and dynamics simulation.

  • Problem

    Recognition-based physical reasoning benchmarks can be answered without committing to an explicit physical hypothesis, leaving genuine causal understanding difficult to assess.

  • Method

    VisPhyWorld interprets visual observations as executable simulation code, while VisPhyBench evaluates reconstructed appearance and physically plausible motion across 209 scenes from 108 physical templates.

  • Results

    MLLMs achieve strong semantic scene understanding but struggle to infer precise physical parameters and reproduce physically consistent dynamics.

  • Takeaways & Limitations

    Executable hypotheses expose a consistent gap between semantic scene parsing and physically grounded simulation, motivating representations that move beyond purely statistical pixel-space pattern matching.

  • Takeaways & Limitations

    Experiments use synthetic simulator-driven scenes, so generalization to high-resolution in-the-wild videos remains untested.

Abstract

from arXiv · show

Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed videos in 97.7% of benchmark runs before fallback. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics. Our code is available https://github.com/TIGER-AI-Lab/VisPhyWorld

1 Introduction

VisPhyWorld addresses the difficulty of testing genuine physical reasoning by requiring MLLMs to reconstruct visual scenes as executable, testable simulations. VisPhyBench evaluates both appearance reconstruction and physically plausible motion, revealing a gap between semantic recognition and accurate physical dynamics.

  • Motivation: Existing VQA- and VoE-style evaluations can be answered through memorized priors or surface-level visual patterns without an explicit physical hypothesis.These protocols therefore make it difficult to determine whether correct outputs reflect coherent causal understanding.
  • Findings: Current state-of-the-art MLLMs perform well at semantic scene recognition but struggle to parameterize simple Newtonian dynamics accurately.The reported limitation persists even in simple 2D settings and is expected to be more challenging in 3D environments.
  • VisPhyWorld: VisPhyWorld requires MLLMs to interpret video frames and generate executable simulation code for predicting future motion.Making object states and dynamics explicit provides an interpretable view of the model’s physical hypothesis.
  • VisPhyBench: VisPhyBench comprises 209 scenes derived from 108 physical templates and evaluates appearance reconstruction alongside physically plausible motion in 2D and 3D.The protocol integrates metrics covering complementary aspects of physical understanding.
  • Findings: Models can describe scene contents accurately yet fail to reconstruct motion that conforms to physical laws, indicating reliance on superficial visual pattern matching rather than grounded physical causality.The executable reconstruction setup makes this discrepancy directly inspectable and falsifiable.

2 Related Work

Prior intuitive-physics benchmarks evaluate plausible dynamics through videos, generative predictions, or MLLM-based judgments, while executable visual representations provide a structured route to specifying motion. VisPhyWorld distinguishes itself by making physical reasoning an executable hypothesis and supporting multimetric diagnostic evaluation.

  • Intuitive physics: Intuitive-physics research studies whether models infer object dynamics, interactions, and causal relationships from visual input.Benchmarks include video prediction and violation-based evaluations of physically plausible outcomes.
  • Intuitive physics: Existing datasets and benchmarks such as Phyre, Physion, IntPhys, PhysicsIQ, PhyGenBench, and WorldModelBench evaluate plausible or temporally consistent dynamics using physics-generated or predicted videos.MLLM-based evaluators such as VideoPhy and VideoScore provide another assessment route.
  • Robustness: Across engines, scene-level motion rankings remain stable despite changes in absolute visual scores, reducing concern that conclusions are driven only by backend-specific coding artifacts.The primary motion metric has an average pairwise Spearman correlation of 0.84 across engines.
  • Executable world representations: Executable visual representations use structured programs to specify objects, motion, and physical interactions as interpretable and controllable world representations.Earlier work primarily targeted static visualizations, while later methods extended code generation to animations and trajectories.
  • VisPhyWorld: VisPhyWorld uniquely turns physical reasoning into an executable hypothesis and enables multimetric diagnostic evaluation beyond relative scoring.The table distinguishes methods by whether they require future visual generation, evaluate MLLM outputs, or require an executable physical hypothesis.

3 VisPhyWorld

VisPhyWorld reconstructs visual observations as executable physical simulations, producing interpretable artifacts that can be rendered and evaluated. VisPhyBench covers 209 videos from 108 templates across 2D and 3D physical scenes.

  • VisPhyWorld interprets visual observations and reconstructs the underlying physical scene as executable code.
  • Input: The model receives early and later frames, optionally augmented with a structured object list containing categories, bounding boxes, and coarse attributes.
  • Outputs: The pipeline outputs motion analysis, a first-frame JSON specification, executable code, and a rendered video obtained by executing that code.
  • Executable code serves as an explicit, falsifiable physical hypothesis whose rendered rollout separates hypothesis construction from execution.
  • Dataset Construction: VisPhyBench contains 108 templates and 209 videos spanning 2D PHYRE scenes and 3D Three.js/Cannon.js rigid-body simulations.
  • Evaluation: Evaluation combines reconstruction, perceptual, semantic, language, motion, and physical-plausibility metrics, while rendering backends differ in physics-engine support.

4 Experiments

Experiments show strong reconstruction and semantic alignment but a persistent gap between visual plausibility and faithful physical dynamics. Code backends, input context, and holistic evaluation materially affect how that gap is diagnosed.

  • Most models achieve strong reconstruction and perceptual scores while maintaining reasonable visual-semantic consistency on VisPhyBench.
  • Overall leaderboard: Three.js generally outperforms p5.js; for GPT-5, it reduces LPIPS by nearly 40% and improves SSIM from 0.74 to 0.94.
  • Overall leaderboard: Code-driven models expose executable states for controlled interventions, whereas pixel-space baselines offer less interpretable physical failure modes.
  • Motion and holistic physical plausibility: RAFT-EPE measures trajectory agreement but can misrepresent physical plausibility when used without holistic perceptual or physics judgments.
  • Robustness and Ablation Studies: Removing detection context mainly harms LPIPS and Gemini scores, suggesting that it supports object discovery and state initialization more than direct physical reasoning.
  • Robustness and Ablation Studies: Removing the later frame changes visual metrics little but lowers Gemini physical-plausibility scores, making correct physical outcomes less likely.
  • Robustness and Ablation Studies: Models recover gravity relatively well but make non-trivial contact-parameter errors and larger initial-speed errors when evaluated by parameter MAE.
  • Case studies: GPT-5 correctly simulates collision dynamics with Gemini 10.0 and DINO 0.926, while Veo-3.1 reaches DINO 0.835 but Gemini 2.0.

5 Conclusions

VisPhyWorld evaluates physical understanding by requiring executable scene reconstructions, separating visual mimicry from physically grounded reasoning. The results reveal strong semantic parsing but weak precise physical parameterization, motivating verifiable hybrid representations.

  • VisPhyWorld requires MLLMs to reconstruct scenes as executable code, decoupling visual mimicry from physically grounded reasoning.
  • Current models excel at semantic scene parsing but struggle with precise physical parameterization and basic Newtonian dynamics.
  • The paper suggests hybrid representations grounded in verifiable, executable physical laws as a direction for more transparent evaluation.

A Limitations and Discussion

VisPhyWorld is evaluated on controlled synthetic scenes and currently targets relatively simple, short rigid-body clips. Generalization to in-the-wild video, long-horizon interactions, and more complex 3D or stylized scenes remains untested.

  • Experiments use synthetic simulator-driven scenes with controlled object layouts and camera motion, leaving in-the-wild generalization untested.
  • The system reliably generates code only for relatively simple rigid-body scenes because current MLLMs and complex engines limit autonomous code production and repair.
  • The current scope excludes long-horizon interactions, complex 3D reasoning, and stylized or heavily cluttered scenes.

C Reproducibility Details

VisPhyWorld uses fixed prompting, structured optional detection context, and deterministic execution to turn visual observations into executable 2D or 3D scene hypotheses. Its benchmark combines stochastic PHYRE-style 2D templates with programmatic 3D rigid-body scenes.

  • Prompting protocol: The single-call prompt asks models to summarize motion between two keyframes and propose an executable scene hypothesis in a fixed output format.The prompt requests object descriptions, motion descriptions, and likely physical causes before code generation.
  • Detection context: Optional detection context D supplies coarse object geometry, appearance, positions, and orientations for the first keyframe in pixel coordinates.Each object may include an identifier, category, RGB color, bounding box, centroid, size descriptor, and orientation.
  • Benchmark construction: VisPhyBench templates are executable PHYRE-style scripts instantiated with sampled seeds, so snapshots do not represent the full scene diversity.Object composition is summarized over the full split using first-frame detection context.
  • 3D scenes: The 3D subset uses Three.js and Cannon.js rigid-body templates with spheres, boxes, ramps, and barriers under a fixed perspective camera.These scenes probe depth-aware contacts and occlusions not present in purely 2D scenes.
  • Deterministic execution: Generated scene hypotheses execute under fixed physics steps, camera settings, recording duration, and deterministic rendering constraints.The 2D subset clamps bodies to a plane, while the 3D subset executes full 3D rigid-body dynamics.

C.5 Robustness: Automatic Retry and Fallback

The robustness protocol repairs executable-program failures once and then uses a minimal fallback to keep evaluation defined. Main-paper success rates count only model-generated clips produced without fallback.

  • Automatic retry: If initial execution fails, the system summarizes diagnostics and gives the model one repair attempt.Diagnostics can include JavaScript console logs and error traces.
  • Rendering protocol: The renderer extracts and canonicalizes model-generated HTML before validating a drawable canvas and finite numeric states.Canonicalization injects fixed library versions and a trusted recording helper.
  • Fallback: A minimal hand-crafted fallback guarantees a valid canvas and finite motion when both model attempts fail.Fallback samples receive correspondingly poor metric scores while preventing pipeline crashes.
  • Success criteria: 97.7% is the model-success rate before fallback, counting only model-generated hypotheses that execute and produce non-empty clips.System-success additionally counts fallback clips solely to keep metric computation well-defined.
  • Evaluation convention: Metrics are computed per scene and averaged over the evaluated split, with frame-wise metrics generally applied after temporal alignment.This establishes the aggregation and alignment conventions for evaluation.

D.1 Default Evaluation Hyperparameters

The evaluation combines aligned frame-level visual, perceptual, semantic, textual, optical-flow, and holistic physics metrics. A Gemini-based judge and model-blind human ratings provide complementary holistic assessments.

  • Frame-wise metrics: Frame-wise PSNR, SSIM, LPIPS, FSIM, VSI, and DISTS compare aligned reference and generated frames for signal, structural, and perceptual similarity.SSIM is averaged across the Y and RGB channels, while perceptual metrics use the piq implementation.
  • Embedding metrics: CLIP-Img and DINO Similarity measure semantic/layout consistency and object structure through cosine similarity of visual embeddings.DINO is described as less biased by text supervision than CLIP.
  • Text and caption metrics: CLIP-Cap, ROUGE, and BERTScore evaluate whether generated motion analysis matches video content or an automatically generated reference description.These metrics assess perception and verbalization of events in the input video.
  • Motion metrics: RAFT diagnostics compare reference and generated optical flow using EPE, flow-magnitude difference, and angular error after temporal alignment.Alignment combines coarse offset search with Dynamic Time Warping to handle temporal shifts.
  • Holistic evaluation: Gemini-2.5-Pro assigns conservative 1–10 holistic physical-plausibility scores, while six STEM graduate students provide model-blind 1–5 ratings.Human rankings show the same broad trend as automatic metrics, with strong code-driven models and Veo-3.1 top-tier.

E.3 Additional Ablation Studies

Additional ablations examine retry behavior, detection context, evaluator choice, and access to the later frame. They indicate that initialization support and later-frame conditioning affect specific aspects of evaluation without resolving physical-dynamics errors by themselves.

  • Iterative self-repair: One retry uses renderer error feedback and can correct surface-level failures such as missing canvas hooks, minor API misuse, or initialization errors.The ablation compares first-attempt success with success after one additional generation-and-render attempt.
  • Detection context: Removing structured detection context worsens LPIPS and Gemini physical-plausibility scores while changing CLIP, DINO, and BERTScore much less.The result indicates that D primarily reduces object-discovery and initialization burden rather than solving physical dynamics.
  • Alternative judges: Across alternative judges, SVD-img2vid remains the weakest baseline and sample-level correlations between judge pairs are significant.The comparison uses GPT-5.4 on uniformly sampled JPEG frames and Qwen3-VL-Plus with video input.
  • Forecast-only setting: Removing the later frame modestly changes visual metrics and RAFT-EPE but lowers Gemini physical-plausibility scores for every tested model.The authors report that models can still produce visually plausible clips while less often recovering the correct physical outcome.

E.4 Cross-Engine Robustness

Cross-engine rankings for the primary motion metric remain stable across rendering backends, while visual-metric rankings are less consistent. The evaluation also separates low-level reconstruction and executability from the broader physical-understanding assessment.

  • Evaluation scope: Table 15 isolates frame-wise pixel fidelity and video-generation success from the full multi-metric physical-understanding assessment.This separation distinguishes reconstruction quality and executability from broader physical reasoning performance.
  • Motion-metric robustness: RAFT-EPE rankings show strong cross-engine stability, with an average pairwise Spearman correlation of 0.84.This supports using the motion metric across different execution backends with limited backend-induced rank variation.
  • Visual-metric robustness: Visual metrics have moderate cross-engine correlations, unlike RAFT-EPE.Backend choice therefore affects visual-metric rankings more than the primary motion ranking.

E.6 Reconstruction & Perceptual Metrics

The reconstruction and perceptual evaluation reports low-level frame fidelity alongside perceptual, semantic, text-consistency, and motion-related measures. Gemini-3-Pro leads the detailed perceptual metrics, while GPT-5 and Gemini-3-Pro align strongly with ground-truth visual semantics.

  • Perceptual metrics: Gemini-3-Pro consistently achieves the best perceptual scores on LPIPS and FSIM, while Three.js generally outperforms P5.js.These results concern the detailed pixel-level and perceptual metric breakdown.
  • Visual semantic consistency: GPT-5 and Gemini-3-Pro show strong alignment with ground truth on CLIP and DINO semantic-consistency scores.These metrics assess semantic understanding beyond exact pixel reconstruction.
  • Additional evaluation dimensions: Text–video and analysis-text consistency, alongside motion and physical plausibility, are reported as additional metric families.Tables 18 and 19 provide the remaining text-analysis and motion-fidelity measures.
  • Reconstruction metrics: The evaluation includes frame-wise pixel fidelity and generation success rate as distinct reconstruction measures.Table 15 reports average PSNR/SSIM together with whether configurations successfully produced valid videos.
Loading 2602.13294v3…