PaperlayerView exact revision on arXiv

Source-linked AI summary

MPMWorlds: Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamics

Žiga Kovačič, Kevin Ellis

arXiv:2606.01538v2cs.GRcs.CVcs.LG

TL;DR

The paper asks whether models can infer physical dynamics from videos and extrapolate them forward, a capability relevant to planning and to working with simulation code. It builds MPMWorld and evaluates code-generation and video-diffusion approaches under varying side information, finding complementary strengths and weaknesses.

  • Problem

    The paper asks whether physical dynamics can be recovered from partial scene information, such as video, to predict future states and support planning or work with simulation code.

    Source: §1 · p. 1, §1.1 · p. 2

  • Method

    The authors assemble MPMWorld and evaluate code-generation and video-diffusion models for simulation inference and extrapolation under varying levels of physically relevant side information.

    Source: §1 · pp. 1–2

  • Results

    Code generation produces consistent long-horizon dynamics but struggles with geometry, while video diffusion captures geometry better but hallucinates dynamics and degrades over long horizons.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The analysis offers guidance for future modeling through side-information studies, and the complementary model strengths motivate hybrid approaches combining visual state estimation with structured dynamics.

    Source: §1 · p. 2, §6 · p. 11

    The benchmark is limited to 2D MPM simulations and may favor code-generation approaches because its videos come from ground-truth simulation code.

    Source: §6 · p. 11

Abstract

from arXiv · show

To study the ability to infer physical dynamics from videos and extrapolate them forward in time, we assemble a dataset of 2D Material Point Method (MPM) physical simulations covering rich physical phenomena such as deformable objects, fluids, kinetic objects, and emitters. We study code generation and video diffusion approaches on this dataset, identifying their strengths and weaknesses by varying the amount of physically relevant side information. The code generation model, beyond giving a working demonstration of automatic synthesis of MPM simulations, reveals that such an approach struggles with inferring physical parameters from visual input, but relative to video diffusion, produces physically and temporally stable extrapolations forward in time, while the video diffusion model more strongly identifies geometric properties from visual input but produces physically implausible extrapolations.

1 Introduction

The paper asks whether generative models can infer physical dynamics and extrapolate them, including richer phenomena than the rigid-body settings emphasized in prior video work. It introduces MPMWorlds to compare simulation-code generation with direct video generation and finds complementary strengths and weaknesses.

1 Introduction

  • Physical simulation authoring requires costly programming and domain expertise, motivating tests of whether code or video generators can produce simulations with less hand-programming.The motivating questions are whether generated code can replace hand-authored simulations and whether video generation can avoid simulation code altogether.Source: §1 · p. 1
  • Prior video-model studies of physics extrapolation focused on simple rigid-body dynamics rather than richer phenomena such as fluids and deformable materials [Kang et al., 2024].The paper frames inference and future extrapolation as a fundamental test of physical reasoning.Source: §1 · p. 1
  • The study compares VLM-generated simulation code and video diffusion outputs under different amounts of past-frame and privileged physical information.The privileged information can include physical parameters or initial positions.Source: §1 · pp. 1–2
  • Both approaches achieve partial success: code generation yields more consistent long-horizon dynamics but struggles with geometry, while video diffusion captures geometry better but hallucinates dynamics and deteriorates over time.The analysis also identifies material classes that are particularly challenging or easy for each approach.Source: §1 · p. 1
  • MPMWorlds pairs 2D simulations of diverse physical phenomena with their source code, scene configuration, and resulting video.The dataset includes fluids, deformable objects, rigid bodies, emitters, and motorized objects.Source: §1 · p. 2, Figure 1 · p. 2

1.1 Problem Definition: Inferring and extrapolating physical dynamics.

  • The paper defines inference as recovering a forward simulation from partial scene information, with extrapolation requiring predictions at future time points.The recovered simulation may be represented as source code or neural activations, but must support future prediction.Source: §1.1 · pp. 2–3

2 Related Works

Prior work studies physical reasoning through human-inspired simulation, benchmarks, learned dynamics, executable programs, and generated videos. This paper distinguishes its focus as long-horizon continuation of deformable, multi-material scenes through executable or implicit models.

Intuitive Physics and Physical Reasoning

  • Physical-reasoning research includes controlled benchmarks, relational and object-centric dynamics models, latent physical-property inference, and program-like scene representations [Ullman et al., 2017; Bear et al., 2021; Tung et al., 2023; Battaglia et al., 2016; Wu et al., 2015, 2016; Liu et al., 2019].This work instead asks whether modern generative models recover executable or implicit forward models of complex physical scenes.Source: §2 · p. 3

Programmatic Physical Reasoning and Simulation Code Generation

  • Programmatic physical reasoning includes systems that infer scene programs and benchmarks for code-driven simulation, while this work pairs executable MPM programs with rendered rollouts and structured configurations [Xie et al., 2026].That pairing supports controlled comparisons of code-space and pixel-space prediction for deformable multi-material dynamics.Source: §2 · p. 3

Video Generation Models as World Models

Physical Reasoning Benchmarks

3 MPMWorlds Dataset

MPMWorlds pairs executable 2D MPM programs, structured scene configurations, and rendered videos to study physical inference and extrapolation. Its generation pipeline produces a large collection of varied physical scenes and numerical perturbations.

3 MPMWorlds Dataset

  • MPMWorlds pairs executable simulation code, structured scene configurations, and rendered videos, enabling controlled study of dynamics extrapolation and physical-state inference.Configurations describe scene properties while code implements the corresponding physical process.Source: §3 · pp. 3–4
  • Scene configurations are YAML object records specifying object type, geometry, material model, initial state, appearance, and optional kinematic or emitter behavior.Dynamic-body entries can include constitutive model, material parameters, position, and velocity; emitters specify spawn region, material, and emission rate.Source: §3 · p. 4
  • The simulations use 2D MPM, with diverse materials and interactions including liquids, elastic and plastic materials, snow, sand, viscoplastic and viscoelastic materials, rigid and kinematic objects, and emitters.The scenes use high-contrast backgrounds to minimize texture and lighting confounds.Source: §3 · p. 4

3.1 Dataset Creation Pipeline

  • The dataset pipeline starts from 35 manually written seed simulations, then uses LLM-generated programs and configurations with automated execution checks and consistency validation to produce about 1k valid parent simulations.Flagged validation failures are manually inspected.Source: §3.1 · p. 4
  • LLM-generated scene modifications and constrained numerical perturbations expand the collection by varying objects, materials, geometries, interactions, and initial parameters.Perturbations are intended to reduce bias from LLM-chosen initial configurations and aid generalization.Source: §3.1 · p. 5

3.2 Dataset Statistics

  • MPMWorlds contains 95,805 rendered simulations, including 9,204 scene templates and associated perturbations.Base scenes define physical setups; perturbations vary numerical parameters.Source: §3.2 · p. 5
  • Scenes have a median of 3 objects and mean of 2.9, and 22.5% contain at least two material types while 5.3% contain three or more.The dataset includes multiple material classes, and object types such as moving boundaries and emitters broaden the interaction mechanisms and long-horizon prediction challenge.Source: §3.2 · p. 5

4 Experimental Setup and Evaluation

The experiments vary the physical information supplied to code-generating and video-diffusion models, evaluate generalization across held-out scenes, and use metrics designed for visual and temporal behavior. The evaluation also accounts for the lack of direct physical-state access and the difficulty of tracking deformable shapes.

4.1 Input Conditions

  • The frames-only condition supplies an approximately 2.5-second video prefix, while full configuration adds the complete scene YAML and no-materials or no-positions conditions remove those respective fields.These regimes test what physical information models can infer from visual evidence.Source: §4.1 · pp. 5–6

4.2 Models

  • The VLM generates executable MPM code from prefix frames and optional configuration, whereas the VDM predicts future frames directly in pixel space.The VLM receives a program scaffold; the VDM is conditioned on the observed prefix and, when available, matching textual scene information.Source: §4.2 · p. 6

4.3 Training and Inference

  • The primary held-out split withholds all children and perturbations of test parent simulations, testing generalization to novel physical scenes; a modification-validation split preserves partial template overlap.The latter changes layouts or materials while withholding modified descendants of training base scenes.Source: §4.3 · p. 6

4.4 Evaluation Metrics

  • Long-horizon rollouts require evaluation beyond pixel similarity, but direct particle states are unavailable and deformable materials make tracking-based correspondence metrics ill-defined.The study therefore evaluates spatial correspondence, appearance, motion, stability, and temporal coherence using complementary aspects of continuation.Source: §4.4 · pp. 6–7
  • Object Collapse Score tracks color-consistent visible area from initial object regions and measures maximum relative area loss over time.Temporal Anomaly Rate flags rollouts whose maximum local appearance jump exceeds a threshold relative to ground truth.Source: §4.4 · p. 7
  • W-MAE measures temporal activity magnitude independent of exact spatial alignment, while CTV compares foreground color distributions between predicted and ground-truth rollouts.W-MAE uses frame-to-frame intensity changes; CTV compares per-frame RGB histograms using total variation distance.Source: §4.4 · p. 7
  • Pixel-level MSE and sliced Earth Mover’s Distance are supplementary rather than primary metrics because they correlate poorly with long-horizon physical consistency.Metrics also support best-of-N program selection using only the observed prefix, with the future suffix hidden at test time.Source: §4.4 · p. 7

5 Results and Analysis

Across the evaluated input conditions, executable VLM simulations generally provide more stable long-horizon dynamics than VDM predictions, while VDMs lead on geometric overlap. Performance also varies with material information, material family, scene familiarity, and the availability of structured spatial inputs.

5 Results and Analysis

  • VLM code generation outperforms VDM prediction on most metrics and input conditions, especially temporal stability and long-horizon consistency, while VDMs lead on mIoU.The largest differences are in anomaly rate and object collapse.Source: §5 · p. 8
  • Removing structured scene information substantially degrades VLM performance, while VDM performance changes comparatively little; both families degrade in the frames-only regime.VLM failures tend to involve incorrect physical implementation or unstable programs; VDM failures involve accumulated drift, object instability, and incoherent motion.Source: §5 · p. 8
  • Removing material information has little effect on VLM liquid scenes but substantially degrades elastic/plastic, snow, and sand scenes; VDMs slightly improve across material categories.The authors suggest fluid behavior may be more readily inferred visually or less sensitive to exact parameters, while coherent or history-dependent materials require more explicit information.Source: §5 · pp. 8–9
  • VLM approaches perform strongest on liquid and snow scenes but struggle on sand, while VDMs perform comparatively poorly on elastic and plastic scenes.The paper distinguishes granular sand dynamics from coherent deformable bodies because sand exhibits flow, frictional yielding, and loss of coherent shape.Source: §5 · p. 9
  • VDM motion-activity error rises quickly after the observed prefix and stays above VLM error, while VLM error degrades more gradually over the horizon.VDM collapses accumulate through rollout and reach a substantially higher final rate; VLM rollouts show fewer collapses and mostly stabilize after initial frames.Source: §5 · p. 9, Figure 8 · p. 9
  • Both model classes improve on modified descendants of training scene families, but VLM gains are larger, especially in motion fidelity and temporal stability.VDM collapse and anomaly rates remain high on the modification split, and the VLM–VDM gap is not explained solely by scene novelty.Source: §5 · p. 9
  • Without positional information, VLMs struggle with visual state estimation, whereas VDMs fail on compositional multi-material interactions despite preserving coarse geometry.The VLM may hallucinate an incorrect obstacle layout; the VDM may merge distinct materials into a static average.Source: §5 · p. 10, Figure 9 · p. 10
  • A prefix-quality gate improves over either model alone, routing most samples to VLMs for temporal and stability metrics and predominantly to VDMs for mIoU.The reported metrics include W-MAE, anomaly rate, collapse score, CTV, and mIoU.Source: §5 · p. 10, Table 1 · p. 10

6 Discussion

The 2D MPM simulations challenge current approaches with rich material interactions, and the models show complementary strengths in extrapolation. The authors identify structured scene information and hybrid architectures as directions suggested by their analysis.

6 Discussion

  • Privileged scene information benefits VLMs, suggesting they struggle to recover materials, positions, and physical parameters from vision alone.Source: §6 · p. 10
  • VLMs excel at long-horizon stability but struggle to estimate geometry without explicit positional data, while VDMs change little with the same information.The authors suggest VDMs’ main bottleneck is maintaining coherent long-horizon dynamics, rather than access to scene state.Source: §6 · p. 10
  • The authors propose hybrid architectures combining visual state estimation with structured dynamics, and report a simple gating mechanism as proof of concept.Source: §6 · p. 11
  • The dataset is limited to 2D MPM simulations and omits real-world material and visual richness, including hair, cloth, and highly stiff bodies.Because videos come from ground-truth simulation code, the benchmark may favor code-generation approaches in ways that do not transfer to natural video.Source: §6 · p. 11

A.1 Example Results

The examples illustrate object-collapse failures, material misidentification, and spatial reasoning errors across both model classes.

A.1 Example Results

  • Figures 10 and 11 provide qualitative examples of object collapse, material misidentification, and spatial reasoning errors across both model classes.Source: §A.1 · p. 13

B Evaluation Metrics

The evaluation computes foreground-mask Intersection-over-Union frame by frame and averages across the video.

B Evaluation Metrics

  • Each frame’s RGB image is thresholded into a binary foreground mask, and the frame-wise Intersection-over-Union is averaged across all video frames.When both masks are empty, that frame’s IoU is 1.Source: §B.1 · p. 13

B.2 Object Collapse Score (OCS)

The appendix describes object-collapse and temporal-anomaly metrics, then a threshold-gating procedure that selects between VLM and VDM continuations. Its qualitative examples contrast model failures, while the supplied table caption identifies metrics but gives no values.

B.2 Object Collapse Score (OCS)

  • OCS tracks foreground regions initialized early in a rollout and reports the worst relative area loss across regions and frames for each video.A score near one indicates that at least one generated region nearly disappears; 0.8 means the worst object lost 80% of its initial pixel count at some point, averaged over test samples in the group.Source: §B.2 · pp. 13–14
  • VDM extrapolation can dissipate a rigid block into fluid, while VLMs can misestimate obstacle geometry when positional coordinates are limited.In the illustrated high-energy scene, VLM preserves object permanence and fluid volume; VDM preserves geometry in the limited-coordinate example.Source: Figure 11 · p. 14

B.3 Temporal Anomaly Rate (RTSJ)

  • Temporal anomaly detection compares localized HSV-histogram jump magnitudes in predictions and ground truth, flagging rollouts whose maximum excess jump exceeds a threshold.The reported anomaly rate is the fraction of test samples flagged as anomalous.Source: §B.3 · p. 14

D Prefix-Quality Threshold Gating

  • For each metric and task condition, thresholds are learned on validation data to minimize average continuation error, with the inequality reversed for higher-is-better metrics.At inference, the system selects the best-prefix VLM candidate and uses the learned threshold to choose between VLM and VDM continuations.Source: D Prefix-Quality Threshold Gating · pp. 14–15
  • Table 3 covers visual metrics for shared VLM and VDM input regimes on the modification validation set, reporting top-per-metric means and standard errors.The caption marks the best and worst per metric but provides no cell values in the supplied passage.Source: Table 3 · p. 15

F Compute Resources.

Training each VLM and VDM model took approximately 12 hours on a server with four NVIDIA RTX A6000 GPUs; additional compute supported dataset generation, rendering, and evaluation.

F Compute Resources.

  • Each VLM and VDM model required approximately 12 hours of training on a server with 4 NVIDIA RTX A6000 GPUs.Source: F Compute Resources · p. 15
  • Dataset generation, rendering, and evaluation used additional compute beyond model training.Source: F Compute Resources · p. 15
  • Under the full-configuration setting, VDM rollouts collapsed earlier and more often than VLM rollouts.Source: Figure 13 · p. 16

Related papers

Loading 2606.01538v2…