PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction

Shuai Shao, Jinchen He, Cunxin Fan, Zihan Guo, Shijian Wang, Qihan Ren, Kangning Zhang, Zunnan Xu, Haoyuan Li, Hao Wang, Jianghao Lin, Weiwen Liu, Weinan Zhang

arXiv:2610.17288v1cs.AI

TL;DR

Coding agents must turn video observations into executable models, but visual agreement alone does not establish physical fidelity. PhysCodeGym evaluates agents on video-based Blender reconstruction with visual and physical checks. Agents often capture recognizable objects yet struggle with motion and interactions, with completion and successful-task quality trading off across reasoning effort.

  • Problem

    Assessing coding agents’ physical fidelity matters because matching observed appearance and motion does not ensure correct behavior under altered physical conditions.

    Source: §1 · p. 2

  • Method

    PhysCodeGym evaluates Blender scene programs on 156 tasks using visual and physical metrics, including program inspection and counterfactual checks.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Agents often reconstruct recognizable objects but struggle with motion and physical interactions; low effort completes more tasks and leads overall under a fixed budget, while max effort scores higher among successful tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Object and event checks can pass while native-scene diagnostics identify physical inconsistencies, exposing a gap between matching visible behavior and constructing physically valid scenes.

    Source: §6 · p. 9

    Physical parameters and hidden structure may not be identifiable from video, so the benchmark does not require recovering their true values or exact structure.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Building physical simulations from video requires coding agents to translate observations into executable models and refine them through tool use. We introduce PhysCodeGym, a benchmark for physically verifiable 4D reconstruction with $156$ tasks covering rigid-body, liquid, deformable, and wind-driven motion. Agents autonomously inspect videos, write and run Blender programs, and revise their scenes using simulation feedback. Six metrics combine rule-based measurements with rubric-based judgments to assess visual and physical reconstruction quality. Program inspection, scene measurements, and counterfactual interventions test physical consistency under original and modified conditions. Experiments with frontier coding agents reveal a tradeoff between artifact quality and completion across reasoning-effort settings, while harness comparisons and agent traces link verification behavior with reconstruction performance. Case studies show that matching visible objects and events can coexist with physical defects. Counterfactual tests further expose latent contact and support failures in reconstructions that pass ordinary physical checks, highlighting the gap between reproducing observations and building reliable physical models. Code is available at https://github.com/ShaoShuai0605/PhysCodeGym.

1. Introduction

PhysCodeGym addresses the need to assess whether coding agents can reconstruct physical processes from video, not just produce plausible-looking scenes. Its benchmark combines broad interaction coverage and visual and physical checks; experiments reveal persistent motion and physics errors alongside tradeoffs in agent performance.

1. Introduction

  • PhysCodeGym evaluates coding agents on 156 tasks spanning rigid-body, liquid, deformable, and wind-driven dynamics, including collisions, suspension, pouring, and cloth deformation.Tasks use object checklists, 578 timestamped event annotations, and task-specific physical requirements.Source: §1 · p. 2
  • The benchmark combines rule-based measurements and rubric-based judgments, with counterfactual checks to test physical consistency under altered conditions.Table 1 frames benchmark comparison across task type, input, visual evaluation, behavioral fidelity, physical implementation, and execution harness.Source: §1 · p. 2, Table 1 · p. 3
  • Agents reconstruct recognizable objects but often struggle with motion and physical interactions; low effort completes more tasks and leads overall under a fixed budget, while max effort scores higher among successful tasks.Reconstructions with perfect object and event scores can still violate physical constraints, including energy conservation.Source: §1 · p. 2
  • PhysCodeGym evaluates coding agents’ visual and physical reconstruction, and studies how results vary with harness choice and configured reasoning effort.The authors present the benchmark as a way to examine agents’ strengths, failure modes, and shortcut behaviors.Source: §1 · p. 2

2. Related Work

Prior work reconstructs simulatable scenes, builds dynamic scenes from code, or benchmarks video-to-program reconstruction. PhysCodeGym positions itself as an evaluation task that jointly tests agreement with video observations and whether required physical mechanisms produce the motion.

2. Related Work

  • Video-based physical reconstruction methods recover scene geometry, properties, and dynamics for simulation, whereas PhysCodeGym evaluates coding agents’ generated programs against observations and physical requirements.Related approaches include differentiable continuum, mesh-free reduced-order, and spring–mass simulation, as well as ∆YNAMICS and OVOW.Source: §2 · p. 3
  • Programmatic scene benchmarks span graphics editing, 3D modeling, and inverse rendering; dynamic-scene systems add code iteration or motion critique, while physical benchmarks audit internal physics or physical assertions.The cited systems include Thinking in Blender, VIGA, Code2Worlds, 4DBuildBench, and PhysCodeBench.Source: §2 · p. 3
  • Video-to-program benchmarks cover layout, motion, event, and mesh reconstruction, but 4DCodeBench permits keyframes, prescribed motion, or simulation as alternatives and primarily scores reconstruction fidelity.BVB prohibits physics simulation, while VisPhyWorld and MPMWorlds study executable simulations from synthetic observations.Source: §2 · p. 3
  • PhysCodeGym requires physical implementation of core motion and tests mechanism dependence through independent scene evaluation and matched interventions on contact, support, or actuation.The comparison framework distinguishes event evaluation, motion alignment, internal physics checks, and mechanism dependence.Source: §2 · p. 4

3. PhysCodeGym: Dataset and Task

PhysCodeGym builds its dataset around visible physical causes expressible in Blender and annotations grounded in what each video shows. It defines reconstruction as an executable scene whose modeled physics produces the observed motion, rather than a rendered match alone.

3.1. Dataset

  • The dataset contains 156 tasks across rigid-body, liquid, deformable, and wind-driven motion, collected from eight-second video segments standardized to 3840×2160 resolution, 30 FPS, and 240 frames.After deduplication and manual review, each retained clip defines one task.Source: §3.1 · p. 4
  • Tasks cover visible physical processes expressible in Blender, exclude processes such as tearing and fracture, and annotate only what the reference video shows.Tasks have mutually exclusive main categories, with rigid-body and liquid tasks further divided by dominant motion mechanism.Source: §3.1 · p. 4
  • Each task has object, event, and physical-requirement annotations that support object accuracy, event scoring, and physical-validity checks.Event annotations specify actions, temporal order, and final states with timestamped evidence; selected tasks use force or contact interventions and re-simulation.Source: §3.1 · p. 4

3.2. Task Definition

  • A reconstruction submission is a Blender scene program and the native scene it produces, with the rendered sequence serving as one presentation of the process.The program builds scene elements and sets physical properties, initial conditions, constraints, and forces so the native scene evolves over time.Source: §3.2 · p. 4
  • A valid reconstruction must match appearance, layout, and visible events while producing motion through the modeled physical process rather than scripted trajectories or hidden supports.Explicitly modeled external actuators are allowed when their objects respond through physical interaction.Source: §3.2 · p. 5
  • The benchmark does not require recovering true physical parameters or exact hidden structure when video observations do not identify them.For example, friction may be inferred in a free-fall and inclined-sliding scenario while mass remains unobservable.Source: §3.2 · p. 5
  • Agents observe reference-video frames and clips, build and run self-contained Blender programs, and submit a scene program that produces a 240-frame sequence.The program must bake its simulation cache before rendering; a checking tool reports format and policy issues before submission.Source: §3.2 · pp. 5–6
  • Visual scoring measures rendered resemblance, while physical scoring measures the scene’s dynamic process; only physical validity reads the program and native scene.The metrics are reported individually to distinguish failures such as scripted motion that matches frames from physically valid scenes that miss observed events.Source: §3.2 · p. 6

4. Experiments

The experiments run coding agents in a bounded terminal-and-Blender workflow and assess rendered similarity, event completion, motion alignment, and physical validity separately. The evaluation aggregates visual and physical metrics while retaining component scores to distinguish failure types.

4.1. Setup

  • Agents typically iterate through video observation, scene-program writing, simulation runs, preview comparison, and revision, with up to 20 Blender launches and a three-hour limit per task.Unless stated otherwise, agents use high reasoning effort; the latest run with a rendered sequence is scored, and tasks without one receive zero.Source: §4.1 · p. 6

4.2. Models

  • The main comparison uses Codex, Claude Code, and Kimi Code harnesses, and separately tests DeepSeek-V4.1-Flash at low, high, and max reasoning effort in Claude Code.The harness comparison includes GPT-5.6-Sol, GPT-5.6-Luna, GPT-6-Luna, GPT-6-Astra, Claude Sonnet 5, GLM-5.3-Flash, DeepSeek-V4.1-Flash, Qwen3.8-Flash, and Kimi K2.8 Preview.Source: §4.2 · p. 6

4.3. Evaluation

  • Visual metrics combine frame-level SSIM and LPIPS with checklist-based object accuracy, while the physical score combines event score, motion score, and physical validity.Object accuracy summarizes vision-language judgments against the checklist; pixel- and perceptual-level similarity complement object-level correctness.Source: §4.3 · p. 6
  • Event scoring uses three independent item-level judgments and a majority verdict, while motion scoring averages spatial and duration-weighted overlap of motion masks.A failed key event caps the event score at zero; motion masks are formed from blurred grayscale frames, a running-average background, and thresholding.Source: §4.3 · pp. 6–7
  • Physical validity checks task rubrics by inspecting the program and native scene, measuring them, and, on selected tasks, re-simulating after interventions on forces, contacts, or initial conditions.Visual, physical, and overall scores are reported separately; the overall score averages the displayed visual and physical scores.Source: §4.3 · p. 7

5. Results & Analysis

Across nine configurations, GPT-6-Astra leads overall, visual, and physical scores, while object scores exceed event scores and motion remains weak. Effort and harness comparisons show that completion and verification behavior relate to aggregate performance, while physical audits expose defects beyond visible matches.

5.1. Benchmark Results

  • 0.663 overall, 0.693 visual, and 0.633 physical scores make GPT-6-Astra the leader; GPT-5.6-Sol ranks second overall at 0.570.The benchmark also reports that all nine configurations score higher on objects than events, with GPT-6-Astra reaching only 0.324 on motion.Source: §5.1 · p. 8
  • GPT-5.6-Sol and GPT-6-Astra lie on the estimated-cost Pareto frontier: Sol scores 0.570 at approximately $3.40 per task, while Astra scores 0.663 at $11.57.Astra's estimated cost is 3.4 times Sol's; Figure 3 compares overall score against estimated API cost and recorded token use, and greater resource use does not consistently yield better reconstruction.Source: §5.1 · p. 8, Figure 3 · p. 8
  • 16.98M cumulative tokens per task yields a 0.450 score for DeepSeek-V4.1-Flash, versus GPT-5.6-Sol's 5.70M tokens and 0.570 score.DeepSeek's estimated cost is $0.72 per task; cumulative tokens include repeated context and cannot directly measure effective reasoning.Source: §5.1 · p. 8

5.2. Analysis

  • 0.476 overall makes low effort the leader when missing artifacts receive zero, while max scores 0.547 versus low's 0.527 among produced artifacts.Under the same 3-hour and 20-launch limits, the aggregate advantage for low effort is linked mainly to completing more tasks; max has longer runtimes and more recorded output tokens. Figure 4 compares visual, physical, and aggregate scores.Source: §5.2 · p. 8, Figure 4 · p. 8
  • 0.479 overall and 0.550 on events for Codex exceed Claude Code's 0.450 and 0.446, respectively, at high effort.Codex has slightly lower motion overlap; traces show it identifying problems, revising, rerunning, and inspecting outputs, while Claude Code's final revisions lack successful execution verification within the launch budget. Figure 5 compares the harnesses.Source: §5.2 · p. 8, Figure 5 · p. 9
  • Six reconstructions with perfect object and event scores still show physical failures, including unexplained energy increases and interpenetration.Program inspection and scene measurements reveal these defects, while interventions changing support, contact, or initial conditions expose further latent problems.Source: §5.2 · pp. 8–9

6. Conclusion

PhysCodeGym benchmarks video-to-Blender reconstruction across 156 tasks using visual and dynamic metrics alongside program inspection, measurements, and selected counterfactual checks. The conclusion emphasizes that visible matches can coexist with physically inconsistent scenes, and bounds the benchmark to Blender-expressible processes and rubric-defined checks.

6. Conclusion

  • PhysCodeGym is a 156-task benchmark for coding agents reconstructing physical processes from video as executable Blender programs.Visual and dynamic metrics assess rendered outcomes, while verification agents inspect scene programs and measure against task rubrics, including counterfactual checks on selected tasks.Source: §6 · p. 9
  • Object and event checks can pass while native-scene diagnostics identify physical inconsistencies.The conclusion frames this as a gap between matching visible behavior and constructing physically valid scenes.Source: §6 · p. 9
  • The benchmark covers only processes Blender can express and physical checks specified by its rubrics.The conclusion identifies extending to more simulation engines and physical interactions as future work.Source: §6 · p. 9

A. Dataset and Execution Framework

The appendix describes PhysCodeGym's task composition, illustrative reference clips, and Harbor-based execution and evaluation framework. It details category coverage, the run limits, and separate verification of submitted scenes and outputs.

A.1. Composition and Sources

  • The dataset contains 156 tasks and 578 event items, spanning 95 rigid-body, 40 liquid, 12 deformable, and 9 wind-driven tasks.Table 3 presents task and event-item counts by main category and subcategory.Source: §A.1 · p. 14, Table 3 · p. 14
  • Rigid-body tasks cover seven mechanisms, while liquid tasks cover pouring and filling, jets and surface flow, and liquid–object interactions.Deformable and wind-driven categories contain few tasks and are not subdivided; reference videos come from Pexels, Physics-IQ, BAAI-DataCube Physics-aware Videos, and IRIS.Source: §A.1 · p. 14

A.2. Reference Video Examples

  • Figures 6–9 show 32 selected reference clips, with each row displaying four frames sampled at approximately 0%, 33%, 67%, and 100% of its clip.The selection illustrates category and subcategory range; task frequencies are reported separately in Appendix A.1.Source: §A.2 · p. 15, Figure 6 · p. 15, Figure 7 · p. 16

A.3. Harbor Integration

  • Each Harbor task packages instructions, a reference video, and environment configuration for agents to observe, program, and check reconstructions in a container.The environment provides Blender and command-line tools for video observation, scene execution and inspection, and preview rendering.Source: §A.3 · p. 19
  • Harbor normally limits runs to 3 hours and 20 Blender launches, shared across scene execution, inspection, and preview rendering.Video observation does not consume Blender launches but counts toward the time limit.Source: §A.3 · p. 19
  • After agent execution, a separate verification container checks the submitted program, scene, rendered outputs, and image and motion measurements.Subsequent jobs assess objects and events, and a verification agent checks annotated physical criteria, including counterfactual tests for selected tasks.Source: §A.3 · p. 19

B. Evaluation Protocol

Physical validity is assessed against task-specific rubrics through program and scene inspection, executable measurements, and controlled counterfactual interventions. These procedures distinguish missed reference events from physical violations and test whether motion depends on required interactions.

B.1. Task Rubrics and Executable Verification

  • Each task rubric specifies permitted setup and actuation, mechanisms to check, and evidence needed for judgment.Criteria include numerical stability, gravity, mass, and actuation, plus task-specific contact, support, and structural-integrity requirements; the same rubric applies to every candidate for that task.Source: §B.1 · p. 20
  • The verifier maps rubric roles to scene objects, writes implementation-specific probes, and measures geometry, states, contacts, support, and motion in Blender.Python tools analyze measurements, and the report gives criterion-level judgments with source references and measured evidence; missed reference events are scored as events, not automatically treated as physical violations.Source: §B.1 · p. 20

B.2. Counterfactual Tests

  • Counterfactual tests compare an independently modified scene with an unchanged baseline to check whether motion depends on a required physical interaction.Interventions may alter force, remove contact, or change initial conditions while holding other relevant conditions fixed; the verifier checks that the intervention takes effect and uses control scenes to validate measurements.Source: §B.2 · p. 20

C. Additional Results and Analysis

The analysis compares scores on aligned task subsets, reconstruction quality when agents produce artifacts, and the relationship between reasoning effort and quality.

C. Additional Results and Analysis

C.1. Scores on Aligned Task Subsets

The aligned-subset analysis recomputes scores on each configuration’s physical-evaluation tasks, while a separate analysis reports quality among produced artifacts.

C.1. Scores on Aligned Task Subsets

  • Table 5 recomputes all metrics on each configuration’s physical-evaluation task subset, with scores aligned within rows but task sets differing across configurations.Video metrics remain event and motion measurements; validity uses the corresponding physical reports.Source: Table 5 · p. 21, §C.1 · p. 21
  • Restricting GPT-6-Astra’s visual mean from 156 to 155 tasks changes Svis from 0.693 to 0.694 and Stot from 0.663 to 0.664.Individual task scores do not change.Source: §C.1 · p. 21, Table 5 · p. 21

C.2. Scores on Produced Artifacts

  • Table 6 reports scores conditional on rendered outputs, excluding tasks without outputs while retaining zero scores for existing outputs.Validity and physical scores use available physical judgments; the full-task results also account for missing outputs.Source: Table 6 · p. 21, §C.2 · p. 21

C.3. Reasoning Effort, Completion, and Reconstruction Quality

Reasoning-effort comparisons distinguish aggregate scores that include missing outputs from quality on produced artifacts and shared tasks. Max effort leads on several conditional measures, while low effort’s aggregate advantage is mainly associated with completion.

C.3. Reasoning Effort, Completion, and Reconstruction Quality

  • Max effort has the highest overall score on produced artifacts (0.547), compared with 0.527 for low and 0.525 for high.Max also has the highest conditional event and physical scores, 0.615 and 0.492.Source: §C.3 · p. 22
  • Max again has the highest visual, physical, and overall means on shared tasks, although the paired overall-score difference from low is 0.030 with a 95% interval of [−0.001, 0.061].The interval includes zero, and the bootstrap does not estimate repeated-run variability.Source: §C.3 · p. 22
  • Asymmetric missing outputs contribute +0.036 to low’s mean-score advantage over max, while tasks with outputs from both contribute −0.020, for a net +0.016.The analysis identifies completion outcomes as the main source of low’s aggregate lead; original-condition-only validity reduces the gap by approximately 0.003.Source: §C.3 · p. 22
  • Max uses 2.46× low’s recorded output tokens on average, and mean execution time rises from 93.3 to 110.7 minutes under the fixed three-hour budget.Additional missing outputs concentrate in timeouts and final submissions that fail execution or lack required files.Source: §C.3 · p. 22

D. Case Studies

  • The case studies connect evaluation scores with reconstructed behavior and executable evidence, including metric interpretation, physical failures, counterfactual checks, and harness verification.Source: D. Case Studies · p. 23

D.1. Metrics and Reconstruction Quality

The case studies show that visual, event, motion, and physical-validity measures can disagree. Original-scene audits and counterfactual controls expose defects that visible reconstruction scores or ordinary trajectories can miss.

D.1. Metrics and Reconstruction Quality

  • Video metrics assess structural and perceptual similarity, object accuracy, event completion, and motion, while physical validity evaluates the submitted program and Blender scene.The case studies pair synchronized reference and candidate frames with component and summary scores.Source: §D.1 · p. 23

D.1.1–D.1.2. Two Pendulums and Rolling Quail Egg

  • GPT-5.6-Luna has the highest motion IoU but an event score of 0 and validity 0.5, while GPT-6-Astra scores 1 on events and validity.On the egg task, DeepSeek-V4.1-Flash has motion IoU 0.235 versus Astra’s 0.225, but validity is 0.5 versus 1 because stationary eggs interpenetrate.Source: Figure 11 · p. 24, Figure 12 · p. 25

D.1.3–D.1.5. Wine Pouring, Avocado Halves Entering Water, and Thai Flag

  • All six wine-pouring candidates score 1 on events despite appearance differences; Claude Sonnet 5’s SSIM is 0.774, but its perceptual similarity is below Astra’s, 0.511 versus 0.589.In the avocado case, Luna has validity 1 but an event score of 0; the Thai-flag case has event score 1 despite cloth self-intersections in 14 frames and validity capped at 0.5.Source: Figure 13 · p. 26, Figure 14 · p. 27, Figure 15 · p. 28

D.2. Physical Validity Failures

  • Six selected reconstructions have perfect object and event scores yet physical-validity failures, including excess energy, embedded eggs or crate geometry, gripper-block overlap, and support from an oversized collider.The cases report unchanged-scene measurements, including a 9.441 mJ gravitational-potential value against an approximately 4.971 mJ fitted initial energy in the Newton cradle.Source: §D.2 · p. 29, Figure 16 · p. 29, Figure 17 · p. 29
  • The flag case also detects nonadjacent self-intersections in unchanged scene frames 54 and 60, with 41 and 352 intersecting triangle pairs, respectively.The inspected frame counts are geometric pair counts, not independent events or a claim about the crossing’s first onset.Source: Figure 21 · p. 31

D.3. Counterfactual Mechanism Checks

  • All five selected counterfactual cases score 1.0 on original-condition checks but fail a counterfactual rubric criterion; failures require measured intrusion or unsupported motion together with a responsive control.Examples include a shifted release entering dish material and a cue ball remaining suspended beyond the finite table support.Source: §D.3 · p. 32, Figure 22 · p. 32, Figure 23 · p. 33

D.3.4–D.3.5. A Different Reconstruction Omits Contact with a Chess Piece; A Solid Token Behaves as if It Were Absent

  • Changed releases reveal omitted contact with a chess piece or solid token that the original rolling trajectories never reach.In the chess-piece case, removing the knight leaves the moving piece’s positions and velocities unchanged; adding passive contact avoids intrusion.Source: Figure 25 · p. 35, §D.3.4 · p. 35, Figure 26 · p. 36

D.4. Cross-Harness Verification

  • In two tasks, Codex’s completed checks and reruns accompany event-score gains from 0 to 1 for rolling and 0 to 0.857 for pouring, while the latter still misses a glass-support event.The comparison discusses proactive-verification guidance, but the historical full Claude Code system prompts were not retained; that part relies on local reproductions.Source: §D.4.1 · p. 37, §D.4.2 · p. 38, §D.4.3 · p. 39
Loading 2610.17288v1…