PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement

Tianyidan Xie, Peiyu Wang, Hu Jiaxin, Yuyi Qian, Yuxuan Wang, Shenyi Wang, Rui Ma, Yanlun Peng, Lanjun Wang, Ying Tai, Jian Yang, Zili Yi

arXiv:2604.23580v2cs.ROcs.AI

TL;DR

Generated simulation code can execute while still representing the wrong physics, and existing benchmarks do not directly assess that correctness. PhysCodeBench and PhysCodeEval provide expert-validated examples and engine-state evaluation, while SMRF separates generation, correction, and refinement. SMRF raises physical-assertion pass rates over the best proprietary baseline and retains performance across engines.

  • Problem

    Existing code-generation benchmarks and visual checks do not directly establish whether executable simulations obey the intended physics, motivating direct physical-correctness measurement and cross-engine evaluation.

    Source: §1 · p. 2

  • Method

    PhysCodeBench supplies expert-validated simulation examples and PhysCodeEval checks engine-state physical correctness; SMRF divides generation, error correction, and refinement among specialized agents.

    Source: §4 · p. 7, §5 · p. 8, §7 · p. 13

  • Results

    70.6% vs. 23.8% physical-assertion pass rates for SMRF and the best proprietary baseline, respectively; SMRF retains 87.9% of its performance across engines.

    Source: §7 · p. 13

  • Takeaways & Limitations

    Ablations identify targeted error correction, rather than generic refinement, as the key driver of physical accuracy in SMRF.

    Source: §6.4 · p. 11

    The benchmark targets high-level APIs rather than solver implementation, and its physical-correctness metrics cover the evaluation split rather than all 1,200 examples.

    Source: F Limitations · p. 29

Abstract

from arXiv · show

Translating natural-language descriptions of physical phenomena into executable simulation code requires both programming expertise and physical reasoning. Current large language models (LLMs) lack this combination: they frequently produce code that runs but simulates the wrong physics. We introduce PhysCodeBench, the first benchmark for this task, with 1,200 expert-validated examples spanning four physical domains. Its evaluation suite, PhysCodeEval, goes beyond executability and visual fidelity to measure physical correctness directly from the engine state via conservation-law residuals and expert-written assertions, and supports cross-engine evaluation to disentangle physics reasoning from API fluency. As a reference method, we propose the Self-Corrective Multi-Agent Refinement Framework (SMRF), which decouples physics-aware error correction from code generation through specialized agents. This design is motivated by our finding that targeted correction, rather than generic iterative refinement, is the key driver of physical accuracy. SMRF nearly triples the physical-assertion pass rate of the best proprietary baseline (70.6\% vs.\ 23.8\%) and retains its advantage under cross-engine transfer.

1 Introduction

Physics-aware symbolic simulation requires both coding and physical reasoning, yet existing benchmarks do not directly measure whether generated code simulates the intended physics. PhysCodeBench, PhysCodeEval, and SMRF address this gap with expert-validated examples, engine-state checks, and specialized agents.

1 Introduction

  • Symbolic simulation requires code to execute, implement the intended laws and parameters, and respect boundary conditions and numerical stability.The task places physics in executable scene code rather than a learned generative model.Source: §1 · p. 2
  • Existing code-generation benchmarks lack direct measures of physical correctness, while visual plausibility can conceal violated conservation laws.Cross-engine evaluation is also needed to distinguish physics reasoning from API fluency.Source: §1 · p. 2
  • PhysCodeBench is presented as the first benchmark for physics-aware symbolic simulation of 3D scenes, with 1,200 expert-validated examples across four domains.The benchmark includes seed-level train/test isolation and a three-fold contamination audit.Source: §1 · p. 3
  • PhysCodeEval measures physical correctness directly from engine state through conservation-law residuals and expert-written assertions, and supports cross-engine evaluation.SMRF separates physics-aware error correction from code generation through specialized agents.Source: §1 · p. 3

2 Related Work

Prior work addresses physical reasoning, code generation, or iterative refinement, but the paper identifies a gap in executable physics simulation code whose physical correctness is measured directly.

2 Related Work

  • Prior physical-reasoning work studies video prediction or domain-specific solvers, while LLM systems generate related programs or specifications rather than complete physics simulations evaluated for correctness.The paper positions its task as authoring a complete executable simulation and measuring its physical correctness.Source: §2 · p. 3
  • General code-generation benchmarks use input-output tests, but passing syntax checks can still produce a simulation of the wrong physical world.PhysCodeEval addresses the missing evaluation of physically correct code through state-space checks.Source: §2 · p. 3
  • Generic iterative refinement lacks physics-specific knowledge, and the paper reports a 26-point gap between generic self-refinement and specialized correction.Preference alignment is applied to physics-specific agent roles using expert simulation preferences.Source: §2 · p. 3

3 The PhysCodeBench Benchmark

PhysCodeBench pairs natural-language instructions with expert-validated Genesis implementations and metadata, using staged curation and controls to protect test-set integrity.

3.1 Overview

  • PhysCodeBench contains 1,200 examples across rigid-body, soft-body, fluid-dynamics, and mechanics domains, split into 1,000 training and 200 testing examples.Examples include expert-validated Genesis implementations, difficulty, physical-law, object-type, and preference metadata.Source: §3.1 · pp. 3–4

3.2 Dataset Curation Process

  • Curation expands 50 expert seed prompts into 2,000 candidates, filters a pool of 1,400, then validates generated code for execution and simulation faithfulness.Experts modified failed candidates up to three times; 1,120 pooled instructions were retained, alongside 80 expert-authored test instructions, for 1,200 total examples. Metadata and expert preference judgments were also collected.Source: §3.2 · pp. 4–5

3.3 Test-Set Integrity

  • Test-set integrity uses held-out seed prompts and expert-authored instructions, with no test-derived artifact used to train the agents.Semantic, lexical, and code-level audits found no evidence of near-duplicate prompts or memorized test solutions.Source: §3.3 · pp. 5–6

4 The PhysCodeEval Evaluation Suite

PhysCodeEval combines code-execution and visual-fidelity scores with direct physical-correctness measurements from engine state, and supports evaluation on a MuJoCo-native subset.

4 The PhysCodeEval Evaluation Suite

  • PhysCodeEval reports a 100-point total from code quality and visual fidelity, alongside a separate physical-correctness axis.Code quality scores execution and expected artifacts; visual fidelity scores semantic alignment and motion realism.Source: §4 · p. 6
  • Expert-written executable assertions measure whether simulations satisfy scenario-specific physical conditions, with failed executions counting as assertion failures.Assertions are written from instruction text, reviewed by another expert, and reported separately because counts vary across scenarios.Source: §4 · p. 6
  • Conservation diagnostics compute invariant residuals on runs with trajectories, while dissipative scenarios use conditions such as energy dissipation, terminal-state stability, and non-interpenetration.The residual checks use a 5% tolerance and are reported on the subset where invariants are defined.Source: §4 · p. 7
  • A set of 120 test instructions has MuJoCo-native references and assertion sets, enabling evaluation across engines independently of a method’s adaptation strategy.Source: §4 · p. 7

5 Methodology

SMRF separates code generation, execution-failure correction, and physical-parameter refinement across specialized agents. The Error Corrector acts only after execution fails; the Simulation Refiner acts after execution succeeds.

Agents and control flow

  • The Simulation Generator produces initial code from the instruction and is trained on PhysCodeBench’s 1,000 training pairs with supervised fine-tuning.All agents are initialized from DRDQ-32B and specialized through role-specific training.Source: §5 · p. 8
  • On execution failure, the Error Corrector diagnoses and repairs code using 520 training tetrads, with up to three correction attempts before returning failure.Its training errors span API misuse, parameter miscalibration, boundary conditions, temporal discretization, and other issues.Source: §5 · p. 8
  • After execution succeeds, the Simulation Refiner optimizes physical parameters while preserving code structure, using supervised fine-tuning followed by preference alignment.DPO uses 600 majority-labeled preference pairs comprising 1,980 judgments; β = 0.1.Source: §5 · p. 8

6 Experiments

Experiments compare SMRF with proprietary, open-source, and single-agent baselines across physical correctness, difficulty, ablations, engine transfer, human ratings, and robustness checks. The reported advantages include higher assertion pass rates, better hard-task performance, and cross-engine transfer.

6.1 Experimental Setup

  • Baselines span proprietary and open-source models as well as single-agent and SMRF variants; role-specific LoRA training used two A100 GPUs, and inference scores average five runs.The main configuration required approximately 34 GPU-hours.Source: §6.1 · pp. 8–9

6.2 Quantitative Analysis

  • 35.9 out of 100 was the best zero-shot model’s total score, while Claude-3.5-Sonnet passed only 23.8% of physical assertions despite full engine documentation.Table 1 reports results on 200 test examples using avg@5; its Code and Visual scores are each out of 50, and Total is out of 100.Source: §6.2 · p. 9, Table 1 · p. 9
  • Vanilla-agent SMRF lifts Sphys from 16.2% to 30.4% for 6.1 total points; DPO adds 5.6 points, while conservation checks pass 81.4% versus Claude-3.5-Sonnet’s 35.3%.The strongest fine-tuned single agent trails SMRF by 29.7 total points and 38.5 points on Sphys.Source: §6.2 · pp. 9–10

6.3 Performance Across Difficulty Levels

  • SMRF + SFT + DPO loses 31.6% of its easy-task score on hard tasks (77.5→53.0) and retains a 28.5-point lead over the strongest single agent.Claude-3.5-Sonnet loses 48.9% (45.2→23.1); hard tasks involve three or more interacting laws or multi-phase scenarios.Source: §6.3 · p. 10, Table 2 · p. 10

6.4 Ablation Studies

  • Removing the Error Corrector costs 11.3 total points and 17.8 Sphys points, while removing the Simulation Refiner costs 8.8 total points and 10.8 Sphys points.DPO adds 5.6 total points and 7.1 Sphys points over the SFT-only refiner; every ablation costs more on Sphys than on the total.Source: §6.4 · p. 11, Table 3 · p. 11

6.5 Cross-Engine Evaluation

  • SMRF transfers from Genesis to MuJoCo at 41.2 points zero-shot, above every zero-shot baseline, and reaches 58.3 versus 66.3 Genesis points with 250 API-mapping samples.That adapted result retains 87.9% of Genesis performance on the same 120 instructions.Source: §6.5 · p. 11, Table 4 · p. 12

6.6 User Study and Qualitative Analysis

  • In a 10-participant study, SMRF received the highest ratings for physical accuracy, code readability, and overall usefulness.Automatic metrics correlated with human ratings at the system level, but the authors caution that they are not substitutes for state-based measurements.Source: §6.6 · p. 12, Table 5 · p. 12

6.7 Robustness and Integrity Checks

  • Bootstrap intervals did not overlap; prompt-style scores differed by 1.8 points, and equal-budget single-agent refinement remained 26.2 points below SMRF.Seed-grouped cross-validation yielded a total-score standard deviation of 1.4 points.Source: §6.7 · p. 13

7 Conclusion

PhysCodeBench evaluates physics-aware symbolic simulation with expert-validated examples and engine-state measurements, while SMRF is the reference framework. SMRF improves physical-assertion pass rates and retains performance across engines, with the stated scope limited to high-level APIs.

7 Conclusion

  • PhysCodeBench is introduced as the first benchmark for physics-aware symbolic simulation of 3D scenes.It contains 1,200 expert-validated examples, reads conservation residuals and expert assertions from engine state, and includes a MuJoCo-native subset for cross-engine evaluation.Source: §7 · p. 13
  • SMRF reaches 67.2 points, raises physical-assertion pass rates from 23.8% to 70.6%, and retains 87.9% of performance across engines.The framework’s error corrector fires only on execution failure.Source: §7 · p. 13
  • The benchmark targets high-level APIs rather than solver implementation.The conclusion also identifies the execution-failure-only trigger of the error corrector as a limitation.Source: §7 · p. 13

A Dataset Details

PhysCodeBench combines expert-validated Genesis simulations with metadata and evaluation spanning dataset construction, code execution, visual quality, physical correctness, and cross-engine transfer. Its examples are curated and annotated to support assessment beyond whether generated code runs or looks plausible.

A.1 Examples and Composition

  • The 1,200-example benchmark covers four physical domains; its 200-example test set contains 68 rigid-body, 52 soft-body, 46 fluid, and 34 mechanics examples.The test set includes 116 easy and 84 hard examples; 376 examples also carry a secondary domain label.Source: §A.1 · p. 17

A.2 Curation Funnel and Success Rates

  • Only 430 of 1,400 pooled candidates (30.7%) passed both validation stages on the first attempt; expert revision produced 1,120 validated pool-derived examples.The final set adds 80 expert-authored test instructions to 1,000 training and 120 pool-derived test examples.Source: §A.2 · p. 17

A.3 Generation Prompt Template

  • Generation prompts pair each scenario instruction with retrieved Genesis documentation and official repository examples, then request Python code to initialize the environment, create physical objects, configure interactions, set a camera, run, and save video.They also request realistic physical properties, conservation-law adherence, correct collision handling, and appropriate forces and constraints.Source: §A.3 · p. 18

A.4 Annotation Guidelines and Agreement

  • Experts annotate difficulty, physical-law tags, preferences, and executable assertions; the reference implementations pass 98.6% of assertions and all execution and artifact checks.Three annotators label difficulty (mean pairwise Cohen’s κ = 0.78); five experts provide preferences for 600 implementation pairs, yielding 600 DPO training pairs.Source: §A.4 · pp. 19–20

B.1 Code-Level Scoring

  • Code-level scoring allocates 25 points each to execution, artifact compliance, CLIP instruction similarity, and motion smoothness.Execution errors score zero for the execution component; the artifact rubric checks video existence, size, format, resolution, frame rate, and duration.Source: §B.1 · p. 20, §B.2 · pp. 20–21

B.2 Visual Scoring

  • Motion scoring uses RAFT optical flow and second-order velocity differences, with jitter, teleportation, and interpenetration lowering the score.Videos are uniformly resampled to 25 fps, and benchmark-wide calibration constants are fixed rather than tuned per system or scenario.Source: §B.2 · pp. 21–22

B.3 Direct Physical-Correctness Scoring

  • Physical checks query engine state rather than rendered video, using conservation residuals for scenarios with defined invariants and necessary conditions for dissipative regimes.The conservation subset contains 102 rigid-body and mechanics test cases; Table 10 reports pass rates at 5% tolerance, with check-specific denominators.Source: §B.3 · p. 22, Table 10 · p. 23

B.4 Cross-Engine Protocol

  • Cross-engine evaluation ports assertions and expert references for 120 test instructions, including fluid cases limited to MuJoCo-supported particle approximations.With 250 API-mapping adaptation samples, SMRF passes 63.4% of ported assertions versus 20.1% for zero-shot Claude-3.5-Sonnet.Source: §B.4 · pp. 22–23

C Training and Inference Details

The training and inference setup gives models engine documentation and example code, then uses separate agents for code generation, error correction, and refinement. The Error Corrector is trained on failure examples drawn from training instructions, while inference applies bounded correction rounds before refining successful programs.

C Training and Inference Details

  • For 32K-context models, inference uses about 20K tokens of BM25-retrieved documentation and three similar official examples; a 4,096-token generation cap constrains reasoning-style open models.The cap does not affect comparisons against Claude-3.5-Sonnet or the fine-tuned single agent, which emit code directly; context ablation finds retrieval recovers most of Claude’s full-context score.Source: C Training and Inference Details · pp. 23–24

D.1 Agent Loop

  • On runtime failure, the Error Corrector uses feedback for up to three correction rounds; after execution succeeds, the Simulation Refiner adjusts physical parameters and code quality while preserving structure.The generator, Error Corrector, and Simulation Refiner are separately fine-tuned LoRA specializations sharing one backbone.Source: §D.1 · p. 24

D.2 Error-Corrector Tetrads

  • The Error Corrector trains on 520 training-only tetrads of instruction, failing code, runtime feedback, and expert-corrected code.Training failures span API misuse (43%), parameter miscalibration (28%), boundary conditions (18%), temporal discretization (7%), and other issues (4%).Source: §D.2 · p. 24

D.3 Semi-Automated Benchmark Expansion

The paper pairs a prompt evaluator for reducing expert review effort with domain-level benchmark comparisons. SMRF’s reported advantage over proprietary and fine-tuned single-agent baselines holds across all four physical domains.

D.3 Semi-Automated Benchmark Expansion

  • In a held-out trial of 200 fresh candidates, a prompt evaluator reduced expert filtering time by 64% at equal acceptance quality.The evaluator is packaged with the validation harness so community contributions require one expert pass rather than full pipeline review.Source: §D.3 · p. 25

E.1 Performance by Physical Domain

  • SMRF leads Claude-3.5-Sonnet by 29.5–32.8 points and DRDQ-32B+SFT+DPO by 27.9–31.2 points in every domain.All systems find fluid dynamics hardest and mechanics or rigid-body scenarios easiest.Source: §E.1 · p. 25, Table 12 · p. 26

E.2 Error Correction Analysis

The paper examines correction behavior, compares specialized refinement against generic alternatives, and reports qualitative cases and contamination checks. It also treats model-judged plausibility as supplementary to state-based physical checks.

E.2 Error Correction Analysis

  • Error Corrector diagnoses include API-name mismatches, camera configurations that miss the phenomenon, and physically inadmissible values.Table 13 reports inference-time error frequencies and correction success rates.Source: §E.2 · p. 26

E.3 Single-Agent Iterative Refinement and Best-of-k

  • 41.0 for the best single-agent self-refinement and 40.4 for Claude-3.5-Sonnet best-of-5 remain below SMRF by 26.2 points for the strongest alternative.Best-of-5 costs five times as much generation; Table 14 compares these alternatives with SMRF.Source: §E.3 · p. 26, Table 14 · p. 27

E.4 VLM-Based Physical Plausibility

  • VLM rankings and margins agree across two judges, but both judges are evaluated baselines, so these ratings remain supplementary to state-based physical checks.The non-circular physical check uses engine-state measurements rather than a model judge.Source: §E.4 · pp. 26–27

E.5 User-Study Protocol

  • The user study uses ten physics-simulation-experienced participants who rated anonymized implementations, with each participant rating all 200 implementations across four sessions.The metric-validity correlation uses Claude-3.5-Sonnet, the strongest single agent, and SMRF.Source: §E.5 · p. 27

E.7 Additional Qualitative Comparisons

  • For cloth falling over a sphere and liquid poured over a duck, SMRF produces contact-consistent folds and a continuous interacting stream where baselines do not.Figure 6 contrasts these outcomes with bare-sphere or rigid cloth and sparse disconnected liquid particles.Source: Figure 6 · p. 28

E.6 Case Studies

  • In the cloth case, the Error Corrector corrected elasticity after unbounded stretching, and the stability check then passed with bounded strain.In the rain-droplet case, refinement produced ripple propagation satisfying the assertion of at least three ripples in 0.3 seconds.Source: §E.6 · p. 28

E.8 Contamination Audit Details

  • The contamination audit reports no near-duplicate instruction pairs above 0.85 similarity and a 1.8-point hard-subset gap between held-out-seed and expert-authored instructions.It also reports non-overlapping domain bootstrap confidence intervals between SMRF and baselines and a total-score standard deviation of 1.4 across seed-grouped cross-validation.Source: §E.8 · p. 29

F Limitations

The paper’s scope and evaluation are bounded by the covered physics domains, test-split metric coverage, and partial cross-engine evaluation; inference cost and a fixed generation budget also constrain how results should be read.

F Limitations

  • The benchmark targets high-level engine APIs, excludes several physics domains, and does not measure whether models can implement stable solvers or numerical methods.Its checks for dissipative scenarios are necessary rather than sufficient, and the Error Corrector only activates on execution failure.Source: F Limitations · p. 29
  • Cross-engine evaluation covers MuJoCo for 120 of 200 test cases, with approximate fluid coverage.The dissipative-regime checks are scenario-specific and are not summarized as a standalone rate.Source: F Limitations · p. 29
  • Physical assertions cover the 200 test examples, while aggregate conservation residuals cover only 102 rigid-body and mechanics cases.Extending assertion coverage to the training split remains the main outstanding curation cost.Source: F Limitations · p. 29
  • A fixed 4,096-token budget binds hardest on open models using long reasoning chains and on SMRF (base), so those scores are not their ceilings.The authors state that the two headline comparisons are unaffected: Claude-3.5-Sonnet fits the budget, and the strongest single agent and SMRF agents use it similarly.Source: F Limitations · pp. 29–30
  • SMRF triples inference cost relative to a single agent, which matters for interactive use.Source: F Limitations · p. 30

G LLM Usage Statement

  • LLMs helped expand prompts and draft implementations, served as some evaluation baselines, and were used for language polishing only in writing.Retained examples passed two-stage validation and expert rewriting, and 80 test instructions were authored without LLMs.Source: G LLM Usage Statement · p. 30

Related papers

Loading 2604.23580v2…