4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

1. Introduction

• Agents receive a reference video and write code to generate 3D geometry over time, with no prescribed implementation for dynamics.The benchmark makes agents’ choices of representation and implementation part of the investigation.Source: §1 · p. 2
• 4DCODEBENCH comprises 200 dynamic scenes—100 real-world videos and 100 synthetic scenes—and benchmarks visual fidelity, geometry, and dynamics.Synthetic scenes include 4D ground truth; agents must produce executable graphics programs that expose geometry and dynamics over time.Source: §1 · pp. 2–3
• Across 18 frontier models, reconstruction of dynamics consistently lags behind appearance and static geometry, even when static reconstruction is strong.Leading models also differ substantially in their reconstruction strategies.Source: §1 · pp. 2–3
• The benchmark pairs its evaluations with a human preference study and automated VLM- and reconstruction-based metrics that closely track human judgments.These measurements support analysis of which scene properties and physical regimes expose different reconstruction failures.Source: §1 · p. 3

2. 4DCODEBENCH

• The 200-scene benchmark combines realistic real-world videos, which lack 4D ground truth, with simulated synthetic scenes that provide per-frame geometry and motion ground truth.The two sources offer complementary roles: realistic appearance and complex dynamics versus direct evaluation in 3D and 4D space.Source: §2 · p. 3

2.1. Task Definition: 4D Inverse Graphics Through Code Generation

• For each video, an agent writes executable 4D code that constructs 3D geometry and its evolution, exposing geometric states for rendering at evaluated timesteps.The code organization and dynamics implementation are unrestricted; the implementation is simulator-agnostic.Source: §2.1 · pp. 3–4

2.2. Benchmark Overview and Construction

• The benchmark combines curated real footage and author-designed simulated scenes across rigid, deformable, co-dimensional, and flowing materials and interactions.Real clips prioritize continuous footage, stationary cameras, and limited occlusion; synthetic scenes vary geometry, materials, initial conditions, and rendering settings.Source: §2.2 · p. 4

2.3. Dataset Statistics

• Scenes span four matter classes; 83% contain multiple dynamic objects and 66% contain multiple materials, with passive and externally driven motion represented.Figure 3 depicts matter-family examples and scene-property distributions across the real and synthetic splits.Source: §2.3 · p. 5, Figure 3 · p. 5

2.4. Metrics and Evaluation: What Is a Good Reconstruction?

• Evaluation averages five metric families, using video-reference measures for perceptual fidelity, 2D dynamics, and 2.5D geometry alongside 3D geometry and dynamics against world ground truth.Failed runs receive the worst score on affected metrics.Source: §2.4 · pp. 5–6
• Video-reference measures assess perceptual and semantic fidelity and moving-object coverage; on real videos, they also compare estimated flow, trajectories, and depth.Reconstruction-side quantities are computed analytically from the 4D world rather than estimated from rendered pixels.Source: §2.4 · p. 6
• VLM evaluation uses scene-specific questions and pairwise comparisons of reference videos with two reconstructions, aggregating preferences into model-level Elo ratings.VQA questions cover initial state, final state, key events, and contact.Source: §2.4 · p. 6
• Synthetic-scene evaluation compares aligned geometry and motion to 4D ground truth, while mesh-validity checks are reported separately from the Overall score.Motion metrics capture both persistent 3D paths and frame-to-frame displacement distributions.Source: §2.4 · p. 6

3.1. Experimental Setup

• The study evaluates 18 multimodal coding models once on each of the 200 scenes, using both proprietary and open-weight models.Multiple reasoning-effort settings are tested for Astra.Source: §3.1 · p. 6
• Agents receive only the reference RGB video and task prompt, then must return executable code and a reconstructed scene that can be regenerated without the reference.The environment supports Blender, numerical and simulation libraries, and custom code; dynamics implementation is unrestricted.Source: §3.1 · pp. 6–7

3.2. Human Evaluation

• The human study collected 3,587 pairwise judgments from 76 participants across 17 models, with model preferences aggregated into Elo ratings.Trials showed the reference and two reconstructions; 209 comparisons had multiple participants and 916 were also judged by the VLM.Source: §3.2 · p. 7

4.1. Overall Performance and Human Alignment

• Astra [Max] leads the 18-model Overall ranking, followed by Opus 5.5 [High], Astra [High], Fable [High], and Astra [Low]; open-weight models generally trail proprietary ones.90.1% of submissions are fully executable.Source: §4.1 · p. 7, Figure 4 · p. 8
• Human and VLM Elo correlate strongly across models (Spearman ρ=0.980), and their judgments agree on 89.3% of 916 shared comparisons, near the 92.1% human-annotator agreement.Human Elo also correlates with Overall score (ρ=0.96); individual VLM judgments are less reliable for closely matched models.Source: §4.1 · pp. 7–8

4.2. Performance Across Metric Families

• VQA accuracy reaches 87.6% for Astra [Max], 78.8% for Fable [High], and 29.5% for GLM [Max], while first-frame questions are generally easier than final-frame and key-event questions.Astra [Max] answers every question on 59% of scenes, whereas Mistral answers none on 94%.Source: §4.1 · p. 8, §4.2 · p. 9, Figure 5 · p. 9
• Even Astra [Max] scores 0.91 on static families versus 0.67 on dynamic families, showing that scene evolution is harder to reconstruct than appearance and geometry.The gap persists when appearance is compared with 2D motion and 3D geometry with 3D motion separately.Source: §4.2 · p. 8, Figure 8 · p. 10
• Five additional reconstruction metrics broadly agree with the leaderboard, but mesh-quality checks can reward simple, inaccurate geometry and are reported separately as structural-defect diagnostics.Metrics rank models similarly overall but agree less when ranking reconstructions of the same scene.Source: §4.2 · pp. 8–9

4.3. Performance Across Scene Types

• Real and multi-matter scenes score lower on VQA and Perceptual measures; co-dimensional scenes score lower on 2D Dynamics and 2.5D Geometry, while driven scenes show lower VQA but higher 3D Geometry.Flowing scenes also show lower 3D Dynamics (−0.21), but that effect does not meet the figure’s consistency criterion.Source: §4.3 · p. 10, Figure 7 · p. 10

4.4. How Do Agents Model Dynamics?

• Solutions use analytic motion most often (67%), followed by custom simulation (19%), Blender physics (10%), and keyframing (3%), with substantial variation across models.Opus 5.5 [High] uses custom simulation in 61% of solutions, while Astra [Low] uses analytic motion in 85% and Terra [High] in 100%.Source: §4.4 · p. 10

4.5. Effect of Inference-Time Reasoning

• Raising Astra’s reasoning from Low to Max increases Overall from 0.73 to 0.79, 2D Dynamics from 0.48 to 0.60, 3D Dynamics from 0.63 to 0.73, and VQA from 78.2% to 87.6%.These gains accompany greater output-token use and cost.Source: §4.5 · pp. 10–11

5. Related Work

• 3DCodeBench tests 3D object modeling in code from text or images, while PhysCodeBench tests executable physics simulations from text descriptions.These benchmarks address spatial and physical coding tasks.Source: §5 · p. 11
• VisPhyWorld reconstructs synthetic physics videos as simulator code, and MPMWorlds covers synthetic 2D videos with deformation, material changes, and topology-changing events.BVB instead reconstructs primarily static indoor scenes observed by a moving camera.Source: §5 · p. 11
• 4DCODEBENCH uses videos with minimal camera motion to test reconstruction of scene-geometry changes and physical motion, with direct 3D-ground-truth evaluation on synthetic scenes.This contrasts with reconstruction evaluated solely by rendered frames in image space.Source: §5 · p. 11

6. Conclusion

• The evaluation measures reconstruction fidelity but leaves physical mechanisms underconstrained; prescribed trajectories can score highly, while simulation’s generalization benefits remain largely untested.The authors propose interventions on initial conditions and external forces, plus longer-horizon prediction, as future tests of transferable physical understanding.Source: §6 · p. 11
• Leading models use markedly different strategies, and greater token expenditure does not consistently yield better reconstructions.The discussion raises whether simulation improves reconstruction quality and whether requiring physics-based simulation preserves relative model performance.Source: §6 · p. 11
• Strongest models perform well on appearance and geometry, but a clear gap remains in their ability to abstract dynamics.Automated rankings closely agree with human preferences across the models studied.Source: §6 · p. 11
• 4DCODEBENCH offers a way to track progress in 4D inverse graphics and identify where models still fail across diverse interactions.The discussion points to physical intuition and targeted feedback as possible ways to form plausible hypotheses and refine solutions with fewer render-and-compare iterations.Source: §6 · p. 11

• The agent builds and renders a 4D scene program from the input video, compares its render with the video, and revises the program in a loop.Source: Figure A1 · p. 18

B. Additional Related Work

• Physical-reasoning research has used explicit object-centric states and simulation, while many contemporary benchmarks instead test indirect outcomes, plausibility, or question answering.Formulating scene understanding as world-program generation requires an executable 4D scene that can be simulated, inspected, and verified.Source: B. Additional Related Work · p. 18
• Classical visual program induction recovered scenes as programs but was largely limited to hand-crafted domain-specific languages; recent coding agents produce general-purpose graphics code for static imagery.Dynamic reconstruction adds the need to represent how physical scenes evolve over time.Source: B. Additional Related Work · pp. 18–19
• Existing video-conditioned benchmarks cover narrower regimes, including controlled simulations, 2D scenes, rigid or articulated systems, and static indoor scenes; Code as Worlds fits trajectories kinematically.The section characterizes these benchmarks as not inferring the underlying physics in Code as Worlds (Wang et al., 2026).Source: B. Additional Related Work · p. 19
• 4DCODEBENCH spans real-world rigid, deformable, co-dimensional, and flowing matter, and evaluates reconstructed 3D geometry and motion against synthetic-scene ground truth.Source: B. Additional Related Work · p. 19

C. Dataset

• Figure A2 reports scene counts by material category and distributions of clip length and frame rate.Source: C. Dataset · p. 19

C.1. Dataset Samples / C.2. Real-World Dataset Composition and Sources

• The dataset appendix shows sample scenes and material, clip-length, and frame-rate distributions; its 100 real-world clips come from physics benchmarks, robot-manipulation datasets, and web footage.Figures A3–A6 show 40 scenes; real-world source counts are 51, 21, and 28 clips, respectively.Source: §C.1 · p. 19, Figure A2 · p. 20, §C.2 · p. 20

C.3. Synthetic-Dataset Simulators

• Synthetic scenes use simulation methods suited to different materials and interactions, including IPC, ABD, SPH, MPM variants, and the PPF Contact Solver.The authors extend Genesis with viscoplastic, snow, and Drucker–Prager models.Source: §C.3 · p. 25, Table A3 · p. 25

D.1. Executability and Scoring

• Submissions must produce a valid video and, for geometry metrics, a valid 4D world; failed outputs receive worst metric values, and 90.1% of submissions are executable.Every proprietary model reaches at least 96.5% executability; open-weight models range from 29.0% to 95.5%.Source: §D.1 · p. 26, Figure A7 · p. 26

D.2. Metrics on All Scenes / D.3. Metrics on Real Videos / D.4. Metrics on Synthetic Scenes

• Evaluation combines video-based appearance and dynamic-object coverage with real-video depth, flow, and trajectory measures, plus synthetic-scene 3D surface and motion measures.For real videos, reconstruction quantities are analytically rasterized from the predicted 4D world; synthetic scenes provide ground-truth geometry for direct 3D evaluation.Source: §D.2 · p. 27, §D.3 · p. 27

D.5. VQA

• VQA asks scene-specific questions about initial and final states, key events, and contact; gated and missing-video scenes count as wrong, leaving all models with the same 1,251 questions.Across models, 81.0% of scenes reach the judge cleanly; 3.6% produce no video, 8.5% fail the first check, and 7.0% fail the presence question.Source: §D.5 · p. 29

D.6. VLM-Based Pairwise Rankings / D.7. Metric Concordance

• Sampled pairwise VLM preferences are aggregated with Bradley–Terry ratings; metric rankings correlate positively (ρ=0.39–0.76), while Trajectory DTW is the most independent metric.Trajectory DTW uniquely tests whether the correct matter follows the correct 3D path.Source: §D.6 · p. 29, §D.7 · p. 31

D.8. Additional Metrics and Geometry Quality

• Five additional reconstruction metrics broadly corroborate the primary rankings, including alternative semantic similarity, trajectory regularity, 3D embedding similarity, and occupancy sequence matching.The accompanying table scores failed runs at the penalty value; higher scores are better.Source: §D.8 · p. 31, Figure A10 · p. 32
• Geometry checks score watertightness, manifoldness, non-degeneracy, self-intersection freedom, and interpenetration only for submissions that deliver a valid 4D world.The figure notes that weaker models can score higher on these checks despite lower reconstruction quality.Source: §D.8 · p. 32, Figure A10 · p. 32
• Mesh quality does not correlate positively with reconstruction fidelity: Muse Glimmer [High] leads or ties on all four integrity checks, while Astra [Max] achieves 0.830 watertightness.Stronger models attempt intricate geometry that is challenging to mesh; weaker models often use simple primitives that pass structural checks more easily.Source: §D.8 · pp. 32–33

E. Human Study

• Participants compare two randomly ordered reconstructions against a reference on geometry, layout, and motion; adaptive sampling targets uncertain model pairs, and Bradley–Terry ratings correct for nonuniform sampling.Participants gave the same answer on 95.5% of repeated comparisons, and human Elo ratings correlate with the aggregate automated ranking at Spearman’s ρ=0.96.Source: E. Human Study · p. 33

F.1. Sample Agent Results

• Figures A13–A15 show results from six models on six benchmark tasks, while A16–A17 summarize scene-level VQA outcomes and score distributions.A16 divides scenes into none, some, or all questions answered correctly; A17 shows score densities and means by model type and scene source.Source: Figure A13 · p. 34, Figure A14 · p. 35, Figure A15 · p. 36

F.2. VQA Results

• Figure A18 reports category-wise accuracy ranges and medians, A19 expands family scores into metrics, and A20 splits the static–dynamic gap by space.The gap panels compare Perceptual with 2D Dynamics and 3D Geometry with 3D Dynamics.Source: Figure A18 · p. 37, §F.2 · p. 37, Figure A19 · p. 38
• Among VQA results, Astra [Max] answers every question on 59% of its scenes, while Mistral answers none on 94%.Figure A18 breaks accuracy down by question category; Figure A19 expands family scores into individual metrics, and Figure A20 separates the static–dynamic gap by space.Source: §F.2 · p. 37

F.3. Comparison Between Real and Synthetic Splits

• GPT-6 Astra [Max] scores 0.85 VQA on real scenes versus 0.90 on synthetic scenes, while Claude Opus 5.5 [High] drops from 0.88 to 0.78.Most models score lower on real scenes, using 17% fewer tokens there on average; GPT-6 Astra [Max] shows the strongest real-world generalization.Source: §F.3 · pp. 38–39, Figure A21 · p. 39

F.4. How Do Models Describe Motion?

• 318 custom simulations identify established schemes; analytic solutions include pose splines (539), vertex displacement (312), and ballistic formulas (256).The table covers solutions from all 18 models, with some rows not summing to 100% because a few use other approaches.Source: §F.4 · pp. 39–40, Table A4 · p. 40
• For the bread tear, Fable 5.1 uses MLS-MPM, DeepSeek v4.1 Flash snaps XPBD ligaments, and Astra [Max] advances an analytic fracture front.For the dam break, every model uses Blender Mantaflow FLIP; nearly all use position-based rod simulations for pasta strands.Source: §F.4 · p. 40

G.1. Execution Environment

• Proprietary models run in their providers’ native command-line agents, while other models use Stirrup with models served via SGLang.Runs use isolated containers with the reference video and task materials mounted read-only; runs that stall or exceed six hours are terminated.Source: §G.1 · p. 41

G.2. Agent Prompt

• The prompt allows motion to be computed in Blender or external code, but requires Blender rendering and persistent material-point correspondence across frames.Rendered mesh topology may change over time, while each dynamics position column tracks the same piece of material.Source: §G.2 · p. 42

G.3. Submission Format

• Submissions must include a deterministic build script that regenerates the explicit world representation; among runs passing the video gate, 5.4% fail 4D world validation.The format is designed to be lightweight, typically requiring about 100 lines of standard Python export code.Source: §G.3 · p. 43

G.4. Running Statistics

• Open-weight models average 247 agent steps and 39.6M tokens per task, versus 101 steps and 11.8M tokens for proprietary models, yet cost less: $2.01 versus $6.65.Across 3,600 runs, total compute was 5,983 hours and $16,495; 97% of tokens were cache reads.Source: §G.4 · p. 44, Table A5 · p. 44
• Raw step counts and token volume have negligible correlation with Overall score (|ρ| ≤0.10), while cost correlates moderately (ρ=0.65).Qwen [XHigh] averages 752 steps and ranks eleventh of 18; increasing GPT-6 Astra reasoning from [Low] to [High] to [Max] steadily improves its Overall score from 0.726 to 0.791.Source: §G.4 · p. 44
• Figure A22 summarizes per-task cost, token, output-token, step, and peak-context distributions, plus mean context compactions with bootstrap 95% intervals.Antigravity CLI does not record peak context.Source: Figure A22 · p. 45