SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation

Motivation

• Dynamic scenes require coordinating spatial layout and multiple mechanisms, including rigid-body, cloth, fluid, particle, force-field, deformer, and keyframed systems, within a shot.The target is an editable Blender artifact with geometry, materials, lighting, cameras, animation, and physics explicit and controllable.Source: §1 · p. 2
• Rendered videos can conceal whether motion comes from a physics solver or visually identical keyframes, so dynamic-scene generation requires mechanism correctness as well as visual plausibility.A solver-based tablecloth remains editable, composable, and physically meaningful as the scene changes, unlike hand-authored alternatives.Source: §1 · p. 2
• Image critique can miss whether a fluid setup has configured flow and effector objects and a baked cache, or whether geometry is merely animated.As objects, interactions, and temporal phases multiply, unchecked failures can leave a visually plausible scene unusable as a 4D asset.Source: §1 · p. 2

Contributions

• SimWorlds turns prompts into editable Blender projects through a planner-coder-reviewer workflow, ordered construction stages, a layered scene protocol, deterministic verification, and runtime-state inspection.A Blender-specific knowledge base supplies procedural detail on demand; local retries limit early errors from contaminating later behavior.Source: §1 · p. 2

Evaluation

• SimWorlds improves scene-level correctness and physical integrity over visual-only baselines on both text-only generation and multi-step Blender editing, with the gap widening on complex inputs.The complex cases involve interacting objects, long temporal structure, nontrivial solver configuration, or edits spanning several objects or physics changes.Source: §1 · p. 2

Contributions

• 4DBuildBench assesses 50 scenes across five solver categories and three difficulty levels, with a static category, using engine-state audits for mechanism correctness and an itemized VLM judge for visual delivery.The five solver categories are cloth, fluid, rigid body, particle, and soft body.Source: §1 · p. 2

Code-Driven and Procedural 3D Scene Generation

• Code-driven Blender agents, procedural scene generators, and layout methods target editable files but focus on static objects or single-room layouts, not end-to-end dynamic scenes from text.Examples include SceneCraft [4], 3D-GPT [5], LL3M [6], Infinigen [10], and ProcTHOR [11] [12] [13].Source: §2 · p. 3

Render-Inspect Loops and Editing Benchmarks

• VIGA [8] uses reference-image-conditioned render-inspect iteration and reports 4D results qualitatively, while BlenderBench [8] provides 27 editing tasks; SimWorlds adds engine-state signals and text-only 4D generation.BlenderGym [22] identifies scenes that match goal pixels through the wrong mechanism as failures uncaught by its photometric and CLIP-based metrics.Source: §2 · p. 3

Neural Text-to-3D and Text-to-4D

• Neural text-to-3D and text-to-4D methods produce learned representations rather than Blender projects that can be opened, resimulated, or composed with downstream tools.Kubric [61] renders programmatic Blender physics, but is configured by code rather than natural language.Source: §2 · p. 3

LLM Agents and Long-Horizon Execution

• SimWorlds applies long-horizon agent principles through stage-scoped context, runtime-state tools, and plans whose physics commitments are checked against the engine after each step.The paper frames these as an instantiation for Blender 4D generation, rather than a contribution of the general principles themselves.Source: §2 · p. 3

3 Method

• The pipeline converts a prompt into a scene plan, then runs each construction stage through a coder, deterministic verifier, and reviewer, with bounded local retries before rendering a frame sequence.The planner’s global plan specifies objects, layout, physical roles, motion phases, and rendering intent.Source: §3 · p. 4, Figure 2 · p. 4

3.1 Staged Construction Pipeline

• Construction follows a fixed dependency-aware order—modeling, UV, texture, deformation, motion, camera, light, and render—with stages optionally skipped when unnecessary.The global scene plan remains fixed while the planner emits a tactical plan at each stage entry.Source: §3.1 · p. 4
• At each stage, the coder extends the live scene; verification and review determine whether to advance, retry, replan, or abort, with retries and replans bounded.Hard verifier failures and blocking reviews cannot advance the stage; final review can trigger a rollback and one coder fix before re-rendering.Source: §3.1 · p. 5
• Edit mode loads an existing .blend file, uses its scene state and optional reference image as planner context, and decomposes instructions into edit-only additions, modifications, or removals.It avoids rebuilding static elements that already exist correctly.Source: §3.1 · p. 5

3.2 Scene Protocol and Verifier

• The protocol adds a three-level hierarchy: systems group related objects, while each logical object gathers its meshes beneath a root Empty; declared contact and co-movement relations make assembly intent explicit.Collections are tagged as system or object groupings, while ordinary untagged collections remain outside the protocol.Source: §3.2 · p. 6
• A bipartite plan graph records objects and typed spatial relations, and deterministic rules check structure, parent chains, declared contacts, interpenetrations, and realized geometry against the plan.Checks also cover motion timing, including whether a settle phase comes to rest in its final frames.Source: §3.2 · p. 6

3.3 Tool Suite

• Runtime tools expose live collections, modifier stacks, physics caches, and animation channels alongside multi-angle previews and per-actor motion strips with phase boundaries and bake state.These readouts provide evidence about what was built that rendered frames alone cannot supply.Source: §3.3 · p. 6

3.4 Knowledge

• The blender_docs tool retrieves Blender-version-specific schemas, operator signatures, API guides, and manual content from a knowledge base derived from upstream sources.The knowledge base regenerates when the Blender version changes.Source: §3.4 · p. 6

4.1 Setup

• 4DBuildBench covers 50 prompts: 45 dynamic scenes across five solvers at three mechanism-complexity levels, plus five static scenes.Dynamic difficulty progresses from a single solver actor to within-category complexity and then cross-category interaction.Source: §4.1 · p. 7
• Evaluation pairs a runtime-state audit of mechanism and structure with itemized VLM scoring of prompt-visible content and visual quality.The audit reads solver state, caches, collision partners, constraints, force fields, and keyframe density; the VLM judge scores objects, relations, actions, quality, and aesthetics from five sampled frames.Source: §4.1 · p. 7
• VIGA is the only prior published from-scratch Blender 4D system directly runnable on the benchmark, enabling comparison of SimWorlds’ added components.Both systems use a dual-agent shape, so the comparison isolates the staged pipeline, scene protocol and verifier, and engine-level tools.Source: §4.1 · p. 7

4.2 Results

• MPR is 0.87 vs 0.67 for SimWorlds and VIGA, while VLM is 0.82 vs 0.78; SPR is +0.19 but its cross-system fairness is limited.SPR is treated as supporting evidence because it scores spatial structure against object groupings SimWorlds emits and baselines do not.Source: Table 1 · p. 8, §4.2 · p. 8
• Across four dynamic sequences, SimWorlds realizes the correct mechanism while VIGA fails it.The examples include a domino cascade, paper airplane landing, wind-blown curtain, and pouring hourglass.Source: §4.2 · p. 8, Figure 3 · p. 8
• On BlenderBench, SimWorlds improves over VIGA across all difficulty levels using staged decomposition and planning-coding-reviewing.The comparison uses a matched six-iteration budget and PL, N-CLIP, and VLM metrics.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Ablations

• On the 15-scene subset, verifier removal most hurts SPR, staging removal most hurts VLM, and MPR remains robust to both.Two multi-object D3 scenes drop to zero SPR without the verifier; a single construction pass loses the per-stage review that helps maintain visual polish.Source: §4.3 · p. 9, Table 3 · p. 9

• Prompt-following, sensible arrangement, and composition remain LLM/VLM judgments rather than deterministic checks, making these perceptual assessments less reliable.Source: §5 · p. 9

• SimWorlds emits editable .blend files whose dynamics use physics solvers rather than hand-authored animation, alongside 4DBuildBench for visual quality and mechanism correctness.Source: §6 · p. 9
• A fixed staged pipeline verifies each scene stage and repairs errors locally, while a lightweight scene protocol keeps assets organised and verifiable.Source: §6 · p. 9
• The VLM reviewer handles perceptual judgments such as composition and prompt alignment that deterministic checks cannot make.Source: §6 · p. 9
• The intended output is an editable, physics-driven asset that others can pick up, perturb, and build on.Source: §6 · p. 9

• Claude Opus 4.7 serves all three agent roles; evaluation uses a deterministic engine-state audit and separate VLM judges for scene generation and BlenderBench editing.Each judge scores each scene once at the provider’s default temperature, without multi-seed averaging; end-to-end runtime is approximately 70–120 minutes per scene on an Apple M4 with 16 GB unified memory.Source: A Implementation Details · p. 15
• The planner writes strategic and per-stage tactical JSON plans, while object properties declare deformation_kind and motion_kind for verifier checks.The planner does not specify bpy implementation details, which are left to the coder.Source: A Implementation Details · p. 15
• Each stage loops through coder, deterministic verifier, and reviewer, with up to 10 coder retries and 5 tactical replans before aborting.Closed stages save checkpoints, enabling rollback after a later-stage coder failure.Source: A Implementation Details · p. 15
• The planner persists across the run, while coder and per-stage reviewer sessions are stage-scoped and a final reviewer runs once on rendered output.Role prompts are composed at runtime with shared scene-protocol documentation.Source: A Implementation Details · p. 15
• Blender runs as a long-lived process behind a local MCP server; inspect and preview tools read scene state without re-rendering.Each closed stage checkpoints the live .blend, allowing rollback to the prior stage after failure.Source: A Implementation Details · p. 15
• The coder queries a version-specific Blender reference generated from class hierarchies, property signatures, enum values, and curated pages, with raw JSON fallback.The passage identifies this as the only knowledge source and says no hand-curated reference material is shipped.Source: A Implementation Details · p. 15
• The implementation lists Blender 5.1, Claude Opus 4.7, GPT-5.5, and BlenderBench among its third-party assets, with their respective licenses or terms stated.Source: A Implementation Details · p. 15
• Table 4 groups the exposed tools by category and highlights engine-state inspection tools that provide mechanism-level evidence unavailable from rendered images.Source: Table 4 · p. 16

• The in-loop verifier runs after every stage and is separate from the external benchmark audit, which scores outputs from any method.Source: B Scene Protocol and Verifier Ruleset · p. 16
• Objects use a three-level collection tree, and custom contact declarations are accepted only when the corresponding surfaces actually meet.Each object collection holds one logical object’s meshes and exactly one Empty root; contact relations are expressed through custom properties and protocol-graph relations.Source: B Scene Protocol and Verifier Ruleset · p. 16
• Rules are grouped into structural, geometric, soft, and plan-versus-state families; any hard-rule failure sets ok=false, while warnings do not.Source: B Scene Protocol and Verifier Ruleset · p. 16
• Structural rules enforce collection placement, naming, ground presence, and mesh-to-root parent relationships, with failures treated as hard fails.Source: B Scene Protocol and Verifier Ruleset · p. 16
• Geometric rules hard-fail when declared contacts exceed the distance tolerance or when overlapping meshes penetrate beyond numerical noise without a declaration.Source: B Scene Protocol and Verifier Ruleset · pp. 16–17
• Soft rules require grounded object collections, connected meshes unless exempted, no scene elements inside object collections, a camera, and lighting or an emissive world.Source: B Scene Protocol and Verifier Ruleset · p. 17
• Plan-versus-state rules compare protocol-graph relations with realized geometry and check motion against planned phases, including near-zero velocity at the end of a settle phase.Source: B Scene Protocol and Verifier Ruleset · p. 17
• Separate contact and penetration checks catch both declared contacts that fail geometrically and undeclared intersections, preventing well-organised but broken scenes from passing.Source: B Scene Protocol and Verifier Ruleset · p. 17

• The engine-state audit measures mechanism correctness from Blender runtime state, while the itemized VLM judge measures whether prompt content is visually delivered.Source: C Evaluation Protocol: Engine-State… · p. 17
• The audit does not inspect pixels, and the judge does not inspect modifiers.Source: C Evaluation Protocol: Engine-State… · p. 17
• A visually convincing scene built with the wrong mechanism can score high on the visual track and low on the audit track rather than passing both.Source: C Evaluation Protocol: Engine-State… · p. 17

Engine-state inspection

• The system-agnostic audit inspects Blender runtime data, including modifiers, physics caches, rigid-body settings, constraints, force fields, and location/rotation keyframe density.It scores any method’s .blend file without relying on the SimWorlds-specific scene protocol.Source: §C.1 · p. 17

Audit specification

• Hand-authored YAML specifies expected actors, mechanism predicates, spatial relations, and scene-level anti-cheat assertions for each scene.Actors are resolved by name and matching hints such as size, topology, and collection role.Source: §C.1 · p. 17

Predicate library

• The 42-check predicate library covers universal, cloth, fluid, rigid-body, particle, and soft-body requirements, including solver presence, configuration, and baked caches.Anti-cheat checks reject dense keyframes on solver actors and shape keys used as cloth or soft-body motion sources.Source: §C.1 · pp. 17–18

Reported metrics

• MPR averages per-actor fractions of mechanism predicates passed, while SPR measures the per-scene fraction of declared spatial relations that hold.SPR checks nearest-surface distance on evaluated meshes at start, middle, and end frames.Source: §C.1 · p. 18

SPR cross-system fairness

• Geometric assembly inference raises VIGA’s macro SPR from 0.62 to 0.70 while leaving SimWorlds unchanged and MPR unaffected; the authors still treat the +0.19 SPR gap as supporting evidence.The adjustment addresses ungrouped baseline meshes; structural scoring across grouping conventions remains difficult to make fully fair.Source: §C.1 · p. 18

Judge protocol

• The judge receives the prompt and five uniformly sampled video frames, or a single still for a static scene, and returns binary verdicts on atomic prompt items.It does not see the planner’s intent or the runtime audit.Source: §C.2 · p. 18

Judged dimensions

• Its five dimensions are object presence, spatial relations, visible actions, technical visual quality, and aesthetics, with action checks comparing the first and last frames.The actions dimension is excluded for static scenes.Source: §C.2 · pp. 18–19

Scoring and scope

• The reported VLM score averages item-positive fractions over dimensions exercised by a scene, while mechanism realism is excluded from the judge and covered by the audit.The passage states that vision models are unreliable at distinguishing real simulation from keyframed motion [82].Source: §C.2 · p. 19

D Benchmark Details

• 4DBuildBench contains 50 scenes: 45 dynamic scenes across five solver categories and three difficulty levels, plus five static scenes.Dynamic prompts span cloth, fluid, rigid body, particle, and soft body; static scenes test object inventory, materials, and layout without physics state.Source: D Benchmark Details · p. 19
• Difficulty progresses from a single solver actor, to within-category complexity, to cross-category interactions requiring two or more solvers.The levels are defined by mechanism complexity rather than clip length or raw object count.Source: D Benchmark Details · p. 19
• Benchmark difficulty is fixed by hand-authored actor predicates and interactions, and VIGA receives the same text-only prompts before both systems’ scenes undergo the same external audit and VLM judge.VIGA uses its native 15-round budget, while SimWorlds uses per-stage bounded retries; the budgets are reported as configured rather than forced equal.Source: D Benchmark Details · pp. 19–20

E BlenderBench Setup

• BlenderBench has 27 editing tasks split evenly among camera adjustment, multi-step attribute editing, and compositional editing that combines attribute and camera changes.Each task supplies a start .blend, goal render, and natural-language description; the reference edit script is not exposed to the agent.Source: E BlenderBench Setup · p. 20
• BlenderBench reports photometric loss, N-CLIP distance, and GPT-4o VLM scores against the reference render using VIGA’s metric definitions.The VLM judge scores task completion, visual quality, spatial accuracy, and detail accuracy.Source: E BlenderBench Setup · p. 20
• Both methods’ final renders are rescored end-to-end with identical metric code for the Table 2 head-to-head means.Per-level comparisons are reported in the main-text table.Source: E BlenderBench Setup · p. 20