PaperlayerView exact revision on arXiv

Source-linked AI summary

Code2Worlds: Empowering Coding LLMs for 4D World Generation

Yi Zhang, Yunshuang Wang, Zeyu Zhang, Hao Tang

arXiv:2602.11757v1cs.CV

TL;DR

Code2Worlds targets the gap between static code-generated scenes and physically grounded 4D simulation, where monolithic generation entangles scales and open-loop scripting can hallucinate dynamics. It uses factorized object and environmental generation with PostProcess Agent actuation and VLM-Motion feedback, and reports stronger Code4D performance with physics-aware dynamics. Its physics engines and iterative VLM feedback impose computational overhead and hinder real-time generation.

  • Problem

    Static code generation struggles with multi-scale scene structure and open-loop dynamics that can violate physical laws, limiting physically grounded 4D simulation.

  • Method

    Code2Worlds factorizes generation into retrieval-augmented object modeling and hierarchical environmental orchestration, followed by PostProcess Agent actuation with VLM-based closed-loop refinement.

  • Results

    Code2Worlds outperforms baselines on Code4D, with a 41% SGS improvement and 49% Richness increase, while uniquely generating physics-aware dynamics absent in prior static methods.

  • Takeaways & Limitations

    The framework bridges static code generation and 4D physical simulation by combining structural fidelity with dynamic consistency.

  • Takeaways & Limitations

    Rigorous physics and iterative VLM feedback create a computational bottleneck that hinders real-time generation.

Abstract

from arXiv · show

Achieving spatial intelligence requires moving beyond visual plausibility to build world simulators grounded in physical laws. While coding LLMs have advanced static 3D scene generation, extending this paradigm to 4D dynamics remains a critical frontier. This task presents two fundamental challenges: multi-scale context entanglement, where monolithic generation fails to balance local object structures with global environmental layouts; and a semantic-physical execution gap, where open-loop code generation leads to physical hallucinations lacking dynamic fidelity. We introduce Code2Worlds, a framework that formulates 4D generation as language-to-simulation code generation. First, we propose a dual-stream architecture that disentangles retrieval-augmented object generation from hierarchical environmental orchestration. Second, to ensure dynamic fidelity, we establish a physics-aware closed-loop mechanism in which a PostProcess Agent scripts dynamics, coupled with a VLM-Motion Critic that performs self-reflection to iteratively refine simulation code. Evaluations on the Code4D benchmark show Code2Worlds outperforms baselines with a 41% SGS gain and 49% higher Richness, while uniquely generating physics-aware dynamics absent in prior static methods. Code: https://github.com/AIGeeksGroup/Code2Worlds. Website: https://aigeeksgroup.github.io/Code2Worlds.

1. Introduction

Code2Worlds addresses the transition from static 3D generation to physically grounded 4D simulation by separating object and environment generation and adding iterative visual-physical refinement. On Code4D, it improves SGS and Richness while generating physics-aware dynamics unavailable in prior static methods.

  • Motivation: Static code-to-scene methods struggle to balance fine-grained object structures with globally coherent environmental layouts.This multi-scale entanglement can produce coarse objects unsuitable for fine-grained physical actuation.
  • Motivation: Open-loop code generation creates an execution gap between semantic motion descriptions and precise simulation parameters.The resulting physical hallucinations can include deforming rigid bodies or particles ignoring gravity.
  • Approach: Code2Worlds uses a dual-stream architecture that disentangles retrieval-augmented object generation from hierarchical environmental orchestration.The factorization is intended to preserve detailed target-object structure while coordinating the global environment.
  • Approach: A PostProcess Agent and VLM-Motion Critic form a closed-loop mechanism that evaluates rendered dynamics and iteratively corrects physical hallucinations.This replaces blind open-loop scripting with self-reflective refinement aligned to user intent.
  • Evaluation: 41% improvement in SGS and 49% increase in Richness are reported on the Code4D benchmark, alongside physics-aware dynamics absent in prior static methods.The benchmark evaluation describes Code2Worlds as consistently outperforming prior code-to-scene frameworks.

2. Related Work

Prior work spans text-driven procedural modeling, 4D content generation, and multi-agent reflection, but existing approaches remain limited in editability, dynamics, temporal consistency, or 4D physical feedback. Code2Worlds is positioned as a parallel multi-agent architecture for these 4D requirements.

  • 3D and 4D Content Generation: Text-to-4D methods such as MAV3D face high computational costs and limited editability.The related-work passage describes MAV3D as optimizing dynamic NeRFs with video diffusion priors.
  • LLM-driven Procedural Modeling: LLM-driven procedural modeling has enabled text-to-scene generation, planning, specialized agents, and retrieval-based reduction of syntactic errors.Despite these developments, programmatic modeling remains predominantly optimized for static 3D environments.
  • Code2Worlds Positioning: Code2Worlds combines object detail, environmental orchestration, and feedback-driven refinement within a 4D scene-generation pipeline.Its execution pipeline uses an Object Stream, Scene Stream, and PostProcess Agent with self-reflection.
  • Multi-Agent Coordination and Reflection: Multi-agent systems and closed-loop self-correction support complex task decomposition and iterative refinement in prior work.The cited examples primarily target static or symbolic domains rather than 4D physical discrepancies.

3. The Proposed Method

Code2Worlds generates 4D scenes through separate object and scene streams, then integrates them with physics-aware dynamic actuation and feedback-driven refinement. Retrieval and hierarchical planning translate sparse language into detailed procedural structures and coherent environments.

  • Architecture: The Scene Stream handles procedural environmental layout while the Object Stream generates detailed 3D objects.A PostProcess Agent later integrates both components and scripts temporal dynamics under VLM supervision.
  • Object Stream: Retrieval-Augmented Parametric Generation maps semantic instructions into procedural parameters and executable code for high-fidelity objects.The method uses procedural priors and retrieves structured parameter definitions with semantic exemplars.
  • Object Stream: Dynamic object selection identifies entities requiring specific interactions while deferring global environmental changes to unified post-processing.The selected target object maximizes dynamic necessity among entities parsed from the instruction.
  • Object Stream: Object self-reflection renders a snapshot, obtains VLM feedback against the instruction, and triggers parameter regeneration when alignment is insufficient.The feedback is propagated to the object-parameter agent to refine semantic alignment.
  • Scene Stream: Hierarchical environmental orchestration converts sparse instructions into structured scene specifications and concrete, logically consistent parameters.The process infers atmospheric context, terrain morphology, vegetation density, magnitudes, incompatibilities, and coupled variables.
  • Physics-Aware 4D Scene Generation: Dynamic scene integration maps instructions and feedback to physics parameters, actuates unified static geometry, and enforces collision constraints.Examples include translating “peacefully” into a wind coefficient and allowing branches to sway while anchoring roots.

4. Experiments

Code2Worlds is evaluated across object, scene, and dynamic generation using semantic, stylistic, temporal, and physics-related measures. It outperforms prior methods in structural fidelity, environmental complexity, temporal stability, and physics-aware dynamics, with ablations supporting retrieval and iterative reflection.

  • Benchmark and Metrics: Code4D evaluates object, scene, and dynamic generation against code-centric methods and text-to-video models.The protocol uses multidimensional metrics for semantic alignment, style, temporal consistency, and physical behavior.
  • Object Generation: 61.4 SGS establishes Code2Worlds as the strongest object-generation method across the reported metrics, exceeding ImmerseGen’s 43.5.The comparison covers procedural, reconstruction-based, and agent-centric baselines.
  • Scene Generation: 62.3 Richness surpasses ImmerseGen’s 35.5 and 3D-GPT’s 41.7, while S-CLIP reaches 0.2432 for scene semantic alignment.The results indicate dense environmental detail alongside semantic consistency.
  • Dynamic Generation: 55.4 HRS and a 10% physics Failure Rate demonstrate temporally dynamic scenes that remain textually aligned and respect physical laws.The framework includes environmental effects such as relighting, water spills, leaf fall, jellyfish movement, and fire.
  • Video Generation: 0.9952 Motion Smoothness and 0.9949 Temporal Flickering show stable video generation through deterministic 3D rendering rather than latent-space interpolation.AnimateDiff’s reported Failure Rates reach 70%, with texture boiling and physical artifacts during transitions.
  • Ablation Study: Removing retrieval reduces SGS to 23.5, while removing the VLM-Critic lowers O-CLIP from 0.2655 to 0.2388 and SGS from 61.4 to 58.6.Removing the VLM-Motion agent increases physics Failure Rate from 10% to 60% and reduces HRS to 47.0.

5. Conclusion

Code2Worlds bridges static code generation and 4D physical simulation through dual-stream generation and VLM-driven closed-loop refinement. On Code4D, it outperforms baselines in generating diverse, physics-aware environments.

  • Code2Worlds bridges static code generation and 4D physical simulation.
  • Its dual-stream architecture targets structural fidelity, while VLM-driven closed-loop refinement targets dynamic consistency.
  • Experiments on Code4D show significant improvement over baselines for diverse, physics-aware environment generation.

Impact Statements

Code2Worlds is presented as supporting safer sim-to-real transfer in embodied AI by enabling physically consistent 4D simulations. The work also notes computational overhead and potential LLM-related bias.

  • Code2Worlds is intended to facilitate safer sim-to-real transfer in embodied AI.
  • The framework enables creation of physically consistent 4D simulations.
  • Its integration of rigorous physics entails substantial computational overhead, while reliance on large language models introduces potential biases.

A. Limitation and Future Work

Code2Worlds faces a fidelity–latency trade-off because rigorous physics engines and iterative VLM feedback create a computational bottleneck. Future work proposes neural physics distillation to accelerate simulation.

  • Code2Worlds encounters a trade-off between fidelity and latency.
  • Rigorous physics engines and iterative VLM feedback cause a computational bottleneck that hinders real-time generation.
  • Future work will explore neural physics distillation to accelerate simulations through learned approximations.

B. Ablation Study

The ablation study evaluates scene-composition components by removing explicit parameter reasoning or global environmental orchestration. Removing the Planner and Solver produces the lowest reported S-CLIP score.

  • The ablation compares variants without Planner & Solver and without the Scene Stream.
  • Removing the Planner and Solver yields the lowest S-CLIP score, 0.2251.
  • The reported result confirms the critical role of the Planner and Solver in semantic alignment between abstract instructions and executable constraints.

C. Implementation Details

The implementation uses Gemini 3 as the framework’s reasoning engine and Blender 4.3 with bpy for asset creation and 4D simulation. Rendering uses Cycles for photorealism with scene-specific resolutions, frame counts, sampling, and denoising.

  • Gemini 3 serves as the core reasoning engine across the framework’s agents and processing stages.It is used for critics, object generation, environmental planning, parameter solving, scene realization, and post-processing.
  • Blender 4.3 and the bpy Python API execute all 3D asset generation and 4D simulations.
  • Cycles path tracing provides high-fidelity photorealistic rendering, with OpenImageDenoise applied for noise reduction.
  • Nature scenes render at 1920 × 1080 for 240 frames with 128 samples per frame, while indoor scenes use 120 frames with 196 samples per frame.

D. Benchmark Details

The Code4D benchmark is designed to evaluate physically grounded 4D generation beyond static scene structure, while the release plan supports reproducible comparison. The section also documents prompt examples and a procedural parameter library used for semantic-to-parameter mapping.

  • Code4D evaluates temporal evolution, physical interactions, and atmospheric changes rather than static 3D structure alone.The benchmark is organized across three dimensions for comprehensive 4D-generation assessment.
  • The authors plan to release Code4D prompts and evaluation scripts to support consistent benchmarking against future text-to-simulation models.
  • Table 6 provides examples of prompts used in the framework.
  • A Procedural Parameters Library maps linguistic categories to procedural values using analyses of high-quality Infinigen scripts.The library supports semantic descriptions being translated into procedural parameters during generation.

F. Additional Qualitative Results

Additional qualitative results present dynamic simulations across natural and indoor scenes, together with procedural asset parameters, generated code, prompts, and evaluator instructions. The examples cover environmental motion, relighting, weather, fluids, rigid-body motion, fire, and scene richness assessment.

  • The qualitative results show key frames from 10 dynamic scenes, illustrating scene evolution and physical interactions over time.Examples include wind, relighting, falling leaves, rain, desert sand, jellyfish motion, fire, spilling water, rolling bottles, and coffee cups.
  • Procedural assets: The examples include procedural parameter and code specifications for leaves, jellyfish, cups, and bowls.Leaf, jellyfish, and cup parameters control geometry, precision, materials, shape, scale, and thickness.
  • Dynamic scenarios: Prompt examples describe motions such as autumn trees swaying, leaves falling, desert sand cascading, and a burning tree casting shifting shadows.
  • Environmental planning: The Environment Planner infers latent season, weather, and lighting variables when expanding sparse scene instructions.It also enforces biome-consistent terrain and populates ecosystems with details such as bushes and rocks.
  • Evaluation prompts: The VFX evaluator scores generated videos on physics plausibility, including gravity, collision, inertia, clipping, and floating objects.A separate richness evaluator focuses on visual richness and object diversity without considering prompt matching.
Loading 2602.11757v1…