Source-linked AI summary
Code World Model: Coding Agent as World Brain
Yiwen Chen, Guosheng Lin, Chi Zhang
TL;DR
Video-based world models learn observable outcomes but not the hidden rules and mechanisms needed for persistent, coherent evolution. Code World Model assigns evolution to a coding agent and executable state, then uses a proxy to condition a video model for visual realization. Preliminary prototype results show proxy adherence in simple interactive worlds, while limited compute and coding-agent reliability constrain the current scope.
Problem
Video observations reveal outcomes rather than the knowledge, rules, relations, and consequence mechanisms governing world evolution, limiting persistent long-horizon modeling.
Method
A coding agent maintains executable world state and revisable code, while a deterministically compiled proxy video conditions a video model to render visual observations.
Results
The prototype provides preliminary evidence that coding agents can modify world state and mechanisms, while fine-tuned MiniMax-H3 follows proxy-specified visual structure in simple interactive worlds.
Takeaways & Limitations
The framework demonstrates a potential path toward combining persistent, rule-consistent world evolution with flexible high-fidelity visual realization.
Takeaways & Limitations
Experiments use small training scale with limited generation quality, omit autoregressive real-time generation, and do not demonstrate complete open-world construction.
Abstract
from arXiv · showhide
World models aim to simulate how complex environments evolve under actions and events, yet existing video-based world models primarily learn dynamics from visual observations, which reveal outcomes rather than the underlying knowledge, rules, and mechanisms governing world evolution. This makes it difficult to maintain persistent consequences and support coherent, open-ended evolution. We introduce Code World Model, a framework that separates world evolution from visual realization by combining the reasoning and coding capabilities of language models with the generative priors of video models. A coding agent serves as the world brain, reasoning about events and their consequences and generating executable code to maintain persistent world state and perform rule-consistent evolution. To connect executable state with visual generation, we introduce a proxy representation that encodes frame-wise spatiotemporal constraints and is compiled into a proxy video, which conditions a video model to render high-fidelity visual observations. We further develop data pipelines for constructing aligned proxy-observation pairs from gameplay and real-world videos. After fine-tuning on paired gameplay data, MiniMax-H3 follows proxy-based spatiotemporal specifications from simple interactive worlds built by the coding agent while preserving rich visual details and dynamics. These results demonstrate the potential of combining code for persistent world evolution with video models for flexible visual realization, providing a new path toward open-ended world models.
1. Introduction
Code World Model addresses the gap between visual outcomes and the hidden knowledge, rules, and mechanisms governing persistent world evolution. It uses a coding agent and executable state for rule-consistent evolution, a proxy interface for spatiotemporal control, and a video model for visual realization.
- Motivation: Video observations expose outcomes but not the world knowledge, rules, relations, and consequence mechanisms that generate them.This makes persistent, coherent evolution difficult when consequences extend across entities and long time horizons.
- Framework: The framework combines a coding agent as world brain with executable code that maintains state and governs long-horizon evolution.The agent handles complex decisions and code revision, while code performs frequent deterministic updates such as positions, attributes, and collisions.
- Proxy interface: A deterministic compiler converts executable world state into a proxy video that supplies frame-wise spatiotemporal constraints to the video model.The proxy specifies coarse spatial relations and trajectories, while learned priors provide appearance, local motion, and fine-grained interactions.
- Data pipeline: Aligned proxy–observation pairs are constructed from synchronized gameplay recordings and from real-world videos using 3D reconstructions and object annotations.The real-world pipeline does not require action labels, reducing annotation burden.
- Results: MiniMax-H3 qualitatively follows proxy-specified positions, motion, layout, and camera trajectories after LoRA adaptation on a small paired gameplay dataset.The result is reported for simple interactive worlds constructed by the coding agent.
- Results: The prototype provides preliminary evidence that coding agents can modify world state and mechanisms while video models render proxy-expressed states into visual observations.The framework unifies knowledge-driven, rule-consistent evolution with learned visual, motion, and interaction priors.
2. Related Work
Prior work improves interactive video generation through visual control, memory, streaming, and persistent representations, while generative 3D systems build explorable scenes. Code World Model instead uses persistent code and explicit state to govern evolution, with a proxy conditioning visual realization.
- Interactive video world models: Interactive video world models primarily carry state and interaction outcomes through generated observations, latent context, or visual memory.Recent systems improve control, long-horizon consistency, persistent memory, and responsive interaction around learned visual rollouts.
- Interactive video world models: Code World Model places a coding agent and persistent code in charge of explicit world state, including consequences that continue affecting later interaction.The video model receives relevant state through the proxy rather than internalizing the entire evolution process.
- Generative 3D worlds: Generative 3D world models use generated geometry, assets, and rendering pipelines as the visual world from which observations are produced.These systems target scene exploration, navigation, spatial editing, and free-viewpoint rendering.
- Generative 3D worlds: The Code World Model proxy is a coarse programmatic condition for camera motion, entity layout, trajectories, and interaction-relevant state, not a final 3D world.Simple 3D primitives may express the proxy, while the video model generates final appearance and fine-grained dynamics.
- Coding agents and world models: Prior program-based approaches generate or refine environment dynamics, transition models, or POMDP representations under language-model guidance.Code World Model differs by using persistent, revisable code to govern an open visual world rather than primarily producing a game artifact.
3. Method
Code World Model separates executable world evolution from visual realization: a coding agent maintains and revises persistent state through code, while a video model renders observations through text and proxy conditions. The method addresses long-horizon consequences, precise spatiotemporal control, and the trade-off between proxy expressiveness and agent-manageability.
- 3.1. Code World Model: The continuing agent-code loop lets events trigger state reading, code execution or local program changes, and feedback-driven revision while repetitive updates proceed without additional model calls.World state can advance event-driven or faster than visual generation, and code changes can affect future world operation.
- 3.1. Code World Model: Code World Model uses a coding agent as the world brain, governing world state, operating mechanisms, and subsequent evolution through executable, modifiable code.Code performs dense, reusable low-level updates while the agent makes higher-level decisions and revisions.
- 3. Method: The implementation pays higher marginal computation and latency because every new visual observation requires generative inference, a cost accepted for visual fidelity and open-ended generation.The paper expects more efficient video-model inference to make this trade-off increasingly favorable.
- 3.2. World-State Conditioning for the Video Model: A proxy organizes camera, entity positions, poses, trajectories, and spatial relations into a deterministic spatiotemporal visual condition, complementing text’s semantic intent.Both text and proxy originate from world state, but the proxy expresses evolving state as frame-wise constraints for the video model.
- 3.2. World-State Conditioning for the Video Model: Proxy condition bandwidth trades stronger grounding against construction difficulty: richer representations require the coding agent to maintain more primitives and parameters, while sparse proxies may underconstrain generation.Joint-level articulated motion is identified as difficult for current coding agents to implement reliably.
4. Experiments
The experiments implement proxy-conditioned video generation and connect coding-agent-built interactive worlds to visual inference. Qualitative results show close adherence to proxy-specified spatial and temporal constraints, with precise control over motion and cameras.
- Training: The system fine-tunes MiniMax-H3 Ref2VA on aligned proxy–RGB gameplay clips for proxy-conditioned video generation.Training uses 9,420 five-second clips from approximately 5.6 hours of gameplay, with temporally aligned proxy sequences as conditions.
- Coding-agent inference: During inference, a coding agent extends existing game-engine code and scenes to construct executable, player-controllable worlds with coarse proxy geometry.The resulting proxy video is recorded under user control and supplied frame-wise to the downstream video model.
- Video-model inference: Video inference uses overlapping 124-frame windows and appearance anchors to generate temporally continued visual observations from proxy sequences and text.Subsequent windows use continuation RGB frames from the preceding window together with the shared appearance anchor.
- Visual quality results: Even with LoRA adaptation on a small paired dataset, the video model closely follows proxy-specified entity positions, motion, scene layout, and camera trajectories.The qualitative examples use proxies recorded from simple interactive worlds constructed by the coding agent.
- Visual quality results: Proxy conditioning provides more precise and responsive control over character motion, actions, and camera movement than action- or camera-conditioned video world models.The comparison isolates control quality and does not consider inference latency.
5. Limitations and Conclusions
The prototype is constrained by limited compute and by current coding agents’ difficulty implementing complex game mechanisms from scratch. Consequently, it does not yet demonstrate autonomous construction of a complete open-world game or simulator.
- Limitations: Limited compute keeps training scale small, limits generation quality, and prevents autoregressive real-time generation in the current system.These constraints are identified as primary limitations of the prototype.
- Limitations: Current coding agents still struggle to implement highly complex game mechanisms reliably from scratch.The prototype therefore does not demonstrate autonomous construction of a complete open-world game or simulator.
- Conclusions: Code World Model separates persistent world evolution from visual realization by using executable code for state maintenance and a video model for observations.A proxy supplies frame-wise spatiotemporal conditions between these components, supported by aligned proxy–observation data pipelines.
A. Text Prompt Examples
The prompt examples specify distinct protagonists, environments, and temporally ordered actions for proxy-conditioned visual generation. They include camera-only motion, uphill walking and falling, and solitary environmental scenes.
- The Rainkeeper’s Steps: The Rainkeeper’s Steps features a solitary reed-masked rainkeeper standing on a monsoon-washed terraced valley spillway while the camera arcs around the stationary subject.The timeline keeps the rainkeeper planted and isolates camera following from entity translation.
- The Rainkeeper’s Steps: The courier scene places a candle-headed marionette on a polygonal hillside thoroughfare surrounded by cliffs, Halloween-town buildings, pumpkins, and amber-lit windows.The courier’s timeline describes an uphill stride across uneven slate facets, followed by a continuous fall through an open ravine.
- Prompt structure: The examples maintain solitary protagonists and static environmental elements while specifying detailed appearance, spatial layout, camera framing, and action timelines.These constraints include the absence of other visible actors and persistent scene elements throughout the clips.
- Hollowfacet Hill: Hollowfacet Hill features a solitary adult shoebill courier walking downhill through a biological market with fungus awnings, produce displays, and termite-spire storehouses.Its action prompt contains temporally ordered motion states beginning with an opening stance and restrained weight shift.