Source-linked AI summary
Programmable World Model
Zheng-Hui Huang, Guixu Lin, Jiacheng Lin, Yi-Chuan Huang, Ruihan Yu, Muyao Niu, Siqi Yang, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang, Zhixiang Wang
TL;DR
Interactive video world models lack reliable persistent state and programmable rules over extended interactions. Programmable World Model separates executable state evolution from generative rendering through state-augmented 3D OBBs and a deterministic compiler; on CombatStateBench, it achieves 94% Count Accuracy and 98% State Accuracy while supporting coherent long-horizon generation.
Problem
Existing video world models lack reliable mechanisms for maintaining persistent global state and enforcing programmable rules over extended interactions.
Method
The framework uses an agent-written executable world program and lightweight state engine, then compiles state-augmented 3D OBBs into structured controls for a pretrained video renderer.
Results
94% Count Accuracy and 98% State Accuracy are achieved on CombatStateBench, substantially outperforming existing interactive video world models.
Takeaways & Limitations
Explicit, executable state evolution separated from rendering supports more reliable persistent interactions and remains effective across long-horizon generation and varied entities, styles, and domains.
Takeaways & Limitations
The evaluation uses coarse, globally observable metrics, while competing methods lack external instance-level state interfaces for direct object-level correspondence comparison.
Abstract
from arXiv · showhide
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.
1 Introduction
Programmable World Model addresses persistent-state and rule-programming gaps in interactive video world models by separating executable world-state evolution from visual rendering. It uses state-augmented 3D OBBs and a deterministic compiler to connect programmable state with generation, and evaluates consistency with CombatStateBench.
- Motivation: Existing video world models generate responsive environments but offer limited entity-level control, persistent global state, and programmable rules for world evolution.These missing capabilities include off-screen entities and nonvisual information such as inventory, task progress, and interaction history.
- Approach: The framework translates natural-language instructions into executable programs defining entity states and world rules, while a renderer generates observations from the evolving state.A state executor applies player actions and events, and users can revise behavior through further instructions to the coding agent.
- Approach: State-augmented 3D OBBs provide a compact interface for explicit world-space control while leaving fine geometry, articulation, and secondary visual dynamics to the renderer.The representation augments geometry with persistent identity, semantic category, dynamic state, and appearance information.
- Evaluation: CombatStateBench evaluates consistency between generated videos and engine-maintained states using visible alive-character counts and visual death-state realization.The benchmark focuses on diverse camera and entity motion.
- Approach: The deterministic state compiler converts updated OBB-based world state under a target camera into pixel-aligned controls for video generation.The contribution connects editable world state to view-consistent spatiotemporal conditioning signals.
- Contributions: The work also contributes a data-curation pipeline that extracts spatial and semantic supervision from videos for training programmable generative worlds.
2 Representation Trade-offs
The paper frames representation choice as a trade-off between explicit, fine-grained control and the cost of supervision and inference-time state evolution. It selects state-augmented 3D OBBs as a compact, view-independent scaffold that preserves world-space structure while delegating fine dynamics to generation.
- Representation spectrum: Lightweight text descriptions convey categories, attributes, and high-level states but do not impose geometric constraints on position, extent, or orientation.2D boxes and masks provide more direct control but remain tied to image-space viewpoints.
- Representation spectrum: Greater structural detail enables finer-grained control over geometry, spatial relationships, and temporal evolution, but requires richer supervision and more complex inference.Detailed scenes, articulated models, dynamic geometry, and G-buffers increase structural and data-acquisition costs.
- Training–inference asymmetry: Training and inference obtain structural representations through different paths, creating an asymmetry between observing realized dynamics during training and constructing evolving structure from high-level transitions at inference.This mismatch becomes more demanding as representations encode more articulated motion or dynamic 3D geometry.
- Design principle: A representation that is too weak leaves world-space structure and state-dependent changes to the generative model, whereas one that is too strong requires extensive low-level geometric and dynamic specification.The design boundary separates explicit structural control from generative dynamic completion.
- Chosen representation: State-augmented 3D OBBs encode position, extent, orientation, identity, semantic category, and dynamic state in a compact shared world coordinate system.They provide a view-independent, reprojectable scaffold without requiring meshes, materials, skeletons, or predefined animations.
- Chosen representation: OBBs restrict explicitly evolved structure to a compact entity-level space while leaving fine geometry, articulation, and secondary visual dynamics to the generative renderer.This balances controllability with the burden of constructing and evolving world state.
3 Related Work
Related work spans observation-centric interactive video models, explicit state prediction, and renderer-facing structured representations. The proposed architecture combines executable world programming, OBB-based control compilation, and generative rendering with temporal and spatial memories for long-horizon generation.
- Interactive video world models: Recent video world models support action-conditioned generation, controllable cameras, real-time interaction, and increasingly long-horizon rollouts, but primarily optimize pixel-level plausibility.Such objectives do not explicitly require persistent structured world state over time.
- Explicit state modeling: StatePlay jointly predicts observations and game-state variables, but model-predicted state errors can directly produce incorrect updates and accumulate over long interactions.
- Generative rendering: Generative rendering methods condition visual synthesis on structured scene representations ranging from dense G-buffers to lighter geometric proxies.These approaches trade stronger structural constraints against delegating more visual and dynamic realization to the generative prior.
- Proposed architecture: The proposed architecture instantiates executable world state from natural-language specifications, advances it with a lightweight engine, and compiles OBBs into pixel-aligned identity, semantic, and motion-direction controls.A pretrained camera-controlled video model then synthesizes the next video chunk from these controls and visual history.
- Proposed architecture: Completed video chunks are incorporated into temporal and geometry-aligned spatial memories to support long-horizon generation.
4 Method
The method maintains an explicit canonical world state governed by executable programs, then compiles its visually relevant contents into camera-aligned controls for generative rendering. Persistent entities, relations, attributes, and rules support programmable interaction while projected OBB controls preserve identity, semantics, and object motion.
- World-state execution: The interaction loop updates a canonical state with a lightweight engine, projects it through a deterministic compiler, and renders the next observation.The engine applies player actions and triggered events before compilation and generation.
- World representation: The canonical state stores persistent entities and poses, semantic and functional attributes, inter-entity relations, and executable world rules.These components jointly specify the current configuration and how it evolves under interaction.
- World programming: Natural-language descriptions are converted into engine-readable programs specifying initial states, relations, supported actions, update rules, triggers, constraints, and objectives.In combat scenarios, the program can define factions, health, attacks, damage, death, and objective updates.
- Persistent state: The engine explicitly maintains nonvisual facts such as health, inventory, and faction membership, which can govern future transitions and rendered outcomes.These variables remain part of the canonical state even when they have no direct visual realization.
- Control compilation: State-augmented 3D OBBs provide a compact world-space scaffold with projected geometry, persistent identity, semantic category, and camera-relative object motion.The compiler projects OBBs under the target camera and rasterizes aligned control maps for these attributes.
5 Experiments
Experiments evaluate world-state consistency and visual quality on CombatStateBench, alongside qualitative tests across cameras, entities, and long-horizon interactions. The method more reliably preserves engine-maintained states while retaining strong video quality and broad qualitative generalization.
- Benchmark and evaluation: CombatStateBench contains 50 clips with synchronized entity states, 3D boxes, camera parameters, projected controls, and masks validated by automatic consistency checks.The benchmark evaluates whether generated videos reflect engine-maintained world states under combat scenarios with diverse camera and entity motion.
- Metrics: Count Accuracy compares VLM-counted visibly alive characters against engine-maintained counts across eight sampled frames per generated video.State Accuracy samples three post-transition frames for each engine-recorded death event and checks whether a dead character is visibly depicted.
- Quantitative results: 94.00 Count Accuracy exceeds LingBot-World-V2 by 53.25 percentage points and YUME by 62.00 percentage points.The result indicates more reliable preservation of the intended number of visibly alive characters throughout the rollout.
- Quantitative results: 98.00 State Accuracy exceeds LingBot-World-V2 by 90.00 percentage points and YUME by 40.00 percentage points.Both world-state metrics evaluate coarse globally observable properties rather than instance-level correspondence.
- Video quality: Ours achieves the best performance across all four VBench video-quality metrics, including 94.74 Subject Consistency, 96.98 Background Consistency, and 99.00 Temporal Stability.Imaging Quality is 67.62, compared with 67.46 for LingBot-World-V2 and 64.10 for YUME.
- Qualitative results: Qualitative results follow specified death events, camera motions, spatial layouts, and entity states across dynamic scenes, heterogeneous objects, and an 897-frame autoregressive sequence.The tests include novel minotaur and racing scenes, large-angle camera rotation revealing initially hidden characters, and progressively entering NPCs.
6 Conclusion
Programmable World Model explicitly maintains world state while using video generation primarily for appearance, making persistent interactions more reliable without sacrificing generative flexibility.
- Explicit state maintenance separates what remains true about the world from visual generation.The framework uses the video model primarily as a renderer.
- The framework achieves stronger state consistency across long-horizon generation, unseen entities, visual styles, and interaction domains.
- The results support executable and verifiable state as a direction for world models while keeping appearance generative.