Source-linked AI summary
From Pixels to States: Rethinking Interactive World Models as Game Engines
Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, Kaipeng Zhang
TL;DR
Interactive game world models still struggle to represent evolving game conditions, preserve consequences, and operate in real time. This paper organizes the field around the action-state-observation loop, reviews approaches across four dimensions, and introduces a 90-hour state-annotated gameplay dataset; it concludes that game state remains the central unresolved challenge.
Problem
Interactive game worlds require rule-governed outcomes, persistent consequences, and real-time generation, but existing models often keep game state implicit.
Method
The paper reviews interactive game world modeling through the action-state-observation loop across four dimensions and builds a scalable Black Myth: Wukong data engine.
Results
Existing methods provide natural input control, explorable scenes, and generation approaching real-time rates, while accumulated-condition outcomes and persistent rule-timed effects remain difficult.
Takeaways & Limitations
Integrating explicit game state, state-transition-grounded memory updates, and rule-aligned outcome timing remain open challenges for interactive game world modeling.
Takeaways & Limitations
Efficient and reliable update decisions for accumulated interaction changes lack a principled grounding.
Abstract
from arXiv · showhide
Building interactive worlds that respond coherently to player actions has long been a shared goal of computer graphics, games, and artificial intelligence. Recent video generative models provide a data-driven route toward this goal by predicting future observations conditioned on user actions, and are increasingly regarded as potential next-generation game engines. Realizing a genuinely interactive game world, however, requires interaction outcomes that follow rules over evolving game conditions, consequences that persist over long horizons, and a generation loop that operates in real time. Conventional game engines realize these properties through a recurrent action-state-observation loop, in which player actions update an explicit game state according to predefined rules and observations are rendered from the resulting state. Taking this loop as an organizing lens, this paper examines interactive game world modeling along four dimensions: player action control, game state dynamics, state-observation persistence, and real-time interactive generation. For each dimension, we start from the capabilities required by an interactive game world, group existing approaches into representative families, and discuss the strengths and trade-offs of each family. Complementing this analysis, we present a scalable data engine for Black Myth: Wukong that collects over 90 hours of gameplay with frame-aligned player actions, ground-truth game states, and visual observations, together with structured and semantic annotations, as a resource for state-aware game world modeling. We hope this paper offers a clear picture of where the field stands and fosters progress toward interactive game worlds.
1 Introduction
The paper frames interactive game world modeling as a recurrent action-state-observation loop, organized around four dimensions required for coherent, persistent, real-time interaction. It also introduces a scalable Black Myth: Wukong data engine and dataset with aligned actions, game states, observations, and annotations.
- Action-State-Observation Loop: Interactive worlds update internal game state according to rules and render observations from that state in response to player inputs.The paper uses this recurrent action-state-observation loop as its organizing lens.
- Four Dimensions: The framework examines player action control, game state dynamics, state-observation persistence, and real-time interactive generation.These dimensions define the capabilities required for interactive game worlds.
- Systematic Review: The paper systematically reviews existing approaches by grouping them into representative families and analyzing each family’s strengths and trade-offs.The review is conducted along the four framework dimensions.
- Data Engine and Dataset: Over 90 hours of Black Myth: Wukong gameplay include frame-aligned player actions, ground-truth game states, structured annotations, and semantic annotations.The scalable data engine collects boss-encounter gameplay from crowdsourced players with diverse skill levels and play styles.
2 Related Work
Related work frames game world models as systems that map player inputs to evolving worlds and visual outcomes, positioning them as potential next-generation game engines. It also situates these systems within the rapid progress of video generation models, which commonly combine language-based conditioning with latent-space diffusion transformers.
- Video Generation Models: Video generation models have advanced rapidly, typically coupling a language model for conditioning with a diffusion transformer that synthesizes video in latent space.Existing models are grouped into three families according to how language-model outputs interact with the diffusion transformer.
- Game World Models: Game world models take player inputs, evolve the underlying world, and produce visual observations, motivating their role as potential next-generation game engines.Early systems include action-conditional Atari prediction, GameGAN, world models for agent training, and neural game engines such as GameNGen, Oasis, MineWorld, and WHAM.
3 Interactive Game World Modeling as Game Engine
Interactive game world modeling is organized around the action-state-observation loop of conventional game engines. The section examines player action control, game state dynamics, state-observation persistence, and real-time interactive generation as the requirements for coherent, persistent, responsive worlds.
- Framework: The framework structures interactive game world modeling along four dimensions: player action control, game state dynamics, state-observation persistence, and real-time interactive generation.Interactive worlds map player inputs through evolving game conditions to real-time visual feedback.
- Player action control: Player action control represents heterogeneous player intent as geometric trajectories, motor signals, or semantic events.These categories cover navigation and viewpoint control, device operations, and explicit natural-language game events.
- Game state dynamics: Game state dynamics differ in whether state is entangled in observations, compressed into learned latents, or maintained as explicit descriptions.Explicit game state variables such as health, stamina, cooldowns, and equipment are continually updated from player actions according to game rules.
- State-observation persistence: State-observation persistence requires generated observations to remain consistent with evolving game state over long horizons, including changes that occur off screen.Stored-observation memories retrieve history through temporal or spatial indexing, while newer methods estimate how the present changes dynamically.
- State-observation persistence: High-frequency interactions can accumulate into substantial divergence, making efficient and reliable memory updates an open requirement in game world modeling.The challenge is not only retaining observations but grounding update decisions as the world evolves out of view.
- Real-time interactive generation: Real-time interactive generation must minimize control latency while preserving accurate consequence latency for rule-governed outcomes.Responsiveness depends on both faster sampling and conditioning that tracks changing user intent during a rollout.
4 Data Engine
The paper introduces a Black Myth: Wukong data engine to address the scarcity of gameplay paired with explicit state annotations. Its dataset aligns player actions, game states, and visual observations frame by frame for state-aware interactive world modeling.
- Motivation and scope: The data engine targets scarce gameplay datasets that pair videos with explicit state annotations, focusing on boss encounters with frequent attacks and dodges.It is built for the photorealistic action role-playing game Black Myth: Wukong, developed on Unreal Engine.
- Dataset contents: Over 90 hours of gameplay are recorded at 1280 × 720 resolution and 30 FPS with frame-aligned actions, states, RGB frames, and depth maps.Actions are raw mouse and keyboard inputs; states describe the evolving world, including camera, player, and boss poses, animations, and active skills.
- Data collection: The engine exports every-tick interaction data as structured JSON records, while a ReShade-and-OBS split-screen scheme captures RGB frames and depth maps.Each frame carries its complete interaction context, and the rendering buffer supplies the visual modalities.
- Data processing: Recorded streams are timestamp-aligned, segmented into encounter samples, re-encoded, and filtered by discarding samples with dropped frames, stuttering, or cross-stream inconsistency.Numerically encoded actions and states are converted into training-ready formats during processing.
5 Conclusion
The paper organizes interactive game world modeling around the action-state-observation loop and four dimensions, finding that current methods support natural control, explorable scenes, and near-real-time generation. Remaining challenges center on implicit game state, while the proposed Black Myth: Wukong data engine provides over 90 hours of aligned gameplay data.
- Conclusion: The analysis examines player action control, game state dynamics, state-observation persistence, and real-time interactive generation through the recurrent action-state-observation loop.Existing approaches are organized into representative families along these four dimensions.
- Conclusion: Current methods already provide natural input control, explorable scenes, and generation approaching real-time rates.These capabilities represent the areas where interactive game world modeling has made progress.
- Conclusion: The hardest capabilities involve accumulated-condition outcomes, consequences beyond the current view, and rule-defined timing of effects, all centered on implicit game state.Most models keep game state implicit, limiting these forms of coherent interaction.
- Conclusion: The paper presents a scalable Black Myth: Wukong data engine contributing over 90 hours of gameplay with frame-aligned actions, engine-exported states, and visual observations.The resource is designed to support state-aware game world modeling.