PaperlayerView exact revision on arXiv

Source-linked AI summary

AlayaWorld: Long-Horizon and Playable Video World Generation

AlayaWorld Team, Kaipeng Zhang, Chuanhao Li, Yifan Zhan, Yongtao Ge, Yuanyang Yin, Jiaming Tan, Kang He, Liaoyuan Fan, Ruicong Liu, Xiaojie Xu, Xuangeng Chu, Zhen Li, Zhengyuan Lin, Zhixiang Wang, Zian Meng, Zihui Gao

arXiv:2607.06291v1cs.CVcs.HC

TL;DR

Traditional game worlds require labor-intensive production and are difficult to extend or modify after deployment. AlayaWorld combines autoregressive generation with camera and spatial conditioning and prompt-driven actions in an open-source interactive-world framework. Its generated sequences demonstrate camera control, consistent leave-and-return scenes, and stable long-horizon rollouts.

  • Problem

    Conventional game worlds require labor-intensive production, with components specified in advance, making environments costly and difficult to extend or modify after deployment.

    Source: §1 · p. 2

  • Method

    AlayaWorld combines an autoregressive DiT with camera conditioning, a 3D cache, history compression, and chunk-level prompt switching in an open-source interactive-world framework.

    Source: §1 · p. 2, §3 · p. 3, §3.1.1 · p. 4

  • Results

    Generated sequences show camera-controllable viewpoint changes, consistent leave-and-return regions, and maintained visual quality, object identity, and motion continuity in long-horizon rollouts.

    Source: §4.1 · p. 8, §4.3 · p. 8, §4.4 · p. 8

  • Takeaways & Limitations

    AlayaWorld is presented as a full-stack, open-source foundation for future work on video world models.

    Source: §1 · p. 2

Abstract

from arXiv · show

Game worlds have traditionally been built through labor-intensive production pipelines, making them costly to develop, difficult to customization, and expensive to modify after deployment. Recent advances in video world models offer a fundamentally different paradigm. Rather than explicitly authoring every component of a virtual environment, these models autoregressively synthesize future observations conditioned on the current world state and user interactions, enabling playable worlds to be generated online. Trained on both gameplay recordings and real-world videos, they can capture diverse visual appearances and physical dynamics, opening new opportunities for interactive applications beyond gaming, including embodied intelligence. In this paper, we present \textbf{AlayaWorld}, a full-stack open-source framework for building interactive generative worlds. AlayaWorld enables open-ended real-time interaction, allowing users to freely navigate and perform diverse actions such as combat, spell casting, and monster summoning. The framework unifies the complete development-from data preparation model architecture, model training, inference acceleration, and deployment-within a modular and extensible architecture. Alongside the framework, we release reproducible pipelines, reference implementations, evaluation tools, and comprehensive documentation, establishing a practical foundation for future research and real-time applications of generative world models.

1 Introduction

AlayaWorld presents an open-source framework for playable video-generated worlds, addressing the limits of labor-intensive world production and the control, consistency, stability, and runtime challenges of generative approaches.

1 Introduction

  • Video world models [45] [60] [71] [87] [99] predict future observations from user interactions, learning content, behavior, and rendering from data.Training on diverse videos can support generated environments beyond gaming, with controllable visual appearance, dynamics, and physics.Source: §1 · p. 2
  • Playable video world models face four challenges: open-ended control, spatial and temporal consistency, long-horizon stability, and low-latency real-time generation.Stability specifically concerns visual drift during long-horizon generation.Source: §1 · p. 2
  • The paper reviews interactive generative world modeling and introduces AlayaWorld, an autoregressive DiT combining prompt switching, camera control, 3D caching, history compression, an error bank, and few-step distillation.The components are presented as addressing the challenges of control, consistency, stability, and runtime.Source: §1 · p. 2
  • AlayaWorld is an open-source, long-term project intended as a foundation for future video world model research, with its technical details, results, and codebase slated for mid-July release.Source: §1 · p. 2

2 Related Work

Related work frames video-generation models as backbones for world generation and surveys how interactive systems extend them into playable, user-driven worlds.

2.1 Video Genration Models

  • Diffusion and latent-diffusion models with transformer denoisers underpin large-scale video generation, with Sora establishing the “video as world simulator” paradigm.Open and proprietary text- and image-to-video systems, alongside physical and embodied video foundation models, provide candidate generative backbones for interactive worlds.Source: §2.1 · pp. 2–3

2.2 Interactive World Models

  • Interactive world models include action-conditioned environments, neural game engines, and systems seeking greater controllability and generalization beyond single-game settings.Examples range from Genie [8] and Genie 2 to GameNGen [65], DIAMOND [3], Oasis, GameGen-X [9], The Matrix [14], GameFactory [84], and MineWorld [17].Source: §2.2 · p. 3
  • For free exploration, Yume [45] generates explorable worlds from a single image using keyboard-style controls, while Yume 1.5 improves quality and controllability.Source: §2.2 · p. 3
  • Recent systems target real-time streaming and longer consistency, while gaming-oriented WHAM/Muse and WHAMM jointly model environments and human actions.Genie 3 remains closed-source; Matrix-Game [21] and Hunyuan-GameCraft [32] release weights, with extensions addressing long-horizon memory and instruction control [71] [59]; Lingbot-World [60] scales diffusion context for geometric consistency.Source: §2.2 · p. 3

3 AlayaWorld

AlayaWorld is a full-stack interactive generative-world framework fine-tuned from LTX-2.3 with designed modules. It supports navigation and prompt-driven actions, with methods for consistency, stability, and responsive runtime interaction.

3.1.1 Navigation

  • AlayaWorld combines a rendered 3D cache with lightweight AdaLN-style camera modulation to improve trajectory following and cross-view consistency, including when players revisit regions.The cache supplies spatially grounded appearance and geometry, while camera modulation provides trajectory awareness with small parameter and computation overhead.Source: §3.1.1 · p. 4

3.1.2 Prompt-driven Action

  • Chunk-level prompt switching enables spell-casting, weapon combat, and monster summoning while changing future generation without regenerating existing content.Text conditions can be replaced at chunk boundaries, so the next chunk reflects the new prompt without affecting previously generated content.Source: §3.1.2 · pp. 4–5

3.2 Consistency

  • AlayaWorld combines a 3D cache reprojected into the queried viewpoint with compressed recent-frame history to support spatial persistence on revisits and temporal persistence.The cache anchors previously observed regions, while the history embedding captures recent dynamics that the static cache cannot represent by itself.Source: §3.2 · pp. 5–6

3.3 Stability

  • AlayaWorld trains on drifted histories and injects error-bank samples into both memory conditions and target segments to prevent autoregressive errors from compounding.This trains the model to generate from imperfect memory while correcting errors in the next segment.Source: §3.3 · p. 7

3.4 Runtime

  • AlayaWorld reduces runtime latency with DMD-based few-step distillation, short temporal chunks, and prompt updates at chunk boundaries.Short chunks lower per-call latency and let user commands affect output after a short delay without regenerating the full sequence.Source: §3.4 · pp. 7–8

4 Qualitative Results

Qualitative results show that AlayaWorld follows camera and changing-prompt controls, preserves consistency across revisited regions, remains stable over extended rollouts, and retains scene content across visual styles.

4.1 Camera Control

  • AlayaWorld follows requested viewpoint changes and translations while preserving scene identity and geometric plausibility.Figure 2 shows generated sequences under varied camera and action commands.Source: §4.1 · p. 8, Figure 2 · p. 8

4.2 Open-ended Action

  • Switching text prompts at chunk boundaries produces a short-delay transition with visual continuity, without regenerating the preceding sequence.This supports responsive control of changing user intent at interactive latency.Source: §4.2 · p. 8, Figure 3 · p. 9

4.3 Consistency

  • Revisited regions remain consistent in geometry, layout, and texture, while representative prior models show visual degradation, inaccurate camera control, or revisit inconsistency.The explicit spatial cache and compressed temporal history jointly support reliable loop closure.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 Long-Horizon Generation

  • Extended autoregressive rollouts maintain visual quality, object identity, and motion continuity without pronounced artifact accumulation during forward exploration.Figure 5 presents one-minute generation results.Source: §4.4 · p. 8, Figure 5 · p. 9

4.5 Diverse Styles

  • Across realistic, Minecraft, ink painting, oil painting, cyberpunk, pixel art, and Zelda-inspired styles, scene geometry, camera trajectory, and semantic content remain consistent.Figure 6 renders the same navigation trajectory in these styles.Source: §4.5 · pp. 9–10, Figure 6 · p. 10

Related papers

Loading 2607.06291v1…