PaperlayerView exact revision on arXiv

Source-linked AI summary

AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report

AlayaWorld Team, Kaipeng Zhang, Chuanhao Li, Yifan Zhan, Yongtao Ge, Yuanyang Yin, Jiaming Tan, Kang He, Liaoyuan Fan, Mingliang Zhai, Ruicong Liu, Xiaojie Xu, Xuangeng Chu, Zhen Li, Zhengyuan Lin, Zhixiang Wang, Zian Meng, Zihui Gao

arXiv:2607.18367v1cs.AI

TL;DR

Interactive world models must jointly support responsive control, persistent consistency, stable long-horizon generation, and efficiency. AlayaWorld combines autoregressive generation with visual memory and few-step distillation, achieving the best overall performance on iWorld-Bench.

  • Problem

    Interactive world models must jointly provide responsive control, persistent consistency, stable long-horizon generation, and efficiency, while conventional world creation requires lengthy, costly production pipelines.

    Source: §1 · p. 2

  • Method

    AlayaWorld combines autoregressive chunk generation under camera and text controls with bounded visual memory, drift-aware training, and few-step distillation.

    Source: §1 · p. 2, §5 · p. 14

  • Results

    AlayaWorld achieves the best overall performance on iWorld-Bench across generation quality, trajectory following, and memory ability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The unified framework supports low-latency interaction while preserving responsiveness to evolving camera trajectories and text prompts.

    Source: §5 · p. 14

    Its understanding of object state, physical causality, and long-term task structure remains limited to visible consequences of visual observations, estimated geometry, and visual memory.

    Source: §1 · p. 2

Abstract

from arXiv · show

Unlike conventional video game development, which relies on labor-intensive pipelines for asset production, animation, physics, and programming, video world models generate interactive environments from user inputs instantly. It enable us to create customized, explorable, and continuously evolving virtual world from text, an image, or video. Realizing this vision requires four tightly coupled capabilities: interaction, persistent spatiotemporal consistency, stable long-horizon generation, and efficient response. We present AlayaWorld, an interactive long-horizon video world model that generates 24-fps video at 540p and 720p. Built on a 15B video diffusion transformer, AlayaWorld generates short latent chunks autoregressively under camera trajectories and switchable text prompts. Its bounded visual context combines a persistent sink frame, compressed temporal history, geometry-aligned spatial memory, and recent-frame conditioning. To reduce long-term drift, the model is trained with corrupted histories and prediction residuals collected from its own roll-outs. We further introduce a discrete autoregressive distillation formulation that combines distribution-matching distillation, self-forcing++, and consistency distillation, reducing inference from approximately 30 sampling steps to four steps per chunk. On iWorld-Bench, AlayaWorld achieves the best performance over long-horizon generation. Conceived as a full-stack, open-source, and long-term project, AlayaWorld is intended to provide an extensible foundation for future research on interactive video world models.

1 Introduction

Video world models [20] [10] offer an alternative to labor-intensive game production, but interactive long-horizon generation requires coordinated control, consistency, stability, and efficiency. AlayaWorld combines autoregressive generation, bounded visual memory, and training and distillation methods for this task.

1 Introduction

  • Video world models generate worlds from text, images, videos, and user controls, offering an alternative to game development’s prolonged, labor-intensive production pipeline [20] [10].Interactive world models must coordinate response to camera trajectories and user intent, preservation of spatial structure and identity, stable long roll-outs, and low latency; these capabilities can undermine one another.Source: §1 · p. 2
  • AlayaWorld uses a 15B video diffusion transformer to generate 24-fps video at 540p and 720p in autoregressive chunks controlled by camera trajectories and switchable prompts.Its bounded context combines a global scene anchor, compressed recent history, geometry-aligned spatial memory, and recent-frame conditioning, keeping chunk-generation cost approximately constant as roll-outs grow.Source: §1 · p. 2
  • AlayaWorld trains on corrupted histories and its own roll-out residuals, then combines distribution-matching, self-forcing++, and consistency distillation to reduce sampling from approximately 30 steps to four per chunk.The three-stage process adapts a bidirectional video prior, converts it to an autoregressive generator with camera control and memory, and applies discrete autoregressive distillation while preserving that control and memory stack.Source: §1 · p. 2
  • On iWorld-Bench [5], AlayaWorld achieves the best overall performance across generation quality, trajectory following, and memory ability.Qualitative results also demonstrate controllable navigation, consistent revisitation, prompt-driven actions, and stable long-horizon generation across diverse scenes.Source: §1 · p. 2
  • AlayaWorld’s understanding of object state, physical causality, and long-term task structure remains limited to their visible consequences.The project is conceived as a full-stack, open-source, long-term effort intended to provide an extensible foundation for research on interactive video world models.Source: §1 · p. 2

2 Training Data

AlayaWorld’s training data combines heterogeneous real and synthetic trajectories with camera- and time-aligned supervision, then applies shared curation and hierarchical captions to support controllable learning.

2 Training Data

  • 222 147 clips from seven sources are normalized into records containing video, available or recoverable per-frame camera parameters, and timeline-aligned hierarchical captions.The corpus spans visual domains, motion geometries, and supervision fidelities; Figure 2 illustrates sample trajectories and their domain coverage.Source: §2 · p. 3

2.1 Data Mixture

  • Real-world captures supply physical appearance, natural camera shake, sensor artifacts, and scene layouts; GameVerse contains 124 116 clips of approximately 66 s each, while GenEvent has 6490 event-centric clips averaging 10.6 s.GenEvent clips are synthesized by a generative video model. For real-world sources without camera metadata, ViPE [11] recovers per-frame intrinsics and poses.Source: §2.1 · p. 3

2.2 Curation Pipeline

  • A shared manifest runs ingest, dependency-ordered feature evaluation, and all-active-gates selection, using cached single-decode and RAFT features to support six filtering gate classes.Gates cover technical and photometric quality, shot boundaries, motion, text and interface suppression, and person control; trajectory-stability checks also apply where camera paths are estimated. Thresholds are calibrated on roughly 200 clips per source, and only clips passing every active gate are retained.Source: §2.2 · pp. 4–5

2.3 Training-Oriented Caption Annotation

  • Timestamped vision-language annotation separates compact clip-level context from segment-level subject, environment, scene, and camera tracks, producing full_prompt and short_prompt text.Frames are sampled at 1–2 fps with timestamps; the global-attribute vocabulary is reduced from 59 to 26 values. Segment tracks distinguish subject motion from camera motion, and the prompts support text encoding, caption dropout, and multi-caption augmentation.Source: §2.3 · p. 5, Table 2 · p. 5

3 AlayaWorld

AlayaWorld adapts an LTX-2.3 video backbone into a chunk-wise autoregressive world model conditioned on camera trajectories and optional text prompts. Its persistent context combines temporal and geometry-aligned spatial memory, while staged training addresses long-rollout drift and reduces inference to four steps per chunk.

3 AlayaWorld

  • AlayaWorld removes the audio module from the public 22B LTX-2.3 checkpoint, leaving an approximately 13B video diffusion transformer, and trains it through pre-training, autoregressive integration, and acceleration.The autoregressive stage adds history compression, spatial memory, camera control, and anti-drift training before the final distillation stage.Source: §3 · p. 6

3.1 Formulation

  • Generation proceeds in four-latent-frame chunks conditioned on camera trajectories and optional boundary-switched prompts, using a fixed sink, compressed history, spatial memory, and the latest frame.The bounded rolling context uses a six-latent-frame history and at most ten rendered cache frames, keeping compute per chunk constant as the horizon grows.Source: §3.1 · pp. 6–8
  • A depth-and-pose cache retrieves and warps up to ten past frames into the target view, providing long-range consistency beyond the six-frame temporal window.Its coverage mask suppresses attention to unobserved regions, and generated frames with estimated depth and camera pose are appended to the cache.Source: §3.1 · pp. 7–8

3.2 Bidirectional Model Pre-Training – Establishing the General Video Prior

  • Bidirectional pre-training fully fine-tunes the base video prior at 24 fps on variable-length clips up to 20 seconds, mixing image-, video-, and text-conditioned objectives.Training mixes scene and camera-pose data with gameplay, walkthroughs, and magic-event clips, and drops camera-tag descriptions so the model does not over-rely on text for camera control.Source: §3.2 · p. 8

3.3 Autoregressive Model Training – Control and Memory Integration

  • Full-stack autoregressive fine-tuning trains the backbone with history compression, AdaLN camera control, spatial memory, and a next-forcing head for frame-to-frame continuity.History compression is first pretrained with the backbone frozen; the subsequent full-parameter phase trains the backbone and dedicated modules together.Source: §3.3 · pp. 8–9
  • Anti-drift training corrupts temporal, spatial, and nearby context while keeping the sink clean, using simulated artifacts and replayed reconstruction residuals from the model’s own rollouts.The error bank takes priority after warm-up, while Helios drift corruption is applied before then and is mutually exclusive with error-bank replay.Source: §3.3 · p. 9

3.4 Post-Training – Inference Acceleration

  • Four-step distillation preserves 24-fps generation and the teacher’s camera-control and memory stack while reducing sampling from approximately 30 steps per chunk.The discrete formulation combines distribution-matching distillation, self-forcing++, and consistency distillation, avoiding Jacobian-vector-product computation.Source: §3.4 · pp. 9–10

4 Results

On iWorld-Bench, the distilled autoregressive AlayaWorld performs best on most metrics, with reported strengths in visual consistency, trajectory following, and memory. Qualitative results show camera-conditioned navigation, prompt-driven interactions, consistent revisits, and stable extended rollouts.

4.1 Experimental Setup

  • The evaluation uses iWorld-Bench action-control and memory tasks and WorldMark blind human-preference comparisons across visual quality, control alignment, and world consistency.WorldMark votes are aggregated into Elo ratings.Source: §4.1 · p. 10

4.2 Quantitative Results

  • AlayaWorld achieves the best performance on most iWorld-Bench metrics using four sampling steps per chunk at 480p.The evaluation covers Generation Quality, Trajectory Following, and Memory Ability; the benchmark resolution matches its initial-frame resolution.Source: §4.2 · pp. 10–11, Table 3 · p. 11
  • AlayaWorld leads in brightness consistency, color-temperature constraint, and sharpness retention, but not in image-quality score.Its overall generation quality remains competitive with existing video and interactive world models.Source: §4.2 · p. 11
  • AlayaWorld achieves the best results in motion smoothness and trajectory accuracy and leads on both Memory Ability metrics.The results indicate that it preserves visual appearance and spatial structure when revisiting previously observed regions.Source: §4.2 · p. 11

4.3 Qualitative Results

  • Under diverse translation and rotation commands, AlayaWorld follows user-specified camera trajectories while maintaining coherent scene geometry and appearance.Figure 5 presents camera-controlled generation results.Source: §4.3 · p. 11, Figure 5 · p. 12
  • During autoregressive rollouts, switchable text prompts introduce new actions and events while the existing scene context remains visually coherent.Examples include spell casting, combat, object appearance, and scene transformation.Source: §4.3 · p. 11, Figure 6 · p. 12
  • On leave-and-return trajectories, AlayaWorld preserves scene layout, object identity, textures, and structural details when revisiting earlier viewpoints.The results demonstrate consistency beyond the recent context window.Source: §4.3 · p. 11, Figure 7 · p. 13
  • Over extended autoregressive rollouts, AlayaWorld maintains coherent scene evolution, stable visual quality, and smooth temporal transitions with limited blur, illumination shifts, color drift, and structural degradation.The reported results attribute this stability to the drift-aware training strategy.Source: §4.3 · pp. 11, 13, Figure 8 · p. 13

5 Conclusion

AlayaWorld presents an open-source interactive video world model that synthesizes evolving environments from user inputs through autoregressive generation. Its unified design combines controllable navigation and persistent memory with long-horizon robustness and low-latency responsiveness.

5 Conclusion

  • AlayaWorld offers an open-source alternative to conventional game-development pipelines, synthesizing interactive worlds directly from user inputs through continuous autoregressive generation.Source: §5 · p. 13
  • Its unified autoregressive framework jointly addresses interaction, consistency, stability, and efficiency through bounded visual context, drift-aware training, and discrete few-step distillation with short chunks.These design choices support controllable navigation and persistent memory, long-horizon robustness, and low-latency responsiveness to camera trajectories and text prompts.Source: §5 · pp. 13–14
  • Experiments on iWorld-Bench demonstrate strong performance in generation quality, trajectory following, and memory ability, and AlayaWorld is intended as an extensible foundation for future research.The project is conceived as full-stack, open-source, and long-term.Source: §5 · p. 14

Related papers

Loading 2607.18367v1…