Source-linked AI summary
AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
AlayaWorld Team, Kaipeng Zhang, Chuanhao Li, Yifan Zhan, Yongtao Ge, Yuanyang Yin, Jiaming Tan, Kang He, Liaoyuan Fan, Mingliang Zhai, Ruicong Liu, Xiaojie Xu, Xuangeng Chu, Zhen Li, Zhengyuan Lin, Zhixiang Wang, Zian Meng, Zihui Gao
TL;DR
Interactive video world models must jointly support controllable interaction, persistent consistency, stable long-horizon generation, and efficient response. AlayaWorld addresses these requirements with bounded visual memory, drift-aware training, and few-step distillation, achieving the best overall performance on iWorld-Bench.
Problem
Interactive video world models need to jointly support controllable interaction, persistent spatiotemporal consistency, stable long-horizon generation, and efficient response.
Method
AlayaWorld autoregressively generates short video chunks using bounded temporal and spatial memory, drift-aware training, and discrete distillation for four-step inference.
Results
AlayaWorld achieves the best overall iWorld-Bench performance across generation quality, trajectory following, and memory ability, despite using four sampling steps.
Takeaways & Limitations
AlayaWorld provides an open-source full-stack foundation for interactive video world-model research.
Takeaways & Limitations
AlayaWorld’s understanding of object state, physical causality, and long-term task structure remains limited to their visible consequences.
Abstract
from arXiv · showhide
Unlike conventional video game development, which relies on labor-intensive pipelines for asset production, animation, physics, and programming, video world models generate interactive environments from user inputs instantly. It enable us to create customized, explorable, and continuously evolving virtual world from text, an image, or video. Realizing this vision requires four tightly coupled capabilities: interaction, persistent spatiotemporal consistency, stable long-horizon generation, and efficient response. We present AlayaWorld, an interactive long-horizon video world model that generates 24-fps video at 540p and 720p. Built on a 15B video diffusion transformer, AlayaWorld generates short latent chunks autoregressively under camera trajectories and switchable text prompts. Its bounded visual context combines a persistent sink frame, compressed temporal history, geometry-aligned spatial memory, and recent-frame conditioning. To reduce long-term drift, the model is trained with corrupted histories and prediction residuals collected from its own roll-outs. We further introduce a discrete autoregressive distillation formulation that combines distribution-matching distillation, self-forcing++, and consistency distillation, reducing inference from approximately 30 sampling steps to four steps per chunk. On iWorld-Bench, AlayaWorld achieves the best performance over long-horizon generation. Conceived as a full-stack, open-source, and long-term project, AlayaWorld is intended to provide an extensible foundation for future research on interactive video world models.
1 Introduction
AlayaWorld addresses the challenge of building interactive, persistent, stable, and efficient video-generated worlds without conventional game-development pipelines. It introduces a 15B autoregressive video world model with camera and prompt control, long-horizon memory, self-rollout training, and strong iWorld-Bench performance.
- Motivation: Video world models generate interactive virtual worlds from text, images, videos, and user controls, avoiding manual asset creation and interaction programming.They jointly encode visual appearance, motion, viewpoint changes, and certain interactions in model parameters.
- Challenges: Four coupled capabilities are required: accurate interaction, persistent spatiotemporal consistency, stable long autoregressive roll-outs, and efficient response.These capabilities cover camera trajectories, evolving user intent, scene identity across revisits, resistance to visual and geometric drift, and responsive generation.
- Model: AlayaWorld is a 15B video diffusion transformer generating 24-fps video at 540p and 720p in short latent chunks under camera trajectories and switchable text prompts.The design supports controllable navigation and prompt-driven open-ended actions while preserving scene information through bounded visual context.
- Training: AlayaWorld mitigates long-horizon drift by training an autoregressive generator with corrupted histories and prediction residuals replayed from its own roll-outs.Its three-stage training adapts a bidirectional video prior, adds camera control and spatiotemporal memory, and teaches recovery from imperfect context.
- Evaluation: AlayaWorld achieves the best overall performance on iWorld-Bench across generation quality, trajectory following, and memory ability.Qualitative results show controllable navigation, consistent revisitation, prompt-driven actions, and stable long-horizon generation across diverse scenes.
- Limitations: AlayaWorld’s visual-observation, estimated-geometry, and visual-memory representation limits understanding of object state, physical causality, and long-term task structure.These aspects remain represented primarily through their visible consequences.
2 Training Data
AlayaWorld trains on a heterogeneous corpus combining real-world captures and synthetic renderings, normalized into records with video, camera metadata, and timeline-aligned hierarchical captions. A unified filtering pipeline and two-level annotation schema provide quality control and localized supervision for controllable world modeling.
- Corpus composition: The corpus spans visual domain, motion geometry, and supervision fidelity by combining real-world captures with synthetic renderings.Real data anchors natural appearance, scene layout, and capture artifacts, while synthetic data provides scalable camera and action control.
- Corpus composition: 222 147 clips from seven sources are normalized into records containing video, camera intrinsics and poses when available, and hierarchical timeline-aligned captions.The sources include the internally curated MUGEN and GameVerse datasets.
- Filtering: A shared three-phase pipeline—ingest, run, and select—applies dependency-aware filtering with cached features and an all-stages-must-pass selection rule.One NVDEC decode and one RAFT forward pass provide reusable photometric, frame-difference, border, and motion statistics for downstream gates.
- Filtering: Filtering validates technical quality, photometric stability, shot continuity, motion, camera pose stability, perceptual quality, and near-duplicate exclusion.Technical gates require at least 720p resolution, at least 3 s duration, and 24–65 fps; photometric filtering caps border ratio at 0.10.
- Annotation schema: Each clip receives global context attributes and timestamped segment tracks for subject motion, environmental dynamics, static scene properties, and camera viewpoint or motion.The global vocabulary is reduced from 59 to 26 values, while separate subject and camera tracks distinguish actions from camera movements.
3 AlayaWorld
AlayaWorld is an interactive world model that generates controllable video autoregressively in latent-space chunks, conditioning on camera trajectories, optional text prompts, and bounded visual context. Its training adapts a general video prior, stabilizes long roll-outs with memory and anti-drift mechanisms, and distills approximately 30 sampling steps into four for interactive inference.
- Training: Training proceeds through bidirectional pre-training, autoregressive training with memory and anti-drift training, and post-training acceleration.The initial fine-tuning uses 24 fps video at mixed 540p and 720p resolutions with variable-length clips up to 20 seconds.
- Autoregressive generation: AlayaWorld generates video autoregressively in VAE latent space, using chunks of K=4 latent frames conditioned on camera trajectories and optional chunk-level text prompts.Prompts can switch at chunk boundaries to drive prompt-driven actions.
- Autoregressive generation: Each chunk uses a persistent sink, compressed temporal history, geometry-aligned spatial memory, nearby-frame conditioning, camera control, and optional text as its conditioning context.The visual past enters as an in-context token prefix, while camera trajectories are injected through an AdaLN camera-control module.
- Spatial consistency: The spatial cache provides long-range spatial consistency when revisiting previously seen places, extending consistency beyond the 6-frame temporal window.The full per-chunk conditioning is a function of the past and includes the sink, temporal memory, spatial memory, nearby frame, camera trajectory, and text prompt.
- Spatial consistency: Spatial memory caches generated frames, monocular depths, and camera poses, then retrieves up to 10 frames to render evidence into the target viewpoint.A coverage mask causes uncovered regions to be ignored rather than trusted, and generated chunks update the cache.
- Inference acceleration: 4 sampling steps per chunk replace approximately 30 teacher steps while preserving 24 fps output, full camera control, temporal memory, and spatial memory.The discrete distillation formulation avoids Jacobian-vector-product computation and combines distribution matching with consistency-oriented training.
4 Results
On iWorld-Bench, AlayaWorld is evaluated across generation quality, trajectory following, and memory ability, achieving the best performance on most metrics with four-step chunk generation. Qualitative results further show camera control, prompt-driven interaction, persistent scene consistency, and stable long-horizon rollouts.
- Benchmark and quantitative results: iWorld-Bench evaluates AlayaWorld on Generation Quality, Trajectory Following, and Memory Ability, covering visual consistency, action-conditioned trajectories, and loop-closure consistency.The benchmark assesses visual quality and consistency, trajectory smoothness and accuracy, and preservation of spatial and visual consistency along revisited trajectories.
- Benchmark and quantitative results: With each chunk generated in four sampling steps, AlayaWorld achieves the best performance on most iWorld-Bench metrics despite the reduced sampling budget.Results use the distilled autoregressive model at 480p, matching the benchmark’s initial-frame resolution, with semantics-preserving prompt adaptation before inference.
- Benchmark and quantitative results: AlayaWorld leads in brightness consistency, color-temperature constraint, and sharpness retention, mitigating visual drift, illumination fluctuations, and sharpness degradation during autoregressive generation.It does not obtain the highest image-quality score, but its overall generation quality remains competitive with existing video and interactive world models.
- Benchmark and quantitative results: AlayaWorld achieves the best results in motion smoothness and trajectory accuracy and leads on both Memory Ability metrics, preserving appearance and spatial structure during revisitation.These results support smooth, accurate responses to input action trajectories and validate the proposed spatial and temporal memory mechanisms.
- Qualitative results: Qualitative rollouts demonstrate camera-controlled navigation, switchable prompt-driven actions, persistent scene consistency under leave-and-return trajectories, and stable visual quality over extended generation.Long-horizon videos show limited accumulation of blur, illumination shifts, color drift, and structural degradation, indicating the effectiveness of drift-aware training.
5 Conclusion
AlayaWorld is an open-source interactive long-horizon video world model that synthesizes continuously evolving environments directly from user inputs. Its unified autoregressive framework integrates interaction, consistency, stability, and efficiency through bounded visual context, drift-aware training, and discrete few-step distillation.
- 5 Conclusion: AlayaWorld is an open-source interactive long-horizon video world model that synthesizes environments directly from user inputs and evolves them continuously through autoregressive generation.It presents an alternative to conventional game-development pipelines for constructing virtual worlds.
- 5 Conclusion: The framework defines interactive world modeling through four tightly coupled properties: interaction, consistency, stability, and efficiency.AlayaWorld integrates these properties rather than addressing them independently.
- 5 Conclusion: Bounded visual context supports controllable navigation and persistent world memory, while drift-aware training improves long-horizon robustness.These design choices are integrated within the unified autoregressive framework.
- 5 Conclusion: Discrete few-step distillation together with short chunk-wise generation enables low-latency inference.The conclusion identifies these mechanisms as jointly supporting efficiency within the autoregressive framework.
6 Contributions and Acknowledgments
The section lists authors by contribution category, with names alphabetized by first name within each category. It identifies five core contributors and twelve contributors.
- Within each role category, authors are listed alphabetically by first name.
- The core contributors are Chuanhao Li, Kaipeng Zhang, Yifan Zhan, Yongtao Ge, and Yuanyang Yin.
- The contributors are Jiaming Tan, Kang He, Liaoyuan Fan, Mingliang Zhai, Ruicong Liu, Xiaojie Xu, Xuangeng Chu, Zhen Li, Zhengyuan Lin, Zhixiang Wang, Zian Meng, and Zihui Gao.