Source-linked AI summary
Matrix-game 2.0: An open-source real-time and streaming interactive world model
Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, Baixin Xu, Hao-Xiang Guo, Kaixiong Gong, Size Wu, Wei Li, Xuchen Song, Yang Liu, Yangguang Li, Yahui Zhou
TL;DR
Existing interactive world models face latency from bidirectional processing and lengthy denoising, while auto-regressive models can accumulate errors over time. Matrix-Game 2.0 addresses these challenges with scalable annotated-data production, action-conditioned causal diffusion, and few-step distillation. It generates minute-level interactive videos at 25 FPS while maintaining temporal coherence and action controllability, but remains limited in out-of-domain generalization, resolution, and long-history consistency.
Problem
Existing interactive video models are limited by bidirectional processing and lengthy denoising, while auto-regressive models suffer error accumulation during long generation.
Method
Matrix-Game 2.0 combines Unreal Engine and GTA5 data-production pipelines, frame-level action injection, causal auto-regressive diffusion, Self-Forcing distillation, and KV caching.
Results
25 FPS generation delivers minute-long interactive videos with temporal coherence, visual quality, and precise action controllability across diverse scenarios.
Takeaways & Limitations
The framework provides an efficient approach for real-time world simulation and human-in-the-loop interaction across diverse scenes.
Takeaways & Limitations
Limitations include degraded results in some out-of-domain scenes, 352×640 output resolution, and insufficient explicit memory for preserving consistency over very long generations.
Abstract
from arXiv · showhide
Recent advances in interactive video generations have demonstrated diffusion model's potential as world models by capturing complex physical dynamics and interactive behaviors. However, existing interactive world models depend on bidirectional attention and lengthy inference steps, severely limiting real-time performance. Consequently, they are hard to simulate real-world dynamics, where outcomes must update instantaneously based on historical context and current actions. To address this, we present Matrix-Game 2.0, an interactive world model generates long videos on-the-fly via few-step auto-regressive diffusion. Our framework consists of three key components: (1) A scalable data production pipeline for Unreal Engine and GTA5 environments to effectively produce massive amounts (about 1200 hours) of video data with diverse interaction annotations; (2) An action injection module that enables frame-level mouse and keyboard inputs as interactive conditions; (3) A few-step distillation based on the casual architecture for real-time and streaming video generation. Matrix Game 2.0 can generate high-quality minute-level videos across diverse scenes at an ultra-fast speed of 25 FPS. We open-source our model weights and codebase to advance research in interactive world modeling.
1 Introduction
Interactive world models can support real-time simulation, but existing approaches face latency and error-accumulation challenges. Matrix-Game 2.0 combines action control, causal few-step diffusion, and scalable annotated-data production to generate interactive videos at 25 FPS.
- Motivation: Interactive video generation models can respond causally to continuous camera and keyboard inputs, making them promising world models for real-time interaction.World models can reduce real-world trial costs and facilitate interactive simulation across applications such as game engines, autonomous driving, and spatial intelligence.
- Challenges: Bidirectional diffusion models are unsuitable for streaming because each frame requires processing the entire video, with quadratic frame-length scaling and many denoising iterations.These requirements make long-video generation computationally intensive and economically impractical.
- Challenges: Auto-regressive video diffusion models generate subsequent frames from prior frames but can accumulate errors, degrading video quality over time.This directly limits reliable long-video synthesis.
- Matrix-Game 2.0: Matrix-Game 2.0 integrates action-control modules with Self-Forcing-based causal few-step distillation and KV caching for real-time generation.The architecture is designed for temporal consistency and precise action controllability, including scenes beyond the training distribution.
- Matrix-Game 2.0: 25 FPS generation on a single H100 GPU supports minute-long interactive videos with temporal consistency and precise action controllability.The model is presented as a framework for real-time simulation and interaction, with code and pretrained weights slated for release.
- Data Production: The data pipeline uses Unreal Engine and GTA5 systems to produce large-scale frame-level annotations aligned with visual content, controls, and dynamic in-game interactions.It includes navigation and camera-control modules in Unreal Engine and synchronized interaction capture through Script Hook in GTA5.
2 Related Work
Video generation research has advanced through bidirectional and auto-regressive diffusion models, while world models use video diffusion to learn dynamics and causality. Long and real-time generation remains constrained by data, computation, and sampling efficiency.
- Video Generation: Recent video diffusion models synthesize high-quality, temporally coherent, and substantially longer videos using bidirectional transformers and auto-regressive models.
- World Models: Video diffusion-based world models aim to learn physical laws, object dynamics, and causality for complex environment simulation.
- Controllability: Controllable video generation includes scene control through text, images, or 3D priors and action control through camera angles or trajectories.
- Long Video Generation: Most video generation models remain limited to videos of ≤5 seconds because long-video training data are scarce and computation is costly.Long-context methods use either overlapping short segments or auto-regressive generation.
- Long Video Generation: Auto-regressive diffusion methods such as Diffusion Forcing, CausVid, and Self-Forcing support promising long-video synthesis but remain largely focused on conventional text-to-video generation.
- Real-Time Generation: Real-time generation commonly combines higher VAE compression, diffusion distillation, or KV caching with causal transformers to infer frames auto-regressively.These approaches reduce decoding or sampling costs to achieve generation faster than or closer to the duration of the video.
3 Data Pipeline Development
The data pipeline combines Unreal Engine and GTA5 systems to produce diverse, synchronized interactive video data. It integrates navigation, camera and input control, agent behavior, curation, and concurrent processing for scalable collection.
- Pipeline Overview: The pipeline targets synchronized keyboard and camera annotations while supporting static and dynamic scenes from Unreal Engine and GTA5.It addresses precise alignment between visual content and control signals for interactive training data.
- Unreal Engine Data Production: Unreal Engine receives a navigation mesh and 3D scene, then records automated movement, camera control, visual data, and action annotations.Outputs are exported through integrated MP4 encoding and CSV generation.
- Unreal Engine Data Production: Navigation Mesh-based Path Planning supports deterministic, adaptive NPC movement and diverse trajectories, with customized optimizations reducing average query latency to less than 2 ms.Controlled stochasticity adds varied interaction dynamics while preserving reproducibility.
- Agent Behavior: RL-trained agents are combined with collision-based rules to produce realistic, diverse interaction patterns while maintaining physical plausibility and collection reliability.The reward combines collision avoidance, exploration efficiency, and trajectory diversity.
- Input and Camera Control: The Unreal Engine system synchronizes multiple keyboard inputs with rendered frames and uses double-precision quaternion calculations to reduce camera rotation errors to an effectively negligible level.Input events are buffered against frame timing for training alignment.
- Curation and Acceleration: Data curation removes temporally redundant frames and invalid zero- or negative-velocity samples, while multi-thread execution enables dual-stream production on one RTX 3090 GPU.Trajectory examples illustrate planned agent paths in complex scenarios.
- GTA5 Data Recording: GTA5 recording uses Script Hook integration to capture visual content and user actions synchronously, including frame-accurate mouse and keyboard operations.The system combines agent behaviors, the game environment, video capture, CSV collection, and temporal synchronization.
4 Methods
Matrix-Game 2.0 uses a vision-driven, language-free foundation model with frame-level action control, then distills it into a causal few-step autoregressive diffusion model. Self-Forcing and KV caching support real-time long-video generation while addressing training-inference mismatch and error accumulation.
- Causal Distillation: The bidirectional foundation model is transformed into a causal few-step autoregressive diffusion model through Self-Forcing for real-time long-video synthesis.Student initialization and DMD-based Self-Forcing training form the two distillation phases.
- Design Motivation: Matrix-Game 2.0 removes language input and models spatial structures and dynamic patterns directly from images.The design emphasizes visual and physical understanding without linguistic semantic scaffolding.
- Foundation Model Architecture: The foundation architecture derives from Wan’s image-to-video design, removes the text branch, and predicts next frames from visual content and corresponding actions.A 3D causal VAE compresses video spatially and temporally before DiT generation and decoding.
- Action Control: Frame-level action control injects discrete keyboard movement and continuous mouse viewpoint signals into the diffusion transformer.Mouse actions are concatenated with input latent representations.
- Self-Forcing Training: Self-Forcing conditions each frame on previously self-generated outputs, aligning student and teacher distributions while mitigating training-inference gaps and error accumulation.The student is initialized using ODE trajectories before subsequent distillation training.
- Streaming Generation: A rolling KV cache maintains recent latents and action embeddings, evicting older tokens to support efficient sequential and infinite-length generation.The cache window is constrained in image-to-video inference to reduce reliance on an excluded first frame.
5 Experiments
Experiments evaluate Matrix-Game 2.0 across Minecraft, wild scenes, GTA5 driving, and TempleRun, including qualitative comparisons, quantitative benchmarks, and acceleration studies. The model maintains long-video quality and controllability while achieving 25 FPS, with cache-size trade-offs and domain-specific limitations.
- Minecraft Scene Results: Matrix-Game 2.0 maintains excellent performance throughout extended Minecraft generation, whereas Oasis degrades after several dozen frames.Quantitative results improve across most evaluation dimensions, with marginally lower scene consistency and action smoothness scores.
- Wild Scene Results: Matrix-Game 2.0 maintains stable style fidelity in wild scenes while YUME develops artifacts and color-saturation issues after several hundred frames.YUME’s generation speed is described as slow for interactive world modeling.
- Wild Scene Results: Matrix-Game 2.0 maintains robust action controllability in out-of-domain wild scenes, while YUME shows significantly degraded action control.Static collapsed outputs from YUME may inflate object-consistency and scenario-consistency scores.
- More Qualitative Results: Qualitative results show minimal quality degradation in long videos and domain adaptability across GTA5 driving and TempleRun scenarios.These results are presented as evidence of potential as a foundation framework for world modeling.
- Different KV-cache Local Size: 6-frame KV caches provide superior long-term generation quality, while larger caches introduce earlier visual artifacts through over-reliance on cached information.The study describes a trade-off between richer historical context and distortion or degradation artifacts.
- Comparative Analysis of Acceleration Techniques: 25 FPS throughput is achieved while maintaining comparable generation-quality metrics through combined diffusion and VAE acceleration techniques.The reported modifications include VAE caching, selective action modules, and reducing denoising steps from 4 to 3.
6 Conclusion
Matrix-Game 2.0 combines scalable action-annotated data production with action-conditioned, Self-Forcing-distilled autoregressive diffusion for real-time interactive video generation. It achieves strong visual quality, action controllability, and long-term interaction performance, while retaining limitations in OOD generalization, resolution, and explicit history preservation.
- 6 Conclusion: The data pipeline uses Unreal Engine and GTA5 recording frameworks to produce scalable, high-fidelity action-annotated video data.The authors present this pipeline as addressing limitations in obtaining interactive training data.
- 6 Conclusion: The autoregressive diffusion framework combines action-conditioned modulation with Self-Forcing distillation and reaches 25 FPS for human-in-the-loop interaction.The approach is paired with diffusion-process and VAE optimizations.
- 6 Conclusion: Larger KV-cache local sizes cause artifacts in long sequences, while smaller sizes balance visual quality and content fidelity.The cache-size study identifies a practical trade-off in contextual-history selection.
- 6.1 Limitations: Out-of-domain scenes can produce over-saturated or degraded results under prolonged camera movements.The limitation is illustrated in the reported bad cases.
- 6.1 Limitations: The 352×640 output resolution is below that of state-of-the-art video generation models, and long generations still lack explicit history-preservation mechanisms.The authors identify both issues as future research directions.
- 6.1 Limitations: The authors propose expanded training domains, architecture scaling, and compatible memory retrieval as pathways for improving generalization, resolution, and history preservation.These directions are intended to improve practicality for real-world applications without compromising real-time performance.