Source-linked AI summary
Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
Zile Wang, Zexiang Liu, Jiaxing Li, Kaichen Huang, Baixin Xu, Fei Kang, Mengyin An, Peiyu Wang, Biao Jiang, Yichen Wei, Yidan Xietian, Jiangbo Pei, Liang Hu, Boyi Jiang, Hua Xue, Zidong Wang, Haofeng Sun, Wei Li, Wanli Ouyang, Xianglong He, Yang Liu, Yangguang Li, Yahui Zhou
TL;DR
Interactive world models still struggle to combine memory-enabled long-term consistency with high-resolution real-time generation. Matrix-Game 3.0 co-designs industrial-scale data, error-aware and camera-aware memory modeling, and training–inference-aligned distillation with system acceleration. It achieves up to 40 FPS at 720p with a 5B model while maintaining stable memory consistency over minute-long sequences.
Problem
Existing approaches struggle to achieve memory-enabled long-term temporal consistency and high-resolution real-time generation together.
Method
Matrix-Game 3.0 combines industrial-scale interactive data, an error-aware base model, camera-aware long-horizon memory, aligned few-step distillation, and real-time acceleration.
Results
40 FPS at 720p with a 5B model is achieved while stable memory consistency is maintained over minute-long sequences.
Takeaways & Limitations
Matrix-Game 3.0 provides a practical pathway toward industrial-scale deployable interactive world models.
Takeaways & Limitations
Maintaining long-context sequences during training incurs substantial computational overhead.
Abstract
from arXiv · showhide
With the advancement of interactive video generation, diffusion models have increasingly demonstrated their potential as world models. However, existing approaches still struggle to simultaneously achieve memory-enabled long-term temporal consistency and high-resolution real-time generation, limiting their applicability in real-world scenarios. To address this, we present Matrix-Game 3.0, a memory-augmented interactive world model designed for 720p real-time longform video generation. Building upon Matrix-Game 2.0, we introduce systematic improvements across data, model, and inference. First, we develop an upgraded industrial-scale infinite data engine that integrates Unreal Engine-based synthetic data, large-scale automated collection from AAA games, and real-world video augmentation to produce high-quality Video-Pose-Action-Prompt quadruplet data at scale. Second, we propose a training framework for long-horizon consistency: by modeling prediction residuals and re-injecting imperfect generated frames during training, the base model learns self-correction; meanwhile, camera-aware memory retrieval and injection enable the base model to achieve long horizon spatiotemporal consistency. Third, we design a multi-segment autoregressive distillation strategy based on Distribution Matching Distillation (DMD), combined with model quantization and VAE decoder pruning, to achieve efficient real-time inference. Experimental results show that Matrix-Game 3.0 achieves up to 40 FPS real-time generation at 720p resolution with a 5B model, while maintaining stable memory consistency over minute-long sequences. Scaling up to a 2x14B model further improves generation quality, dynamics, and generalization. Our approach provides a practical pathway toward industrial-scale deployable world models.
1 Introduction
Matrix-Game 3.0 addresses the need for real-time generation with long-horizon spatiotemporal consistency through coordinated advances in data, modeling, and deployment. It reaches 720p real-time generation while maintaining minute-long memory consistency.
- Motivation: Real-time generation with long-horizon spatiotemporal consistency is a foundational requirement for practical interactive world models.Existing short-video diffusion models lack this capability, causing incoherent segments or prohibitive latency in deployment.
- Results: Matrix-Game 3.0 reaches 40 FPS at 720p with a 5B model while maintaining stable memory consistency over minute-long sequences.Scaling to 28B further improves generation quality, dynamics, and generalization.
- Approach: The framework co-designs data, modeling, and deployment to address long-horizon memory, coherence, controllability, and error accumulation.The three factors motivate a coordinated solution rather than an isolated component improvement.
- Data: Its industrial-scale data engine combines Unreal Engine 5 synthesis, automated AAA-game capture, and diverse real-world video corpora.The engine is designed to provide large-scale interactive supervision with paired visual, action, and camera information.
- Deployment: Deployment uses multi-segment distillation inspired by DMD and Self-Forcing together with acceleration techniques for streaming inference.The stated goal is to reduce error accumulation while enabling real-time high-resolution generation.
2 Related Works
Interactive world models extend video generation to action-conditioned state–environment transitions, but existing systems still face challenges in long-horizon stability, controllability, and efficient generation. Recent work demonstrates increasing feasibility while leaving important memory, openness, and deployment limitations.
- Video Generation Models: Diffusion Transformer video models support scalable high-resolution synthesis but are primarily designed for offline generation without explicit action or interaction modeling.Their long-horizon consistency limitations are also noted in the supplied passage.
- Long-Horizon Generation: Autoregressive causal diffusion and KV-cache reuse improve efficiency but do not fundamentally prevent error accumulation and temporal drift.Small prediction errors can compound over time during long-horizon generation.
- Interactive World Models: Interactive world models condition future observations on both past visual context and external actions.This setting jointly requires controllability, long-horizon consistency, and real-time inference.
- Interactive World Models: Genie-3 demonstrates approximately 24 FPS generation at 720p with minute-level coherence, but remains not open source and has unclear details.Earlier Genie systems are described as having short-term memory or long-horizon instability.
- Open-Source Systems: Open-source systems including OASIS, Matrix-Game 2.0, and WorldPlay advance unified interactive modeling and real-time control.Matrix-Game 2.0 uses causal autoregressive diffusion and few-step distillation for streaming interactive video generation.
3 Method
Matrix-Game 3.0 combines error-aware action-conditioned generation, camera-aware memory, aligned few-step distillation, and inference acceleration to support long-horizon interactive video. The framework targets stable memory consistency and real-time 720p generation, with larger scaling improving quality and dynamics.
- Framework Overview: The framework integrates error-aware interaction, camera-aware memory, training–inference-aligned distillation, and real-time acceleration.These four components jointly address long-horizon memory and high-resolution real-time generation.
- Error-Aware Interactive Base Model: The bidirectional base model predicts current latent frames from past latent history while injecting explicit keyboard and mouse action conditions.Keyboard actions use cross-attention, whereas continuous mouse signals use self-attention.
- Error-Aware Interactive Base Model: Training collects prediction residuals and reinjects perturbed historical contexts, teaching the base model to remain robust to imperfect autoregressive histories.The formulation preserves compatibility with downstream distillation and supports self-correction during rollout.
- Long-Horizon Memory: Retrieved memory frames, short-term history, current noisy frames, action conditions, and geometric conditions are jointly processed to preserve long-range spatial consistency.The memory design combines structured retrieval, unified self-attention, geometry-aware conditioning, and self-corrective memory training.
- Few-Step Distillation: The student performs multi-segment self-generated rollouts using prior segment tails and online memory retrieval, aligning few-step training with inference.This alignment provides the foundation for streaming inference in the full pipeline.
- Real-Time Inference: 40 FPS at 720p is achieved with an asynchronous deployment using eight GPUs for DiT inference and one GPU for VAE decoding.The inference pipeline also uses a GPU retrieval approximation that avoids explicit 3D intersection computation and remains efficient for long iterative generation.
- Large Model Scaling Up: Minute-long sequences benefit from MoE-28B scaling through improved frame-to-frame temporal consistency and context preservation.The larger model targets high-fidelity, minute-long video generation.
4 Data System
Matrix-Game 3.0 builds a scalable data system combining synthetic, AAA-game, and real-world video sources with synchronized pose, camera, action, and textual annotations. Automation, filtering, and reusable recording layers provide diverse, high-quality training data.
- Data System: The data system integrates Unreal Engine generation, AAA-game recording, real-world acquisition, and unified annotation and filtering.The resulting data spans static and dynamic scenes across multiple viewpoints.
- Unreal Engine-based Data Production: Tick-level synchronization captures rendered frames, player pose, camera pose, and discrete actions within the same engine callback.This produces exactly zero temporal alignment error, with double-precision quaternion arithmetic for sub-degree rotation accuracy.
- Unreal Engine-based Data Production: More than 108 character variants and randomized camera viewpoints increase visual diversity across recording sessions.The system supports both first-person and third-person perspectives.
- Scalable AAA Game Data Recording System: The AAA-game pipeline uses reusable recording and output layers across multiple titles, exporting per-frame video, actions, camera parameters, and pose information.Adding a new game requires only a game-specific agent layer.
- Scalability & Reliability: Overall data accuracy exceeds 99%, while automated recovery, segmented recording, monitoring, and environment randomization support reliable scaling.The pipeline also combines geometric, motion, camera-speed, and perceptual filtering to curate the training set.
5 Experiments
Experiments evaluate controllability, memory, scaling, distillation, and acceleration. The models maintain stable scene behavior and long-range revisitation, while system optimizations enable high-throughput inference and pruned VAE decoding.
- Base Model: The interactive base model provides action controllability while preserving background stability and camera-consistent zoom relationships.These qualitative results indicate stable responses under camera movement.
- Base Model: Long-range memory enables faithful recovery of previously observed scene structures and appearance when reversed actions revisit earlier viewpoints.Recovered details include local geometry, object configuration, facade patterns, and texture-level cues.
- Scaling and Long-Horizon Generation: The 28B model maintains temporal consistency in scene layout, character identity, and object relations across diverse long-horizon third-person scenes.The examples also preserve vivid motion dynamics under continuous camera and action changes.
- Distilled Model: The distilled model inherits memory capability, reproducing previously occluded content while generating newly emerged scenes without noticeable drift.The evaluation deliberately includes viewpoint and scene-content revisitation.
- Real-Time Inference: The full acceleration configuration reaches up to 40 FPS, with INT8 quantization, VAE pruning, and GPU retrieval contributing jointly to throughput.Removing GPU retrieval causes the largest FPS drop, while H-series GPUs outperform A-series GPUs under matched parallel settings.
- Real-Time Inference: The 50% pruned MG-LightVAE preserves the main scene structure and visual content while reducing VAE decoding cost.Reconstruction quality and efficiency are compared against the original Wan2.2 VAE and a 75% pruned variant.
6 Conclusion
Matrix-Game 3.0 unifies long-horizon consistency, high-resolution generation, and real-time inference through coordinated data, modeling, memory, distillation, and system optimizations. It reaches 40 FPS at 720p with a 5B model while maintaining minute-long spatiotemporal consistency, and larger models improve quality, dynamics, and generalization.
- Conclusion: The framework jointly targets long-horizon consistency, high-resolution generation, and real-time inference through co-designed data, modeling, and deployment components.Its components include error-aware self-correction, camera-aware memory, multi-segment distillation, and system-level acceleration.
- Conclusion: 40 FPS at 720p is achieved with a 5B model while maintaining stable spatiotemporal consistency over minute-long sequences.Scaling to larger models further improves generation quality, dynamic behavior, and generalization capability.