PaperlayerView exact revision on arXiv

Source-linked AI summary

Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

Jihai Zhang, Runjia Qian, Kaiqin Kong, Biao Jiang, Puyi Wang, Siru Chen, Tianle Li, Guanjie Chen, Junyi Wu, Chubin Chen, Yuanhang Li, Kaichen Huang, Haobo Zhang, Yangguang Li, Wei Yu, Guibin Chen, Yu Cheng, Yahui Zhou

arXiv:2610.27999v1cs.CVcs.GR

TL;DR

Video-led interactive worlds need persistent state, executable rules, and low-latency interaction. Matrix-Game 4.0 uses an external game harness with a coding agent and autoregressive renderer; on an RTX 5090 it reports 83 FPS at 480p with one denoising step and 112 ms measured interaction latency.

  • Problem

    Video-generation-led worlds face challenges in persistent state, compounding consistency errors, long-horizon memory, and support for executable gameplay mechanics.

    Source: §1 · p. 2

  • Method

    Matrix-Game 4.0 separates execution from rendering with an engine for state and rules, an agent for game code and captions, and an autoregressive video renderer conditioned on engine proxies and memory.

    Source: §1 · p. 3, §4.3 · p. 9

  • Results

    83 FPS steady-state generation throughput at 480p with one denoising step and 112 ms measured key-press-to-display latency are reported on an RTX 5090.

    Source: §8 · p. 15

  • Takeaways & Limitations

    The system offers a technical basis for research and a preview of natural-language prototyping with engine-enforced mechanics and neural rendering.

    Source: §8 · p. 15

    The authors identify inference cost and hardware requirements, visual fidelity, higher-resolution output, multiplayer consistency, precise visual control, and broader genre coverage as areas for further work.

    Source: §8 · p. 15

Abstract

from arXiv · show

Moving from controllable videos to genuinely playable worlds requires persistent world state, executable game rules, and low-latency interaction. We outline a roadmap for the next generation of Matrix-Game that separates game execution from visual generation through an executable game harness, and introduce Matrix-Game 4.0 as the first system built on this design. A lightweight game engine maintains world state and executes interaction rules, a multimodal coding agent translates player ideas into game code and dynamic scene captions, and a renderer based on autoregressive video model transforms engine-rendered low-poly proxies into detailed visuals, conditioned on dynamic captions and memory references. This framework provides continuous guidance for visual generation and enables natural-language creation and modification of scenes and gameplay. Our renderer is trained on aligned low-poly and high-fidelity video pairs and distilled into a causal generator that generates in four-frame chunks with one or two denoising steps. Together with targeted inference optimizations, Matrix-Game 4.0 enables 480p, 60 FPS gameplay on a single NVIDIA RTX5090 with a key-press-to-display latency of 112ms. Single-step generation achieves 83FPS steady-state throughput with 18.2GiB of peak GPU memory usage. With fixed prompts, steady-state throughput also reaches 67FPS on an RTX5080 and 58FPS on an RTX5070Ti. We demonstrate customizable shooter and driving games with engine-enforced mechanics, and release code, model weights, and low-poly game assets to support research and creation of playable game worlds.

1 Introduction

Video-only interactive worlds face challenges in persistent state, consistency, memory, and gameplay mechanics. Matrix-Game 4.0 separates game execution from visual generation and reports real-time play on a single consumer GPU.

Motivation

  • Video-generation-led systems face four challenges: weak external correction, accumulating errors, growing frame-based memory costs, and limited gameplay mechanics.Their world state is largely implicit in generated pixels and context, and interaction often lacks executable rules, persistent consequences, and explicit objectives.Source: §1 · p. 2
  • The paper argues that visual realism alone does not establish causal or physical understanding, and identifies low-latency interaction, richer mechanics, and shared state as further needs.It frames the video model primarily as a renderer and assigns persistent state and rule execution to an external system.Source: §1 · p. 2

Framework

  • Matrix-Game 4.0 places the video model in an external harness: an engine maintains state and rules, while the agent creates game code and scene descriptions.The engine supplies structural, motion, and semantic conditions for rendering, so the video model need not maintain world state or execute rules.Source: §1 · p. 3

Contributions

  • 83 FPS steady-state generation throughput at 480p with one denoising step and 112 ms key-press-to-display latency are reported on an RTX 5090.The system also reports 60 FPS gameplay on that GPU, with fixed-prompt throughput of approximately 67 FPS on an RTX 5080 and 58 FPS on an RTX 5070 Ti.Source: §1 · p. 3
  • The system is presented as a reusable foundation for researchers and a preview of natural-language game-development workflows for developers.The paper describes natural-language prototyping and customization of scenes, interactions, and gameplay.Source: §1 · pp. 3–4

2 Roadmap

The roadmap divides playable-world generation among an engine that runs the game, a coding agent that builds and describes it, and a video model that renders it. This separation keeps execution distinct from appearance.

2 Roadmap

  • The architecture assigns world management to the engine, idea-to-code translation to the agent, and realistic future-frame generation to the video model.The modules exchange game rules, state, semantic context, structural references, and control signals.Source: §2 · p. 4

2.1 Game Engine: The World’s “Referee”

  • The engine maintains rules and persistent state, including off-screen objects, and supplies structural renderings that constrain layout, motion, and viewpoint.Game state such as health, inventory, collisions, and quest progress is explicitly maintained and enforced in code.Source: §2.1 · p. 4

2.2 Multimodal Coding Agent: The World’s “Scriptwriter” and “Translator”

  • The agent translates engine-defined scenes and outcomes into visual descriptions, and turns player ideas into game code, scene configurations, and mechanics.When players leave details open, it can propose content that the engine registers and thereafter tracks.Source: §2.2 · p. 4

2.3 Video Generation Model: The World’s “Painter”

  • The video model adds materials, lighting, atmosphere, and fine detail to structural renderings, using current controls and past frames in causal autoregressive generation.Few-step distillation and inference optimizations are used to reduce the delay between player actions and visual feedback.Source: §2.3 · p. 5

3 Data

The data pipeline pairs synchronized low-poly structure renders with high-fidelity video and metadata. Dynamic captions update from engine state so visual generation receives current semantic information alongside structural guidance.

3.1 Paired Low-Poly and High-Fidelity Capture

  • The UE5 dataset covers city, desert, castle, and air-combat scenes with synchronized RGB, low-poly, and depth video plus per-frame metadata.Captures are 1280 × 720; held-out city, desert, and castle clips come from complete sessions, while air-combat holds out whole capture batches.Source: §3.1 · p. 6
  • Aligned low-poly proxies preserve scene topology, rigid poses, and motion, while removing material, texture, reflection, and lighting detail.The paired data and semantic metadata provide supervision for mapping geometric structure to high-fidelity visuals and for spatiotemporal consistency in long rollouts.Source: §3.1 · p. 6

3.2 Dynamic Captions

  • Dynamic captions update with visible objects and their states because camera movement and events make a fixed description insufficient.The rule-based template uses engine-reported objects and states; captions change as objects leave view, pedestrians approach, or new objects appear.Source: §3.2 · p. 6, Figure 4 · p. 7
  • The low-poly proxy supplies position, layout, and motion, while captions supply object identity and state to constrain generation.Captions are reorganized when the player changes view, enters an area, or triggers an event.Source: §3.2 · p. 7

4 Rendering Model

The rendering model conditions chunk generation on proxy geometry, dynamic captions, and memory references. It is fine-tuned on paired clips, then adapted from bidirectional generation to causal autoregressive generation with few-step distillation.

4.1 Architecture

  • Each four-frame chunk combines a noisy latent with an encoded low-poly proxy, while causal DiT blocks attend to cached history and dynamic-caption text.One or two flow updates produce the clean latent, which TAEHV decodes into four frames; finished chunks are added to the cache without control input.Source: Figure 5 · p. 8

4.2 Supervised Fine-Tuning

  • The model is fine-tuned per game as a bidirectional video-to-video model on 81-frame paired-data windows conditioned on proxy, caption, first frame, and spatial-memory context.Context frames are selected from at least 40 frames before the window based on view overlap and camera pose; first frame, prompt, and proxy control are dropped probabilistically during training.Source: §4.2 · p. 9

4.3 Causal Few-Step Distillation

  • Causal few-step distillation proceeds from teacher-forced autoregressive training to causal consistency distillation and then distribution matching on the student's own rollouts.The teacher-forced stage trains each frame against preceding clean ground-truth frames; consistency distillation reduces generation to four steps, and the final stage trains two-step chunks with deployment-aligned memory and proxy controls.Source: §4.3 · p. 9

5 Real-Time Inference

Matrix-Game 4.0 reduces real-time rendering overhead through CUDA Graph replay, fused kernels, an efficient KV cache and attention, and asynchronous decoding. Together, these optimizations substantially improve throughput and key-to-frame latency on an RTX 5090.

Launch overhead

  • CUDA Graph replay records the DiT computation for a single runtime launch, while linear layers use FP8 weights.These changes target launch overhead.Source: §5 · p. 10

Fused kernels

  • Fused Triton kernels handle RoPE, normalization, and FP8 quantization; text-condition keys and values are cached until the caption changes.Query, key, and value projections share one input quantization.Source: §5 · p. 10

KV cache and attention

  • The video-history KV cache overwrites the oldest latent frame in a ring buffer, while attention computes window parts in parallel and merges them.Figure 6 also describes cached quantized keys and values across seven window ranges, combined by log-sum-exp.Source: §5 · p. 10, Figure 6 · p. 10

Decoding and I/O

  • Replacing the Wan VAE decoder with TAEHV and running model input and output asynchronously reduces decoding and I/O overhead.The renderer generates one latent frame, or four video frames, per chunk; short chunks keep action response short but increase CPU kernel-launch waiting.Source: §5 · pp. 9–10

Result

  • 4.5 to 83.4 FPS (19×) and 1110 ms to 112 ms (9.9×) are the steady-state rate and key-to-frame latency changes after cumulative inference optimizations.The benchmark uses 480 × 864, four-frame chunks, a 21-latent-frame KV window, and one RTX 5090.Source: §5 · p. 10, Table 2 · p. 11
  • At two steps, inference reaches 57 FPS; peak memory is about 18.2 GiB in both cases, and one-step inference denoises one four-frame chunk in 48 ms.Source: §5 · p. 11
  • With fixed prompts at 480 × 864, throughput reaches approximately 67 FPS on an RTX 5080 and 58 FPS on an RTX 5070 Ti.Memory use is approximately 12.8 GiB and 10.9 GiB, respectively.Source: §5 · p. 11

6 Evaluation

The evaluation tests held-out autoregressive rollouts, compares interactive-system latency and hardware, and demonstrates both visual consistency and coding-agent gameplay customization. The examples pair engine-enforced rules with video-model rendering.

6.1 Setup

  • Held-out rollouts run autoregressively for 961 frames (40 seconds at 24 FPS) at 480 × 864, starting from the ground-truth first frame and conditioned on proxy and caption.The section evaluates distilled causal models of two shooters and city driving, each using two denoising steps per chunk.Source: §6.1 · p. 11

6.2 Comparison with Recent Interactive Systems

  • Matrix-Game 4.0 has the lowest end-to-end latency among four compared systems—about one fifth that of the next system—and is the only one running on a single consumer GPU.Table 3 reports latency, hardware, and openness for recent interactive generation systems.Source: §6.2 · p. 11, Table 3 · p. 12

6.3 Qualitative Results

  • In three held-out rollouts shown at four time steps, generated frames retain proxy layout and camera path, add materials and combat effects, and keep scene and character appearance stable over 40 seconds.Source: §6.3 · p. 11

6.4 Gameplay Customization with the Coding Agent

  • The coding agent lets players customize terrain, objects, NPC behavior, interaction logic, and game modes through natural language.A shooter example adds health and ammunition controls, shield durability, a faster shotgun, a game-over condition, and sound effects.Source: §6.4 · pp. 11–12
  • Driving examples add a checkpointed time trial with collision penalties and a pursuit mode requiring interception of a target vehicle.The engine evaluates the rules, while the video model renders the engine output.Source: §6.4 · p. 12

7 Related Work

Related work spans interactive video world models, executable code-defined worlds, structure-conditioned neural rendering, and real-time autoregressive generation. Matrix-Game 4.0 places game dynamics in code and uses a video model for visuals.

7.1 Interactive Video World Models

  • Interactive world models have progressed from learned latent actions and autoregressive dynamics to playable environments, richer controls, and longer real-time video horizons.Examples include Genie, GameNGen, DIAMOND, Oasis, Hunyuan-GameCraft, and Matrix-Game 2.0 and 3.0.Source: §7.1 · p. 12

7.2 Code-Defined Worlds Rendered by Video Models

  • Code-defined world systems assign state and rules to executable code and use video models to render engine or program outputs.Matrix-Game 4.0 follows this division, targets interactive latency on one consumer GPU, and adds an agent that edits rules and prompts during play.Source: §7.2 · pp. 13–14

7.3 Neural Rendering and Structure-Conditioned Video Generation

  • Neural-rendering work uses graphics structure or semantic controls to guide learned visual generation; Matrix-Game 4.0 conditions its renderer on low-poly engine proxies and trains on pixel-aligned UE5 pairs.Prior approaches include G-buffer-conditioned rendering and structure-conditioned video generation.Source: §7.3 · p. 14

7.4 Real-Time Autoregressive Video Generation

  • Real-time autoregressive video generation draws on few-step distillation methods and systems components such as CUDA Graphs, split-KV decoding, FP8, Triton kernels, and TAEHV.The paper follows Causal Forcing++ to distill its bidirectional rendering model.Source: §7.4 · p. 15

8 Conclusion and Future Work

Matrix-Game 4.0 separates executable game logic from visual generation, while a coding agent supports natural-language creation and refinement of scenes and gameplay. The conclusion reports real-time rendering performance and outlines future work to reduce inference costs and expand system capabilities.

8 Conclusion and Future Work

  • A lightweight engine maintains persistent world state while a coding agent connects user ideas to executable game code for natural-language creation and iterative refinement.Source: §8 · p. 15
  • 83 FPS steady-state generation throughput at 480p with one denoising step and 112 ms key-press-to-display latency are reported on a single NVIDIA RTX 5090.Source: §8 · p. 15
  • Future work targets lower inference costs and hardware requirements, higher fidelity and resolution, multiplayer view consistency, finer visual control, and broader genres and scenes.Source: §8 · p. 15

Related papers

Loading 2610.27999v1…