PaperlayerView exact revision on arXiv

Source-linked AI summary

The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, Hongyang Zhang

arXiv:2412.03568v1cs.AI

TL;DR

Prior world simulators faced limitations in realistic fidelity, continuous generation, and real-time control. The Matrix adapts a pre-trained video DiT with methods for infinite-length generation and faster sampling; it produces high-quality, controllable worlds, with SCM raising rendering speed from 0.8 FPS to 8 - 16 FPS.

  • Problem

    Prior world simulators were limited in real-world fidelity, sequence length, or real-time generation, and continuous movement data can be difficult to collect in real-world settings.

    Source: §1 · p. 2

  • Method

    The Matrix adapts a pre-trained video DiT, using Swin-DPM for infinite-length generation and a fine-tuned Stream Consistency Model to accelerate inference.

    Source: §3 · p. 3, §3.1 · p. 4

  • Results

    The Matrix generates infinitely long worlds whose visual quality surpasses previous world simulators while retaining strong control; SCM raises rendering speed from 0.8 FPS to 8 - 16 FPS.

    Source: §4.2 · pp. 8–9

  • Takeaways & Limitations

    The Matrix demonstrates zero-shot generalization to scenarios absent from both gaming data and real-world sources, supporting game-data-based world models for unseen settings.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

We present The Matrix, the first foundational realistic world simulator capable of generating continuous 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives, enabling immersive exploration of richly dynamic environments. Trained on limited supervised data from AAA games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains -- deserts, grasslands, water bodies, and urban landscapes -- in continuous, uncut hour-long sequences. Operating at 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting--an environment present in neither gaming data nor real-world sources. This approach showcases the potential of AAA game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.

1. Introduction

The Matrix addresses world-simulation gaps in fidelity, continuous generation, and speed with a controllable 720p simulator. Its contributions include Swin-DPM, GameData, and real-time, frame-level control with domain generalization.

1. Introduction

  • World models can reduce manual configuration and support scalable workflows against AAA game development’s high cost and extensive asset preparation.Traditional engines require expertise, intensive asset preparation, and tuning, while new titles often need comprehensive redesigns.Source: §1 · pp. 1–2
  • Prior world models often lacked real-world fidelity, continuous long-form generation, or real-time speed, motivating The Matrix’s combined high-fidelity, real-time approach.The introduction contrasts short video sequences and slow generation with the paper’s 1280×720 simulator and cites Genie at 1 FPS.Source: §1 · p. 2

1.1. Our Contributions

  • The Matrix combines infinite-length 720p video, real-time interactive control, and domain generalization, alongside Swin-DPM and GameData for video generation and paired training data.The Matrix has 2.7B parameters; GameData captures in-game states from CPU memory alongside video frames.Source: §1.1 · p. 2

1.2. Technical Advantages of The Matrix

  • The Matrix targets consistent, infinitely long AAA-quality video at 1280 × 720, with frame-level control at 8 - 16 FPS and generalization to real-world settings.Its training uses small amounts of supervised AAA game data and large amounts of unsupervised internet video.Source: §1.2 · p. 2

2. Related Work

Related work distinguishes world models for agent learning from interactive world simulation. Prior simulators often addressed only some of video length, resolution, and domain generalization, whereas The Matrix is presented as combining them with real-time frame-level control.

World Model for Agent Learning

  • World models for agent learning reconstruct observations, rewards, and continuation signals, then predict future states for reinforcement-learning policy optimization.Prior work reports sample-efficiency gains over direct learning from real environments and includes recurrent models, Dreamer, TD-MPC, and MuZero.Source: §2 · p. 2

World Simulation

  • World simulation focuses on human interaction through rendering, control, and domain generalization, with video-level signals and frame-level adjustments as distinct approaches.Video-level approaches may stitch independently generated clips, while frame-level control allows adjustments every few frames.Source: §2 · p. 2
  • Prior simulators typically addressed only video length, high resolution, or domain generalization, while The Matrix is presented as combining these with real-time frame-level control.Table 1 compares recent game-simulation models; its caption identifies infinite video, AAA quality, high resolution, frame-level control, and domain generalization as distinguishing features.Source: §2 · p. 3, Table 1 · p. 3

3. Methods

The Matrix adapts a pre-trained video DiT with Swin-DPM for continuous generation and a Stream Consistency Model for real-time sampling. Its interactive control and source-data pipeline combine labeled game data with unlabeled internet video.

3. Methods

  • The Matrix adapts a pre-trained video DiT with Swin-DPM for indefinitely long video generation and fine-tunes a Stream Consistency Model to accelerate inference to real time.The approach leverages the DiT’s pre-existing knowledge and generation quality.Source: §3 · p. 3

3.1. Model Components

  • The three components are an Interactive Module for user intentions, Swin-DPM for infinite-length generation, and SCM for faster sampling; training proceeds through three stages.The first two stages use labeled gaming and unlabeled internet video data; final SCM training focuses on labeled gaming data.Source: §3.1 · pp. 3–4
  • The Interactive Module translates keyboard inputs into natural-language guidance, embeds that guidance with video tokens, and applies cross-attention for frame-level control.Cross-attention runs after each odd-numbered self-attention step; unlabeled data receives a default movement description.Source: §3.1 · p. 4
  • Swin-DPM uses a sliding temporal window and stride-1 token generation; cached low-noise tokens rejoin denoising to maintain continuity across windows.Training and inference warm up on the first w tokens; those tokens are discarded at inference.Source: §3.1 · p. 4

3.2. Construction of the Source Dataset

  • The source dataset comprises six-second continuous-scene clips captioned with GPT-4o, yielding 750k labeled and 1.2 million unlabeled samples at 60 FPS.Labeled game data is refined to balance possible game states.Source: §3.2 · p. 5
  • The GameData Platform aligns in-game status and keyboard actions with recorded video, filters stuck or irregular frames, and removes game UIs and HUDs.Cheat Engine captures status from CPU memory, Reshade standardizes the visual data, and OBS records frames.Source: §3.2 · p. 5, Figure 4 · p. 5

4. Experiments

The experiments evaluate frame-level interaction, long-horizon generation, real-time rendering, and generalization to unseen scenes and objects. The Matrix generates hour-long or longer worlds and improves rendering speed to 8–16 FPS, while retaining strong control precision and demonstrating generalization beyond its Source data.

4. Experiments

  • The experiment training sequence warms up the base DiT, trains the Interactive Module, fine-tunes with Swin-DPM, and then distills with SCM.Training details are reported as an experimental setup rather than a comparative result.Source: §4 · pp. 5–6

4.1. Precise Frame-Level Interactions

  • The Interactive Module is evaluated on three scenarios using visual-quality and movement-control metrics, with keyboard inputs and prompts drawn from a fixed test set for control evaluation.The metrics include FID, FVD, CLIP Score, Move-PSNR, and Move-LPIPS.Source: §4 · p. 6, §4.1 · p. 6
  • The Interactive Module produces vivid, dynamic outputs across the tested scenarios and models physical effects such as desert dust and river splashes.The evaluation covers Forza Horizon 5 driving, Cyberpunk 2077 walking, and a DROID robotic-arm task.Source: §4.1 · p. 6

4.2. Infinete-Horizon World Generation

  • One-minute examples span desert, river, grassland, snow, and day-to-night transitions, while supplementary videos include half-hour examples and the authors report truly infinite-length generation.Changing the DiT prompt adapts the environment during generation.Source: §4.2 · p. 8
  • Rendering speed rises from 0.8 FPS to 8 - 16 FPS with SCM, alongside a slight visual-quality trade-off and minimal loss in control precision.The reported comparison concerns SCM integration with The Matrix.Source: §4.2 · p. 9

4.3. Generalization to Out-of-Distribution Worlds

  • The Matrix controls a car in indoor scenes absent from the Source dataset and makes a prompted human move in response to keyboard inputs.The paper also shows long videos generated from ordinary prompts with the Interactive Module disabled.Source: §4.3 · p. 9

5. Conclusion

The conclusion presents The Matrix as a simulator for infinitely long, high-fidelity video with real-time control and zero-shot generalization. It emphasizes game data, complemented by real-world footage, as a route toward robust world models with minimal supervision.

5. Conclusion

  • The Matrix is presented as a real-world simulator for infinitely long, high-fidelity video streams with precise real-time control.The conclusion describes immersive exploration across diverse terrains at 8 - 16 FPS.Source: §5 · p. 9
  • Trained on AAA game data blended with real-world footage, The Matrix demonstrates zero-shot generalization to unseen scenarios.Source: §5 · p. 9
  • The conclusion highlights game data's potential for building robust world models with minimal supervision.Source: §5 · p. 9

A. Details in Experiments

The DiT backbone combines patch, caption, and timestep embeddings with repeated transformer blocks and a linear output head. Its modules process video latents, text captions, timestep information, and network features.

A.1. DiT Backbone

  • The backbone is adapted from publicly available DiT models and comprises patch, caption, and timestep embeddings, 32 DiT blocks, and a linear output head with LayerNorm.Source: §A.1 · p. 10
  • The backbone uses a 3D convolution and reshape to turn video latents into 2,048-feature tokens while capturing spatial and temporal information.The convolution kernel is 1 × 2 × 2.Source: §A.1 · p. 10
  • The caption module processes T5-encoded caption tokens through a two-layer FFN, producing 2,048-dimensional output features.Source: §A.1 · p. 10
  • The timestep module applies sinusoidal embedding followed by a two-layer FFN, with 2,048-dimensional hidden and output features.Source: §A.1 · p. 10
  • Each DiT block combines self-attention, cross-attention linking conditions with self-attention outputs, and a two-layer GELU feed-forward network.Source: §A.1 · p. 10

A.2. Training Details

Training proceeds through four stages: unlabeled-source warm-up, Interactive Module training, Swin-DPM fine-tuning, and SCM distillation. The shared setup uses batch size 32, learning rate 1 × 10−5, and bfloat16 mixed precision.

A.2. Training Details

  • The four training stages are unlabeled-source warm-up, Interactive Module training, Swin-DPM fine-tuning, and SCM distillation.Source: §A.2 · p. 10
  • Shared settings use an overall batch size of 32, learning rate 1 × 10−5, and bfloat16 mixed-precision training.Source: §A.2 · p. 10
  • Warm-up fine-tunes all base-DiT linear layers with LoRA on unlabeled Source data for 20,000 steps at rank 128.Source: §A.2 · p. 10
  • The Interactive Module is trained for 20,000 steps while base-DiT parameters are frozen, with one module placed after each pair of consecutive DiT blocks.There are 16 Interactive Modules in total.Source: §A.2 · p. 10
  • Swin-DPM fine-tunes all model parameters for 60,000 steps, followed by 10,000 steps of one-stage guided consistency distillation using the fine-tuned model as teacher.Distillation uses Classifier-Free Guidance and an Euler solver with a single-step size of 25/1000.Source: §A.2 · p. 10

B. The Source Dataset

The GameData Platform collects game video paired with control signals through coordinated control, simulation, and observation components, using different control systems for simpler and more complex game environments.

B.1. The GameData Platform

  • Automatic controls detect when a game is stuck using coordinate movement and restart recording, while manual controls support movement through keyboards, mice, or gamepads.The manual pipeline translates these signals through a socket server into keyboard events on the cloud server.Source: §B.1 · p. 11
  • Games run on cloud servers whose images can be copied to create instances; low-quality recordings are filtered, NPCs and moving vehicles removed, and scenes adjusted with Reshade.Source: §B.1 · p. 11
  • OBS records gameplay at 2560 × 1600, while a Reshade plugin removes GUI-related shaders without altering the native video.Source: §B.1 · p. 11
  • Forza Horizon 5 uses automatic random movement and steering, while Cyberpunk 2077 uses manual control because its terrain complexity makes random signals unsuitable.Manual inputs are sent through cloud servers; recorded control signals and observed actions were generally no more than three frames apart.Source: §B.1 · p. 11

B.2. The Source Dataset

The Source dataset combines recordings from Forza Horizon 5 and Cyberpunk 2077, covering varied landscapes and urban scenes with game-specific control signals.

B.2. The Source Dataset

  • The dataset description covers basic information, conversion of platform data into the desired format, and filtering of undesirable data.Source: §B.2 · p. 11

B.2.1. Basic Information

  • The Source contains approximately 1,200,000 video-control pairs from Forza Horizon 5 and around 1,000,000 from Cyberpunk 2077, with videos about six seconds long at 60 FPS.Source: §B.2.1 · p. 12
  • Forza Horizon 5 recordings cover deserts, oceans, water bodies, grasslands, and fields, while Cyberpunk 2077 recordings focus on urban environments with tall buildings.Source: §B.2.1 · p. 12
  • Forza Horizon 5 has three forward-driving controls, whereas Cyberpunk 2077 has forward, left, right, look-up, and look-down controls.Source: §B.2.1 · p. 12

B.2.2. Annotation Methods

The annotation process segments long recordings into short clips, aligns their control signals, and adds captions generated from sampled video frames and manually corrected.

B.2.2. Annotation Methods

  • Original GameData Platform recordings last around ten minutes, so FFmpeg segments them into six-second clips for training.Source: §B.2.2 · p. 12
  • The corresponding control signals are extracted from each complete recording after its video is segmented.Source: §B.2.2 · p. 12
  • InternVL captions each clip from 12 uniformly sampled key frames, after which people manually correct obvious errors.Source: §B.2.2 · p. 12

B.2.3. Filtering Methods

The filtering methods remove problematic training data, including stuck or collision-affected clips and clips dominated by the most frequent control signal.

B.2.3. Filtering Methods

  • The filtering section introduces five methods for removing undesirable data that could disrupt training.Many of the listed issues do not arise for Cyberpunk 2077 because its data was collected by people rather than automatically.Source: §B.2.3 · p. 12
  • Control balancing repeatedly removes clips containing the highest proportion of the most frequent signal until the distribution is relatively balanced.Source: §B.2.3 · p. 12
  • Forza Horizon 5 collision-affected clips are discarded using significant short-term acceleration variations as an indicator of collisions.Source: §B.2.3 · p. 12
  • Forza Horizon 5 clips are discarded when the car travels less than a threshold distance, indicating it is stuck.Source: §B.2.3 · p. 12

Detect and Remove the Data with Mismatched Motion

The supplied appendix material describes data-cleaning and curation procedures, then points to additional demonstrations of precise control, generalization, and long-video generation.

Detect and Remove the Data with Mismatched Motion

  • The data pipeline removes samples with large acceleration–movement direction mismatches and filters collision artifacts using consecutive-frame pixel changes.Horizon5 scenario quantities are relatively balanced; Cyberpunk 2077 has a relatively high share of forward-movement signals, while Horizon5 control signals are evenly distributed.Source: Detect and Remove the Data with… · p. 13, Figure A2 · p. 13

B.3.1. Basic Information

  • DROID contains 76k demonstration trajectories spanning 350 hours, 564 scenes, and 86 tasks, collected over 12 months by 50 collectors.Collection used the same hardware setup across 13 institutions, including a Franka Panda arm, stereo cameras, and teleoperation equipment.Source: §B.3.1 · p. 13

Remove Overly Complex Scenes and Balance Different

  • DROID curation selects 50 less-complex scenes per category, removes semantic outliers, balances scene samples, excludes failed trajectories, and discards videos with over 20% arm-absent frames.Gripper position is used as a condition for each training frame.Source: Remove Overly Complex Scenes and… · pp. 13–14

C.3. Long Video Generation

  • The appendix provides long-video generation demonstrations in supplementary video files, which are heavily compressed to meet the file-size limit.Source: §C.3 · p. 14

C.1. Precise Frame-Level Control

C.2. Generalization

Loading 2412.03568v1…