Source-linked AI summary
Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory
Runjia Qian, Zile Wang, Jihai Zhang, Kai Zou, Wei Yu, Jiaxing Li, Zexiang Liu, Yaokun Li, Fei Kang, Kaichen Huang, Mengyin An, Haobo Zhang, Biao Jiang, Jiahua Wang, Haofeng Sun, Yang Liu, Yangguang Li
TL;DR
Interactive world models must maintain geometry, subject identity, and camera control during real-time long-horizon generation. Matrix-Game 3.5 addresses this with geometry-aware patch memory, static-dynamic disentanglement, and progressive causal distillation, achieving strong performance across scene recall, control, consistency, and interaction tasks while leaving persistent dynamic states and internal geometry grounding open.
Problem
Interactive world models must causally generate responsive future observations while preserving scene geometry, subject identity, and controllability over long horizons.
Method
Matrix-Game 3.5 unifies geometry-aware patch memory and tiled PRoPE with static-dynamic representation and two-stage progressive distillation into a few-step causal generator.
Results
Matrix-Game 3.5 achieves strong performance in long-horizon scene recall, geometry-consistent camera control, subject consistency, prompt-driven world evolution, and stable real-time interaction.
Takeaways & Limitations
Explicitly unifying memory, geometry, and dynamic world representation provides an effective foundation for scalable, controllable, persistent interactive world simulation.
Takeaways & Limitations
Dynamic entities lack persistent states of their own, and patch-memory geometry currently comes from an external estimator.
Abstract
from arXiv · showhide
Interactive world models extend video generation from offline clip synthesis toward persistent simulation of interactive virtual worlds, enabling applications in games, robotics, embodied agents, and XR. Achieving stable long-horizon interactive generation, however, remains challenging, as the model must simultaneously preserve scene geometry, dynamic consistency, and camera control while supporting real-time autoregressive generation. Building upon Matrix-Game 3.0, we present Matrix-Game 3.5, as shown in Figure 1, which advances real-time interactive world generation toward geometry-aware and long-horizon consistent simulation through three key improvements. First, we propose a unified geometry-aware memory framework, whose patch-memory and tiled-PRoPE components introduce no additional learnable parameters, combining explicit 3D patch retrieval with projective camera conditioning to enable geometry-consistent camera control and faithful long-horizon scene recall. Second, we introduce a static-dynamic disentangled world representation that separately models static scene geometry and dynamic subjects, preserving both geometric consistency and subject identity throughout long-horizon generation. Third, we develop a two-stage progressive real-time distillation framework that converts a bidirectional diffusion model into a few-step causal generator through Perceptual Flow Matching and curriculum based Self-Rollout DMD, enabling minute-long real-time interactive generation. Extensive experiments demonstrate that, with a unified training corpus spanning Unreal simulation environments, open-world games, and internet videos, MatrixGame 3.5 achieves strong performance in long-horizon scene recall, precise camera control, subject consistency, prompt-driven world generation, and stable real-time open-world interaction.
1 Introduction
Matrix-Game 3.5 targets persistent, controllable interactive worlds by unifying geometry-aware memory, dynamic-subject modeling, and real-time causal generation. Experiments report strong performance across scene recall, camera control, subject consistency, prompt-driven evolution, and real-time interaction.
- Motivation: Interactive world models must causally respond to camera motions, user actions, and prompts while preserving scenes, identities, and lasting environmental changes.These capabilities support gaming, embodied agents, robotics, XR, and interactive content creation.
- Challenges: Closed-loop autoregression accumulates visual and geometric errors, while limited context and viewpoint changes make distant scene retrieval and correspondence difficult.The resulting challenges include scene drift, identity inconsistency, degraded controllability, and difficult scene revisitation.
- Approach: Matrix-Game 3.5 stores historical observations as explicit geometry-aware image-patch memory and combines it with tiled PRoPE for viewpoint-consistent retrieval and attention.The framework introduces no additional learnable parameters on top of the backbone.
- Approach: The model disentangles static environments from dynamic entities, using patch memory for stable scene structures and lightweight multi-view reference tokens for movable subjects.This representation addresses persistent geometry and evolving subject identity within one world model.
- Results: Extensive experiments on Unreal environments, open-world games, and internet videos show strong performance across long-horizon recall, camera control, subject consistency, prompt-driven evolution, and stable real-time interaction.The paper frames explicit unification of memory, geometry, and dynamic representation as a foundation for persistent interactive simulation.
2 Method
Matrix-Game 3.5 combines pose-aware sequencing, unified memory, and progressive distillation into a real-time interactive world model. These components jointly preserve scene geometry and dynamic-subject identity over long-horizon interaction.
- Pose-aware representation: Matrix-Game 3.5 uses a pose-aware sequence representation to unify heterogeneous inputs with shared spatiotemporal and camera-aware coordinates.The sequence representation is one of three tightly coupled components in the model.
- Unified memory: Its unified memory combines geometry-aware patch memory, context memory, and reference-token memory to preserve scene geometry and dynamic-subject identity.The memory system addresses both persistent environments and evolving subjects.
- Progressive distillation: A progressive distillation framework converts a bidirectional video diffusion model into a few-step fully causal generator for real-time interaction.This component completes the model’s streaming generation pipeline.
2.1 Pose-aware Sequence Representation
The pose-aware representation places all conditioning and target tokens in one sequence carrying both temporal coordinates and camera poses. Tiled PRoPE preserves pretrained spatiotemporal RoPE while adding geometry-aware attention across viewpoints.
- Sequence layout: All token types are arranged in one pose-aware sequence, with each token receiving a camera pose and a spatiotemporal RoPE coordinate.The sequence includes anchor, clean context, retrieved memory, and noisy target tokens.
- Camera-aware attention: Camera-aware attention encodes relative projection through each frame’s world-to-image matrix, combining relative rotation, translation, and intrinsics.The projection matrix is built from camera intrinsics and world-to-camera extrinsics.
- Tiled PRoPE: Tiled PRoPE multiplies camera projection across all head channels and the full spatiotemporal RoPE, retaining the pretrained layout and frame axis.Unlike native PRoPE, it does not reserve disjoint head-dimension blocks for camera and spatial coordinates.
- Pose normalization: Camera poses are recentered to the first target frame, and translation magnitude is compressed as log(1 + ∥t∥)/4 while direction is preserved.This design addresses numerical instability from large camera transformations.
- Token assignment: Patch-memory tokens reuse the target frame’s temporal coordinate and pose, while reference tokens occupy fixed negative times and inherit the anchor pose.These assignment rules place different memory sources into the shared pose-aware sequence.
- Tiled PRoPE: Relative attention between frames jointly carries temporal difference and relative pose, while values are transported through the corresponding relative transformation without learnable parameters.The figure describes q and k rotation followed by camera-projection multiplication before the shared softmax.
2.2 Unified Memory System
The unified memory system combines geometry-aware patch retrieval, complementary context frames, and reference tokens to preserve static scene structure and dynamic subject identity during long-horizon generation.
- Static-Dynamic Disentangled Memory: The system separates static scene geometry from dynamic subjects, using patch memory for stable regions and reference-token memory for persistent identity cues.Motion-aware filtering masks dynamic regions before patch-memory fusion.
- Patch Memory: Patch-level reference conditioning combines geometric correspondence with flexible synthesis of novel, dynamic, and unseen content.Unlike hard reconstruction targets, retrieved patches remain conditioning inputs for the diffusion transformer and support sparse retrieval and localized updates.
- Patch Memory: Historical latent patches are back-projected into 3D and reprojected into the target view to establish explicit patch-wise correspondence.The process uses metric depth, camera intrinsics, and camera poses.
- Patch Memory: A z-buffer keeps the surface closest to the target camera when multiple historical patches project to the same target location.Selected features populate an aligned memory canvas, while invalid or unreliable projections are discarded.
- Patch Memory: Coverage-aware selection greedily chooses complementary historical frames that add the most target-grid coverage, with a budget of five frames per target.Selection stops when marginal coverage gain vanishes or the budget is reached.
- Context Memory: Context memory supplies clean, unwarped historical latent frames selected for complementary viewpoint coverage rather than nearest-frame redundancy.These frames preserve faithful past observations alongside target-aligned patch content.
2.3 Progressive Distillation
Matrix-Game 3.5 uses two progressive distillation stages to convert bidirectional diffusion into a few-step causal generator for real-time interactive streams. Teacher-forced perceptual flow matching initializes the model, while curriculum-based self-rollout DMD adapts inference-time autoregression with control and memory conditions.
- Self-Rollout Distribution Matching: A condition curriculum progressively distills classifier-free guidance, camera control, patch memory, and context frames.The curriculum first omits online memory, then gradually introduces patch memory and context frames while retaining a clean anchor.
- Causal Adaptation: Teacher-forced perceptual flow matching first produces a high-quality few-step causal initializer.The causal-adaptation stage reformulates generation as chunk-wise causal denoising and constrains flow matching in pretrained perceptual feature space.
- Causal Adaptation: The perceptual objective combines causal denoising and few-step generation in a single training objective.It uses a frozen VAE decoder and selected frozen perceptual-model feature blocks to measure feature distance.
- Self-Rollout Distribution Matching: Self-rollout DMD removes teacher forcing and matches the student distribution to a bidirectional teacher along the student’s own autoregressive trajectory.Patch memory and context frames are retrieved online from causally visible generated history.
- Self-Rollout Distribution Matching: Stable supervision shares the initial memory, anchor frame, text prompt, and camera trajectory while keeping scorer memory fixed as the student updates online memory.This avoids forcing incompatible internal memory trajectories to match.
- Progressive Distillation: The combined distillation framework yields a three-step causal generator for minute-long, real-time interactive streams while preserving quality and control.The final generator is designed to preserve quality and control over extended interactive generation.
3 Data Infrastructure
The data infrastructure augments raw videos with geometry, semantic, and identity supervision for persistent interactive world modeling. It uses metric-scale geometric reconstruction, window-level dynamic scene prompts, tracked multi-view subject references, and quality-aware curation.
- Data Infrastructure: The infrastructure provides geometry, semantic, and identity supervision because these signals are unavailable in large-scale raw video corpora.These forms of supervision target memory retrieval and camera control, coherent world generation, and persistent dynamic-subject modeling.
- Geometric Annotation: The geometric annotation pipeline processes overlapping video chunks with VGGT-Omega, metric anchoring, and Sim(3) alignment to produce globally consistent scene geometry.It outputs metric depth, camera-to-world poses, camera intrinsics, and scale and quality diagnostics.
- Geometric Annotation: Metric anchoring uses Depth Anything 3, while neighboring chunks are stitched through Sim(3) alignment and aggregated into a single global metric scale.The alignment allows each chunk its own corrective scale while maintaining a continuous stitched trajectory.
- Semantic Annotation: Semantic annotation generates one temporally stable prompt per window from sparsely sampled multi-image frames.Each prompt describes the static environment and dynamic objects’ identities, motions, trajectories, pose changes, and interactions.
- Identity Annotation: The identity pipeline extracts detections, segmentation masks, track identities, and diverse multi-view reference images for dynamic-subject modeling.DINO features preferentially select mutually dissimilar references to provide complementary identity and appearance cues.
- Quality-Aware Curation: Quality-aware curation retains clips only when they pass hard criteria across reconstruction, camera behavior, visual quality, and scene suitability.Video length and camera exploration terms correct biases favoring short fragments and nearly static cameras.
4 Experiments
Experiments evaluate Matrix-Game 3.5’s memory, camera control, subject preservation, prompt-driven evolution, and real-time generation across quantitative and qualitative settings.
- Implementation Details: Training jointly optimizes camera conditioning and patch memory with up to five geometrically retrieved historical candidates per target latent.Camera intrinsics and extrinsics enter every DiT block through tiled PRoPE, while depth-aware z-buffering fuses retrieved observations into target-aligned memory.
- Quantitative Results: Matrix-Game 3.5 achieves the best camera accuracy on every pose metric across both SANA-WM trajectory splits.On the Hard split, rotation error is 2.70 versus 8.34 for the runner-up, while T and CMC are 1.25 and 1.33.
- Quantitative Results: The model achieves the highest revisit SSIM on both splits, with values of 0.439 and 0.414, indicating strong structural scene recall.Pixel-aligned PSNR and LPIPS can penalize valid changes from characters, vegetation, weather, and other evolving content.
- Qualitative Results: Patch memory preserves recognizable scene structure and local appearance across large viewpoint changes in both game-style and real-world environments.Tiled PRoPE produces smooth camera motion, while revisited content remains consistent with initial observations.
- Qualitative Results: VLM prompt rewriting carries user-triggered volcano, dog, train, and UFO events across segments while retaining surrounding scenes and controllable camera motion.The rewritten prompt uses rollout context, retrieved pose-ordered frames when available, actual camera motion, and the user-specified event.
- Real-Time Inference: Reference tokens reduce dynamic-subject identity and appearance drift under viewpoint changes, while the three-step distilled model retains coherent minute-long rollouts.The distilled model preserves sharp local detail and scene geometry without abrupt resets across approximately one minute.
5 Related Work
Related work spans latent-memory, frame-memory, object-control, streaming-generation, and diffusion-distillation approaches, motivating Matrix-Game 3.5’s unified treatment of these capabilities.
- World Memory: Latent-memory methods compress history implicitly, whereas frame-memory methods retrieve whole observations using field-of-view overlap or reconstructed spatial surfaces.These memory units determine how efficiently systems model long context and how explicitly they support scene revisitation.
- Object Control: Object-controllable video generation uses trajectory-conditioned spatial controls or reference-conditioned appearance features to improve motion control and subject identity.The two directions address complementary aspects of controllable video generation.
- Streaming Generation: Streaming systems generate frames or chunks causally for low latency, but transferring a high-quality bidirectional video model into a fully causal generator remains challenging.Prior systems combine short- and long-term memory, chunkwise autoregression, efficient latent diffusion, and online processing.
- Distillation: DMD-based methods convert bidirectional diffusion models into causal or few-step generators, with self-rollout training reducing exposure bias for longer generation.CausVid, Self-Forcing, and Self-Forcing++ establish progressively longer-horizon distillation strategies.
6 Conclusion
Matrix-Game 3.5 unifies geometry-aware memory, dynamic-subject representation, and progressive distillation for persistent real-time interactive world generation, while identifying important open extensions.
- Conclusion: Patch memory and tiled PRoPE combine persistent 3D patch retrieval with projective camera conditioning in the backbone’s existing self-attention stack.Neither component adds learnable parameters or requires architectural modification.
- Conclusion: Static patch memory preserves stable scene structure, while lightweight multi-view reference tokens carry movable subjects to suppress ghosting and identity drift.This separates static geometry from dynamic-subject appearance during extended rollouts.
- Conclusion: The two-stage progressive distillation framework converts the bidirectional world model into a few-step real-time generator using perceptual Flow Matching and curriculum-based Self-Rollout DMD.The conclusion presents this as the final component of the real-time streaming system.
- Open Directions: Future work includes persistent states for off-screen dynamic entities, internally predicted geometry, richer embodied actions, larger backbones, and longer horizons.The current geometry grounding comes from an external estimator, and dynamic subjects do not yet retain independent persistent states.
A Text-Prompt Annotation Examples
Prompt annotations describe both the static scene and each dynamic object’s motion using four evenly spaced frames within an annotation window.
- Annotation Format: Each annotation case pairs four evenly spaced frames with a window-level prompt describing the static scene and dynamic-object motion.The examples span game and real-world sources.
B Preservation of RoPE’s Temporal Encoding under Tiled PRoPE
Tiled PRoPE preserves the temporal structure encoded by native spatiotemporal RoPE while adding geometry-aware attention through relative camera viewpoint. Its camera transformation re-weights existing positional channels rather than replacing their temporal information.
- Design: Tiled PRoPE multiplies camera transforms onto channels already carrying spatiotemporal RoPE, preserving temporal information while adding camera conditioning.This design addresses the mismatch between unordered multi-view PRoPE and video generation, where frame ordering is encoded temporally.
- Design: After RoPE rotations, tiled camera projections transform queries and keys, producing attention logits governed by relative projection between token poses.The construction uses tile(P_i)^⊤R_i for queries and tile(P_j)^−1R_j for keys.
- Design: Values are transported through the same relative camera transformation, then mapped back to the destination frame’s coordinate system.This keeps value transport consistent with the pose-aware attention computation.
- Evidence: A matched-content probe shows that tiled PRoPE leaves the RoPE-only temporal kernel essentially unchanged and modulates attention through relative viewpoint.Identical frame content isolates the effect of positional and camera encodings.
C Camera Collision Avoidance via a Progressive 3D Occupancy Map
Matrix-Game 3.5 avoids camera collisions by building a cumulative 3D occupancy map from registered depth and pose observations, then correcting planned trajectories before generation. The corrected trajectory remains consistent with memory retrieval, camera conditioning, and subsequent map updates.
- Problem: Camera control maps interaction inputs to trajectories but does not itself reason about collisions with generated scene surfaces.A separate geometric correction mechanism is needed when a planned path enters a wall, object, or other surface.
- Occupancy mapping: Metric depth and camera pose are back-projected into a unified world frame and accumulated as voxels, which become occupied after enough surface observations.The occupancy map is initialized from the anchor and incrementally updated across rollout segments.
- Trajectory correction: Before each segment, the planned trajectory is aligned with the occupancy map and corrected when its camera centers approach occupied regions or safety margins.The correction adjusts the planned path before generation rather than reacting only after a collision.
- Trajectory correction: The correction pushes motion away from obstacles while preserving tangential and retreating components, producing lateral motion along scene structure.Camera rotation and intrinsics remain unchanged; only the camera position and extrinsic translation are recomputed.
- Closed-loop consistency: The corrected trajectory drives both patch-memory gathering and tiled PRoPE, while generated frames update the occupancy map in a closed registration-correction-generation loop.This keeps historical reprojection, pose conditioning, and current generation mutually consistent across segments.