Source-linked AI summary
MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines
Ryan Po, David Junhao Zhang, Amir Hertz, Gordon Wetzstein, Neal Wadhwa, Nataniel Ruiz
TL;DR
Video world models lack reliable user control over environment structure and shared inference across players. MultiGen addresses this with persistent external memory and Memory, Observation, and Dynamics modules, enabling editable level-conditioned generation and coherent multiplayer interaction. The approach supports controllable long-horizon gameplay, while remaining limited by unrepresented scene properties, accumulated pose errors, and training-distribution coverage.
Problem
Video world models struggle to provide reproducible, editable environment control and shared interaction through a common underlying state.
Method
MultiGen uses persistent external memory updated by user actions and decomposes generation into Memory, Observation, and Dynamics modules.
Results
MultiGen enables level-conditioned gameplay generation and real-time multiplayer interaction in a shared world with consistent first-person views conditioned on common state.
Takeaways & Limitations
Coarse layouts can support controllable level design and structural adherence, while shared memory provides an interface for multi-player roll-outs.
Takeaways & Limitations
Scene properties absent from map M may not be preserved on revisits, pose errors can accumulate, and appearance may not generalize beyond the training distribution.
Abstract
from arXiv · showhide
Video world models have shown immense promise for interactive simulation and entertainment, but current systems still struggle with two important aspects of interactivity: user control over the environment for reproducible, editable experiences, and shared inference where players hold influence over a common world. To address these limitations, we introduce an explicit external memory into the system, a persistent state operating independent of the model's context window, that is continually updated by user actions and queried throughout the generation roll-out. Unlike conventional diffusion game engines that operate as next-frame predictors, our approach decomposes generation into Memory, Observation, and Dynamics modules. This design gives users direct, editable control over environment structure via an editable memory representation, and it naturally extends to real-time multiplayer rollouts with coherent viewpoints and consistent cross-player interactions.
1 Introduction
MultiGen addresses limited environment control and single-user interaction in video world models with persistent external memory. Its modular design supports editable level layouts, long-horizon rollouts, and shared multiplayer worlds.
- Video world models make interactive rollouts practical, but implicit state limits reproducibility, environment authoring, and reliable shared interaction.These systems generate worlds on the fly, making long rollouts difficult to steer and align with user intent.
- MultiGen introduces external memory that persists beyond the model context window and is updated by user actions.The memory provides a persistent state that agents can read and write during generation.
- Shared memory enables multiple agents to interact through a common state while rendering coherent observations from multiple viewpoints.The approach supports real-time multiplayer interaction with cross-player effects and consistent viewpoints.
- Users author or edit a coarse top-down 2D minimap that anchors detailed first-person observations and interactive sessions.The minimap serves as an external-memory blueprint while the model synthesizes visual detail and moment-to-moment interactions.
- The system decomposes generation into Memory, Observation, and Dynamics modules rather than relying on a single model.Memory maintains persistent state, Observation generates visual observations, and Dynamics updates state from actions and observations.
- The paper demonstrates editable environment design and real-time multiplayer interaction as applications of its external-memory formulation.Both applications are evaluated against relevant baselines.
2 Related Work
Prior work spans bidirectional diffusion, autoregressive video generation, diffusion game engines, and external-memory video systems. MultiGen extends diffusion game engines with persistent, user-editable memory and modular state handling for long-horizon and multiplayer interaction.
- Bidirectional diffusion models generate fixed-length videos through simultaneous spatiotemporal token denoising.
- Autoregressive video models generate frames sequentially, aligning with interactive simulation, but discretized approaches often lag behind diffusion models.
- Diffusion game engines use diffusion models for real-time gameplay, while conventional systems commonly condition next-frame generation on short frame histories and actions.MultiGen builds on this setup with explicit external memory for authored layouts.
- Existing video-memory approaches use explicit 3D representations, compressed context windows, or KV caching, whereas MultiGen's memory persists beyond the context window.
- The external memory is updated by user actions, supports shared read-write state across agents, and lets users govern global structure.
3 Method
The method augments diffusion game-engine rollouts with persistent map and pose memory, separating state maintenance, frame generation, and pose dynamics. This supports geometry-conditioned observations, action control, and coherent authored-map rollouts.
- Interactive rollout: The rollout alternates actions and visual observations, predicting each next observation from recent history and the current action.Closed-loop simulation repeatedly feeds sampled observations back into the next step.
- Modular architecture: The system decomposes the game engine into Memory, Observation, and Dynamics modules instead of relying only on frames as state.Memory stores structured map geometry and pose; Observation predicts frames; Dynamics updates state from actions and observations.
- Memory module: The explicit state combines an L-frame visual context with a static level map and player pose, separating persistent low-dimensional information from observations.The map is represented by 2D vertices and line segments, while pose contains player coordinates and facing direction.
- Memory module: Persistent map memory provides a stable reference that can be consulted throughout long rollouts, reducing reliance on layout information remaining in a finite visual buffer.The map is time-invariant and supplies a coarse description of the current level.
- Observation module: The Observation module predicts the next frame from recent context, a memory-derived geometric signal, and the next action.Geometry is supplied through a ray-traced disparity representation, while actions enter through learned cross-attention tokens.
- Dynamics module: The Dynamics module predicts an incremental pose update from the current pose, action, geometric signal, and aggregated UNet features, then applies angle wrapping.Noised-context training exposes the observation model to imperfect histories to improve robustness during autoregressive rollouts.
- Rollout examples: Example rollouts show viewpoints evolving coherently with action inputs while adhering to an authored map layout.The figure pairs the minimap and pose with generated first-person observations and actions.
4 Application I: Level Design
MultiGen uses an editable top-down map as persistent external memory to anchor level-conditioned gameplay and preserve layout structure during long rollouts. Compared with implicit-state baselines, this external reference improves structural consistency, especially later in generation.
- Evaluation: SSIM/PSNR/LPIPS comparisons show MultiGen consistently outperforms baselines, with the largest gains in later rollout stages.Later stages are where consistent memory is most important.
- Level Design: Users directly influence environment structure by authoring coarse map geometry before inference begins.The map consists of vertices and wall segments and conditions interactive first-person rollouts.
- Level Design: The model queries the authored map at every timestep, maintaining a persistent geometric reference throughout generation.This contrasts with engines that infer global structure from a finite visual history.
- Evaluation: Over 10 million gameplay frames from 100 procedurally generated maps support training across varied corridor and room configurations.The maps were generated with randomized structures while preserving valid Doom geometry.
- Results: Designed maps anchor long rollouts to corridor structure and represented turns while avoiding structural drift from frame-history inference.Representative rollouts show viewpoint evolution consistent with action inputs and the specified layout.
5 Application II: Multiplayer Interaction
MultiGen represents multiplayer state explicitly in shared external memory, allowing independently generated player viewpoints to remain synchronized. The system supports dynamic interactions, arbitrary player counts, and real-time inference without requiring multi-view training data.
- Shared Memory: All players read and write a shared external memory, so one player’s actions can influence another player’s observations.This avoids entangling world state with each player’s local observation history.
- System Design: Each player runs separate Observation and Dynamics modules while shared memory stores the map, poses, and active-player state.Generated frames are produced from shared-memory readouts, then actions update the common state.
- System Design: The distributed design supports arbitrary player counts and handles death and respawn by removing or reintroducing poses in shared memory.Unlike a fixed split-screen baseline, adding players does not require changing the model interface.
- Evaluation: Over 10 million gameplay frames were collected from Doom deathmatch sequences for the multiplayer study.The sequences record actions, active-player poses, and map layouts.
- Results: Two-player rollouts preserve cross-viewpoint consistency through repeated kills, removal, respawn, and reappearance events.Players appear with correct pose and location when visible and disappear from the other camera when dead or out of view.
- Results: The full system runs at approximately 20 FPS using one NVIDIA A100 per player, while three viewpoints remain aligned with shared state.Each player’s model runs independently, so additional players add per-player instances rather than slowing a single joint inference process.
- Evaluation: MultiGen consistently outperforms all baselines on opponent-presence accuracy, precision, and recall judged against ground-truth visibility labels.The evaluation uses a VLM-based judge on generated multiplayer frames.
6 Discussion and Conclusions
The system combines explicit memory, observation, and dynamics modules to support level-conditioned gameplay and multiplayer rollouts. Context length improves fidelity, while the approach remains limited by unrepresented scene properties, accumulated pose errors, and training-distribution coverage.
- Discussion: Increasing context length consistently improves fidelity, with higher SSIM/PSNR and lower LPIPS.The ablation varies the number of conditioning frames from L ∈ 2, 4, 8, 16, 32 while holding other settings fixed.
- Limitations: Scene properties omitted from map M, such as textures or small objects, are not explicitly preserved across revisits and may appear inconsistently.The approach relies on explicit state for long-horizon consistency.
- Limitations: Small pose errors may accumulate during long rollouts, and visual appearance may not generalize to styles far outside collected VizDoom trajectories.The authors report that actions remain aligned with plausible motion and overall gameplay experience is preserved.
- Conclusions: MultiGen stores map geometry and evolving player poses in external memory, alongside observation and dynamics modules.The observation model uses ray-traced disparity and actions, while the dynamics model updates pose during rollout.
- Conclusions: This decomposition supports controllable level design from coarse layouts and consistent first-person views for multiplayer rollouts sharing one underlying state.Each player generates a view conditioned on the same shared state.