PaperlayerView exact revision on arXiv

Source-linked AI summary

Solaris: Building a Multiplayer Video World Model in Minecraft

Georgy Savva, Oscar Michel, Daohan Lu, Suppakit Waiwitlikhit, Timothy Meehan, Dhairya Mishra, Srivats Poddar, Jack Lu, Saining Xie

arXiv:2602.22208v2cs.CV

TL;DR

Existing video world models generally simulate only one agent’s perspective, leaving multiplayer interactions and shared-world consistency under-modeled. Solaris combines a coordinated Minecraft data system, a multiplayer-adapted video diffusion architecture, staged training, and Checkpointed Self Forcing. The resulting system generates consistent multi-view observations and demonstrates coherent multi-agent world simulation, while remaining limited by synthetic data and the absence of persistent world state.

  • Problem

    Single-agent video world models do not simulate all agents’ perspectives, although accurate multi-agent world states require consistent observations across agents.

  • Method

    Solaris combines a multiplayer Minecraft data system with a video diffusion model adapted for multiple perspectives and Checkpointed Self Forcing for memory-efficient long-horizon training.

  • Results

    Solaris generates consistent multi-view observations from coordinated multi-agent interactions and demonstrates coherent multi-agent world simulation.

  • Takeaways & Limitations

    The paper demonstrates that coherent multiplayer world simulation is achievable with scalable data collection, staged training, and memory-efficient long-horizon training.

  • Takeaways & Limitations

    Synthetic training data limits generalization, while the model lacks persistent memory and loses shared context when players leave one another’s field of view.

Abstract

from arXiv · show

Existing action-conditioned video generation models (video world models) are limited to single-agent perspectives, failing to capture the multi-agent interactions of real-world environments. We introduce Solaris, a multiplayer video world model that simulates consistent multi-view observations. To enable this, we develop a multiplayer data system designed for robust, continuous, and automated data collection on video games such as Minecraft. Unlike prior platforms built for single-player settings, our system supports coordinated multi-agent interaction and synchronized videos + actions capture. Using this system, we collect 12.64 million multiplayer frames and propose an evaluation framework for multiplayer movement, memory, grounding, building, and view consistency. We train Solaris using a staged pipeline that progressively transitions from single-player to multiplayer modeling, combining bidirectional, causal, and Self Forcing training. In the final stage, we introduce Checkpointed Self Forcing, a memory-efficient Self Forcing variant that enables a longer-horizon teacher. Results show our architecture and training design outperform existing baselines. Through open-sourcing our system and models, we hope to lay the groundwork for a new generation of multi-agent world models.

1. Introduction

The paper introduces Solaris, a multiplayer video world model for consistent multi-agent perspectives, supported by a scalable Minecraft data system and memory-efficient long-horizon training.

  • Single-agent video world models cannot simulate all agents’ perspectives, limiting their ability to represent accurate multi-agent world states.
  • Multiplayer modeling requires temporal and cross-view consistency, including reflecting one agent’s movements or block placements in every other viewpoint.
  • 12.64 M frames, including 6.32 M per player, support evaluation of multiplayer movement, memory, grounding, building, and view consistency.
  • Solaris adapts a single-agent pretrained video DiT to simulate multiple perspectives and autoregressively generates future observations from past observations and actions.
  • Checkpointed Self Forcing extends Self Forcing with a longer-context teacher while mitigating the excessive memory use of sliding-window generation.

2. Related Work

The paper situates Solaris among video world models, autoregressive video-generation methods, and Minecraft agent platforms, emphasizing the unmet need for multiplayer-capable simulation.

  • 2.1. World Models and Video World Models: Video world models use video diffusion models as simulators across robotics, games, self-driving, and physical simulation, while Solaris focuses on multiplayer shared-world modeling.
  • 2.1. World Models and Video World Models: Multiverse is identified as the only prior video world model capable of simulating multiple agents, but it targets a simpler Gran Turismo 4 race track than Minecraft’s 3D open world.
  • 2.2. Autoregressive Video Generation: Diffusion Forcing, CausVid, and Self-Forcing provide techniques for autoregressive generation, causal conversion, and reducing autoregressive train-test mismatch.
  • 2.3. AI Agents in Minecraft: Existing Minecraft agent frameworks primarily support reinforcement learning or single-player data and cannot simulate multiplayer gameplay with visuals.
  • 2.3. AI Agents in Minecraft: SolarisEngine addresses this gap through Docker orchestration of servers, cameras, and controller bots, plus reusable skill primitives for episodes.

3. SolarisEngine: A Framework for Multiplayer Gameplay at Scale

SolarisEngine is a scalable system for collecting controlled, coordinated multiplayer Minecraft gameplay with synchronized visual and action data. Its modular bot, communication, rendering, and automation infrastructure produces diverse episodes at large scale.

  • System capabilities: SolarisEngine combines fine-grained programmatic control, multiplayer gameplay, and rich Minecraft visual observations, unlike prior frameworks that provide only subsets of these capabilities.RL-based systems offer visual observations but limited controllability, while high-level systems such as Voyager and Mineflayer lack visual output.
  • Training dataset: 12.64 M frames were collected across 9,240 episodes, with 6.32 M frames per player spanning building, combat, movement, and mining scenarios.Most episodes contain 128–512 frames, corresponding to 6.4–25.6 seconds at 20 fps.
  • Cooperative gameplay: The system uses Mineflayer primitives and a communication layer to coordinate bots performing building, scaffolding, tool use, navigation, and other predefined goals.High-level primitives are translated into low-level actions compatible with human-player action datasets such as VPT.
  • Visual recording: Controller bots pair with GPU-rendered camera bots whose synchronized state and actions provide visual recordings for each logical player.A server-side plugin mirrors controller-bot states, actions, and animations in real time.
  • Robust and scalable execution: Automated Docker orchestration runs multiple workers in parallel, while failure detection aborts stuck episodes and starts fresh ones without manual intervention.Controller bots, camera bots, and servers run in isolated containers managed by Docker Compose.

4. Solaris Model Design

Solaris extends a pretrained single-agent video DiT to model joint multiplayer observations and actions. Its architecture exchanges information across players, while staged training adapts bidirectional modeling to causal, long-horizon autoregressive generation.

  • Multiplayer formulation: The model represents observations and actions for multiple agents as joint tensors and models future states conditioned on prior joint observations and actions.For this work, the framework restricts the number of players to P = 2 while remaining generalizable to more players.
  • Training objective: The model is trained with conditional Flow Matching using a shared noise level for bidirectional modeling and independent player-frame noise levels for causal generation.The causal variant adopts Diffusion Forcing to support autoregressive generation.
  • Network architecture: Solaris adapts Matrix Game 2.0 by extending its action space, applying player-specific processing, and exchanging cross-player information through multiplayer self-attention.Learned player-ID embeddings and independently applied per-player modules preserve player identity while enabling interaction.
  • Training pipeline: The training pipeline progresses from a pretrained bidirectional model to single-player and multiplayer finetuning, causal masking, and Self Forcing for stable long-horizon generation.This staged sequence adapts bidirectional video diffusion to autoregressive use.

5. Multiplayer Training Pipeline

Solaris uses staged training to transition from single-player initialization to multiplayer causal generation, then applies Checkpointed Self Forcing for efficient long-horizon training. The method replaces memory-intensive rolling-window backpropagation with recomputation and parallel masked processing.

  • Stages 1–2: Solaris begins from Matrix-Game 2.0, expands its limited action space through VPT pretraining, and uses that single-player initialization for multiplayer modeling.The single-player model is trained bidirectionally for 120K steps on 33-frame contexts before multiplayer adaptation.
  • Stage 3: The multiplayer model is trained with full-sequence diffusion for 120k steps, producing the teacher checkpoint used during Self Forcing.Training branches at 60k steps so the causal model can be initialized while the bidirectional model continues toward its teacher role.
  • Stage 4: Checkpointed Self Forcing extends Self Forcing with a teacher context longer than the student’s, using a sliding-window student during generation.The approach addresses the excessive memory usage caused by retaining overlapping context windows during backpropagation.
  • Stage 4: The method first rolls out clean estimates without gradients, then recomputes generator outputs in parallel with cached states and a causal sliding-window attention mask.The mask reproduces inference-time conditioning by restricting noisy frames to preceding clean context frames.
  • Stage 4: The parallel formulation reduces memory from O(L_t·L_s) to O(L_t), making backpropagation through recomputed KV representations feasible.The implementation enables gradients through KV layers, unlike standard Self Forcing’s stop-gradient treatment.

6. Evaluation Benchmark

The evaluation benchmark measures multiplayer world-model behavior across movement, grounding, memory, building, and cross-view consistency. It combines held-out episode types with FID and task-specific VLM judgments.

  • Benchmark design: The benchmark uses completely unseen episode types covering Movement, Grounding, Memory, Building, and Consistency capabilities.FID measures overall visual quality, while a VLM judge answers verifiable task-specific questions about generated videos.
  • Movement: Movement tests whether both views consistently render translation and camera rotation when one bot moves and the other observes.The VLM judges the moving player’s position from the observer’s view.
  • Grounding: Grounding tests whether a player’s world position remains recoverable from the other player’s observation after temporarily turning away.The episode uses two nearby agents in a Superflat world, with one stationary observer.
  • Memory: Memory tests whether both agents remember the other player and surrounding visibility conditions after turning away and then back.The VLM evaluates visibility during both the turned-away interval and the return orientation.
  • Building: Building tests whether the observer’s view reflects a structure constructed by the other agent during an ongoing block-building episode.The builder creates a predefined square or strip while the observer watches.
  • Consistency: Consistency tests whether nearby agents see the same previously unseen region after simultaneously turning 90 degrees in the same or opposite directions.The VLM judges agreement between their resulting views in a Normal world.

7. Experiments

Experiments show that Solaris maintains coherent long-horizon multiplayer gameplay and outperforms the compared architectures across evaluation categories, while ablations identify benefits and trade-offs from KV backpropagation. Qualitative examples further demonstrate state tracking, weather, mining, building, and PVP capabilities.

  • Qualitative Results: Solaris qualitatively simulates building, mining, fighting, multiplayer viewpoints, inventory tracking, weather, torch placement, and mining animations.The evaluated action sequences come from bot gameplay not present in the training dataset.
  • Architecture Experiments: Solaris produces superior qualitative and quantitative results to frame concatenation and training without single-player pretraining across the evaluation categories.The comparison uses the only previously available multiplayer world-model baseline and a no-pretraining variant.
  • Architecture Experiments: The frame concatenation baseline outperforms Solaris on Movement but exhibits action hallucinations when actions are no-ops.This result qualifies the broader across-category comparison.
  • Self-Forcing Ablations: KV backpropagation achieves better FID than the other Self Forcing variants, while reducing action-following performance in some categories.The method remains competitive overall and excels on Building and Consistency VLM tasks.

8. Conclusion

The paper concludes that Solaris demonstrates consistent multi-view simulation for coordinated Minecraft interactions through scalable data collection, staged training, and Checkpointed Self Forcing. It also identifies synthetic-data distribution gaps and missing persistent world state as key limitations.

  • Conclusion: Solaris generates consistent multi-view observations from coordinated multi-agent interactions in Minecraft.The conclusion attributes this result to a scalable data system, staged single-player-to-multiplayer training, and memory-efficient long-horizon training.
  • Future Directions: SolarisEngine supports more than two concurrent players and downstream work involving multimodal data, unified perception-action models, policies, and 3D benchmarks.These directions include planning, memory, and neurosymbolic reasoning research.
  • Limitations: Synthetic training data leaves action and visual distribution gaps that limit generalization.The paper suggests investigating how abundant single-player data could help close this gap.
  • Limitations: Without persistent memory or an underlying persistent world state, players’ trajectories diverge after leaving each other’s fields of view.The broader environment is specified only through two initial frames and cannot be controlled or maintained like a true game engine.

A.2. GPU Data Collection

GPU-based rendering produced smoother Minecraft videos and enabled hardware encoding, while the dataset uses filtered episode types and records a broad but incomplete action space with skewed camera-motion coverage.

  • Rendering: GPU-based rendering removed frame repetition seen with CPU-based headless rendering in graphics-intensive scenes.The CPU renderer produced up to four repeating frames at 20 fps in open Normal-world scenes.
  • Episode filtering: 340 of 6000 Normal-world episodes were underwater and excluded because underwater mechanics break most episode logic.A bubble-icon classifier filters underwater episodes from episode frames with reported 100% accuracy.
  • Action space: SolarisEngine records nearly the full Minecraft action range but omits inventory interaction, crafting-related GUI actions, and raw mouse movement.It stores semantic actions rather than the raw keyboard and mouse events used by VPT.
  • Action distribution: Camera-action coverage is heavily skewed toward fast movements originating from the Pathfinder navigation plugin.Pathfinder operates at a maximum camera speed of 178 degrees per second, despite manually sampled lower-range camera movements.

C. Model Training

This section provides the training hyperparameters for Solaris’s different training stages and specifies the TPU hardware used for the models.

  • Training hardware: The bidirectional single-player model uses v5p-128 TPUs, while all other models use v5p-64 TPUs.This hardware distinction applies across the reported model-training configurations.
  • Training configuration: Training hyperparameters are provided across all training stages in Table 6.The supplied passage identifies the table but does not report individual hyperparameter values.

D. Evaluation Details

The evaluation details describe a VLM-based judging procedure and summarize the Self Forcing denoising process used for autoregressive video generation.

  • VLM evaluation: VLM evaluation requires the expected answer across all queries and player perspectives for an episode to count as accurate.Ground-truth episode accuracy and prompt details are provided in Table 7 and Figure 14.
  • Action analysis: Figure 13 shows the distribution of mouse-action magnitudes in the training dataset.The supplied figure passage identifies the plotted quantity but does not state a specific distributional result.
  • Self Forcing: Self Forcing autoregressively generates frames while truncating each frame’s denoising process at a randomly sampled noise level.The next frame is conditioned on clean estimates of prior frames derived from the truncated states.
  • Self Forcing: Gradients flow only from the final truncation step, giving supervision across noise levels while keeping memory complexity constant in the number of diffusion steps.The original implementation also applies stop-gradient to the KV cache.

E.2. Teacher Forcing Mask Implementation

The teacher-forcing mask re-simulates denoising steps in parallel while enforcing sliding-window causality, so frames attend only to permitted noisy or past clean frames.

  • Mask design: Checkpointed Self Forcing uses a teacher-forcing mask to re-simulate the final denoising step across all frames in parallel.The mask enforces sliding-window causality and restricts frames to attend to past clean frames.
  • Evaluation queries: The evaluation table distinguishes query modes by whether the VLM asks separately about each player or jointly about both perspectives.“Each Player” uses two queries, while “Both Players” uses one query.
  • Implementation: The teacher-forcing implementation builds query and key index grids, computes frame indices, and separates noisy from clean frames.These operations provide the indices used to construct the attention mask.
  • Attention constraints: The mask allows noisy queries to attend to same-frame noisy keys and earlier clean keys, while clean queries attend causally to clean keys.This specifies the attention relationships encoded by the implementation.
  • Sliding window: The sliding-window constraint permits a key frame only when it lies within the student context window preceding the query frame.The implementation expresses this as kv_frame > q_frame - L_s.
Loading 2602.22208v2…