Multiplayer Interactive World Models with Representation Autoencoders

• The model conditions on multiple agents’ action streams to attribute scene changes correctly, represent interactions, and remain coherent under arbitrary joint actions.Single-agent world models treat other agents as part of the environment, limiting explicit control for self-play, multi-agent training, counterfactual evaluation, and human interaction.Source: §1 · pp. 2–3
• Multiplayer world models substantially outperform single-player counterparts, while warm-starting from a single-player model works better than direct multiplayer training.The additional views and action streams reduce uncertainty about the future and improve prediction accuracy.Source: §1 · p. 3
• Pretrained feature-extractor latents improve rollout quality and temporal consistency while matching the reconstruction quality of a feature extractor trained from scratch, and diffusion forcing with few-step distillation stabilizes long horizons and speeds inference.These findings summarize the codec and generative-objective design studies.Source: §1 · p. 3
• The authors present the first interactive multiplayer world model for a highly dynamic environment with complex physical interactions, controllable by four concurrent agents and operating at 20 fps in real time.It is trained on 10,000 hours of multiplayer Rocket League data and contains 5 billion parameters, running on a single Nvidia B200 GPU.Source: §1 · pp. 3–4
• The study evaluates physical understanding, visual fidelity, and human preference, while examining scaling from 100M to 5B parameters and from 100 to 10,000 training hours.The scaling analysis identifies emergent capabilities and characteristic failure modes, while training runs show action-consequence refinement after visual plausibility emerges.Source: §1 · p. 4
• The authors release the training data, full training and inference codebase, and a live demo to support continued research on multiplayer world models.The live demo is available at https://mira-wm.com.Source: §1 · p. 4

2.1 World models for control

• Control-oriented world models prioritize compact latent dynamics for planning or policy learning, whereas this model decodes its compact latent predictions into evaluated video frames.JEPA-style approaches may omit rendering entirely, while this work retains generative visual prediction.Source: §2.1 · p. 4

2.2 Interactive and playable game world models

• Interactive game world models generate action-responsive frames, but prior systems generally condition on one player’s or camera’s action stream rather than simultaneous controls from multiple players.Related systems span DOOM, Atari, CS:GO, Minecraft, open-world games, AAA titles, and driving video.Source: §2.2 · p. 5

2.3 Multiplayer and multi-agent world models

• Rocket League requires jointly predicting four players’ views from simultaneous low-level controls in a continuous-physics, fully three-dimensional environment with fast collisions.Unlike prior multi-agent systems focused on grid worlds, voxel environments, planar navigation, or combat animation, this setting emphasizes tightly coupled rigid-body interaction.Source: §2.3 · p. 5

2.5 Visual representations for generation

• Prediction-space design trades off dimensionality, unpredictable texture synthesis, latent structure, and rollout stability; this work builds its codec’s prediction space on the self-supervised DINOv3 encoder.The paper treats latent versus pixel prediction as an empirical design choice because pixel-space models remain competitive in some domains.Source: §2.5 · p. 6

2.6 Long-horizon, real-time rollout and the training objective

• Autoregressive rollout creates exposure bias and error accumulation, motivating diffusion forcing, teacher forcing, and few-step objectives for coherent real-time generation.The model predicts at 10 Hz in latent space and renders video at 20 fps, with diffusion forcing and teacher forcing compared empirically.Source: §2.6 · p. 6

2.7 Evaluating generative world models

• The evaluation framework complements visual fidelity with physical consistency, action obedience through Action Recoverability Ratio (ARR), and long-horizon stability validated against human judgment.Reference-based metrics suit known short-rollout targets, whereas stochastic world models require distributional, physical, and temporal-drift measures.Source: §2.7 · p. 7

3.1 Data collection

• Each match uses four independent Nexto bot instances in 2v2 games, with synchronized video, action streams, and frame-aligned recordings collected through a centralized VM orchestration system.The model trains on mapped keyboard actions, while intended and engine-consumed actions provide sanity checks; failed matches are discarded.Source: §3.1 · pp. 7–8, §3 · p. 7
• ≈10,000 match-hours comprise 82,983 clean self-play matches, evenly split across three Rocket League arenas and yielding 331,932 per-view recordings.The dataset contains no human gameplay; each match produces four synchronized first-person recordings.Source: §3 · p. 7, §3.1 · p. 8
• The dataset limits behavioral diversity to one bot policy, visual diversity to three fixed maps, and predictive uncertainty because the game is nearly deterministic given all players’ actions.Each car is controlled by its own Nexto process, but all instances share the same policy.Source: §3.1 · p. 8

3.2 Physics game state

• Privileged physics state is logged exactly at 120 Hz from the simulation and used only as evaluation ground truth, covering match, ball, and four-car dynamics.The world model trains on pixels and actions alone; logged state includes scores, timing, poses, velocities, boost, and gameplay flags.Source: §3.2 · p. 8

4.1 Problem setup

• The world model predicts all players’ next latent frames jointly from past latents and every player’s actions, learning solely from recorded play without engine access or privileged state.For 2v2 matches, P = 4, with one video and action stream per player; predicted latents are rolled out autoregressively and decoded into frames.Source: §4.1 · p. 9

4.2 Codec

• The codec uses frozen DINOv3-L features, a learned linear 2×2×2 bottleneck, and a causal spatio-temporal decoder to reconstruct video, reducing channels from 1024 to 32.It emits latents at 10 Hz from 20 fps video, achieving about a 192× reduction in values relative to raw RGB frames.Source: Figure 3 · p. 9, §4.2 · p. 10

4.3 Generative objective

• Flow matching trains the model to transport Gaussian noise to target latents, while diffusion forcing exposes it to imperfect contexts and progressive self-distillation enables one- or two-evaluation sampling.Diffusion forcing assigns independent flow times to frames, and distillation trains average-velocity predictions over larger integration intervals.Source: §4.3 · pp. 10–11

4.4 World model architecture

• The world model is a diffusion transformer with factorized spatial and causal temporal attention that predicts one latent frame at a time during autoregressive rollout.Each block uses adaptive layer normalization driven by summed action and flow-time embeddings, while past latents enter through causal temporal attention.Source: §4.4 · pp. 11, 13

4.5 Multiplayer conditioning

• The multiplayer model tiles four player views into one grid and binds fixed-order per-player action embeddings to those views, allowing spatial attention to render a shared world consistently.Action dropout lets the model predict players with withheld controls, and two-stage training warm-starts from single-player pretraining before joint four-player training.Source: §4.5 · p. 13

Streaming design

• The online demo conditions each new frame on recent player actions and prior generated frames, producing a 10 Hz latent stream that decodes to 20 fps video on a single Nvidia B200 GPU.This fixed-rate streaming loop supports browser-based interactive play.Source: §5 · p. 13

Autoregressive rollout with a streaming cache

• A key-value cache and fixed window of T = 20 latents keep autoregressive generation and three-frame codec decoding at constant per-frame cost.The window rolls by one latent per step while the decoder uses the three most recent latent frames.Source: §5 · pp. 13–14

Priming the rollout

• Rollouts are primed with encoded real gameplay frames that prefill the cache, providing a coherent in-distribution context before fully generated decoding begins.The demo displays decoded ground-truth latents during prefill and switches to model-generated frames afterward.Source: §5 · p. 14

Few-step sampling

• Few-step sampling denoises each new frame using a linear-quadratic schedule and the distilled sampler from Section 4.3.This reduces the sampling work required for real-time generation.Source: §5 · p. 14

Decoupled production and display

• A producer-consumer pipeline separates GPU generation from display, using a bounded queue to pace delivery and prevent the model from running ahead of the client.Each world-model step emits two video frames, which are evenly spaced by the sender.Source: §5 · p. 14

Real-time rollout

• 70 ms per two-frame step enables roughly 35 ms per frame, keeping the 5B model ahead of the 50 ms budget for 20 fps on one Nvidia B200 GPU.The extra speed provides headroom for scheduling jitter and smooth delivery.Source: §5 · p. 14

6.1 The flagship model

• The flagship 5B model simulates four-player Rocket League in real time at 20 frames per second on one Nvidia B200 GPU while remaining controllable and physically coherent.It reproduces contact dynamics, demolitions, goals, and game-state trajectories across viewpoints, with rollouts remaining coherent far beyond the training window.Source: §6.1 · pp. 14–15

6.2 Evaluation metrics

• Experiments combine distributional, pixel-level, perceptual, physical-state, action-recoverability, and human-preference metrics to evaluate realism, dynamics, controllability, and long-rollout drift.Action Recoverability Ratio normalizes generated-video action precision against codec reconstruction, with ARR 1 indicating equally recoverable actions.Source: §6.2 · pp. 15, 17

6.3 Designing the codec's latent space

• A learned latent is essential: pixel-space models trail the latent baseline by an order of magnitude in generation quality and are markedly less controllable despite matched training budgets.The pixel-space rollout also drifts into warped, unstructured texture within a second.Source: Table 2 · p. 19, §6.3 · p. 20
• Pretrained DINOv3 features make latents easier to generate and substantially more stable, while learned bottlenecks add only a small margin and 10 Hz temporal latents preserve generation quality.From-scratch features reconstruct more sharply but drift more; halving the latent rate shortens sequences and enables real-time 20 fps rollout without generation-quality loss.Source: Figure 7 · p. 20, §6.3 · pp. 21–22
• LPIPS and P-DINO are complementary and sufficient without GAN loss, while decoder reconstruction quality saturates at the Large 0.3B scale.Removing both perceptual losses collapses generation, and larger decoders provide diminishing reconstruction returns beyond Large.Source: §6.3 · pp. 22–23

6.4 World model objectives

• Diffusion forcing improves long-rollout stability over teacher forcing: beyond the 4 s training window, teacher forcing degrades sharply and remains about 10× worse in gFID.Diffusion forcing noises every frame independently, exposing training to imperfect context analogous to rollout conditions.Source: Figure 9 · p. 23, §6.4 · p. 23
• Diffusion forcing outperforms teacher forcing across generation metrics and limits long-rollout drift, while self-distillation preserves quality with far fewer flow-matching steps.Context noise is unnecessary for pretrained and from-scratch codecs but rescues the distilled codec, whose clean-context quality collapses at long horizons.Source: §6.4 · pp. 23–24, Figure 10 · p. 24

6.5 Controllability

• ARR closely matches human controllability judgments, improves after image quality saturates, and increases with action frequency, leaving rare air-roll and reverse inputs less faithfully modeled.ARR and human preferences rank models almost identically, while aggregate ARR steadily approaches the reconstruction ceiling during training.Source: Figure 12 · p. 25, Figure 13 · p. 25, §6.5 · p. 26

6.6 Going from single-player to multiplayer

• Pretraining on single-player data before multiplayer training avoids fixed-budget collapse, whereas a larger budget enables scratch training and favors a mostly-multiplayer mix.At 3.2M player-views, multiplayer from scratch reaches gFID 9.9 at 4 s, while a 75% multiplayer and 25% single-player mix reaches gFID 9.4.Source: §6.6 · p. 26, Figure 14 · p. 27

6.7 Scaling behavior

• More unique data mainly improves controllability beyond the data-starved regime, while larger models improve generation quality, convergence speed, and physical-state encoding with diminishing returns.Ball-position probe error falls from 2130 at 100M parameters to 1448 at 5B, with the largest improvement between 100M and 300M.Source: Figure 15 · p. 27, §6.7 · p. 28, Figure 16 · p. 28

6.8 Emergent properties

• The multiplayer model infers plausible behavior for uncommanded players, maintains mutually consistent views, generalizes across action distributions, and can recover desynchronized views.These properties emerge without explicit supervision for withheld-player policies, cross-view recovery, or out-of-distribution action patterns.Source: §6.8 · pp. 29–30, Figure 17 · p. 29, Figure 18 · p. 30

6.9 Failure cases

• Failures are rare but reflect short context and imbalanced data: resting balls drift, cars may act uncommanded, HUD state loses track, replays diverge, and single-player rollouts mishandle off-screen cars.Over roughly 48,000 human-play frames, the model produced about 80 uncommanded boosts and 30 jumps; multiplayer largely avoids single-player cars disappearing or merging with the ball.Source: §6.9 · pp. 30–31

• MIRA models four simultaneous Rocket League players by conditioning latent diffusion on every player’s actions and past video context, generating consistent views at 20 frames per second.It renders each player’s own view, maintains consistency across all four views, and remains coherent over hour-long rollouts on a single Nvidia B200 GPU.Source: §7 · p. 32
• A frozen pretrained feature extractor, diffusion forcing with few-step distillation, and all-four-player action conditioning respectively improve rollout stability, interactive speed, view consistency, and shared-state inference.The pretrained extractor helps prevent drift without improving reconstruction; teacher forcing collapses over long horizons, while all-player conditioning outperforms a single-player model.Source: §7 · p. 32
• Despite next-frame-only supervision, MIRA maintains consistent goals, demolitions, and HUD states across views, responds coherently to human control, and recovers when views become out of distribution.These behaviors were not directly supervised and differ from the bot behavior present in its training data.Source: §7 · p. 32
• MIRA is limited by training on one game and one bot-policy family, short context that loses match state, and data imbalance that under-models rare events such as a resting ball.These limitations can cause clock and score errors, forgotten off-screen cars, and inherited stylistic bias; future work targets varied human-like behavior, longer memory, and real environments.Source: §7 · p. 32
• The authors release MIRA’s dataset, complete training and inference code, and a live demo to support research on world models for interacting agents.The stated goal is to advance models faithful enough to train and evaluate interacting agents.Source: §7 · p. 32

A Implementation Details

• The baseline representation codec is a frozen, temporally downsampled DINOv3 RAE whose full architecture, optimization, and loss settings are documented in Table 9.The codec is trained for 250k steps, while codec ablations use its 125k-step checkpoint.Source: Table 9 · p. 41, A Implementation Details · p. 41
• Baseline single-player and multiplayer world models share the latent world model, optimizer, and frozen codec, differing only in player count and multiplayer conditioning.Their complete training and architecture configuration is provided in Table 10, with shared and differing values separated across columns.Source: A Implementation Details · p. 41, Table 10 · p. 42

A.1 Live-demo model configuration

• The live-demo system is a 5B-parameter, four-player, action-conditioned flow-matching transformer trained with diffusion forcing in the RAE codec’s latent space.It uses the largest configuration in the paper and is trained on approximately 10,000 clean match-hours from the multiplayer corpus.Source: §A.1 · p. 42
• 20 frames per second are emitted in real time on a single Nvidia B200 GPU by the live-demo model.The demo checkpoint is warm-started from a single-player model pretrained for 30,000 steps, then trained for a further 100,000 multiplayer steps, below the 2M-step schedule ceiling.Source: §A.1 · p. 42
• The demo checkpoint is warm-started from a 30,000-step single-player model and receives 100,000 additional multiplayer-training steps, below the 2M-step schedule ceiling.This training procedure adapts the pretrained single-player model to multiplayer data rather than using the full configured schedule.Source: §A.1 · p. 42
• The live-demo codec is a temporally downsampled DINOv3 RAE with 288 × 512 inputs at 20 fps, temporal patch size 2, a 10 Hz latent, and 32 latent channels.It shares the baseline codec’s architecture and loss, but is trained for 400k steps on 4 nodes with 32 B200 GPUs and a global batch of 32 clips.Source: §A.1 · p. 42
• The demo codec uses the AdamW optimizer and baseline warmup schedule with torch.compile enabled, alongside distributed data-parallel training across 32 B200 GPUs.Its 400k-step training run uses 4 nodes and a global batch of 32 clips.Source: §A.1 · p. 42

B Data Details

• Policy outputs become nine-key actions by thresholding axes at ±0.5 and conditioning mappings on whether the car is grounded or airborne.Buttons fire on any non-zero value, and the key vocabulary has different in-game meanings on the ground and in the air.Source: B Data Details · p. 44, Table 13 · p. 44, Table 12 · p. 44
• Optional action noise affects 16.52% of matches, while 83.48% remain fully clean; models train only on approximately 10,000 clean match-hours.Noisy players are independently selected with 50% probability, and recordings retain per-view noise metadata for filtering or reweighting.Source: B Data Details · p. 45
• 99,408 matches yield roughly 12,454 match-hours across three near-uniformly represented maps, including 9,998 clean and 2,456 noisy match-hours.The processed corpus contains 397,632 per-view recordings and was collected over about 26 days across 120 concurrent VMs.Source: B Data Details · p. 45

B.1 Data processing

• Each match is aligned across four player views using the end of the first kickoff countdown as a shared master-timeline anchor.The anchor is a networked event presented at the same logical instant across clients, despite separate VM clocks.Source: §B.1 · p. 46
• Usable gameplay excludes roughly 8 seconds of recorder warm-up, retains kickoff countdowns and goal replays, and is tiled into frame-aligned 4-second chunks.Chunk identity uses (match_id, player_id, chunk_idx), with matching chunk_idx values covering the same interval across players.Source: §B.1 · p. 46
• Videos are standardized to 720p at 20 fps with H.264 encoding and no B-frames, using frame-accurate two-stage seeking and retaining no audio.The source recordings are 720p at 30 fps; B-frames are disabled to avoid non-causal temporal leakage.Source: §B.1 · p. 46
• Each chunk contains synchronized video, per-frame action labels, and per-frame physics state, linked by a shared (match, player, chunk_idx) key.Actions and physics use hold-last-seen step-function sampling, ensuring every video frame has exactly one corresponding action and physics row.Source: §B.1 · p. 46

C Physics Game State Schema

• The schema records one physics-state record per 120 Hz tick, containing match-level state, ball state, and one state record for each of four cars in 2v2.These components correspond to Tables 16–18.Source: C Physics Game State Schema · p. 47, Table 16 · p. 47, Table 17 · p. 47
• Coordinates use a Z-up world frame, with positions in Unreal units, velocities in uu/s, angular velocities in rad/s, times in seconds, and boost normalized to [0, 1].Vec3 and Quat contain three and four floats, respectively; dimensionless fields use “–” in the unit column.Source: C Physics Game State Schema · p. 47
• The is_demolished flag is always false because of a broken collection hook; demolitions are instead identified through attacker_player_id until it reverts to −1.The attacker_player_id stores the demolishing player’s id while the car remains demoed.Source: C Physics Game State Schema · p. 47
• Table 19 provides engine caps and arena geometry for normalization and sanity-checking, but these are limits and bounds rather than empirical recording extrema.The values apply to vanilla 2v2 without mutators and come from the RLBot community wiki.Source: C Physics Game State Schema · p. 47
• Each client records the full state from its viewpoint, so client-side prediction can cause minor cross-view differences; the host is the lowest-player_id client, and replay physics are held constant.Downsampling from 120 Hz to frame timestamps uses the same hold-last-seen rule as actions.Source: C Physics Game State Schema · p. 47

Action probe for controllability

• A game-state probe reads ball and car positions and velocities from the world model’s pre-head activations during multiplayer rollouts.The action recoverability ratio relies on a learned action probe whose architecture and held-out-video accuracy are reported for calibration.Source: D Action Probe for Controllability · p. 47, Figure 21 · p. 49

Architecture and training

• The probe uses frozen DINOv3-B features with an approximately 4M-parameter attention-pooling head to predict nine multi-label action logits from 8-frame chunks.Controls are positive if pressed in any frame, and training uses per-action binary cross-entropy on the single-player split, enabling transfer across real, reconstructed, and generated frames.Source: D Action Probe for Controllability · p. 50

Accuracy on real data

• 0.838 mAP and 0.896 mean AUROC on 35,000 held-out windows establish the probe’s real-video calibration across nine controls.Accuracy is high for common controls and lower for rare controls such as reverse, matching action-frequency trends revealed by ARR.Source: Table 20 · p. 50, D Action Probe for Controllability · p. 50

E Multiplayer Data Allocation at a Larger Budget

• At 3.2M player-views, a 75%-multiplayer/25% single-player blend achieves the lowest gFID, outperforming pure multiplayer and pure single-player.With the doubled budget, multiplayer trains adequately even from scratch; the result is reported in Figure 22.Source: E Multiplayer Data Allocation at a… · p. 50

F Codec latent-space ablations

• Most single-player/multiplayer data mixes improve on pure single-player training, while multiplayer-heavy mixes achieve the best gFID at the 4 s horizon.With sufficient multiplayer steps, multiplayer training also works from scratch using 0% single-player data.Source: Figure 22 · p. 51
• Smaller DINOv3 variants and EUPE-B trail DINOv3-L only slightly, while all feature extractors remain equally low-drift over long rollouts.The DINOv3-B comparison is biased on P-DINO and FDD because those metrics use a DINOv3-B backbone.Source: F Codec latent-space ablations · pp. 51–52, Table 21 · p. 51
• Intermediate-layer aggregation beats the final DINOv3-L block on every metric because earlier blocks preserve spatial detail, consistent with RAEv2 (Singh et al., 2026).The baseline aggregates seven DINOv3-L blocks and adds the deepest as a residual.Source: F Codec latent-space ablations · p. 52, Table 22 · p. 52
• Temporal 2× compression can move between codec and world model, but spatial 2× compression belongs in the codec because moving it worsens every generation metric.The final design therefore performs all compression in the codec.Source: F Codec latent-space ablations · p. 52, Table 23 · p. 52
• Adaptive gradient-norm loss balancing beats fixed weights on nearly all reconstruction and generation metrics despite its small computational cost.The comparison fixes every perceptual-term weight to 1 when removing adaptive rescaling.Source: F Codec latent-space ablations · p. 52, Table 24 · p. 53
• Upsampling at the decoder input far outperforms upsampling after the ViT blocks, making late upsampling the study’s worst codec.The preferred arrangement expands the /32 latent to the feature-extractor’s /16 grid before attention.Source: F Codec latent-space ablations · p. 53, Table 25 · p. 53

G Pixel-space rollouts

• The pixel-space world model holds together for only a few frames before drifting into warped, unstructured texture, unlike latent-space models.The rollout rapidly degrades, illustrating why pixel-space generation trails the latent baseline.Source: Figure 23 · p. 53, G Pixel-space rollouts · p. 53

H Context Noise Across Metrics

• Across six codecs, gFID, gFVD, and gFDD remain flat with past noise except for the distilled codec, whose quality collapses during clean-past rollouts but is rescued by a small amount of noise.This pattern extends the gFID result to the full codec set and the additional metrics.Source: H Context Noise Across Metrics · p. 53

I Human Evaluation

• Human evaluation complements automatic metrics through pairwise quality and action-adherence preference studies on a subset of comparisons.Quality is evaluated with pairwise Elo on accuracy and visual quality, while adherence compares action-conditioned predictions against context.Source: I Human Evaluation · p. 53, Figure 27 · p. 57, Figure 28 · p. 58
• Inference-time context noise is evaluated with gFID across six codecs as a function of past-noise standard deviation and rollout time.The visualization clips color values at 30.Source: Figure 24 · p. 54
• Action-adherence rankings correlate strongly with ARR, validating action adherence as a controllability proxy.The study reports an Elo delta between action-conditioned predictions and context across a subset of comparisons.Source: I Human Evaluation · p. 54, Figure 28 · p. 58

J Out-of-distribution recovery

• A multiplayer rollout recovers autonomously after being driven far out of distribution: views diverge into noise, then resynchronize on a scripted goal replay and resume coherent play.The model is not trained for this recovery behavior.Source: J Out-of-distribution recovery · p. 54, Figure 29 · p. 59
• Inference-time context noise is also evaluated with gFVD across six codecs over past-noise standard deviation and rollout time.The visualization clips color values at 450.Source: Figure 25 · p. 55
• Inference-time context noise is further evaluated with gFDD for each codec over past-noise standard deviation and rollout time.The visualization clips color values at 2.2.Source: Figure 26 · p. 56
• Recovery restores multiplayer consistency by respawning players in each view at the post-goal kickoff and snapping the last mis-colored car back to its correct color.These steps follow view resynchronization on the scripted goal replay.Source: Figure 29 · p. 59, J Out-of-distribution recovery · p. 59