PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiplayer Interactive World Models with Representation Autoencoders

Anthony Hu, Václav Volhejn, Adrien Ramanana Rahary, Chris Mulder, Aditya Makkar, Alyx Liao, Amélie Royer, Manu Orsini, Adam Jelley, Eloi Alonso, Florian Laurent, Fredrik Norén, James Swingos, Jan Hünermann, Kent Rollins, Lucas Hosseini, Matthieu Le Cauchois, Maxim Peter, Pim de Witte, Tim Brown, Vincent Micheli, Moritz Böhle, Gabriel de Marmiesse, Viktoriia Sharmanska, Lucia Specia, Michael Black, Patrick Pérez

arXiv:2607.05352v2cs.CVcs.AIcs.LG

TL;DR

Multiplayer world models must predict how several interacting agents jointly change a shared environment rather than treating other agents as part of it. MIRA addresses this gap with a latent diffusion model for four-player Rocket League, achieving real-time generation and coherent long-horizon behavior while systematically studying design choices and scale. The paper also releases its data, code, and demo, while reporting persistent failure cases involving rarely observed resting balls and occasional uncommanded car actions.

  • Problem

    Existing world models mostly adopt a single-agent perspective, limiting explicit control over multiple participants in multi-agent learning and interaction.

  • Method

    MIRA predicts future video in a representation-codec latent space, conditioned on every player’s actions and past video context to render coherent views of four-player Rocket League.

  • Results

    MIRA generates four-player matches at 20 frames per second on one Nvidia B200 GPU, with distributional quality steady to five minutes and observed rollouts continuing for hours.

  • Takeaways & Limitations

    The study finds that multiplayer conditioning improves prediction over single-player counterparts and that the model develops cross-view game-state consistency and recovery under human control.

  • Takeaways & Limitations

    Rare resting-ball states can cause the ball to gain speed and roll toward a goal, while human play also revealed occasional uncommanded boosts and jumps.

Abstract

from arXiv · show

We introduce the first multiplayer world model for highly dynamic environments governed by complex physical interactions. Whereas single-player world models treat the other agents as part of the environment, ours conditions on the action streams of multiple agents, learning to attribute changes in the scene to the correct player and to stay coherent under arbitrary combinations of their actions. We study this problem in the game of Rocket League, where players compete and cooperate under fast, tightly coupled dynamics. Trained on 10,000 hours of gameplay collected with publicly available bots, our 5-billion-parameter latent diffusion model generates four-player matches in real time, producing 20 frames per second on a single Nvidia B200 GPU. Although trained only on short clips, its rollouts stay stable far beyond the training horizon: distributional quality holds steady out to five minutes, the longest horizon we measure, and in practice we observe rollouts continuing for hours with no sign of collapse. We systematically investigate the central design choices: the video codec, the generative objective, and the multiplayer conditioning scheme. In addition, we characterize how behavior changes with model and data scale, including the capabilities that emerge and the failure modes that persist. We further develop targeted evaluations that probe the model's physical understanding rather than visual appearance alone. To support continued research on multiplayer world models, we release our dataset, our full training and inference codebase, and a live demo.

1 Introduction

The paper introduces MIRA, a multiplayer world model for Rocket League that predicts shared scene evolution from multiple players’ actions. It addresses the need for agent-attributable, coherent dynamics and studies model design, scaling, evaluation, and release resources.

  • World models support controllable simulation by predicting future latent states from actions, enabling training, evaluation, and action consequence imagination.
  • MIRA conditions on multiple players’ action streams to model four-player Rocket League dynamics involving fast, tightly coupled physical interactions.
  • Multi-agent world models must attribute scene changes to the correct players, represent interactions, and remain coherent under arbitrary action combinations.
  • The authors release 10,000 hours of training data, full training and inference code, and a live demo to support further multiplayer world-model research.
  • MIRA operates in a video-codec latent space and generates 20 frames per second in real time on a single Nvidia B200 GPU.
  • The study systematically examines representation, generative objective, multiplayer conditioning, scaling, emergent capabilities, failure modes, and evaluations of physical understanding, visual fidelity, and human preference.

2 Related Work

Related work spans latent control models, controllable video generators, and generative video architectures. The paper positions MIRA as a real-time multiplayer model that must combine action control, visual fidelity, physical correctness, and long-horizon coherence.

  • World models include latent dynamics systems for planning or training agents and controllable video generators whose rendered frames are the prediction objective.
  • Generative world models range from view synthesis without dynamics to interactive game models whose frames change in response to player actions.
  • Multiplayer dynamics are especially difficult because several agents jointly reshape a shared environment, while prior systems may condition on only one player’s controls.
  • Video-generation backbones include diffusion, latent diffusion, flow matching, transformers, video diffusion, and token-autoregressive alternatives, but realism alone does not ensure causality or action control.
  • Latent prediction reduces dimensionality and unpredictable texture burden, yet pixel-space prediction remains competitive in some domains, leaving the best prediction space unresolved.
  • Long-horizon real-time generation must address exposure bias, while evaluation requires physical plausibility and temporal criteria beyond frame-level visual fidelity.

3 Data

The dataset consists of large-scale, self-play 2v2 Rocket League recordings pairing each player's video with synchronized actions and physics state. It uses reproducible bot-driven matches across three arenas, while limiting behavioral and environmental diversity.

  • Game environment: Every car is controlled by an independent Nexto bot instance, producing reproducible matches without human data.Nexto emits discrete axis commands and binary actions at 15 times per second.
  • Recorded signals: Each player recording synchronizes gameplay video, physics state, and bot actions on a shared timeline.All four recordings share a match identifier and kickoff-countdown alignment.
  • Recorded signals: The model trains on issued keypress actions, while intended and engine-consumed actions provide sanity checks.Processing resamples raw streams into 20 fps video and per-frame action labels.
  • Scale: ≈10,000 match-hours comprise 82,983 matches and 331,932 per-view recordings, evenly split across three arenas.The dataset contains four player recordings per match.
  • Limitations: Behavioral diversity is limited because all four players use the same bot policy, while environmental diversity is limited to three fixed maps.The nearly deterministic setting also contains less predictive uncertainty than partially observed environments.
  • Physics game state: Privileged physics state is excluded from model inputs and reserved as evaluation ground truth for physical understanding.Logged quantities include ball and car dynamics; state capture occurs exactly at 120 Hz without interpolation.

4 Method

The method combines a representation codec with a latent world model that predicts all players’ futures from their action streams. It uses flow matching, diffusion forcing, and few-step distillation to support stable, fast autoregressive rollouts.

  • World model: The model predicts all four players’ next latent states jointly from past latents and every player’s actions, then decodes them into video.It learns from recorded play without access to the game engine or privileged state.
  • Codec: A frozen DINOv3-L feature extractor supplies semantically meaningful features, while a learned bottleneck compresses them into a compact latent representation.Intermediate feature blocks provide spatial and low-level detail, while the deepest selected block retains semantics.
  • Codec: The bottleneck emits latents at 10 Hz from 20 fps video and reduces the number of values by about 192× relative to raw RGB frames.A single 2×2×2 patchifier reduces channels from 1024 to 32 and coarsens the spatial grid.
  • Codec: The decoder restores spatial resolution with an unpatchify layer, reconstructs video using factorized spatial and causal temporal attention, and upsamples time at the end.Attention operates at the latent 10 Hz rate, while each latent frame expands into two video frames.
  • Codec: The codec combines L1, LPIPS, and DINOv3 feature-consistency losses, with adaptive gradient-norm weighting so neither perceptual term dominates reconstruction training.The latent is deterministic, with no discriminator, injected noise, or KL penalty.
  • Generative objective: Flow matching trains the world model to predict velocity from noise toward target latents, conditioned on past latents and players’ actions.At rollout, the model integrates this velocity autoregressively one latent frame at a time.
  • Generative objective: Diffusion forcing exposes frames to independently sampled noise levels, while progressive self-distillation enables one- or two-evaluation sampling.These choices target robustness to imperfect generated context and real-time inference.

5 Streaming inference

Streaming inference generates latent frames autoregressively while decoding and displaying video continuously. Caching and system-level optimizations allow the 5B model to meet a 20 fps target on one Nvidia B200 GPU.

  • Streaming loop: The online demo conditions each new frame on recent player actions and previously generated frames, producing a 10 Hz latent stream decoded to 20 fps video.A single Nvidia B200 GPU keeps up with the rate of play.
  • Streaming loop: A key-value cache and fixed window of 20 latents keep per-frame world-model cost constant as the rollout progresses.The codec decoder uses the three most recent latent frames for constant-cost frame decoding.
  • Streaming loop: Rollouts are primed with a short real gameplay clip whose encoded latents prefill the cache and provide an in-distribution starting context.After prefill, every displayed frame is model-generated.
  • Sampling: Few-step sampling denoises each new frame using the distilled sampler on a linear-quadratic schedule.This keeps sampling compatible with the fixed per-frame generation budget.
  • Serving: Producer-consumer decoupling smooths delivery by separating GPU generation, host-side JPEG encoding, queueing, and frame sending.Because each world-model step emits two frames, the queue converts bursts into evenly spaced output.
  • Serving: 70 ms end to end produces two video frames, or about 35 ms per frame, within the 50 ms budget required for 20 fps.The resulting headroom absorbs scheduling jitter and supports smooth pacing.

6 Experiments

The experiments evaluate latent representations, training objectives, conditioning strategies, and scaling behavior for interactive multiplayer world modeling. Results show that learned latents, diffusion forcing, perceptual losses, and staged multiplayer training support controllable, stable, real-time rollouts.

  • 6.1 The flagship model: A 5-billion-parameter model renders four-player Rocket League views at 20 fps on one Nvidia B200 GPU while remaining controllable under simultaneous action streams.The flagship model uses tiled four-player conditioning and remains coherent far beyond its short training window.
  • 6.3 Designing the codec's latent space: Latent-space models outperform pixel-space models by an order of magnitude in generation quality and controllability, while pixel rollouts drift into unstructured texture within a second.The comparison uses matched total training budgets, with ARR measuring action recoverability.
  • 6.3 Designing the codec's latent space: Pretrained DINOv3-L features improve generation and long-horizon stability, even though a from-scratch extractor reconstructs more sharply.A learned bottleneck adds a small consistent margin, but random or PCA projections of pretrained features remain predictable.
  • 6.3 Designing the codec's latent space: A 10 Hz latent leaves generation unchanged while halving world-model sequence length, enabling real-time 20 fps rollout without quality loss.Perceptual losses are complementary: removing either LPIPS or P-DINO hurts generation, and removing both collapses it.
  • 6.4 World model objectives: Diffusion forcing improves 4 s generation metrics and prevents long-rollout drift, whereas teacher forcing collapses to much higher gFID.Adding inference-time context noise is unnecessary for stable pretrained or from-scratch codecs but rescues the fragile distilled codec.
  • 6.6 Going from single-player to multiplayer: Action recoverability continues improving after visual quality saturates, and single-player pretraining followed by multiplayer training performs best at the tested budget.The multiplayer model also keeps four viewpoints mutually consistent during salient events such as goals and demolitions.

7 Conclusion

MIRA is a real-time four-player Rocket League world model that conditions on every player’s actions, renders consistent views, and remains coherent during long rollouts. Its evaluations show behavior beyond frame prediction, while its main limitations arise from narrow training diversity and limited memory.

  • MIRA is a real-time world model of 2v2 Rocket League for four simultaneous players.It predicts in a video-representation latent space and renders each player’s view.
  • 20 frames per second on a single Nvidia B200 GPU, with coherent hour-long rollouts and mutually consistent four-player views.The model conditions on every player’s actions and keeps shared events consistent across views.
  • Pretrained latent features, diffusion forcing with few-step distillation, and all-four-player conditioning are the central design choices behind the result.These choices improve rollout stability, inference speed, or multiplayer performance relative to the tested alternatives.
  • The model develops unsupervised multi-view behavior, keeping goals, demolitions, and HUD elements aligned while remaining controllable under human play.It can also recover when a view is pushed out of distribution.
  • The main scope limits are one game, one family of bot policies, a context window too short to retain match state, and imbalanced data.Future work targets more varied behavior, longer memory, and real environments.
  • The authors release the dataset, full training and inference code, and a live demo.

A Implementation Details

The implementation uses a frozen representation autoencoder and shared latent world-model infrastructure for single-player and multiplayer experiments. The live demonstration uses the largest 5B-parameter four-player configuration trained with diffusion forcing on roughly 10,000 clean match-hours.

  • The baseline codec is a frozen, temporally downsampled DINOv3 representation autoencoder used by both world-model variants.Its training configuration is specified in Table 9, with ablations using the 125k-step checkpoint.
  • Both baseline world models share the latent model, optimizer, and frozen codec; they differ in player count and multiplayer conditioning.
  • The live demo uses a 5B-parameter four-player action-conditioned flow-matching transformer trained with diffusion forcing in the RAE latent space.It is trained on approximately 10,000 clean match-hours.
  • The live-demo codec processes 288 × 512 video at 20 fps into 10 Hz latents with 32 channels.It was trained for 400k steps on 32 B200 GPUs across four nodes.

B Data Details

The dataset pipeline synchronizes four player recordings, selects gameplay windows, and produces aligned video, action, and physics chunks. The processed corpus contains nearly 100,000 matches spanning clean and noisy recordings, while the models train on the clean subset.

  • The nine-key action vocabulary is mapped to in-game controls according to whether the car is grounded or airborne.Nexto outputs are converted by thresholding axes at ±0.5 and handling buttons as binary presses.
  • All models train on clean, unaltered-policy recordings, while the released corpus also contains noisy views generated by optional action-noise injection.The full collected corpus is approximately 12,500 match-hours, with 16.52% of matches containing at least one noisy view.
  • 99,408 matches produce 397,632 per-view recordings totaling roughly 12,454 match-hours after processing.The corpus includes approximately 9,998 clean and 2,456 noisy match-hours.
  • Cross-player alignment uses the end of the first kickoff countdown as a shared master-timeline anchor.This compensates for non-comparable local clocks across virtual machines.
  • The pipeline drops about eight seconds of recorder warm-up, retains gameplay through kickoffs and goal replays, and tiles intervals into frame-aligned four-second chunks.Matching chunk indices across players identify the same timeline interval.
  • Videos are standardized to 720p / 20 fps, encoded with H.264 without B-frames, and retained without audio.
  • Each chunk contains synchronized video, per-frame action labels, and per-frame physics state files keyed by match, player, and chunk index.Actions and physics use hold-last-seen sampling on the shared timeline.

C Physics Game State Schema

The physics schema records match, ball, and four-car state in Rocket League’s world frame, with explicit units and coordinate conventions. Client-side prediction creates minor per-view differences, while demolition status requires an indirect field because the dedicated flag is broken.

  • Each physics record contains match-level state, ball state, and one car state for each of the four cars in 2v2.The source telemetry is recorded at 120 Hz.
  • Vec3 and Quat use the game’s Z-up world frame, with positions in Unreal units, velocities in uu/s, angular velocities in rad/s, and times in seconds.Boost is normalized to [0, 1] in the game code but displayed as 0–100 during play.
  • The is_demolished flag is always false because of a broken collection hook; demolitions are recovered from attacker_player_id instead.That identifier holds the demolishing player’s ID while the car is demoed and later returns to −1.
  • Engine caps and arena geometry provide normalization and sanity-checking ranges, not empirical minima and maxima from the recordings.
  • Each client stores the full game state from its viewpoint, so client-side prediction can create minor differences across views; the host is the lowest player_id.During goal replays, physics values remain frozen at the last pre-replay state.

D Action Probe for Controllability

The paper evaluates controllability with learned probes that recover game states and actions from model representations or generated video. The action probe is calibrated on held-out real gameplay before being used to assess action recoverability.

  • Action Probe: The action probe is a learned model used to measure the action recoverability ratio.Its calibration is reported on real held-out video so probe accuracy can be judged.
  • Game-State Probe: The game-state probe reads ball and car positions and velocities from the world model’s pre-head activations.This probes whether internal representations retain physically meaningful state information.
  • Action Probe: The probe processes 8-frame chunks with a frozen DINOv3-B encoder and predicts nine multi-label action logits.A control is positive when pressed in any frame of the chunk.
  • Calibration: 0.838 mAP and 0.896 mean AUROC are achieved on 35,000 held-out windows of real video.Common controls are more accurate than rare controls such as reverse.

E Multiplayer Data Allocation at a Larger Budget

At twice the single-player training budget, multiplayer training is no longer step-starved, and multiplayer-heavy data mixtures perform best. A 75%-multiplayer blend achieves the lowest reported gFID.

  • Larger Budget: At 3.2M player-views, multiplayer trains adequately even from scratch, while multiplayer-heavy mixtures perform best.This larger budget removes the undertraining affecting the single-player-budget sweep.
  • Larger Budget: A 75%-multiplayer blend with 25% single-player matches the lowest gFID and beats both pure multiplayer and pure single-player.The comparison is made in the larger-budget allocation sweep.

F Codec latent-space ablations

The codec ablations favor pretrained, multi-layer latent representations, codec-side compression, adaptive loss balancing, and early decoder upsampling. Pixel-space generation drifts rapidly, whereas latent-space models remain stable.

  • Feature Extractor: Pretrained feature extractors perform similarly, with smaller DINOv3 variants and EUPE-B trailing DINOv3-L only slightly while resisting long-rollout drift.The reported result suggests low drift is shared across these pretrained extractors.
  • Layer Aggregation: Aggregating several DINOv3-L blocks beats using only the final block on every metric.Earlier blocks preserve spatial detail that the deepest layer discards.
  • Compression Placement: The 2× spatial reduction belongs in the codec, while placing the 2× temporal reduction in the codec or world model performs equivalently on generation.The chosen design performs all compression in the codec.
  • Loss Balancing: Fixed perceptual-loss weights worsen nearly all reconstruction and generation metrics compared with adaptive gradient-norm balancing.The adaptive scheme is retained despite its small training cost.
  • Decoder Upsampling: Upsampling the latent at the decoder input, before ViT blocks operate at the finer grid, far outperforms upsampling at the output.Output-side upsampling is reported as the worst codec in the study.
  • Prediction Space: Pixel-space rollouts remain coherent for only a few frames before drifting into warped, unstructured texture.Latent-space models do not share this failure.

H Context Noise Across Metrics

Across the codec suite and generation metrics, context-noise behavior is consistent: most codecs remain stable across noise levels, while the distilled codec requires a small amount of noise to avoid rollout collapse.

  • Cross-Metric Pattern: Across gFID, gFVD, and gFDD, every codec is flat in context noise except the distilled codec.For the distilled codec, clean past context causes quality to collapse over rollout, while a small amount of noise rescues it.

I Human Evaluation

Human studies evaluate both visual quality and action adherence, while automatic metrics track rollout quality across noise levels, times, and codecs. The model’s action-adherence ranking correlates strongly with ARR, and out-of-distribution views can recover coherence without recovery-specific training.

  • gFID, gFVD, and gFDD are evaluated against past-context noise standard deviation and rollout time across the six codecs.
  • Action-adherence ranking correlates strongly with ARR, validating ARR as a controllability proxy.
  • Human evaluation reports pairwise Bayesian Elo with 95% confidence intervals for accuracy and visual quality.
  • Action-adherence evaluation compares predicted video against context, with higher Elo indicating raters more often prefer the prediction.
  • After an intentional out-of-distribution excursion, four views diverge into noise, then resynchronize during a scripted goal replay and resume coherent play.
  • The recovery sequence respawns players in their own views and restores a mis-colored car’s correct color, despite no training for recovery.
Loading 2607.05352v2…