PaperlayerView exact revision on arXiv

Source-linked AI summary

Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players

Fangfu Liu, Kai He, Tianchang Shen, Tianshi Cao, Sanja Fidler, Yueqi Duan, Jun Gao, Igor Gilitschenski, Zian Wang, Xuanchi Ren

arXiv:2605.28816v1cs.CV

TL;DR

Multi-agent world models must support independently controllable, exchangeable agents while remaining efficient and temporally consistent. γ-World combines symmetric simplex-based agent encoding with hub-mediated communication and streaming distillation, improving fidelity, controllability, and inter-agent consistency while generalizing from two to four players without additional training.

  • Problem

    Multi-agent world models need independently controllable, exchangeable agents and efficient inference for shared interactive environments.

  • Method

    γ-World combines Simplex Rotary Agent Encoding, Sparse Hub Attention, and teacher–student distillation with KV-cached streaming for multi-agent simulation.

  • Results

    γ-World improves video fidelity, action controllability, and inter-agent consistency over baselines, generalizing from two-player training to four-player simulation without additional training.

  • Takeaways & Limitations

    The model supports real-time, action-responsive multi-agent rollouts with consistent behavior across time and agent perspectives.

  • Takeaways & Limitations

    Evaluation focuses primarily on gaming and robotics, while broader heterogeneous, long-horizon validation remains future work; long rollouts may accumulate inconsistencies without explicit geometry or physical constraints.

Abstract

from arXiv · show

World models for interactive video generation have largely focused on single-agent settings, where future observations are generated from a single control signal. However, many generated environments require multi-agent interaction: multiple players, robots, or embodied agents act simultaneously within a shared space. Scaling world models to such settings requires a principled multi-agent design: agents should remain independently controllable, permutation-symmetric, and support efficient inference while maintaining consistency across time and perspectives. In this paper, we present our generative multi-agent world model for interactive simulation. It introduces Simplex Rotary Agent Encoding, a parameter-free extension of 3D RoPE that represents agents as vertices of a regular simplex in rotary angle space. This gives each agent a distinct phase while making all agents permutation-equivalent, enabling scalable agent identity without learned per-slot identities or a fixed agent ordering. To avoid dense all-to-all attention across agents, we further propose Sparse Hub Attention, where learnable hub tokens mediate token interaction across agents, reducing cross-agent attention cost from quadratic to linear in the number of agents. For real-time rollout, we distill a full-context diffusion teacher into a causal student that generates temporal blocks sequentially with KV caching, enabling action-responsive generation at 24 FPS. Experiments in multiplayer virtual environments show that our model improves video fidelity, action controllability, and inter-agent consistency over slot-based and dense-attention baselines, while generalizing from two to four players without additional training.

1. Introduction

γ-World addresses the need for controllable multi-agent world models by combining permutation-symmetric agent encoding with efficient cross-agent communication. It targets consistent, scalable interactive simulation beyond two players while improving fidelity, controllability, and inter-agent consistency.

  • Multi-agent world models must maintain consistency across time and agent perspectives because agents share and act upon the same evolving world.This extends the consistency requirement beyond standard single-agent video generation.
  • Dense joint attention and learned per-player IDs create structural limitations because every agent token interacts with every other token and identity depends on learned slot-specific representations.These limitations become increasingly restrictive as the number of agents grows.
  • Simplex Rotary Agent Encoding places agents at regular-simplex vertices in rotary angle space, preserving permutation symmetry while giving each agent a distinct identity.The encoding extends standard 3D RoPE and avoids scalar indices or learned identity vectors.
  • Sparse Hub Attention uses learnable hub tokens to aggregate and broadcast information across agent streams without dense pairwise interaction.The proposed mechanism reduces cross-agent attention cost from quadratic to linear in the number of agents.
  • Experiments in multiplayer virtual environments with two and four players show improved video fidelity, action controllability, and inter-agent consistency over slot-based and dense-attention baselines.Ablations validate the simplex rotary encoding and sparse hub communication designs.

2. Related Work

Related work spans diffusion and autoregressive video generation, video diffusion models used as visual simulators for world modeling, and distillation methods for real-time synthesis. These approaches motivate efficient visual prediction while highlighting limitations of adversarial acceleration methods.

  • Video generation: Diffusion-based models lead video generation, while latent-space modeling improves synthesis efficiency and autoregressive methods offer arbitrary-length sequence generation.Latent diffusion models enable more efficient high-quality video synthesis; autoregressive approaches are studied for extending generation to arbitrary-length sequences.
  • Video world models: Video world models repurpose video diffusion models as visual simulators, directly predicting future visual observations instead of learning compact abstract states.This family emerged alongside advances in video diffusion transformers and text-to-video and image-to-video generation.
  • Video diffusion model distillation: Distillation accelerates video diffusion sampling through few-step inference for real-time synthesis, whereas adversarial objectives can suffer optimization instability and model collapse.The passage identifies real-time generation as crucial for world models and describes adversarial training as a strategy for reducing denoising steps.

3. Method

The method formulates synchronized multi-agent video generation with an explicit agent axis and jointly predicts future observations from initial observations and per-agent actions. It combines permutation-symmetric simplex-based agent encoding, hub-mediated sparse attention, and teacher–student causal distillation for efficient streaming rollout.

  • Multi-agent formulation: The model represents multi-agent latents with an explicit agent axis and jointly predicts future observations conditioned on initial observations and per-agent action sequences.This formulation is intended to maintain consistency across time and agent perspectives.
  • Causal training and inference: During inference, future latent tokens are denoised block by block under per-agent action sequences, supporting causal multi-agent rollout with KV-cached streaming.The causal formulation avoids direct attention to future frames.
  • Agent encoding: Simplex Rotary Agent Encoding assigns agents regular-simplex vertices in rotary angle space, distinguishing identities without arbitrary ordering or learned per-slot embeddings.The encoding is parameter-free and permutation-symmetric.
  • Sparse attention: Sparse Hub Attention routes cross-agent information through learnable hub tokens while masking direct attention between distinct agent streams.Agent tokens attend within their own stream and to hubs; hubs attend to all agents and other hubs.
  • Causal training and inference: The model uses a three-stage training recipe: a bidirectional teacher, a block-causal student with Sparse Hub Attention and Diffusion Forcing, then a conditional few-step generator.This design targets low-latency streaming inference while addressing train–test mismatch during autoregressive rollout.

4. Experiments

Experiments evaluate γ-World in synchronized multi-agent Minecraft and bimanual robotics settings, comparing multi-agent modeling baselines and Sparse Hub Attention efficiency. Results show shared-world consistency in interactions, zero-shot scaling from two to four players, and coordinated physical-scene prediction.

  • Experimental setup: The evaluation uses synchronized multi-agent Minecraft trajectories, with two-agent episodes as the main setting and four-agent scenes for scaling tests.The data pipeline uses controllable episode scripts, coordinated bots, and aligned visual-action recording.
  • Comparison with multi-agent baselines: γ-World is compared with frame concatenation and Solaris, testing whether agents are modeled as distinct but coupled entities rather than aggregated visual observations.The frame-concatenation baseline merges multiple views into one visual stream, while Solaris explicitly trains on synchronized player trajectories.
  • Efficiency of Sparse Hub Attention: Sparse Hub Attention reduces latency and FLOPs relative to dense cross-agent attention as the number of agents increases from 2 to 8.The comparison reports DiT latency, self-attention latency, and self-attention FLOPs, motivated by quadratic growth in dense all-to-all interaction.
  • Two-agent interaction: Two-agent rollouts keep paired streams synchronized, reflecting one agent’s actions in the other’s observations while preserving object and agent grounding out of view.These behaviors suggest tracking a shared latent world state rather than generating independent single-agent videos.
  • Scaling beyond two players: A model trained only on two-agent data generates synchronized four-agent rollouts without architectural changes.Simplex Rotary Agent Encoding avoids fixed learned slot identities, while Sparse Hub Attention provides shared communication without dense pairwise attention.
  • Real-world robotics applications: The model also predicts coordinated motion for two interacting robot arms in physical bimanual manipulation scenes from the RealOmin-Open Dataset.This evaluation applies the same multi-agent world-modeling framework used for virtual players to real-world robotics coordination.

5. Discussion · A. More Visualizations

The discussion summarizes γ-World’s multi-agent design, combining permutation-symmetric agent identities, efficient cross-agent communication, and streaming inference for consistent, action-responsive rollouts. It also identifies evaluation, scaling, geometry, and visualization limitations or future directions.

  • 5. Discussion: γ-World targets interactive simulation beyond two players with a generative multi-agent world model.The discussion frames the system as extending world modeling beyond two-player settings.
  • 5. Discussion: Simplex Rotary Agent Encoding provides distinct, permutation-symmetric agent identities without fixed ordering.The encoding is paired with Sparse Hub Attention for efficient hub-mediated communication across agents.
  • 5. Discussion: Teacher–student distillation and KV-cached streaming inference support real-time, action-responsive rollouts consistent across time and agent perspectives.These components complement the model’s agent encoding and sparse communication mechanisms.
  • 5. Discussion: Current evaluation focuses primarily on gaming environments and robotics, leaving broader complex, heterogeneous, and long-horizon validation for future work.The stated limitation concerns the scope of empirical validation.
  • 5. Discussion: The simplex pool scales agent counts within a fixed rotary agent band, while very large populations may require larger bands or hierarchical grouping.This limitation concerns scaling the agent representation to larger populations.
  • 5. Discussion: Because γ-World does not explicitly enforce 3D geometry or physical constraints, long rollouts may accumulate inconsistencies.The discussion identifies missing geometric and physical enforcement as a remaining limitation.
  • A. More Visualizations: The supplementary video includes extended 24-second multi-agent rollouts and qualitative comparisons against baselines.These materials provide additional demonstrations beyond the main evaluation.
  • A. More Visualizations: Real-world robotic coordination videos offer an intuitive supplementary demonstration of the method.The supplementary materials complement the extended rollouts and baseline comparisons.

B. Proof of Simplex Equidistance

The proof constructs centered, normalized simplex vertices in the agent angle space, showing that every distinct agent pair is exactly equidistant. Under the stated embedding and small-angle conditions, this separation is approximately equal in complex RoPE space and becomes identical with zero padding.

  • Simplex construction: The construction embeds V simplex vertices into the d_p/2-dimensional agent angle space, requiring V ≤ d_p/2 + 1.Vertices are formed from centered one-hot vectors and normalized within the zero-mean subspace.
  • Angle-space equidistance: All distinct agent pairs are exactly equidistant in the simplex angle space.Scaling the agent angles by α preserves equal pairwise separation for every p ≠ q.
  • Complex RoPE separation: For sufficiently small α, the complex RoPE representation yields approximately equal pairwise separation across agents.The result follows from applying 1 − cos x ≈ x^2/2 when all coordinate-wise angle differences are small.
  • Complex RoPE separation: With zero padding and d_p/2 ≥ V, complex-space distances are identical across distinct agent pairs.All pairwise difference vectors have the same non-zero coordinate pattern up to permutation.

C. More Architecture · C.1. Action Design

The model uses synchronized, explicit per-agent action traces as additional conditioning signals, with separate sequences for each agent. Action representations are domain-specific: games use player controls and camera motion, while robots use continuous end-effector and gripper states.

  • C.1. Action Design: Explicit per-agent action traces are synchronized with video frames and supplied separately for each agent as conditioning signals.The action specification differs between domains.
  • C.1. Action Design: Game and robot actions are provided as temporally aligned, domain-specific signals for conditioning multi-agent video generation.Games use player control commands, whereas robots use continuous end-effector state.
  • C.1. Action Design: Game actions contain 25 fields per agent at each frame: 23 discrete player controls and 2 continuous camera controls.The controls include inventory interaction, hotbar selection, movement, item manipulation, mouse-button actions, and horizontal or vertical view motion.
  • C.1. Action Design: The 23 discrete game controls cover inventory interaction, hotbar selection, movement, item manipulation, and mouse-button actions.The two continuous controls describe horizontal and vertical camera motion.
  • C.1. Action Design: Robot actions contain 10 continuous fields per agent at each frame, describing the end-effector and gripper.The same format is used for both left and right robots.
  • C.1. Action Design: Each robot action includes 3D end-effector position, a 6D orientation field, and the gripper opening value.These fields form one temporally aligned action sequence per robot.

D. Additional Implementation Details

The implementation uses Cosmos-Predict2.5-2B-based teacher and student models with Simplex Rotary Agent Encoding for permutation-equivalent player identity. Training proceeds through bidirectional and causal pretraining followed by Self-Forcing distillation, while inference uses autoregressive latent blocks with KV caching.

  • Architecture: Both models use Cosmos-Predict2.5-2B with D=2048, 28 transformer blocks, 16 attention heads, head dimension 128, MLP ratio 4, and rank-256 AdaLN-LoRA.Simplex Rotary Agent Encoding partitions head dimensions across the t, p, h, and w axes and assigns agents regular-simplex rotation phases.
  • Architecture: A size-4 simplex pool is sampled into 2 active runtime slots with permuted ordering, enabling the same checkpoint to serve up to 4 players at inference.Player identity is disambiguated through simplex markers rather than fixed slot identities.
  • Stages 1 and 2 – Bidirectional teacher and causal student pretraining: Teacher and student training starts from the public Cosmos-Predict2.5-2B TI2V checkpoint on 2-agent gameplay at 320 × 480 per view, using staged clip lengths and AdamW optimization.The teacher uses 93-frame clips for 10,000 iterations and 189-frame clips for 6,000 iterations; the student uses 93-frame clips for 15,000 iterations.
  • Stage 3 – Self-Forcing distillation: Self-Forcing distillation trains the student with frozen real and trainable fake scores using DMD loss on 189-frame clips and four denoising timesteps.The timesteps are {1000, 750, 500, 250}, warped by flow shift 5.0.
  • Inference: At inference, the student autoregressively generates per-view latent blocks with the training schedule, while KV caching uses a 24-frame rolling local-attention window.This decouples generated sequence length from cache memory.

E. Additional Ablations

The additional ablations compare causal, bidirectional, and distilled variants using perceptual-quality and reconstruction-fidelity metrics. They also examine how the number of Sparse Hub Attention hub tokens K affects the capacity of cross-agent communication.

  • Model variants: The ablation compares causal, bidirectional, and distilled model variants.Table 5 reports FVD, FID, LPIPS, PSNR, and SSIM for perceptual quality and reconstruction fidelity.
  • Evaluation metrics: FVD, FID, LPIPS, PSNR, and SSIM evaluate perceptual quality and reconstruction fidelity.
  • Hub-token capacity: The number of hub tokens K controls the capacity of the cross-agent communication bottleneck.Hub tokens provide a compact shared state through which agents exchange information.
  • Hub-token capacity: When K is too small, limited hub capacity can hurt generation quality.Increasing K provides a richer shared state for summarizing multi-agent interactions.

F. More Experimental Results · F.1. Training Stage Comparison

The experiments compare bidirectional teacher, causal student, and distilled training stages. The teacher performs best with full temporal context, while distillation restores much of its quality in a streaming-compatible causal architecture.

  • F.1. Training Stage Comparison: The model is evaluated across three training stages: bidirectional teacher, causal student, and distilled model.These stages are explicitly compared in the training-stage experiment.
  • F.1. Training Stage Comparison: The bidirectional teacher achieves the best overall performance.Its advantage is attributed to access to full temporal context during generation.
  • F.1. Training Stage Comparison: Full temporal context during generation supports the bidirectional teacher’s performance advantage.The comparison identifies temporal-context access as the reason for the teacher’s stronger results.
  • F.1. Training Stage Comparison: The causal variant enables streaming inference with KV caching but shows degraded performance.Its limitation is that it can attend only to past frames.
  • F.1. Training Stage Comparison: The causal variant’s performance degrades because it can only attend to past frames.This restriction contrasts with the teacher’s access to full temporal context.
  • F.1. Training Stage Comparison: The distilled model recovers much of the teacher’s quality while retaining the causal structure.This demonstrates transfer of bidirectional modeling capacity into a streaming-compatible architecture.
  • F.1. Training Stage Comparison: Knowledge distillation effectively transfers bidirectional modeling capacity into a streaming-compatible architecture.The distilled model retains causal structure while recovering much of the teacher’s quality.
Loading 2605.28816v1…