PaperlayerView exact revision on arXiv

Source-linked AI summary

Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers

Sicheng Mo, Yuheng Li, Ziyang Leng, Krishna Kumar Singh, Bolei Zhou

arXiv:2607.21594v1cs.CV

TL;DR

Multi-agent video world models must maintain a shared, evolving world state across agents and views, but autoregressive pipelines primarily carry observation history. WorldWeaver introduces persistent, dynamically updated world-state registers and improves aggregate world score to 105.1 in two-agent Minecraft experiments, with gains in logical consistency and visual quality.

  • Problem

    Existing autoregressive video diffusion models carry limited observation history rather than explicit shared world states across agents and views.

  • Method

    WorldWeaver uses persistent, dynamically updated cross-agent world-state registers grounded by agent status, bird’s-eye views, and scene text supervision.

  • Results

    105.1 aggregate world score, surpassing the previous best, with gains in grounding, building, and consistency over Solaris.

  • Takeaways & Limitations

    Across multi-agent Minecraft experiments, explicit world-state modeling improves visual quality and logical consistency while making state representations more verifiable.

  • Takeaways & Limitations

    The approach relies on additional state supervision, while real-world state is more complex and not directly available.

Abstract

from arXiv · show

Multi-agent interactive world models should not only generate consistent observations, but also maintain world states that persist across agents and evolve across views. Existing autoregressive video diffusion pipelines carry forward observation history as conditioning context, which makes shared state difficult to maintain in multi-agent and multi-view settings. We present WorldWeaver (W^2), a streaming multi-agent video diffusion model that augments rollout with cross-agent world state registers: learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk. We ground these registers with supervision signals spanning individual agent status, global state views including bird's-eye views, and scene text. We further improve the architecture with a Mixture-of-Transformers design that uses separate weights for world state modeling and visual frame modeling. Extensive experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency and generation quality.

1 INTRODUCTION

The introduction identifies persistent world-state maintenance as a central challenge for multi-agent video world models and proposes explicit world state registers updated during autoregressive generation. It further grounds these registers with complementary supervision and separates state modeling from frame modeling through a Mixture-of-Transformers architecture.

  • Single-agent video world models struggle with logical and geometric consistency, a problem amplified when multiple agents observe partial projections of one 3D world.
  • Chunk-based autoregressive diffusion carries limited observation history and must repeatedly re-infer world information because each generated chunk is decoded into pixels.
  • World state registers are learnable tokens that persistently store shared world information and individual agent status while being incrementally updated during generation.
  • After each denoised video chunk, the model refreshes the register using the previous register and new observations, allowing it to condition the next chunk.
  • Register supervision spans individual agent status, bird’s-eye views, and scene text, while the MoT architecture assigns state modeling a distinct pathway coupled to visual rollout.

2 RELATED WORK

Related work spans streaming video diffusion, memory-based generation, and unified models that connect visual generation with world understanding. Streaming methods enable chunked causal generation, while memory mechanisms preserve information beyond current local context.

  • Streaming video diffusion models: Video diffusion models use bidirectional attention to synthesize coherent visual content and smooth temporal transitions under shared text conditions.
  • Streaming video diffusion models: Autoregressive video diffusion introduces chunk-level causal attention and KV caching to support streaming generation.
  • Streaming video diffusion models: Self-Forcing reduces train-test mismatch by rolling out models on their own generated frames during training, while later extensions target minute-level streaming.
  • Memory-based video generation: Memory-based video generation preserves appearance, identity, geometry, object states, and scene layout that current frames or local context do not fully specify.Explicit methods use retrieval buffers, spatial maps, 3D representations, or 4D scene memories consulted by the generator.
  • World understanding and generation in unified models: Unified multimodal models build on evidence that visual generation and understanding are connected, including applications in classification, detection, and segmentation.

3 METHOD

WorldWeaver extends latent video diffusion to synchronized multi-agent generation by adding shared world-state registers that persist across rollout steps. A three-stage curriculum and auxiliary register supervision train these registers to preserve shared scene structure, agent status, and cross-view consistency.

  • Multi-agent formulation: The model attaches a player axis to generated tensors, representing synchronized observations and actions for P agents in a shared latent video space.Each clip is represented as x ∈ R^(P×N×H×W×C), conditioned on first-frame embeddings, masked latents, and per-agent actions.
  • World state registers: World state registers are shared learnable tokens that summarize hidden global information and individual agent status, updating incrementally alongside frame denoising.A single causal-transformer forward pass predicts the next frame while refreshing the cross-agent state; the architecture also uses separate Mixture-of-Transformers weights for world-state and frame modeling.
  • Training curriculum: Training proceeds through bidirectional teacher finetuning, causal student conversion with register supervision, and self-forcing rollout training.Stage 1 learns synchronized scene structure with bidirectional cross-player attention; Stage 2 applies causal flow matching and Stage 3 rolls out the model autoregressively.
  • Causal streaming: Interleaving frame tokens and register groups makes rollout causal: each committed register precedes and conditions the next frame while frame attention remains locally windowed.The sequence is ordered as [x0, r0, x1, r1, . . . , xF−1, rF−1], with register ri conditioning xi+1.
  • Register supervision: Register supervision targets agent status, bird’s-eye-view maps, and scene text alongside the frame-level flow matching loss.The auxiliary prediction heads are discarded at inference, so the supervision does not increase rollout cost; Stage 2 sets λstate to one by default.

4 EXPERIMENTS

Experiments evaluate WorldWeaver on synchronized two-player Minecraft rollouts using VLM accuracy and FID as primary metrics. Results show that persistent registers, explicit world-state supervision, and a supervision-aware Mixture-of-Transformers architecture improve world-state consistency and generation quality.

  • Experimental setup: 126 hours of synchronized two-player videos provide agent status information, bird’s-eye-view footage, and scene-text annotations for training.Scene text is annotated for each 4-frame clip.
  • Experimental setup: The model receives both players’ first-frame observations and complete action sequences, then generates the remaining rollout; VLM accuracy and FID are primary metrics.VLM accuracy measures queried world relations, while FID measures visual realism and distributional fidelity.
  • Main results: 105.1 aggregate world score is achieved by WorldWeaver, with Solaris-category VLM accuracy rising from 81.3 to 93.8 for Grounding, 9.4 to 28.1 for Building, and 57.8 to 76.6 for Consistency.The strongest gains occur on state-sensitive categories, indicating improved logical and cross-agent consistency.
  • Supervision ablation: Registers without explicit targets improve world score from 81.0 to 93.8 over Solaris, while scene-text supervision reaches 103.2 and all three signals combined reach 105.1.The supervised variants outperform the 81.0 baseline because the targets specify information that registers should preserve across rollout steps.
  • Architecture ablation: Under scene-text supervision, the Mixture-of-Transformers backbone reaches 103.2 world score versus 91.3 for the dense model, whereas without explicit supervision MoT does not improve over dense.Separate weights also support distinct register and visual transformers during Stage 3 training.

5 CONCLUSION · APPENDIX

WorldWeaver maintains and updates shared cross-agent world state during streaming interactive video generation, improving visual quality and logical consistency in multi-agent Minecraft experiments. The paper concludes that state supervision is promising but limited by the greater complexity and unavailability of real-world state.

  • 5 CONCLUSION: WorldWeaver uses cross-agent world state registers to maintain and update shared world information during interactive video generation.The model refines remembered scene information instead of repeatedly relying on an expanding frame history.
  • 5 CONCLUSION: Explicit world-state modeling improves visual quality and logical consistency across multi-agent Minecraft experiments.Ablation results support the contribution of explicit world-state supervision, though the supplied passage does not report numerical values.
  • 5 CONCLUSION: The key improvement comes from additional state supervision, but real-world state is substantially more complex and not directly available.This limitation constrains direct transfer from the reported Minecraft experiments to real-world settings.
  • 5 CONCLUSION: Future work could model state spanning low-level 3D-consistent visual detail and high-level semantic information.The conclusion presents broader state awareness as a promising direction for future world models.
  • APPENDIX: The appendix describes data collection, including individual agent statistics, global bird’s-eye views, and cross-modal scene text states.These materials are organized under the appendix’s data-collection overview and state-specific subsections.
  • APPENDIX: The appendix documents pipeline and implementation details for state decoding, including agent statistics, bird’s-eye views, and scene text.It separately lists decoding subsections for each state representation.
  • APPENDIX: Additional appendix sections cover the Mixture of Transformers and training procedures.Training is divided into bidirectional training, causal training with World State Registers, and self-forcing with context frame and state rollout.

A DATA COLLECTION … A.3 GLOBAL STATE: BIRD’S-EYE VIEW

The dataset comprises synchronized two-agent interaction recordings with logged agent states and complementary world-state streams. It combines individual motion statistics with shared overhead-camera views that capture both agents’ spatial relationship.

  • A DATA COLLECTION: The dataset is built on the SolarisEngine simulator.
  • A.1 OVERVIEW: Approximately 126 hours of synchronized two-agent interactions are recorded at 832×480 resolution and 20 fps.
  • A.1 OVERVIEW: Each session pairs Alpha and Bravo acting concurrently in the same episode.
  • A.1 OVERVIEW: Each agent’s state is logged at 20 Hz alongside the video stream.
  • A.2 INDIVIDUAL STATES: AGENT STATISTICS: The individual state stream records each agent’s position, velocity, orientation, and ground-truth controller inputs at 20 Hz.
  • A.3 GLOBAL STATE: BIRD’S-EYE VIEW: A shared fixed overhead camera places both agents in one frame, providing a global view of their relative position and spatial relationship.

A.4 CROSS-MODAL STATES: SCENE TEXT

Scene-text supervision is collected independently from the game engine by annotating synchronized four-frame chunks from overhead and egocentric views. Captions are grounded in temporal context and controller inputs, faithfully describing agent behavior but omitting broader surrounding-scene context.

  • Data collection: Each episode is divided into four-frame chunks spanning 200 ms at 20 fps, matching the video tokenizer’s temporal downsampling factor.Every chunk combines synchronized overhead, Alpha egocentric, and Bravo egocentric views.
  • Annotation: Qwen2.5-VL-72B-Instruct annotates every recording using synchronized overhead and two egocentric views for language-grounded cross-modal supervision.The annotation process is separate from the game engine.
  • Prompting: Caption prompts condition on each agent’s ground-truth movement, action, hotbar or tool selection, and net camera rotation, with the previous chunk supplied as temporal context when available.This grounding targets observed behavior rather than action inference from pixels alone.
  • Prompting: For the first chunk, temporal context is omitted and the change line is fixed to “first block, no previous, none.”This handles episodes without a preceding temporal block.
  • Limitations: The resulting captions faithfully describe per-agent behavior but do not capture broader scene context beyond the two agents’ immediate vicinities.The limitation includes surrounding terrain and events outside either agent’s immediate vicinity.

B PIPELINE AND IMPLEMENTATION DETAILS

WorldWeaver converts a single-player video diffusion prior into a streaming two-player world model by adding an explicit persistent state pathway. Its pipeline combines supervised world-state registers with role-specific processing of register and player-frame tokens.

  • Pipeline overview: WorldWeaver converts a single-player video diffusion prior into a streaming two-player world model with an explicit persistent state pathway in autoregressive rollout.The persistent pathway is the central architectural addition described for streaming multi-agent generation.
  • World-state registers: Committed world-state registers are decoded during training using auxiliary heads for agent statistics, bird’s-eye-view features, and scene text.These auxiliary heads provide supervision across individual-agent, global-view, and textual signals.
  • Mixture-of-Transformers backbone: A Mixture-of-Transformers backbone routes register tokens and player-frame tokens through role-specific parameter branches.The passage identifies separate processing roles for world-state registers and player-frame tokens.

B.1 STATE DECODING … B.2 MIXTURE OF TRANSFORMERS

The appendix details three supervision signals for decoding world state registers and introduces a Mixture-of-Transformers backbone that separates state and visual processing while preserving the streaming rollout structure.

  • B.1 STATE DECODING: The three register-supervision signals use separate prediction heads, targets, distance metrics, and loss weights, with a combined setting summing all signal losses.The heads are discarded at inference.
  • B.1 STATE DECODING: Committed world state register tokens are projected into decoder space and placed in attention sequences with the tokens receiving supervision.Transformer-based heads use full self-attention, while scene text uses projected register prefixes.
  • B.1 STATE DECODING: Each head decodes every committed register step without pooling; combined supervision uses disjoint 64/64/128 register slices, and missing targets are skipped per head.Single-signal ablations may read the full register bank, and Table 5’s per-head weights are used directly.
  • B.1.1 AGENT STATISTICS DECODING: Each player’s agent-state target contains position, velocity, and orientation from the latent-aligned simulator frame in raw units.The implementation applies no per-field normalization, clipping, or temporal smoothing.
  • B.1.2 BIRD’S-EYE VIEW DECODING: The bird’s-eye-view decoder predicts a 16×16 grid of 768-dimensional DINOv2 ViT-B/14 patch features using 256 learnable patch queries.Dense patch tokens are used instead of the CLS token or pooled features; PCA-to-RGB is only for qualitative visualization.
  • B.1.3 SCENE TEXT DECODING: The scene-text module projects selected register slices into 2048-dimensional prefix embeddings for a frozen Llama-3.2-1B scorer that produces next-token logits.Captions use 128 training slots, ignoring padding and BOS in cross-entropy; targets are authored offline by Qwen2.5-VL-72B-Instruct.
  • B.2 MIXTURE OF TRANSFORMERS: Mixture of Transformers changes parameter sharing only: WSR tokens use a state branch, while content frame tokens use a visual branch.The streaming rollout, interleaved token layout, and causal KV-cache remain unchanged.

B.3 TRAINING

The primary combined world state model uses a three-stage training curriculum with two players and shared register, attention-head, causal-window, optimizer, precision, and batch-size settings.

  • Training curriculum: The three-stage curriculum trains with P=2 players throughout, using K=256 registers grouped into 64/64/128 heads, W=6 in causal stages, AdamW, bf16 mixed precision, and batch size 32.Table 6 specifies the three-stage setup; shared choices apply across the primary combined setting, while some entries are stage-specific.

B.3.1 STAGE 1: BIDIRECTIONAL TRAINING

Stage 1 initializes from Solaris’s official bidirectional checkpoint and trains a player-aware observation model without a register pathway. Training runs for 120K steps under Table 6.

  • STAGE 1: BIDIRECTIONAL TRAINING: Stage 1 initializes from the official Solaris bidirectional checkpoint of Savva et al. (2026).
  • STAGE 1: BIDIRECTIONAL TRAINING: A player axis is added to observations, actions, VAE latents, and first-frame conditioning.
  • STAGE 1: BIDIRECTIONAL TRAINING: Training runs for 120K steps under Table 6 without using a register pathway.

B.3.2 STAGE 2: CAUSAL TRAINING WITH WORLD STATE REGISTERS · B.3.3 STAGE 3: SELF-FORCING WITH CONTEXT FRAME AND STATE ROLLOUT

Stage 2 trains a causal student with world state registers, mixture-of-transformers components, and state heads after an initial register-free phase. Stage 3 then self-forces with generated context, state rollouts, and separately trained fake-score modeling.

  • B.3.2 STAGE 2: CAUSAL TRAINING WITH WORLD STATE REGISTERS: Stage 2 first trains a causal student without registers for 60K steps.The register-free phase is followed by continued training with WSR, MoT, and three state heads.
  • B.3.2 STAGE 2: CAUSAL TRAINING WITH WORLD STATE REGISTERS: Stage 2 continues training with WSR, MoT, and three state heads for 20K steps.The combined schedule is 60K+20K steps.
  • B.3.2 STAGE 2: CAUSAL TRAINING WITH WORLD STATE REGISTERS: The 60K+20K schedule matches training with WSR for the full 80K steps in the reported checks.This comparison is stated directly for the training schedule.
  • B.3.2 STAGE 2: CAUSAL TRAINING WITH WORLD STATE REGISTERS: Stage 2 uses flow matching loss and register loss on ground-truth latents under Diffusion Forcing noise.Training uses no live teacher or ODE rollout, and register weights follow Table 5 without an extra global multiplier.
  • B.3.3 STAGE 3: SELF-FORCING WITH CONTEXT FRAME AND STATE ROLLOUT: Stage 3 loads the matching Stage-2 combined checkpoint while freezing sreal and training sfake separately from the Stage-1 bidirectional checkpoint.Both sreal and sfake start from the Stage-1 bidirectional checkpoint.
  • B.3.3 STAGE 3: SELF-FORCING WITH CONTEXT FRAME AND STATE ROLLOUT: Each Stage-3 update performs a no-gradient 1000→750→500→250→0 rollout, then a gradient pass on generated context for DMD and register losses.The rollout uses a shared early exit.
  • B.3.3 STAGE 3: SELF-FORCING WITH CONTEXT FRAME AND STATE ROLLOUT: Stage 3 raises agent-statistics and scene-text weights, leaves the BEV weight unchanged, and trains for 3–4K steps.These weight changes are relative to Stage 2 and refer to Table 5.
Loading 2607.21594v1…