PaperlayerView exact revision on arXiv

Source-linked AI summary

OverSeer: Multi-Agent World Models Beyond What Agents See

Runjia Li, Kam Woh Ng, Minghao Chen, Junlin Han, Jianyuan Wang, Zhaochong An, Juan-Manuel Perez-Rua, Andrea Vedaldi, Shikun Liu

arXiv:2610.28554v1cs.CV

TL;DR

Video world models struggle to maintain a coherent shared world across agents, and reconstructing a world-centric state from partial views remains challenging. OverSeer jointly generates egocentric and fixed auxiliary views, compressing auxiliary streams against static references. It improves consistency when agents separate and reunite, while the authors identify limits tied to camera coverage and compression fidelity.

  • Problem

    Multi-agent world models must generate agent-specific streams that remain consistent over time and between agents, while reconstructing a world-centric state from partial observations remains challenging.

    Source: §1 · p. 2

  • Method

    OverSeer jointly generates agents’ egocentric views and fixed-pose auxiliary views as a proxy for shared world state, with δ-VAE encoding auxiliary changes relative to a static reference.

    Source: §1 · p. 2

  • Results

    When agents separate and reunite, existing models restore the partner or scene correctly in 0–12.5% of episodes, versus 38–60% for OverSeer.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Fixed auxiliary views provide a simple, efficient way to improve multi-agent world consistency beyond what any single agent sees.

    Source: §5 · p. 10

    Benefits depend on auxiliary-camera coverage; distant regions remain harder to keep consistent, and aggressive residual compression can trade fidelity—and, in fast-changing Doom scenes, consistency—for efficiency.

    Source: E Limitations and Future Work · p. 16

Abstract

from arXiv · show

Despite the success of video world models, extending them to multiple agents is difficult because it requires generating several agent-specific video streams that are consistent in time and between agents. To address this gap, we introduce OverSeer, a multi-agent world model that augments egocentric agent views with auxiliary camera views. We suggest that, by fixing the auxiliary views in space, they act as a proxy for a world-centric representation: Unlike the agents' views, which can change entirely due to a simple change in agent position, changes in the auxiliary views directly map to localized changes in the world state. We show that these fixed views substantially help the model to maintain consistency. We also show that, due to their redundant and semi-static nature, the auxiliary views can be efficiently compressed with a delta-VAE that encodes only the changes relative to a static reference image, reducing the number of tokens required to process them. Experiments in Minecraft and Doom demonstrate improved world and agent consistency, with qualitative results on real-world multi-camera captures indicating that the approach is not tied to game engines. Our compression analysis further shows that delta-VAE substantially reduces the computational overhead of auxiliary-view generation. We conclude that these fixed views provide a simple and efficient way to improve multi-agent world models.

1 Introduction

Multi-agent world models must preserve both the shared world’s history and agreement among agents, but changing, partial egocentric views make this difficult. OverSeer adds fixed auxiliary views as a shared world-centric proxy, and δ-VAE compresses their redundant content.

1 Introduction

  • World consistency preserves scene elements and past effects over time, while agent consistency requires different agents’ observations to agree about interactions.Player-centric streams make these constraints difficult because each agent sees only a partial, changing projection of the world.Source: §1 · p. 2
  • Fixed-pose auxiliary views serve as a shared world-centric proxy without requiring explicit global-state reconstruction.They cover regions beyond individual agents’ immediate fields of view and provide a shared visual reference for cross-agent interactions.Source: §1 · p. 2
  • Stationary auxiliary views change with the world rather than camera ego-motion, making temporal redundancies predictable and easier to model.δ-VAE retains an initial reference-frame latent and encodes subsequent frames as compact residuals; six auxiliary views incur 37.5% of the two agent streams’ token overhead.Source: §1 · p. 2
  • 0–12.5% to 38–60%: when agents separate and reunite, existing models restore the partner or scene correctly in 0–12.5% of episodes, versus 38–60% for OverSeer.The evaluation covers Minecraft and Doom, with qualitative results on real-world multi-camera captures; auxiliary views cost a small fraction of full video streams.Source: §1 · p. 2

2 Related Work

Prior multi-agent approaches couple changing egocentric streams or use external state, while OverSeer shifts the output space to jointly predict agent and fixed auxiliary views. Its fixed views also offer active coverage complementary to memory and exploitable temporal redundancy.

2 Related Work

  • OverSeer jointly predicts every agent’s egocentric stream alongside fixed auxiliary views, shifting the focus from input conditioning to the model’s output space.The auxiliary streams are predicted with agent streams rather than supplied as external map inputs.Source: §2 · p. 3
  • Auxiliary streams complement memory by actively covering unobserved space, and their joint prediction avoids reliance on pose estimation.Existing memory structures are built from agents’ past observations, so unobserved regions are not stored.Source: §2 · p. 3
  • Coupled-stream methods use moving, partial reference frames, while editable-map approaches model geometry and remain vulnerable to pose drift.The passage identifies stream concatenation, shared attention, tiling, and permutation-symmetric encodings as coupling approaches, and Po et al. (2026) as an external-map example.Source: §2 · p. 3
  • Generic video-token efficiency techniques allocate computational capacity uniformly across streams, so auxiliary views cost nearly as much as agent streams.OverSeer’s fixed views are presented as an opportunity to exploit redundancy rather than apply uniform compression.Source: §2 · p. 3

3 OverSeer

OverSeer augments agents’ partial, changing viewpoints with fixed auxiliary cameras that provide a persistent spatial reference. It jointly predicts agent views and compressed auxiliary dynamics, using the fixed views to retain shared-world information efficiently.

3.1 Background and Notation

  • Fixed auxiliary cameras provide stable spatial references and reveal changes beyond agents’ partial, moving views, which otherwise leave shared state and cross-agent agreement underconstrained.The cameras remain fixed over time, so their changes track changes in the world rather than agent motion.Source: §3.1 · p. 4

3.2 OverSeer Model Design

  • OverSeer jointly models future agent latents, auxiliary anchor frames, and compressed auxiliary dynamics, conditioned on initial agent frames, actions, and camera poses.Shared self-attention enables cross-stream communication, with causal temporal attention preventing future latents from informing earlier predictions.Source: §3.2 · pp. 4–5

3.3 Efficient Modeling of Auxiliary Dynamics

  • δ-VAE keeps each auxiliary view’s first-frame latent as an anchor and compresses subsequent dynamics into residual codes, reducing tokens by s^2 spatially and τ temporally.Its alignment loss weights video-VAE features toward locations that change relative to the anchor, alongside reconstruction and KL losses.Source: §3.3 · p. 6

4 Experiments

The experiments test whether auxiliary views improve multi-agent consistency across Minecraft and Doom benchmarks, then examine camera-count and δ-VAE ablations.

4 Experiments

  • The evaluation uses Solaris and MAC-Bench in Minecraft, revisit tasks in Doom, and qualitative real-world captures; MAC-Bench includes reunite-at-start, reunite-far, and dynamic-event tasks.Metrics include VLM episode accuracy and frame FID; MAC-Bench results are averaged over five runs.Source: §4 · pp. 7–8

4.1 Analysis of Multi-Agent World Modeling Benchmarks

  • On MAC-Bench, VLM accuracy rises from 0.0%/6.2%/12.5% to 60.2%/37.9%/40.5%, and OverSeer achieves the lowest FID across all tasks.OverSeer leads on grounding, memory, and consistency, but trails the best baseline on building and movement.Source: Table 1 · p. 8, §4.1 · p. 8
  • Solaris can achieve an FID of 17.06 on Reunite at Start yet answer none of its episodes correctly, showing frame-level FID misses structural memory failures.Its generated scenes can look in-distribution while filling existing craters or omitting partners.Source: §4.1 · p. 8
  • Without a shared view, γ-World fails all reunite tasks after agents lose visual contact; denser start-region camera coverage accompanies higher Reunite at Start accuracy than Reunite Far.The reported accuracies are 60.2 and 37.9, respectively.Source: §4.1 · p. 8

4.2 Ablations

  • Removing δ-VAE improves visual quality and Doom consistency, but increases the auxiliary-stream token footprint from 108 to 864 tokens per frame at 576 × 384.Residual codes remain faithful insofar as the scene stays close to its static anchor; Doom's rapid lighting changes and enemy movements create larger residual updates than Minecraft's slower dynamics.Source: §4.2 · p. 9, Table 2 · p. 9
  • Increasing auxiliary camera count improves persistent-state metrics and revisit tasks, while movement and building remain flat and gains plateau as spatial coverage saturates.The reported pattern is consistent with expanded visual coverage rather than generic model-capacity scaling.Source: §4.2 · p. 9, Figure 6 · p. 10
  • In cross-scene transfer, stronger compression lowers PSNR and donor-background leakage, while Import IoU and Mover Id. peak at 8× compression before degrading at 16×.The reported peak values are 26.7 for Import IoU and 0.66 for Mover Id.; the authors adopt 8× compression by default.Source: §4.2 · p. 10, Figure 7 · p. 10

5 Conclusion

OverSeer jointly generates egocentric agent views and fixed-pose auxiliary views as a lightweight proxy for a world-centric state.

5 Conclusion

  • Fixed-pose auxiliary views improve consistency when agents separate and reunite, while δ-VAE reduces their modeling cost by encoding changes relative to a static reference.The authors conclude that fixed views offer a simple, efficient design for modeling shared environments beyond any single agent's view.Source: §5 · p. 10

A Dataset Curation Pipelines

The appendix describes separate datasets for Minecraft, Doom, and real captures, with clips containing synchronized views; it also specifies MAC-Bench tasks and evaluation procedures.

A Dataset Curation Pipelines

  • Each domain is a separate dataset with its own trained model; clips contain 241 synchronized frames at 20 fps, with resolution varying by domain.Minecraft and real captures use 576 × 320 per view, while Doom uses 480 × 320.Source: A Dataset Curation Pipelines · p. 14
  • Minecraft uses two Mineflayer bots, six fixed auxiliary cameras, and 20,000 outdoor-terrain plus 11,000 village episodes; each episode is labeled with one of eleven interaction modes.The cameras are placed in a second pass after terrain is established, and episodes are captured at 1280 × 720 and 20 fps.Source: A Dataset Curation Pipelines · p. 14
  • Doom uses two marines and eight fixed auxiliary cameras across 68 Freedoom maps, with 20,000 traversal training episodes and 4,000 held-out arena episodes.The views are recorded at 1024 × 768 and 30 fps.Source: A Dataset Curation Pipelines · p. 14
  • Real captures use two moving head-mounted views and six static views from Replay; evaluation holds out 45 clips from 35 scenes.Replay provides twelve synchronized, geometrically and color-calibrated cameras; static cameras remain still within 5 mm.Source: A Dataset Curation Pipelines · p. 14
  • MAC-Bench agents start mutually visible, lose sight mid-episode, then encounter reunions or a midpoint event; VLM accuracy is averaged over five independent runs, excluding failed generations and unparsable judge outputs.Minecraft uses 32 episodes per task and Doom uses 100; an episode is correct only if both agents’ answers match the ground truth.Source: A Dataset Curation Pipelines · p. 14

B Auxiliary Camera Pose Sampling

At inference, OverSeer samples auxiliary camera poses from a scene-independent prior and selects cameras to maximize geometric field-of-view coverage.

B Auxiliary Camera Pose Sampling

  • Training uses auxiliary camera poses and videos, but inference requires the poses without auxiliary images.The poses are sampled at inference time.Source: B Auxiliary Camera Pose Sampling · p. 15
  • The scene-independent pose prior samples horizontal offset, height, yaw, and pitch as Gaussian variables fitted to training poses; height is non-negative and roll is fixed at zero.Candidate poses are defined relative to a canonical coordinate system whose origin is the first agent's initial position.Source: B Auxiliary Camera Pose Sampling · p. 15
  • Coverage-based selection greedily chooses K candidate cameras to maximize newly covered cells in a bounded discretized volume, using camera geometry without scene-specific visual or geometric input.The camera frusta do not account for scene occupancy or occlusion.Source: B Auxiliary Camera Pose Sampling · p. 15

C More Ablations

The feature-alignment ablation shows that slightly better δ-VAE reconstruction does not translate into better downstream world consistency. Alignment instead directs residual capacity toward dynamic regions useful to the world model.

C More Ablations

  • Removing feature alignment slightly improves δ-VAE reconstruction but lowers VLM accuracy on all three MAC-Bench tasks by 2.4–4.5 points.The ablation evaluates reconstruction quality and downstream world consistency in Minecraft.Source: C More Ablations · p. 15, Table 4 · p. 15
  • Without alignment, the residual code reconstructs the whole frame; alignment pushes it toward dynamic regions needed to keep revisited scenes consistent.These regions include changes that matter to the world model rather than persistent background geometry.Source: C More Ablations · p. 15
  • The ablation indicates reconstruction fidelity alone is not a sufficient proxy for the downstream usefulness of a compressed representation.Source: C More Ablations · p. 15

D Implementation Details

Implementation combines compressed auxiliary views with jointly generated agent and auxiliary streams. Training and inference use long-horizon rollouts with action conditioning.

D Implementation Details

  • δ-VAE reduces each auxiliary view from 864 tokens per frame to 108, an 8× bandwidth reduction.The stated compression rates are 2 × 2 spatial and 2× temporal at 576 × 384 resolution.Source: D Implementation Details · p. 16
  • OverSeer jointly generates both agents’ egocentric streams and K auxiliary views over a 241-frame horizon, conditioned on initial agent-view frames and both agents’ action sequences.K defaults to 6 for Minecraft and 8 for Doom.Source: D Implementation Details · p. 16
  • The δ-VAE training loss uses λrec = 1, λKL = 10^-5, and λalign = 10^-3.Training uses Adam with a learning rate of 10^-4.Source: D Implementation Details · p. 16

E Limitations and Future Work

OverSeer’s benefits are bounded by auxiliary-camera coverage and scene dynamics. The authors propose broader camera placement and world-scale extensions as future directions.

E Limitations and Future Work

  • Regions far from sampled auxiliary cameras, as in Reunite Far, remain harder to keep consistent.Source: E Limitations and Future Work · p. 16
  • Short-horizon egocentric movement does not benefit from auxiliary views, while aggressive residual compression trades some fidelity and, in fast-changing Doom scenes, consistency for efficiency.Source: E Limitations and Future Work · p. 16
  • Proposed directions include adaptive camera placement, re-anchored views for unbounded worlds, and scaling to larger agent populations.The authors also frame what world models render as a first-class design axis.Source: E Limitations and Future Work · p. 16

F Prompts

The prompts define frame-based judge questions for partner visibility and event states or progression, alongside a language-based action-conditioning vocabulary. They also specify judge handling and action selection.

F.1 The judge

  • Each benchmark question goes to Claude Opus 5 in one turn, and an episode is correct only when every question is answered correctly by exact match.Answers are reduced to their first non-empty line, lowercased, and stripped of quotes and trailing punctuation.Source: §F.1 · p. 16
  • Revisiting questions ask only whether the partner is visible, using one sentence per question.Source: §F.1 · p. 16
  • Partner visibility is queried from individual frames at four trajectory moments, with expected visibility based on view-cone and distance thresholds.The moments are the first generated frame, maximum separation, midpoint of the return, and rendezvous.Source: §F.1 · p. 16
  • Dynamic-event prompts ask about event outcomes, including whether a creeper remains, leaves a crater, or leaves flat ground.Source: §F.1 · p. 16
  • Event progression compares first and return frames, while presence questions test whether an event entity is visible in the first generated frame.Crop and burning-plank questions use their own specified outcome labels.Source: §F.1 · pp. 16–17
  • A sharp-frame prompt clarification distinguishes the viewer’s own arm and held item from another player.Source: §F.1 · p. 17

F.2 Action prompts

  • Actions are represented as English sentences encoded once; each frame selects one vocabulary entry by the highest keyboard or mouse-action score, defaulting to idle.The vocabulary includes entries never pressed in the episodes so it matches the game’s input.Source: §F.2 · p. 17, Table 5 · p. 17

G More Visualizations

Additional MAC-Bench episodes show OverSeer recovering partners or event states where the egocentric-only variant does not. Long-rollout visualizations cover Minecraft, Doom, and real-world multi-camera captures.

G More Visualizations

  • In Minecraft, OverSeer recovers the partner or evolved event state where ground truth does, while the egocentric-only variant does not.The episodes include three benchmark tasks and three further dynamic events.Source: Figure 8 · p. 18, G More Visualizations · p. 18
  • In Doom reunite tasks on three Freedoom maps, OverSeer restores partners after the agents separate, unlike the egocentric-only variant.The long-rollout visualization shows both agent views and all eight auxiliary views over 21 seconds.Source: G More Visualizations · p. 18, Figure 9 · p. 19, Figure 11 · p. 21
  • Long-rollout visualizations show both agent views and all auxiliary views over 30 seconds for Minecraft and real-world captures.The real-world visualization also includes a top-down layout of auxiliary cameras and agent trajectories.Source: G More Visualizations · p. 18, Figure 10 · p. 20, Figure 12 · p. 22

Related papers

Loading 2610.28554v1…