PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction

Dahyun Chung, Siyoon Jin, Hyunwook Choi, Honggyu An, Junyoung Seo, Hyunsung Kim, Seung Wook Kim, Seungryong Kim

arXiv:2610.12299v1cs.CVcs.AI

TL;DR

Egocentric world models mostly focus on single agents, while multi-agent embodied interaction calls for synchronized views that agree on actions, environments, and resulting state changes. ME-World jointly generates ego streams using shared action conditioning and environment memory, and experiments report improvements in consistency, control, identity preservation, and video quality. The results support modeling ego streams as coupled observations of an evolving shared world.

  • Problem

    Most egocentric world models assume a single agent, leaving the challenge of generating multiple views consistent with fine-grained actions, shared environments, and interaction-induced state changes.

    Source: §1 · p. 2

  • Method

    ME-World jointly denoises multiple ego streams and couples them through shared action conditioning and shared environment memory.

    Source: §1 · pp. 2–3, §3.1 · p. 4, §3.2 · p. 4

  • Results

    ME-World improves shared-world consistency, action control, identity preservation, and video quality on real and synthetic benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings support modeling ego streams as coupled observations of an evolving shared world.

    Source: §5 · p. 10

Abstract

from arXiv · show

Egocentric world models predict first-person observations conditioned on an agent's actions, but most focus on a single agent. Real embodied settings often involve multiple agents that act and interact within a shared environment. Existing multi-agent world models rely on coarse actions like locomotion, camera control, or discrete commands, leaving fine-grained embodied interactions underexplored. We formulate multi-agent egocentric world modeling as synchronized ego-stream generation for multiple agents interacting through fine-grained actions in a shared world. This requires cross-view action consistency, shared-environment consistency, and consistent propagation of interaction-induced state updates. We propose Multi-agent Egocentric World Model (ME-World), which jointly denoises multiple ego streams in a shared token sequence, conditions each stream on all agents' target-view poses, and grounds generation with shared environment memory. We train and evaluate on real and synthetic multi-agent data and introduce shared-world consistency metrics for environment, update, and identity consistency. Experiments show ME-World improves shared-world consistency, action control, identity preservation, and video quality over existing methods.

1 INTRODUCTION

Existing egocentric models largely focus on one agent, while multi-agent settings require fine-grained actions, shared environments, and interaction-driven state changes to remain consistent across views. ME-World couples ego-stream generation with action conditioning and shared environment memory; experiments report improved consistency, control, identity preservation, and video quality.

1 INTRODUCTION

  • Most egocentric models assume a single agent, while fine-grained embodied actions must remain consistent across viewpoints and update shared objects and environments.Such actions serve both as first-person controls and as events observable from other agents’ views.Source: §1 · p. 2
  • Multi-agent generation must synchronize cross-view actions, preserve a consistent shared environment, and propagate interaction-induced state changes across ego streams.The paper frames the task as generating multiple views of one evolving shared world.Source: §1 · p. 2
  • ME-World jointly denoises ego streams and conditions each stream on agents’ motions as observed from that view, grounding generation in shared environment memory.The memory warps historical observations into each target view and uses common anchor references to maintain scene appearance.Source: §1 · pp. 2–3
  • The training data combines real and synthetic multi-ego examples spanning diverse agent configurations, environments, and interaction scenarios.Existing egocentric datasets were described as primarily single-agent or lacking coverage of agent number, viewpoint diversity, interactions, and appearance variation.Source: §1 · p. 3
  • The evaluation adds Senv for consistency of generated environment regions and Supdate for interaction-induced changes, with Sid measuring identity preservation on synthetic data.These metrics complement video-quality, control-alignment, and cross-view-consistency evaluations.Source: §1 · p. 3
  • ME-World improves shared-world consistency, action control, identity preservation, and video quality on real and synthetic benchmarks.Ablations support the importance of its three core components for modeling ego streams as coupled observations of one evolving world.Source: §1 · p. 3

2 RELATED WORKS

Prior egocentric world models generally focus on one agent, while multi-agent world models address coherence across controlled views in several domains. This paper instead studies synchronized human egocentric views with fine-grained articulated actions that affect other agents and the shared environment.

2 RELATED WORKS

  • Egocentric world models cover body motion, hand and camera control, manipulation, 3D memory, and goal-directed evolution, but the cited work focuses on a single agent.The paper distinguishes its focus on consistency of multiple embodied agents’ actions and interaction-induced changes across ego views.Source: §2 · p. 3
  • Multi-agent world models address shared-view coherence using joint attention, shared scene representations, or shared memories across games, driving, robotics, and open-domain video.The paper focuses instead on synchronized human egocentric views with fine-grained articulated actions affecting other agents and the shared environment.Source: §2 · p. 3
  • Figure 3 depicts per-agent viewing-ray and shared-action conditions plus shared environment memory feeding joint generation of synchronized ego streams.The caption describes conditioning and memory construction in panel (a), followed by joint generation in panel (b).Source: Figure 3 · p. 4

3 METHOD

ME-World formulates multi-agent egocentric modeling as generating action-following ego streams that remain consistent with one shared world. It couples streams through joint generation, shared action conditioning, and shared environment memory.

3.1 OVERVIEW AND FORMULATION

  • ME-World generates future ego streams for N agents from their actions, prompts, initial frames, and pooled observation histories, with streams consistent with a shared environment and world state.The formulation is general to N agents, though the paper presents the two-agent case used in experiments.Source: §3.1 · p. 4

3.2 JOINT MULTI-AGENT GENERATION

  • A pretrained DiT jointly denoises all agents’ latent streams under shared conditioning, with cross-stream self-attention and stream-specific prompts and agent embeddings.Identical rotary positional embeddings align streams’ space-time positions in a common frame.Source: §3.2 · pp. 4–5

3.3 SHARED ACTION CONDITIONING

  • Shared action conditioning supplies each stream with ego-visible hand and arm motion, other agents’ projected body poses, and head-motion ray maps in a shared canonical frame.These cues target consistency of hands, arms, body pose, and gaze-induced viewpoint changes across ego streams.Source: §3.3 · p. 5

3.4 SHARED ENVIRONMENT MEMORY

  • To provide shared scene references beyond view-specific warped regions, ME-World greedily selects complementary history observations and appends their unwarped anchor features to the joint denoising sequence.All ego streams can attend to the same anchors during generation.Source: §3.4 · pp. 5–6
  • For each target stream, geometric memory selects the history image with greatest valid target-view coverage, warps it using target rays, masks dynamic agent regions, and provides aligned RGB and visibility conditions.The warped RGB and visibility mask are encoded and concatenated with the noisy latent for denoising.Source: §3.4 · p. 6

3.5 TRAINING OBJECTIVE

  • The pretrained model is fine-tuned with rectified flow, using concatenated clean latents and per-agent Gaussian noise in the training objective.Source: §3.5 · p. 6

4 EXPERIMENTS

Experiments assess shared-world consistency and generation quality on real and synthetic multi-agent benchmarks. ME-World improves these measures over existing methods, and ablations associate its components with distinct failure modes.

4.1 SHARED-WORLD CONSISTENCY METRIC

  • Senv measures consistency in environment regions lacking direct geometric-memory guidance, Supdate measures cross-view consistency of interaction-induced changes, and Sid measures counterpart identity.Senv and Supdate are reported on real data; all three metrics are reported on synthetic data, where distinct character appearances enable reference-based Sid evaluation.Source: §4.1 · pp. 6–7

4.2 IMPLEMENTATION DETAILS

  • The evaluation compares multi-view, single-ego, and general world models, plus adapted multi-agent models on the real benchmark.The multi-agent adaptations share a backbone, training data, and Shared Action Conditioning with ME-World while retaining each method’s characteristic mechanisms.Source: §4.2 · p. 8, Table 1 · p. 9, Table 2 · p. 9

4.3 RESULTS

  • ME-World improves shared-world consistency, self- and other-agent action control, and video quality on the real benchmark, and achieves the strongest shared-world consistency on synthetic data.On the synthetic benchmark, it also improves camera control, self- and other-agent action control, and video quality; distinct character appearances enable identity-preservation evaluation.Source: §4.3 · p. 9, Table 1 · p. 9
  • ME-World achieves the best performance across metrics against adapted multi-agent models under the embodied setting.The adaptations show greater degradation in shared-world consistency and fine-grained action control under this setting.Source: Table 2 · p. 9, §4.3 · p. 10

4.4 ABLATION STUDY

  • Ablations associate joint generation with coupled predictions, shared action conditioning with cross-view behavior alignment, and shared environment memory with consistent backgrounds.Removing components produces inconsistencies in agent appearance, embodied actions, or the shared environment across streams.Source: §4.4 · p. 10, Figure 7 · p. 10, Table 4 · p. 10

5 CONCLUSION

The conclusion frames multi-agent egocentric world modeling as agents interacting in a shared world through their own viewpoints. ME-World jointly generates ego streams with shared action conditioning and environment memory, and the paper reports improved shared-world consistency and fine-grained action control while maintaining strong video quality.

5 CONCLUSION

  • Experiments show ME-World improves shared-world consistency and fine-grained action control while maintaining strong video quality.The conclusion highlights modeling ego streams as coupled observations of an evolving world.Source: §5 · p. 10
  • The paper introduces multi-agent egocentric world modeling, in which agents interact in a shared world through their own egocentric viewpoints.Source: §5 · p. 10
  • ME-World jointly generates ego streams using shared action conditioning and environment memory, and the paper constructs real and synthetic benchmarks with shared-world consistency metrics.Source: §5 · p. 10
  • The conclusion proposes jointly modeling actions and observations, efficient cross-agent communication and memory, and efficient video generation as directions toward real-time interactive multi-agent simulation.Source: §5 · p. 10

D Implementation Details

The appendix presents additional qualitative generations from real and synthetic benchmarks, along with demonstrations of multi-agent and extended-horizon generation. The third-person views shown in some figures serve only as visual references, not as model inputs or generated outputs.

D Implementation Details

  • Figure 8 shows jointly generated synchronized ego streams across multiple timesteps for diverse real-benchmark interaction sequences.Source: Figure 8 · p. 18
  • Figure 9 shows jointly generated ego views for diverse synthetic-benchmark interactions, with third-person views included only as visual references.The third-person view is neither generated nor used as input.Source: Figure 9 · p. 18
  • Figure 10 demonstrates three synchronized ego views for three interacting agents and an autoregressive two-agent sequence spanning 221 frames.The third-person view in the three-agent example is only a visual reference and is not generated or used as input.Source: Figure 10 · p. 19

A.1 ADDITIONAL QUALITATIVE RESULTS

Additional qualitative results describe paired ego-view generations on real and synthetic benchmarks, and the appendix extends demonstrations beyond the standard two-agent, fixed-length setting. The extensions include three-agent generation and autoregressive generation over a 221-frame sequence.

A.1 ADDITIONAL QUALITATIVE RESULTS

  • On real interactions, generated paired ego views preserve shared scene content and compatible interaction dynamics despite substantial viewpoint differences.Source: §A.1 · p. 19
  • On synthetic interactions, the model preserves both agents’ appearances and coordinated interactions across generated ego views.Source: §A.1 · p. 19

A.2 GENERALIZATION TO EXTENDED SETTINGS

  • Adding a third ego stream and its agent token, pose conditions, and shared environment memory enables joint generation of three synchronized views for interacting agents.The demonstrated extension preserves a common scene and mutual interactions.Source: §A.2 · pp. 19–20
  • Autoregressive chaining produces a 221-frame two-agent sequence while maintaining the agents and shared environment over the extended horizon.Each chunk conditions on two frames from the preceding chunk; generation within a chunk remains bidirectional, and training perturbs conditional latents to expose the model to imperfect handoff frames.Source: §A.2 · p. 20

B.1.1 METRIC IMPLEMENTATION DETAILS

The metric implementation uses synchronized views and ordered query-target agent pairs to assess shared-world consistency. It defines cross-view correspondences and feature-change regions, with separate measures for environment consistency, interaction-induced updates, and synthetic identity consistency.

B.1.1 METRIC IMPLEMENTATION DETAILS

  • For each synchronized frame, an ordered pair treats agent n as the query stream and agent m as its target; the two-agent metrics average both ordered directions.Senv and Supdate are reported on both benchmarks, while Sid is reported on synthetic data.Source: §B.1.1 · p. 20
  • The implementation extracts DINOv3 ViT-L/16 features on a 32 × 32 patch grid and sets the change threshold to the 80th percentile of valid non-human background feature distances per frame.The threshold is used to account for reprojection artifacts; visibility and change masks share the patch grid.Source: §B.1.1 · p. 20
  • Cross-view correspondences are formed by back-projecting query patch centers with metric depth and ground-truth camera poses, then retaining in-bounds projections with depth agreement within 10 cm.Source: §B.1.1 · p. 20
  • Senv measures consistency of environment content without direct scene evidence, while Supdate measures consistency of interaction-induced changes across ego streams.Metric validation visualizes query regions, co-visible target correspondences, and DINOv3 cosine similarities; consistent examples receive higher scores, while localized cross-view inconsistencies correspond to similarity drops and lower scores.Source: §B.1.1 · pp. 20–21, Figure 11 · p. 21
  • On synthetic data, Sid compares the detected interaction partner’s features with multi-view references of the corresponding ground-truth avatar and averages maximum cosine similarities across valid frames and both ego streams.The references render each benchmark avatar from 12 azimuths with full-body and upper-body framings.Source: §B.1.1 · p. 21

B.1.2 METRIC VALIDATION

The proposed metrics respond differently to controlled environment and interaction-state conflicts, while qualitative comparisons show higher scores for ME-World than independent single-stream generation. Together, the tests support their intended complementary roles.

B.1.2 METRIC VALIDATION

  • Senv = 0.582 and Supdate = 0.495 for ME-World, versus 0.498 and 0.402 for independent single-stream generation.The comparison uses co-visible regions; reported scores use bidirectional evaluation, although the figure visualizes one correspondence direction.Source: §B.1.2 · p. 21
  • Senv falls by 0.090 under environment conflict, while Supdate falls by 0.147 under state conflict, supporting their intended complementary roles.Environment conflict changes Supdate by only 0.019; state conflict changes Senv by 0.059.Source: §B.1.2 · pp. 21–22
  • Temporal misalignment from shifting one stream by five frames decreases both consistency metrics.The controlled test modifies one stream in 100 benchmark clips and leaves the paired stream unchanged.Source: §B.1.2 · pp. 21–22

B.2 EVALUATION METRICS

This material defines evaluation measures and describes baseline configurations and the construction of synthetic and real training data. The data pipelines provide synchronized ego streams with motion, camera, scene, and identity conditions.

B.2 EVALUATION METRICS

  • The evaluation covers shared-world consistency, camera control, self- and other-agent action control, and video quality.Measures include ATE and mean rotation error; Hand-F1 and Hand-mIoU; Full-Body-F1 and Full-Body-PCK@10; and PSNR, SSIM, LPIPS, and FVD.Source: §B.2 · p. 22, Table 5 · p. 22

B.3.1 MULTI-VIEW VIDEO GENERATION

  • GEN3C and AnyView-DVS serve as multi-view video generation baselines, using source-view video and camera or rendered-warp conditions in different configurations.GEN3C also receives a target initial frame and caption; AnyView-DVS does not receive the target initial frame, text, or depth.Source: §B.3.1 · p. 22

B.3.2 SINGLE-EGO WORLD MODELS

  • EgoSim and JointControlVideo are single-ego baselines conditioned on first-person inputs, with hand or body-motion conditions supplied in different forms.EgoSim receives rendered wearer hand tracking; JointControlVideo receives the wearer’s 3D hand-joint trajectories and does not explicitly receive camera trajectories.Source: §B.3.2 · pp. 22–23

B.3.3 GENERAL WORLD MODELS

  • DreamX-World and LingBot-World independently generate each ego stream from its initial frame, caption, and camera trajectory.Both interpolate camera poses to 20 latent frames and express them relative to the initial camera; LingBot-World additionally normalizes translations before Plücker-ray encoding.Source: §B.3.3 · p. 23

B.3.4 MULTI-AGENT WORLD MODELS

  • Multi-agent baselines retain different combinations of joint generation, cross-agent communication, and environment conditioning.MultiWorld uses a VGGT-based global state encoder; Solaris and γ-World omit explicit environment conditioning, while MetaWorld uses cross-view attention and geometric scene evidence.Source: §B.3.4 · pp. 23–24

C DATASET CONSTRUCTION

  • Training data combines synthetic interactions and synchronized real-world recordings, represented with scene, pose, camera, reference-image, and text conditions.The synthetic source provides geometric and motion annotations, while real recordings capture natural interactions and visual complexity.Source: C DATASET CONSTRUCTION · p. 24

C.1 SYNTHETIC DATASET GENERATION

  • The synthetic pipeline retargets paired motions to avatars, places them in compatible shared scenes, and synchronously renders two head-mounted views.It produces 4,736 paired 77-frame clips from 1,822 interaction sequences, 100 avatars, seven environments, and 53 validated scene configurations.Source: §C.1 · pp. 24–25
  • Synthetic conditioning includes target-view scene warps, projected poses for both agents, and shared reference observations selected for complementary coverage.The pose conditions use identity-specific colors, and eight clean reference observations are shared across the two streams.Source: §C.1 · p. 25

C.3 BENCHMARK CONSTRUCTION

The benchmarks comprise filtered real and synthetic synchronized ego-stream clips. The model configuration starts from a multi-view diffusion transformer and conditions two synchronized streams using first-frame input and shared references.

C.3 BENCHMARK CONSTRUCTION

  • The real benchmark samples 100 disjoint validation clips with unseen kitchens and participants, balancing eligible examples across recordings and co-visibility strata.It has mean cross-view co-visibility 0.38; clips with invalid visual conditions, low co-visibility, or too few partner-visible frames are excluded.Source: §C.3 · p. 26
  • The synthetic benchmark contains 100 distinct interaction sequences across six held-out environments, with mean cross-view co-visibility 0.31.It includes 84 Inter-X and 16 InterHuman sequences across 86 avatar pairs, with exact camera, depth, segmentation, pose, and identity annotations.Source: §C.3 · pp. 26–27

D.1 MODEL DETAILS

  • ME-World starts from a Cosmos-Predict2.5 2B multi-view diffusion transformer and trains on two synchronized 77-frame streams at 10 FPS.Each view is encoded into 20 latent frames, and four shared references are selected by greedy coverage maximization over both views.Source: §D.1 · p. 27

D.2 TRAINING DETAILS

Real and Synthetic checkpoints are trained separately, then used with EMA weights to jointly generate two ego streams. Training applies the rectified-flow objective to generated latent frames.

D.2 TRAINING DETAILS

  • Separate Real and Synthetic checkpoints are fine-tuned without transferring weights, using rectified flow loss only on generated latent frames.Training uses AdamW, BF16, FSDP on four A100 80 GB GPUs, and a global batch size of 4.Source: §D.2 · p. 27
  • Inference uses EMA weights, 35 denoising steps, and guidance scale 7.0 to jointly generate two 77-frame ego streams.Source: §D.2 · p. 27
Loading 2610.12299v1…