Source-linked AI summary
MultiWorld: Scalable Multi-Agent Multi-View Video World Models
Haoyu Wu, Jiwen Yu, Yingtian Zou, Xihui Liu
TL;DR
Existing video world models generally focus on single-agent environments and do not adequately handle multi-agent interactions or cross-view consistency. MultiWorld introduces agent-specific action conditioning and a 3D-aware global state encoder for scalable multi-agent, multi-view generation. Experiments on games and multi-robot manipulation show improvements across video quality, action controllability, and multi-view consistency, while large-scale training remains unexplored.
Problem
Existing video world models largely assume a single agent and fail to maintain scene consistency across multiple agents’ viewpoints.
Method
MultiWorld combines a Multi-Agent Condition Module for agent-specific controllability with a Global State Encoder that aggregates observations into a consistent 3D-aware environmental state.
Results
MultiWorld consistently outperforms competitive baselines in video quality, action controllability, and multi-view consistency across game and multi-robot manipulation datasets.
Takeaways & Limitations
The framework supports multi-agent and multi-view generation across variable agent and camera counts while maintaining synchronized simulation.
Takeaways & Limitations
Large-scale training remains unexplored because of computational constraints.
Abstract
from arXiv · showhide
Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present \textbf{MultiWorld}, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/
1 Introduction
MultiWorld addresses the limitations of single-agent world models by jointly supporting controllable multi-agent actions, consistent multi-view generation, and variable agent-view configurations. It combines specialized conditioning and global-state mechanisms, and is evaluated on games and collaborative robotics.
- Existing video world models largely assume one agent and fail to preserve consistency across observations from different viewpoints.
- Multi-agent, multi-view modeling requires agent-specific action assignment, geometrically consistent observations, and scalability across varying agent and camera counts.
- MultiWorld uses the Multi-Agent Condition Module, including Agent Identity Embedding and Adaptive Action Weighting, for multi-agent controllability.Identity embeddings distinguish agents, while adaptive weighting prioritizes active agents over static ones.
- The Global State Encoder aggregates observations into a global 3D-aware environmental state to anchor synthesis across views.
- The framework is evaluated on a dual-agent game dataset and a variable-scale multi-robot manipulation simulator covering changing agent and viewpoint configurations.
- MultiWorld consistently outperforms competitive baselines in video quality, action controllability, and multi-view consistency.
2 Related Work
Prior interactive video world models have explored action-conditioned simulation but remain limited in multi-agent settings and often rely on fixed configurations or costly processing. MultiWorld uses a global-state representation to support scalable multi-agent, multi-view modeling.
- Interactive video world models condition future-state generation on signals such as camera or action controls, alongside properties including physical consistency.
- Existing multi-agent video world models remain underexplored, with challenges in controlling multiple agents and maintaining multi-view consistency.
- Prior approaches assume fixed agent and view counts, while interleaving multiple views can become computationally and memory intensive as views increase.
- MultiWorld’s Global State Encoder compresses cross-view information into a compact latent representation for scalable multi-agent, multi-view modeling.
- The pipeline combines an action-conditioned diffusion backbone, the Multi-Agent Condition Module, and the Global State Encoder, with scalability and autoregressive generation components.
3 Method
MultiWorld combines an action-conditioned diffusion backbone with modules for scalable multi-agent control and 3D-aware multi-view consistency. Its shared global state supports arbitrary agent and view counts, parallel view generation, and autoregressive simulation.
- Backbone and formulation: MultiWorld formulates multi-agent, multi-view simulation as parallel image-action-conditioned video generation tasks sharing a global environment state.The framework considers K agents and C camera views independently, with initial frames supplying scene context.
- Backbone and formulation: Frame-wise causal masking restricts each video token to current and earlier actions, preventing future-information leakage during autoregressive generation.This supports stable long-horizon generation.
- Multi-Agent Condition Module: MACM resolves agent identity ambiguity and models inter-agent interactions before injecting aggregated action tokens into the diffusion backbone.AIE applies agent-specific rotations, self-attention captures interactions, and AAW prioritizes active agents over static ones.
- Global State Encoder: The Global State Encoder uses VGGT features from multiple views, fuses them with an MLP, and injects the resulting representation through cross-attention.The fused representation retains 3D spatial information without explicitly reconstructing a 3D point cloud.
- Scalability and efficiency: The shared global representation improves spatial consistency, supports arbitrary view counts, and enables parallel generation of different views.MultiWorld also supports arbitrary agent counts through extrapolatable relative identity embeddings and variable view counts through global-state compression.
- Scalability and efficiency: Parallelized view generation achieves an approximate 1.5× speedup over sequential generation in double-view simulation.Inference latency remains nearly constant with view count when computational resources are scaled accordingly.
4 Experiment
MultiWorld is evaluated across multiplayer video games and multi-robot manipulation, with experiments testing performance, scalability, architectural components, qualitative behavior, failure trajectories, and long-horizon generation. It consistently outperforms baselines while improving action following, visual quality, physical coherence, and multi-view consistency.
- Main Experiments: MultiWorld generalizes across multiplayer video games and multi-robot manipulation, including scenarios with varying numbers of agents and camera viewpoints.The evaluation compares the framework with baselines in a dual-player game and extends it to multi-robot manipulation for scalability.
- Main Experiments: MultiWorld consistently outperforms all baselines on FVD, PSNR, action-following, and RPE metrics in both evaluated scenarios.Table 1 reports results for the multiplayer video game and multi-robot manipulation settings.
- Ablation Study: MACM improves action-following ability, while GSE improves multi-view consistency; together, both modules improve visual quality.The ablation adds MACM and GSE step-by-step to a standard image-action-to-video baseline.
- Ablation Study: A base frequency of 20 improves Agent Identity Embedding by matching the number of agents, yielding better visual and action-following quality than the default base of 10000.At 10000, adjacent agent embeddings are nearly indistinguishable because their angular separation is minimal.
- Ablation Study: Adaptive Action Weighting improves visual and action-following quality by prioritizing influential or active agents over stationary ones.The mechanism dynamically allocates attention according to the relative importance of each agent’s action.
- Qualitative Results: Qualitative comparisons show that MultiWorld mitigates inaccurate action execution, agent disappearance, and multi-view inconsistency relative to competing methods.The framework produces physically coherent successful coordination and realistic failure modes, including inter-robot racing and collisions.
- Qualitative Results: MultiWorld generates long-horizon robot stacking videos up to 2× longer than its training context without significant quality degradation and up to 4× longer with minimal quality loss.The reported example stacks three cubes in the correct order without collisions.
- Qualitative Results: MultiWorld simulates action-controlled multi-agent interactions while maintaining physical and multi-view consistency during generation.The summary conclusion covers action adherence, agent-environment interactions, physical consistency, and cross-view consistency.
5 Conclusion
MultiWorld is a scalable multi-agent, multi-view world-modeling framework that supports variable agent and camera counts. Experiments demonstrate its effectiveness, while large-scale training and real-time or ultra-long simulation remain future directions.
- MultiWorld supports generation with a variable number of agents and camera views.
- The framework uses MACM for multi-agent controllability and GSE to enhance multi-view consistency.
- MultiWorld demonstrates effectiveness on multi-player video games and multi-robot manipulation.
- Large-scale training remains unexplored because of computational constraints.
- Future work targets real-time generation and memory mechanisms for ultra-long multi-agent simulation.
A Dataset Construction
The datasets combine extensive human gameplay with multi-robot manipulation tasks covering different agent counts and success conditions.
- The video-game dataset contains over 100 hours of human gameplay recorded at 60 FPS with synchronized keyboard and mouse actions.
- Game videos are downsampled from 1440×2560 to 320×640 for training.
- The robotics dataset covers striking, two-robot stacking, three-robot stacking, and four-robot passing tasks.
- Each robotics task includes 1,000 successful and 2,000 failure episodes to reduce success-only bias.
A.1 Dataset Preprocessing for ItTakesTwo
ItTakesTwo gameplay is filtered to retain interactive, side-by-side split-screen footage while removing cutscenes and motion-blurred segments.
- Non-interactive cutscenes where the game controls the players are removed.
- Only side-by-side split-screen frames with one player on each half are retained.
- Segments with large camera motion that may cause motion blur are removed.
A.2 Dataset Preprocessing for the Robotics Dataset
The robotics preprocessing constructs meaningful failure episodes by perturbing correct task plans with controlled execution randomness rather than using purely random actions.
- Failure cases are constructed from correct task plans while preserving the basic operation sequence.
- Controlled randomness is introduced at each step to simulate execution errors that can lead to task failure.
- Meaningful failures are nearly successful episodes, whereas completely random actions produced poor training episodes.
B Metrics
MultiWorld evaluates multi-view geometric consistency with Reprojection Error and action-following performance with an inverse dynamics model. Lower reprojection error indicates better 3D alignment and spatio-temporal stability.
- Reprojection Error: Reprojection Error (RPE) measures multi-view geometric consistency using scene reconstruction from frame features, camera poses, and pixelwise depth maps.DROID-SLAM and differentiable Dense Bundle Adjustment support reconstruction and pose/depth refinement before computing RPE.
- Reprojection Error: RPE averages projection discrepancies over valid feature correspondences between observed pixels and reconstructed 3D points.The projection function maps reconstructed 3D points into camera coordinates for comparison with observed pixel positions.
- Reprojection Error: Lower RPE indicates superior 3D alignment and enhanced spatio-temporal stability in generated videos.
- Action Following Ability: An inverse dynamics model infers discrete and continuous actions from generated video sequences to evaluate action-following performance.The model uses a ResNet-50 backbone and temporal processing layer, trained for 20 epochs with a learning rate of 10^-4.
C Implementation Details
MultiWorld trains a neural network to predict a conditional velocity field for each camera view and samples future videos through a probability flow ODE. Inference uses iterative Euler integration to generate video from noise.
- Training: For each camera view, flow matching trains v_theta to predict the velocity field conditioned on noisy observations, time, and actions.The objective minimizes the squared L2 distance between the predicted velocity and target velocity.
- Training: The flow-matching objective is an expectation over sampled time and Gaussian noise of the squared prediction error.
- Sampling: At inference, MultiWorld follows a probability flow ODE to evolve the video state using the predicted velocity field.
- Sampling: The model iteratively applies a standard Euler solver to sample video from noise.
D Supplementary Visualizations
Supplementary visualizations show MultiWorld controlling multiple agents, preserving physical effects across views, and generating cooperative robot behaviors. They also reveal a limitation: distant or small agents can appear ambiguous because of limited spatial resolution.
- Action Controllability: MultiWorld generates static videos under zero-action commands, avoiding spurious motion associated with action bias.
- Physical Consistency: Multi-view outputs preserve physical effects across viewpoints, including consistent shadows and persistent snow footprints.
- Multi-Agent Interactions: MultiWorld captures coordinated multi-agent interactions, including one agent pushing while another pulls a board, with their combined physical effects on the environment.
- Failure Case Analysis: Agents may appear ambiguous when they occupy small image regions because distant or small objects have limited spatial resolution.
- Multi-Robot Success Trajectory Simulation: MultiWorld generates physically plausible cooperative behaviors in which multiple robots coordinate successfully on manipulation tasks.