Source-linked AI summary
ActionParty: Multi-Subject Action Binding in Generative Video Games
Alexander Pondaven, Ziyi Wu, Igor Gilitschenski, Philip Torr, Sergey Tulyakov, Fabio Pizzati, Aliaksandr Siarohin
TL;DR
Existing video world models are largely restricted to single-agent control and struggle to associate each action with its intended subject. ActionParty jointly models persistent subject states and video latents with attention masking and spatial RoPE biasing. It controls up to seven subjects across 46 environments while improving action following and identity preservation.
Problem
Existing world models are largely limited to single-agent control, while multi-subject video models struggle to associate each control signal with its corresponding subject.
Method
ActionParty jointly denoises video frames with persistent subject state tokens, using attention masking and spatial RoPE biasing to enforce action-subject correspondence.
Results
ActionParty controls up to seven subjects across 46 environments and significantly improves action-following accuracy and identity preservation.
Takeaways & Limitations
Explicit subject grounding enables precise multi-subject control and stable autoregressive tracking through complex interactions.
Takeaways & Limitations
The action space is fixed across environments, and inference assumes the initial frame and initial subject positions are known.
Abstract
from arXiv · showhide
Recent advances in video diffusion have enabled the development of "world models" capable of simulating interactive environments. However, these models are largely restricted to single-agent settings, failing to control multiple agents simultaneously in a scene. In this work, we tackle a fundamental issue of action binding in existing video diffusion models, which struggle to associate specific actions with their corresponding subjects. For this purpose, we propose ActionParty, an action controllable multi-subject world model for generative video games. It introduces subject state tokens, i.e. latent variables that persistently capture the state of each subject in the scene. By jointly modeling state tokens and video latents with a spatial biasing mechanism, we disentangle global video frame rendering from individual action-controlled subject updates. We evaluate ActionParty on the Melting Pot benchmark, demonstrating the first video world model capable of controlling up to seven players simultaneously across 46 diverse environments. Our results show significant improvements in action-following accuracy and identity consistency, while enabling robust autoregressive tracking of subjects through complex interactions.
1 Introduction
Existing video world models are largely limited to single-agent control and struggle to bind actions to the correct subjects. ActionParty addresses this with persistent subject states, spatially biased action binding, and multi-player video generation across 46 environments.
- Existing world models typically accept one control stream and control one subject, limiting their ability to simulate multi-agent dynamics.The stated applications include complex social environments such as self-driving and robotics.
- Textual instructions for multiple actors cause catastrophic action-subject association failures, even for simple two-shape movement sequences.The failure becomes more significant when several actions are described together.
- ActionParty introduces subject state tokens as persistent latent variables for each controllable subject and jointly models them with video latents.The model is designed as a multi-subject world model for multi-player game environments.
- Attention masking enforces action-subject correspondence, while 3D RoPE biases subject tokens toward their current spatial locations.Together, these mechanisms support subject localization and action-controlled updates.
- ActionParty controls up to seven players across 46 distinct environments and outperforms text-only baselines in action-following and subject consistency.The model also maintains co-generated subject states that track moving subjects through complex interactions.
2 Related work
Prior work covers single-actor world models, separately modeled multi-actor views, motion control, and attribute binding, but multiplayer action binding in a shared video remains underexplored.
- Video diffusion world models have mainly advanced single-actor or first-person game control rather than shared multi-agent scenes.Related systems learn actions from videos or gameplay inputs and simulate games such as Doom, CSGO, and Minecraft.
- Multiverse and Solaris model each actor’s view separately, making action binding easier but scaling video-token counts linearly with the number of actors.These methods are reported only for two-player settings in a single game.
- Model-based multi-agent reinforcement-learning approaches build multi-actor world models, but often train small-scale models from scratch.ActionParty is contrasted as applicable to large-scale pretrained video DiT models.
- Motion-control methods guide object trajectories with masks, bounding boxes, reference videos, or textual motion descriptions.These approaches address trajectory or motion generation rather than the paper’s multiplayer action-binding setting.
- Attribute-binding research associates descriptions or spatial relations with entities, while spatiotemporal binding of multiple video entities remains unsolved.Spatial masking and structured scene representations are among the prior approaches discussed.
3 Method
ActionParty extends video world models to multi-subject control by jointly modeling video frames with persistent subject states and explicit action binding. Its masked-attention and spatial-biasing design separates subject-state updates from video rendering while supporting autoregressive generation.
- Subject State: ActionParty jointly denoises video frames and latent subject-state tokens to associate each controllable subject with its corresponding actions.The model extends a video DiT by concatenating flattened video tokens and subject-state tokens, then predicting both future video and states.
- Problem Setup: The task generates future frames from initial context, a global game description, and per-subject action sequences across environments with discrete, shared action spaces.The same abstract action can have different effects depending on the environment and the subject’s current state.
- Subject State: Subject states are defined as 2D coordinates, providing a minimal representation that disambiguates subjects during complex interactions.The method assumes spatial position generally suffices because two subjects cannot occupy the same position in most game environments.
- Generative Game Engines: Masked cross-attention restricts each subject-state token to its own action, enforcing explicit subject-action binding during state updates.The architecture follows a game-engine-like update-and-render paradigm implemented through controlled information flow in DiT attention.
- Generative Game Engines: 3D RoPE biases subject tokens toward nearby video tokens using previous spatial positions, reducing global subject disambiguation to local refinement.The current position is unknown while being denoised, so the previous timestep provides the spatial reference.
- Autoregressive Generation: During autoregressive inference, predicted frames and subject states become context for subsequent predictions, with initial frame and subject positions required for subject identification.The context window is bounded by dropping the oldest frames once it exceeds the maximum length.
4 Experiments
ActionParty is evaluated on 46 Melting Pot games using multi-subject action-binding metrics, qualitative comparisons, autoregressive rollouts, and component ablations. It controls diverse scenes with up to seven subjects while maintaining action following, subject preservation, and localization.
- Setup: ActionParty is trained on gameplay videos from 46 Melting Pot games with a unified 25-action space and up to seven subjects per scene.The training setup includes constrained initial actions to move subjects from their starting positions and at least one “Interact” action later in each sequence.
- Setup: ActionParty is compared with text-conditioned and pretrained autoregressive baselines, including a larger zero-shot image-to-video model.The baselines are designed to test action binding from textual descriptions and provide a random-action lower bound.
- Evaluation: The evaluation measures movement accuracy, effect accuracy, subject preservation, detection rate, coordinate error, and standard visual fidelity.Movement accuracy compares extracted subject movement with input actions, while effect accuracy evaluates context-dependent “Interact” effects using local tile patches.
- Quantitative results: 0.779 movement accuracy, 0.903 subject preservation, and 0.886 detection rate show stronger action binding and subject tracking than the reported baselines.The strongest baseline reaches 0.158 movement accuracy, while the closest subject-preservation baseline reaches 0.668.
- Autoregressive stability: ActionParty maintains stable action binding over autoregressive steps, whereas baseline movement accuracy degrades toward 0.The joint modeling of subject states is reported to support relatively stable inference over time.
- Qualitative results and ablations: The model controls 2-, 4-, and 7-player games with diverse and identical-looking subjects, while ablations show that attention masks and RoPE biasing are essential for binding actions correctly.Removing the cross-attention mask reduces movement accuracy to 5.2%, and removing RoPE bias reduces it to 3.2%; removing self-attention masking reduces it to 58%.
5 Conclusion
ActionParty addresses action binding in multi-player game simulations by grounding individual subjects explicitly. Across 46 environments, it enables simultaneous control of up to seven subjects with improved action following and identity preservation.
- ActionParty jointly denoises subject state tokens with video latents to associate controllable identifiers with individual subjects.Attention masking and spatial RoPE biasing separate subject updates from global frame rendering.
- ActionParty enables precise simultaneous control of up to seven subjects across 46 diverse environments.
- Explicit subject grounding significantly improves action-following accuracy and identity preservation in autoregressive video diffusion.
Supplementary Material
The supplementary material documents additional generation settings and demonstrations, including longer context windows, expanded action spaces, baseline comparisons, and ablation videos.
- ActionParty was trained with a T = 20 context window and 20 unique actions, alongside the T = 5 setup.The supplementary material reports that generation works in both context-window settings.
- The website provides ActionParty generations and baseline comparisons for each MeltingPot game.
- The supplementary material includes ablation videos and long-horizon generation results.
A Fine-grained metrics
Fine-grained metrics are organized by the number of subjects per game to assess performance across sparse, dense, and crowded scenes. The authors report generally consistent action following and visual quality, while noting greater variability in smaller game groups.
- 17 sparse games contain 2–3 subjects, 24 dense games contain 4–5 subjects, and 5 crowded games contain 6–7 subjects.
- ActionParty maintains consistent action following and visual quality across the subject-count categories.
- Metrics may vary more in categories with fewer games because some environments are inherently more difficult to model.
B Implementation details
The implementation and evaluation details define the action categories, detection-based accuracy measurement, subject-count metric breakdowns, and a limitation of ground-truth comparison. Crowded scenes remain a key stress test because all methods degrade as subject count increases, although ActionParty is reported as the most robust.
- Action and metric setup: The dataset contains 7 base movement actions and up to 18 game-dependent Interact actions, grouped into one Interact category for metrics.This yields at most 25 actions, while 25 games use only 1 Interact action for 8 total actions.
- Scene-density analysis: All methods degrade as subject count increases, but ActionParty remains the most robust across metrics.
- Scene-density analysis: ActionParty maintains high accuracy across action types in crowded scenes, whereas baselines degrade significantly.Zero-shot I2V reports Idle and Interact as 0 because it produces only a single next-frame prediction and those actions occur later in validation.
- Action and metric setup: A linear detection and classification model measures subject position and orientation for action-accuracy evaluation.It is trained with ground-truth subject tiles as positives and other tiles as negatives, achieving 100% accuracy on that training task.
- Evaluation caveat: Ground-truth comparison can penalize generations when scene details differ, but still captures drift and insufficient action following across the video.The authors therefore consider it suitable for overall action-binding performance.
C Melting Pot details
ActionParty adapts Melting Pot games for stable video training by fixing resolution and conditioning generation on textual game and action descriptions.
- The adapted Melting Pot suite uses fixed resolution to stabilize video training across games with varied original resolutions and scales.The Wan DiT uses VAE encoding with 8× spatial downsampling, motivating reconstruction-aware preprocessing.
- Player tiles are set to 32×32 pixels so reconstructed videos preserve player orientation needed for action following.
- Text prompts describe each game setup, while baseline prompts also provide subject actions and positions.Initial positions use tile-grid coordinates such as player 1 starts at (col, row).
D Generalization
ActionParty supports long-horizon autoregressive generation and variable player counts, while showing specific drift and cross-game action-leakage boundaries.
- Long horizon: 20-step videos can be generated autoregressively from a model trained on 4 rollout steps using a sliding context window.Predicted coordinates drift from subject positions near the end because of error accumulation.
- The supplied game-description prompts specify visual environment content, with full prompts provided for all 46 games.
- Player count: The Coins game generalizes from training with 2 subjects to rollouts containing 1–8 subjects, including unseen player counts.None of the games included 1 or 8 players during training.
E Limitations
The authors identify limitations in localization, subject persistence, interactivity, and environmental scope, despite preserving action binding under positional error.
- Predicted positions may misalign after wrong moves, although smooth position biasing can preserve action binding to the correct subjects.
- Some subjects can disappear, preventing further interaction with the scene.
- The model is not fully real-time, and the evaluation focuses on 2D games rather than partially visible 3D scenes.The authors leave extension to other game environments for future work.