PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces

Xinyu Wang, Huafeng Shi, Yue Ma, Zijie Meng, Teng Xi, Yuzhuo Chen, Yucheng Wang, Ruicheng Zhang, Guoxin Zhang, Pengfei Wan, Xiu Li, Fangneng Zhan

arXiv:2610.29408v1cs.CV

TL;DR

Existing video world models largely use a single aggregate control stream, limiting their fit to environments where multiple agents act concurrently and controls vary by task. LiveAct combines agent-specific control binding with schema-guided action semantics, and reports strong reconstruction and control across games and robot manipulation, with high agent-attribution accuracy without location, mask, or identity input.

  • Problem

    Most existing systems expose a single aggregate control stream, while multi-agent environments require independent action attribution and task-specific control meanings lack a universal coordinate system.

    Source: §1 · p. 2

  • Method

    LiveAct’s Heterogeneous Action Interface combines language-driven agent-specific spatial modulation with schema-defined semantic tokens modulated by synchronized action values, retaining the interface in causal rollout.

    Source: §1 · p. 3

  • Results

    Across six sources, LiveAct achieves best reconstruction and control, with agent-attribution accuracy of 0.898 on games and 0.942 on robots without location, mask, or identity input.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The causal generator transfers the complete interface unchanged to few-step rollout, advancing action windows with the cached rollout rather than redefining agent assignment or action meaning.

    Source: §1 · p. 3, §5 · p. 9

Abstract

from arXiv · show

Interactive video world models predict how an environment evolves under user actions, but most existing systems assume one controlled agent and one task-specific action interface. We study multi-agent video world modeling, where one model generates a joint visual future for multiple controlled agents while interpreting heterogeneous controls ranging from binary game buttons to continuous robot commands. This task presents two complementary challenges. First, every agent-indexed action stream must control its intended visual entity independently. Second, the model must adapt the meaning of each control variable across different tasks, where the same tensor dimension may denote an attack, steering command, or robot command. We introduce LiveAct, centered on a Heterogeneous Action Interface (HAI) whose two components mirror this decomposition. To support independent control, Agent-Aware Spatial Modulation uses language-driven spatial gates and agent-specific attention to bind each action stream to its intended visual agent. To accommodate heterogeneous controls, Schema-Guided Action Adaptation describes each control variable in language and modulates its semantic token with the synchronized numerical value. We further transfer the complete interface to a blockwise causal generator through teacher forcing and rollout-aware distribution matching, enabling efficient streaming generation while retaining agent-specific control and action semantics. We instantiate the formulation on six multi-agent sources spanning games and multi-robot manipulation, and evaluate visual quality, action following, agent attribution, and latency. Across games and robot manipulation, LiveAct reaches 0.898 and 0.942 agent-attribution accuracy without agent locations or identity crops, and the causal generator attains 17.70 FPS at $512\times 448$.

1. Introduction

Multi-agent world models must keep each action stream bound to its intended visual agent while representing task-specific control meanings alongside synchronized values. LiveAct’s Heterogeneous Action Interface combines agent-specific visual conditioning with schema-guided control tokens, then transfers that interface to causal streaming generation.

1. Introduction

  • The paper formulates multi-agent video world modeling as generating a shared visual future from an initial frame, scene description, and multiple frame-level action streams.Each stream should control its intended agent while the agents and environment remain consistent.Source: §1 · pp. 2–3
  • Existing interfaces trade off numerical precision, semantic grounding, and multi-agent control; none supports all four requirements natively, according to Table 1.Numerical vectors preserve values but use task-specific dimensions, while language actions express semantics without synchronized continuous values.Source: §1 · p. 2, Table 1 · p. 3
  • Independent control requires associating each indexed action stream with its intended visual entity, especially when agents overlap, look similar, or jointly manipulate objects.Sequentially injecting separate action tensors does not ensure their effects remain correctly attributed in shared visual features.Source: §1 · p. 2
  • The action interface must separate what each task-specific control dimension means from the binary or continuous value it takes.The same dimension may encode different behaviors across tasks, while language alone omits frame-synchronized values.Source: §1 · p. 2, Figure 2 · p. 2
  • LiveAct combines language-driven agent binding and value-modulated schema tokens, retaining the complete interface during causal adaptation and rollout-aware distillation.The evaluation spans six multi-agent environments, including games and robot manipulation.Source: §1 · p. 3
  • 0.898 and 0.942 agent-attribution accuracy are reported for games and robots without locations, masks, or identity crops.The benchmark contains 108 clips spanning four games and two robot-manipulation sources.Source: §1 · p. 3

2. Related Work

Related work addresses multi-agent scene modeling and action-to-entity binding, while action interfaces differ in whether they preserve numeric precision or expose control semantics. LiveAct’s HAI combines language-defined meanings with synchronized values and uses agent-aware queries for binding.

2. Related Work

  • Prior multi-agent systems model synchronized observations or shared game and robot environments, while other systems focus on binding independent controls to visual entities.The cited approaches include Solaris [36], MultiWorld [50], Gamma-World [20], ActionParty [30], WanToFight [13], and Incantation [63].Source: §2 · p. 3
  • HAI pairs language-defined control meanings with synchronized binary or continuous values and conditions visual queries on each agent’s description.This addresses control semantics and per-agent binding as distinct parts of the interface.Source: §2 · pp. 3–4
  • Action interfaces commonly choose between task-specific numerical vectors and language actions that make control semantics explicit.Vectors preserve numerical trajectories but may rely on spatial priors and implicit task-specific meanings; language actions can transfer semantics across entities and tasks [13, 30, 50, 63].Source: §2 · p. 4

3. Method

The method separates two interface problems: grounding each action stream in the visual agent it controls, and representing each control’s task-specific meaning together with its value. HAI combines these components through per-agent cross-attention and is retained as the model is adapted for causal streaming rollouts.

3.1. Motivation

  • Existing action tensors preserve stream values but do not identify their visual recipients, while task-specific coordinate indices do not carry consistent meanings across tasks.HAI separates query-side agent binding from key–value-side control representation.Source: §3.1 · p. 4

3. Method

  • The method first trains a bidirectional teacher with HAI, then transfers the interface to causal attention and distills few-step streaming rollouts.Figure 3 presents these as Foundational Ability Acquisition, Teacher Forcing for AR Tuning, and DMD for Streaming Video.Source: §3 · p. 4, Figure 3 · p. 5

3.2. Agent-Aware Spatial Modulation

  • Agent-Aware Spatial Modulation conditions visual queries on each agent’s description and uses a soft learned gate whose strength varies across spatial tokens.The gate is not a segmentation mask or an explicitly supervised object assignment, and no explicit spatial supervision or inference-time spatial inputs are required.Source: §3.2 · pp. 4–5

3.3. Schema-Guided Action Adaptation

  • Schema-Guided Action Adaptation creates a separate token for each control dimension, pairing its language description with its synchronized numerical value.This gives task-specific coordinates named meanings while retaining individual controls for visual queries to select.Source: §3.3 · pp. 5–6

3.4. Assembling the Heterogeneous Action Interface

  • Per-agent cross-attention uses visual tokens as queries and action tokens as keys and values, producing independent control residuals from a shared visual state.Zero-initialized output projections preserve pretrained backbone behavior at initialization.Source: §3.4 · p. 6

3.5. Efficient Causal Rollout

  • Teacher Forcing adapts the model to blockwise causal attention using clean preceding chunks, while DMD trains on autoregressive rollouts to address the exposure gap.The HAI remains at the same transformer locations with aligned conditions; inference reuses preceding chunks in a KV cache.Source: §3.5 · p. 6

4. Experiments

LiveAct is evaluated on a six-source benchmark spanning games and robot manipulation, with tests of visual quality, action following, and agent attribution. It achieves the best results among evaluated methods on both tracks, while ablations show distinct contributions from agent-aware modulation and schema-guided action adaptation.

4.1. Data Collection

  • The benchmark contains 17,951 trajectories from six game and robotic sources, totaling 119.54 hours, with 108 held-out 81-frame clips for evaluation.Each sample includes an initial frame, scene description, synchronized per-agent action streams, agent descriptions, and a task-specific control schema.Source: §4.1 · p. 6, Figure 4 · p. 7

4.2. Implementation details

  • The model inserts HAI into every block of Wan2.2-TI2V-5B and fine-tunes the DiT while freezing the text encoder and VAE.The teacher trains for 10k steps, followed by 6k causal-generator steps and 700 distribution-matching iterations; inference uses four denoising evaluations per chunk.Source: §4.2 · p. 7

4.3. Comparison with Baselines

  • LiveAct leads the evaluated methods on both tracks, with game and robot Attr-Acc of 0.898 and 0.942, respectively.It also records FVDs of 338.5 and 270.2, game Act-F1 of 0.594, robot NRMSE of 0.558, and the best MME and Aesthetic scores.Source: §4.3 · p. 7, Table 2 · p. 8
  • Figure 5 shows LiveAct reproducing the kick and opponent response at the ground-truth frames, while WanToFight-style misses the second agent’s command.The qualitative comparison uses shared initial frames and action streams, with methods in columns and sampled durations in rows.Source: §4.3 · p. 8, Figure 5 · p. 8

Effectiveness of Agent-Aware Spatial Modulation

Effectiveness of Schema-Guided Action Adaptation

  • Replacing Schema-Guided Action Adaptation with a flat action encoder lowers game Act-F1 to 0.225 and Attr-Acc to 0.603, while robot NRMSE rises to 1.120 and Attr-Acc falls to 0.484.Agent modulation remains, but action semantics collapse; the two interface components are complementary.Source: Effectiveness of Schema-Guided Action… · p. 9, Table 3 · p. 9
  • Stage 3 reaches 17.70 FPS, 10.7× faster than Stage 2, with game Act-F1 rising from 0.539 to 0.594 and Attr-Acc from 0.844 to 0.898.On robots, NRMSE rises from 0.430 to 0.558 and Attr-Acc falls from 0.969 to 0.942.Source: Effectiveness of Schema-Guided Action… · p. 9

5. Conclusion

LiveAct combines agent-specific control with task-aware action semantics and transfers this interface to causal generation. The paper situates this design within structured video conditioning and causal streaming generation.

5. Conclusion

  • Across six sources, LiveAct achieves best reconstruction and control, 0.898 and 0.942 agent-attribution accuracy without location, mask, or identity input, and 17.70 FPS at 512 × 448.The conclusion reports these outcomes for games and robot manipulation.Source: §5 · p. 9

A. Additional Related Work

  • The HAI represents each control dimension with a task-specific semantic description and synchronized scalar, while preserving each stream’s provenance until cross-attention.Agent-Aware Spatial Modulation keeps stream provenance intact; Schema-Guided Action Adaptation uses the numerical value to modulate, not replace, the semantic representation.Source: A. Additional Related Work · p. 10

A. Additional Related Work S1

  • LiveAct adopts causal teacher initialization and rollout-aware distribution matching to carry the complete HAI into efficient rollout while keeping physical action intervals aligned.The cited discussion places this training choice among approaches combining bidirectional teachers, causal students, and rollout-aware distillation.Source: A. Additional Related Work · p. 10

B. Method Details

The supplementary method specifies how the interface represents agent-indexed controls, aligns them to video latents, and carries them through causal training and rollout.

B.1. Problem Formulation

  • Each numerical control dimension is paired with an agent identity and task-specific semantic description, with source controls padded to a common width of C = 12.The formulation states that S preserves source-specific control semantics.Source: §B.1 · p. 10

B.2. Frame–Latent–Action Alignment

  • For frame–latent alignment, actions are truncated or padded to the required sequence length, and each future latent receives its corresponding action interval.For the primary setting L = 1, the interval reduces to the main-text interval; boundary handling uses final-value padding.Source: §B.2 · p. 11
  • In the 81-frame example, each future latent uses actions beginning at 1 + 4(k −1); during DMD training, the generator and both score models read the same physical interval.The clean-context/noisy-target shift advances the action interval by one latent during causal teacher forcing.Source: §B.2 · p. 11

B.3. Causal Condition Propagation

  • During teacher forcing, HAI conditions only noisy target tokens; during rollout, the clean initial latent is cached without action injection before aligned streams condition later chunks.This separation prevents context tokens from being optimized as future action responses.Source: §B.3 · p. 11

B.4. Causal Training Objectives

  • The bidirectional model supplies an action-conditioned prior and initializes the causal generator and score networks.Its velocity prediction is trained from noisy future latents under initial-latent, scene, and action conditions.Source: §B.4 · p. 11
  • Teacher Forcing for AR Tuning uses z0:F −2 as context and z1:F −1 as targets, excludes the initial frame from loss, and shares a timestep across each four-latent group.The context–target shift applies a four-raw-frame offset to the actions.Source: §B.4 · p. 11
  • DMD trains a causal generator to produce five future chunks from noise without future ground-truth latents, using real and fake scores conditioned on the same scene and action inputs.The fake score is trained on detached generator samples; the default schedule updates it every iteration and the generator once every five iterations.Source: §B.4 · p. 11

C. Dataset Details

The dataset pairs shared observations with synchronized per-agent actions across game and multi-robot sources. The implementation retains the pretrained conditioning pathways while sharing HAI placement across model components.

C. Dataset Details

  • The dataset comprises 17,951 multi-agent trajectories and 119.54 hours of video, with two to four concurrently controlled agents and one synchronized action stream per agent.The sources include fighting, shared-map competition, shared-track racing, and multi-arm manipulation.Source: C. Dataset Details · p. 11
  • The game sources provide 12,860 trajectories of 30 seconds each, recorded at native 60 fps with independently driven per-agent controllers.The collection excludes menus, transitions, and incomplete rollouts.Source: C. Dataset Details · p. 12
  • RoboTwin and RoboFactory contribute 5,091 robot trajectories across multi-arm tasks, with concurrent continuous action streams aligned to shared observations.RoboFactory retains successful and unsuccessful cooperative cube-stacking trajectories.Source: C. Dataset Details · p. 12
  • Evaluation uses 108 held-out clips, one 81-frame clip from each unseen trajectory; 78 are game clips and 30 are robot clips.Training and evaluation use 81-frame clips drawn from longer trajectories.Source: C. Dataset Details · p. 12

D.1. Backbone and Conditioner Placement

  • HAI leaves Wan2.2-TI2V-5B’s pretrained conditioning pathways unchanged, and the teacher, causal generator, and both score networks share its placement and structure.The initial image latent is fused into the video representation; scene text retains pretrained text cross-attention and diffusion time continues to modulate AdaLN.Source: §D.1 · pp. 12–13

D.2. Training and Inference Configuration

The primary configuration specifies shared flow-matching settings, stage-dependent optimizer choices, and EMA inference weights.

D.2. Training and Inference Configuration

  • All stages use flow matching with 1000 training timesteps and a timestep shift of 5.0.The primary configuration uses description-based, value-modulated control tokens.Source: §D.2 · p. 13
  • Foundational Ability Acquisition uses AdamW with β = (0.9, 0.999), while both causal stages use β = (0, 0.999) and clip gradient norms at 10.Source: §D.2 · p. 13
  • DMD for Streaming Video maintains generator EMA with decay 0.99 from iteration 200 onward and uses EMA weights at inference.Source: §D.2 · p. 13

E.1. Evaluation Protocol

The evaluation protocol tests agent attribution, heterogeneous-control interpretation, and quality retained after causal transfer, using shared benchmarks for games and robot manipulation. Baselines use their released interfaces or stated reproductions, while LiveAct is evaluated without agent locations or identity crops.

E.1. Evaluation Protocol

  • The evaluation asks whether streams reach the correct agents, heterogeneous controls retain their meaning, and causal transfer preserves control and visual quality while improving latency.The two evaluation tracks use the same three questions.Source: §E.1 · p. 13
  • Methods are evaluated on one 81-frame prediction per condition at seed 0, with 16 uniformly sampled future frames used for frame-paired metrics.The conditioning frame is excluded; causal efficiency is evaluated separately because published throughput uses different backbones, resolutions, hardware, and decoding pipelines.Source: §E.1 · p. 13

E.2. Baseline Configurations

  • Released checkpoints are evaluated without further training where available; ActionParty and the WanToFight-style reimplementation are trained on the dataset under the stated protocol.ActionParty receives initial agent locations, while LiveAct uses per-agent schema-numerical controls without locations, masks, boxes, or identity crops.Source: §E.2 · p. 13
  • The baselines cover serialized language actions, per-agent numerical controls, and reference-guided control, with method-specific conditioning configurations.The compared methods are LingBot-World 2.0, Incantation, MultiWorld, ActionParty, and WanToFight-style.Source: §E.2 · p. 13
  • Table S2 catalogs training and inference settings, including FSDP training for Stages 2 and 3 and Stage 3 sampling settings.The caption says Stage 3 uses EMA weights and timesteps [1000, 750, 500, 250].Source: Table S2 · p. 14

E.3. Metrics

The metrics separately assess reconstruction, motion and appearance, control recovery, and agent attribution. Causal-transfer throughput is reported alongside these evaluation details.

E.3. Metrics

  • FVD measures distributional similarity using I3D features, while LPIPS, PSNR, and SSIM measure perceptual, pixel, and structural agreement with the ground-truth future.The frame-paired metrics follow the stated sampling protocol.Source: §E.3 · p. 14
  • Motion Magnitude Error compares generated and ground-truth mean RAFT flow magnitudes, with lower values indicating closer aggregate motion energy.Aesthetic uses the LAION improved predictor with CLIP ViT-L/14.Source: §E.3 · p. 14
  • The inverse-dynamics evaluator is trained only on real clips and predicts controls from surrounding RGB frames without agent identity or location inputs.Game Act-F1 is macro F1 across valid events and agents; for robots, the evaluator predicts continuous control trajectories.Source: §E.3 · p. 14
  • Robot control error is normalized over valid, non-padding dimensions using each dimension’s real-training-set standard deviation; lower values mean more accurate recovery.The recovered control is compared with the commanded control, with a small constant in the denominator.Source: §E.3 · p. 14
  • Attr-Acc compares recovered controls under the commanded agent assignment against a pairwise-swapped assignment, scoring frames only when at least one pair has distinct commands.Centering fixes the chance level at 0.5 for any number of agents; with two agents, the test reduces to swapping their assignments.Source: §E.3 · pp. 14–15
  • Scores are averaged within clips, then across each source, and finally across sources without weighting, giving each source equal weight.Averaging within clips first prevents clips with more scored frames from dominating.Source: §E.3 · p. 15

F.1. Three-Stage Causal Transfer

  • 17.70 FPS at 512 × 448 is reached by Stage 3, which is 10.7× faster than Stage 2.Stage 3 uses four denoising steps per chunk without classifier-free guidance.Source: §F.1 · p. 15

F.2. Detailed Ablations

The detailed ablations isolate the roles of value modulation, schema descriptions, and temporal alignment in Schema-Guided Action Adaptation. A spatial diagnostic also checks whether agent-specific residuals concentrate around their corresponding visual agents.

F.2. Detailed Ablations

  • The SGAA variants isolate schema description, per-dimension tokens, and value modulation while retaining the other stated control and attention components.The no-SGAA variant removes all three factors and reduces each latent’s action memory from rC to r tokens.Source: §F.2 · p. 15
  • Replacing value modulation with addition alone yields control and attribution results comparable to removing SGAA, with robot Attr-Acc near chance.The passage attributes this to addition mixing value into the schema token, while modulation preserves the schema token as the control-identity carrier.Source: §F.2 · p. 15
  • Learned per-dimension IDs fall to the no-SGAA level on games, while retaining part of the gain on robots but remaining below the full model.The passage reports that games reuse indices for different controls across sources, whereas schema descriptions provide their meanings.Source: §F.2 · p. 15
  • Pairing controls with mismatched schema descriptions at inference drops game Act-F1 to 0.207 and game and robot Attr-Acc to 0.504 and 0.496.Action values, per-dimension tokens, and value modulation are unchanged in this test.Source: §F.2 · p. 15
  • Extending the action window from L=1 to L=2 and L=4 monotonically worsens all four reported game and robot control and attribution metrics.Game Act-F1 changes 0.594 → 0.449 → 0.325 and Attr-Acc 0.898 → 0.846 → 0.721; robot NRMSE changes 0.558 → 0.591 → 0.700 and Attr-Acc 0.942 → 0.937 → 0.926.Source: §F.2 · p. 16

F.3. Spatial Localization of Agent-Specific Residuals

  • Agent-specific residual-magnitude heatmaps show high-response regions concentrated around the corresponding visual agents.Figure S2 presents representative heatmaps from the agent-specific branches.Source: §F.3 · p. 16

F.4. Additional Qualitative Comparisons

Additional comparisons show distinct failure patterns among baselines and closer target-arm pose matching by LiveAct in sampled robot frames. Further showcases cover more than two agents and a rollout beyond the quantitative evaluation window.

F.4. Additional Qualitative Comparisons

  • In the added game and robot cases, LingBot-World 2.0’s scene drift can become abrupt, Incantation’s game failures worsen over time, and MultiWorld and ActionParty smear players and arms.WanToFight maintains the scene and completes the action but shows an arm-action mismatch; LiveAct matches the target arm pose more closely at sampled frames.Source: §F.4 · pp. 16–17
  • Figure S2’s red and blue residual heatmaps show high-response regions concentrated around the corresponding agents.Higher intensity denotes a larger response.Source: Figure S2 · p. 17

F.5. Additional LiveAct Showcases

  • The interface is used with two to four concurrently controlled agents, and the described scenes beyond two agents include games with binary controls and robot manipulation with continuous controls.Adding an agent requires one description, one action stream, and one per-agent cross-attention residual, with no architectural change.Source: §F.5 · p. 17, Figure S6 · p. 20

F.6. Long-Horizon Rollout Showcase

  • Figure S7 shows a dynamic rollout beyond the 81-frame quantitative window, with 24 generated frames sampled every ten frames from one 241-frame rollout.The frames are ordered left to right and top to bottom.Source: §F.6 · p. 17, Figure S7 · p. 20

Related papers

Loading 2610.29408v1…