PaperlayerView exact revision on arXiv

Source-linked AI summary

PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents

Siqi Yang, Yimu Wang, Ziqi Cai, Chengxuan Zhu, Shuchen Weng, Erwin Wu, Zhaofei Yu, Kaipeng Zhang, Boxin Shi

arXiv:2610.20240v1cs.CVcs.AI

TL;DR

POV video captures only part of a shared multiplayer world, and existing Minecraft datasets do not pair interacting players’ POV and panoramic video with their actions. PAWA provides such data and a dual-tower model that jointly predicts both views; it outperforms state-of-the-art multiplayer world models in POV prediction while producing high-fidelity panoramas.

  • Problem

    Multiplayer world modeling needs to predict changes beyond each player’s field of view, but no Minecraft dataset pairs interacting players’ POV and panoramic video with aligned actions.

    Source: §1 · pp. 1–2

  • Method

    PAWA pairs world-stabilized panoramas, POV video, and actions in a dataset, then jointly predicts both views with dual towers and ray-aligned cross-attention.

    Source: §1 · pp. 2–3

  • Results

    PAWA outperforms state-of-the-art multiplayer world models in POV prediction and produces high-fidelity panoramic predictions.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The dataset and collection pipeline are open source to support further research on multiplayer world models.

    Source: §1 · p. 3

    PAWA’s computational cost rises rapidly with player count, limiting training and evaluation to sixteen players.

    Source: §6 · p. 10

Abstract

from arXiv · show

A multiplayer world model must predict how the independent actions of several players change a shared environment, including changes outside a player's current field of view. A point-of-view (POV) video captures such changes only after they enter its field of view, whereas panoramic video records the surroundings in every viewing direction and has served as an intermediate representation of the world state in recent world models. Learning this task therefore requires synchronized POV and panoramic videos of interacting players together with their actions, yet no existing dataset provides this combination in Minecraft, an established testbed in which players act independently in a shared, modifiable 3D world. We present PAWA, a dataset and framework for action-conditioned panoramic multiplayer world modeling. The dataset pairs each player's POV video with world-stabilized panoramic video and aligned actions. Its distributed pipeline decouples CPU gameplay capture from GPU rendering, scales beyond 64 simultaneous players, and re-renders each recorded session under new observation configurations without replaying the gameplay. On this dataset, we develop a dual-tower model that jointly predicts POV and panoramic video and couples the two streams through cross-attention that aligns tokens observing the same direction in a shared world frame. We further introduce a quadratic-phase player encoding that provides many more player codes than existing rotary codes at the same embedding dimension. Experiments show that PAWA outperforms state-of-the-art multiplayer world models in POV prediction quality while also producing high-fidelity panoramic predictions. The dataset and collection pipeline are open source to support further research on multiplayer world models.

1 Introduction

POV views miss changes outside a player’s field of view, while panoramas expose the surroundings in every direction. PAWA pairs these observations and actions in a dataset and jointly predicts POV and panoramic video for multiple players.

1 Introduction

  • Figure 1 depicts forecasts of every player’s POV and world-stabilized panorama from initial observations and both players’ actions.The co-player appears in matching directions in the POV and panorama.Source: Figure 1 · p. 2
  • POV-only models miss off-screen changes, while full-sphere panoramic prediction jointly with each player’s POV remains unexplored.Panoramas record surroundings in every direction, including regions no local camera currently observes.Source: §1 · pp. 1–2
  • PAWA pairs each player’s POV video with a world-stabilized panorama and aligned actions, addressing the lack of this combination in Minecraft multiplayer data.The distributed pipeline records sessions with more than 64 simultaneous players and can render saved replays under new observation configurations.Source: §1 · pp. 2–3
  • A dual-tower model jointly predicts POV and panoramic video, coupling the streams through cross-attention that aligns tokens observing the same world-frame direction.Matching directions receive identical rotary phases despite occupying different image coordinates.Source: §1 · pp. 2–3
  • Quadratic-phase player encoding expands the identity pool from m + 1 to m2 codes at a fixed embedding dimension.The encoding provides closed-form correlations between any two codes.Source: §1 · p. 3

2 Related work

Prior work studies action-conditioned video models, synchronized multiplayer perspective views, and panoramic representations, but PAWA combines panoramic prediction with a shared multiplayer world. Its dataset pipeline also separates gameplay capture from offline rendering.

2 Related work

  • Interactive video world models predict action consequences for control and use approaches including autoregressive tokens, latent actions, and video diffusion.Examples cited include Wu et al. (2024), Bruce et al. (2024), and Huang et al. (2026).Source: §2 · p. 3
  • Existing multiplayer models generate synchronized perspective views, while panoramic-coverage approaches still produce perspective video; PAWA predicts both POV and panorama in a shared world.Prior work includes models for Minecraft and Rocket League (Savva et al., 2026; Liu et al., 2026a; Hu et al., 2026a), and methods that stitch directional views or crop panoramas (Zhu et al., 2026; Sun et al., 2026a).Source: §2 · pp. 3–4
  • Panoramas support explorable worlds, long-horizon exploration, perspective rendering, driving, geometric consistency, and spatial memory, while PAWA extends prediction to changes caused by co-players.The cited panoramic applications include HunyuanWorld Team (2025), Tan et al. (2026), and Jiang et al. (2026).Source: §2 · pp. 3–4
  • Figure 2 depicts scripted interactions, CPU capture of sessions with more than 64 players, offline GPU replay rendering, and revision after quality checks.Behavioral QC and VLM review return failed scripts for revision.Source: Figure 2 · p. 4

3 Panoramic multiplayer dataset

PAWA records synchronized POV and world-stabilized panoramic observations with actions and camera annotations. Its pipeline captures programmable interactions without graphics clients, then renders saved replays offline.

3 Panoramic multiplayer dataset

  • Each player’s POV video is paired with an equirectangular panorama from the same camera origin, aligned actions, and camera annotations.The panorama remains oriented to the world as the player turns, and all videos share the server clock.Source: §3 · p. 4
  • Saved replays can produce POV and panoramic videos, relative depth, and custom camera trajectories without repeating gameplay.This extends observation configurations beyond the live perspective view available during play.Source: §3 · p. 4
  • CPU-based capture records interactions among more than 64 simultaneous players without graphics clients, while GPU workers render saved replays on separate hosts.Capture and rendering use separate resources, and a session can be recorded once and rendered afterward.Source: §3 · p. 4
  • Natural-language scenarios are turned into executable scripts and checked for behavioral and visual success before acceptance.Behavioral QC flags inactivity, ineffective movement, and abrupt control changes; a VLM reviews sampled videos against expected outcomes.Source: §3 · p. 4

4 Panoramic multiplayer world modeling

PAWA formulates multiplayer world modeling as predicting each player’s future POV and panoramic observations from synchronized initial views and shared action conditions. Its model combines projection-specific towers with ray-aligned attention, player identity codes, and causal rollout training.

4.1 Problem formulation

  • PAWA predicts future POV and panoramic observations for all players from their initial observations, actions, appearance attributes, and shared scene and camera conditions.Future positions, visibility, and images are not provided as inputs.Source: §4.1 · p. 5

4.2 Dual towers with ray-aligned cross-attention

  • Two towers jointly predict POV and panorama, sharing weights across players and exchanging features through bidirectional cross-attention at selected blocks.The towers are initialized from Gamma-World and PanoWorld, respectively, and the new cross-attention gates are zero-initialized.Source: §4.2 · p. 5, Figure 3 · p. 6
  • Ray-based rotary phases give POV and panorama tokens observing the same world direction zero relative phase, regardless of image coordinates or head orientation.The panorama tower’s vertical rotary coordinate follows the equal-area sine of latitude, compressing rows near the poles.Source: §4.2 · pp. 5–6

4.3 Player identity encoding

  • Quadratic-phase encoding provides m^2 − 1 player codes after reserving the all-zero code, versus m + 1 simplex codes, so the identity pool grows quadratically rather than linearly.Codes within a family are orthogonal, while codes across families have uniform correlation m^-1/2.Source: §4.3 · pp. 6–7

4.4 Training objective

  • Joint flow matching trains both projection towers with a shared noise level and sums their losses across players and forecast frames.Position weights increase loss on co-player regions while preserving unit mean per sample; half of training windows additionally favor co-player occupancy.Source: §4.4 · p. 7

4.5 Causal generation and distillation

  • The causal generator uses block-causal hub attention, restricting each temporal block to its own and earlier blocks while shared hubs mediate cross-player interaction.Noise levels are shared among players and both projections within each block.Source: §4.5 · p. 7
  • Few-step distillation trains the causal generator on its own rollouts using a frozen bidirectional teacher and a learned score model for generated samples.At inference, the distilled model jointly generates both streams block by block with four denoising steps per block.Source: §4.5 · p. 8

5 Experiments

Across held-out multiplayer forecasts, PAWA outperforms prior models in POV prediction while producing high-fidelity panoramas. The experiments compare forecasting quality, test larger player populations, and examine cross-attention ablations.

5.1 Experimental setup

  • The main comparison uses 30 held-out two-player episodes of 12 seconds at 20 Hz, with separate 12-episode sets for four, eight, and sixteen players.Metrics cover each player's 239 forecast frames, all forecast frames, or 16-frame clips, depending on the metric.Source: §5.1 · p. 8

5.2 Comparison with multiplayer world models

  • PAWA (Causal) more than halves POV FVD versus Gamma-World (Bidirectional), while both PAWA variants beat every baseline on every POV metric and panoramas are high-fidelity.PAWA (Distilled) also more than halves POV FVD versus Gamma-World (Distilled) under the same four-step budget.Source: §5.2 · p. 8

5.3 Scaling the player population

  • At 16 players, only PAWA is supported; Gamma-World's bidirectional variant collapses, and its best variant has more than twice PAWA (Causal)'s POV FVD.Table 2 summarizes POV prediction with larger populations.Source: Table 2 · p. 9, §5.3 · p. 9

5.2 Comparison with multiplayer world models

  • In synchronized forecasts, PAWA keeps both co-players consistently visible through 12 seconds, while baselines replace or lose characters or stall.Figure 4 aligns POV and panorama rows by time, allowing co-player comparison against the recorded continuation.Source: §5.2 · p. 9, Figure 4 · p. 9

5.4 Ablations

  • Disabling seven cross-attention sites raises Distilled POV FVD from 138.1 to 179.0 and LPIPS from 0.465 to 0.510; after 3,000 updates, the causal model without them remains worse.The causal model without cross-attention reaches LPIPS 0.511 and FVD 191.7, versus 0.455 and 138.4 with cross-attention.Source: §5.4 · pp. 9–10
  • At late sites, POV queries put up to 57% of attention on panorama regions inside their frustum, versus 10% for uniform attention or re-initialized sites.The results indicate that cross-attention learns to connect matching directions.Source: §5.4 · p. 10

6 Conclusion

PAWA combines a scalable paired-view dataset and a dual-stream multiplayer world model, outperforming prior models in POV prediction while producing high-fidelity panoramas. The conclusion also identifies long-rollout fidelity and scaling beyond sixteen modeled players as limitations.

6 Conclusion

  • PAWA pairs synchronized POV and world-stabilized panoramic videos with aligned actions, and its pipeline scales beyond 64 players while supporting joint prediction of both streams.The model uses ray-aligned cross-attention and quadratic-phase player codes.Source: §6 · p. 10
  • PAWA outperforms state-of-the-art multiplayer world models in POV prediction and produces high-fidelity panoramic predictions.Source: §6 · p. 10
  • Long rollouts lose co-player fidelity, and POV and panorama forecasts gradually diverge because cross-attention aligns directions without enforcing pixel agreement.Source: §6 · p. 10
  • Rapidly increasing computational cost limits current model training and evaluation to sixteen players, despite the pipeline recording more than 64 and the codebook providing 120 identities.Scaling further requires redesigning the model to separate player computation and limit information exchange.Source: §6 · p. 10

B MODEL AND TRAINING SETTINGS

The collection records complete per-player views that bundle POV, panorama, and actions. Streams are synchronized to server time, and saved replays support additional renderings without replaying gameplay.

B MODEL AND TRAINING SETTINGS

  • Each counted player view contains a POV stream, a panorama stream, and a per-frame action record.Source: Table 3 · p. 17
  • POV video is 640 × 360 and panorama video is 1024 × 512, both at 20 Hz; streams are aligned to server world-age ticks and cropped to the interval covered by retained players' actions.A three-tick display-phase correction aligns actions with rendered frames.Source: B MODEL AND TRAINING SETTINGS · p. 17
  • Each replay exports relative depth in both projections and can be rendered along custom camera trajectories; sessions with 64 and 128 simultaneous players have been recorded for on-demand participant and observer views.Source: B MODEL AND TRAINING SETTINGS · p. 17

B.1 Conditioning and architecture

The appendix specifies camera and player conditioning, tower configuration, and the objective and sampling choices for training. Player codes and ray encodings are configured to support multiple players and align viewing directions.

B.1 Conditioning and architecture

  • Camera conditioning totals 22 dimensions, combining yaw and pitch increments, scaled sine/cosine pairs for absolute angles, and a 16-dimensional appearance attribute; keyboard actions are separate.Absolute orientations derive from the initial orientation and angular controls, and angular increments are reindexed as outgoing transitions.Source: §B.1 · p. 17
  • Each tower has 28 blocks of width 2048 with 16 heads; cross-attention sites follow blocks 3, 7, through 27, and causal towers use eight hub tokens per latent frame.The panorama tower uses equirectangular rotary position encoding.Source: §B.1 · p. 17
  • The player-code band uses 11 phase planes for 120 assignable identities, while ray encoding uses 16 phase planes per Cartesian component and a 102.45° × 70° POV field of view.For two to eleven players, code families are sampled mostly from a single family; larger populations span several families.Source: §B.1 · p. 17

B.2 Objectives and window sampling

  • The flow-matching objective excludes supplied context frames and applies backbone noise weighting; causal training shares one noise level across players and projections within each temporal block.Appearance adaptation and joint training use projected co-player support with λ = 4; causal and distilled models expand body boxes and union them over lags of zero to six ticks.Source: §B.2 · p. 17
  • Half of training windows are sampled uniformly, while the other half are selected by a separate criterion.Distillation applies the stated weights to both the generator and learned score.Source: §B.2 · p. 17

C GEOMETRY AND PLAYER-CODE ANALYSIS

This material specifies PAWA’s viewing-ray and player-code geometry alongside its training and evaluation setup. Across additional population tiers, PAWA retains the best POV scores, while distillation improves four-step forecasts.

C GEOMETRY AND PLAYER-CODE ANALYSIS

  • The geometry analysis samples player populations from specified sets, weighting training toward smaller populations.Joint training samples 1, 2, 4, or 8 players; causal training and distillation sample 2, 4, 8, or 16.Source: C GEOMETRY AND PLAYER-CODE ANALYSIS · p. 18

B.3 Optimization

  • Training proceeds from projection-specific towers through appearance adaptation and joint updates, then trains a causal generator and distills it into a four-step sampler.The optimization uses 93-frame clips for the teacher and causal training up to eight players, with shorter clips for sixteen-player and distillation runs.Source: §B.3 · p. 18

C.1 Viewing rays

  • Paired world-frame rays use normalized image coordinates, camera-to-world orientation, and POV fields of view; matching POV and panorama rays have zero relative phase.The panorama axes remain fixed to the world, and its spherical map is continuous at the longitude seam.Source: §C.1 · p. 18

C.2 Simplex capacity and quadratic-phase correlations

  • For odd-prime m, quadratic-phase code families yield m^2 codes across m phase planes, exceeding the m + 1 equidistant simplex vertices.The supplied proof gives cross-family magnitude m^-1/2; the Gram-matrix argument bounds equidistant points in R^m by m + 1.Source: §C.2 · p. 18

D.1 Protocol

  • The protocol evaluates forecast frames per player, averages per-player scores across episodes, and compares models with different supported player counts and samplers.Interaction episodes are selected using recorded co-visibility rather than generated quality; scoring excludes the conditioning frame and covers 239 forecast frames per player.Source: §D.1 · p. 19

D.2 Additional comparisons

  • PAWA again obtains the best POV scores in the low-motion, four-, eight-, and sixteen-player tiers; distillation lowers interaction-tier POV FVD from 438.5 to 138.1 and LPIPS from 0.497 to 0.465.The tier comparisons cover 12 episodes each; the interaction-tier distillation comparison is between direct four-step causal sampling and the distilled model.Source: §D.2 · p. 19, Table 4 · p. 19, Table 5 · p. 20

D.3 Out-of-view changes, action following and player codes

PAWA’s forecasts are tested on co-players entering view, panoramic co-player regions, action following, and player-code families. The results show advantages over baselines across these tests and populations.

Co-players entering the view

  • Across 48 two-player, 111 four-player, and 505 eight-player entry events, PAWA (Distilled) beats every baseline and static copy on co-player LPIPS, including newly revealed regions.For regions first revealed by camera rotation, LPIPS improves over the best baseline by 0.13, 0.10, and 0.09 at two, four, and eight players.Source: §D.3 · p. 20

Co-players in the panorama

  • Panoramic co-player regions have similar LPIPS inside and outside the POV frustum, while outside-POV predictions improve over the static first panorama at two, four, and eight players.The inside-versus-outside difference is +0.009 [−0.013, 0.031].Source: §D.3 · p. 20

Action following and binding

  • PAWA has the lowest camera-action following error at every population, reduces the best baseline’s error by 33–43% for causal and 14–31% for distilled models, and binds 90–98% of players through eight players.When two action streams are swapped while initial observations and codes remain fixed, 95% of generated streams are best explained by the injected actions.Source: §D.3 · p. 21, Table 8 · p. 21

Player code families

  • Using multiple quadratic-phase code families changes POV FVD by −6.8, −11.6, and −17.0 at two, four, and eight players, respectively, compared with single-family codes.This resampling comparison uses the same checkpoint, noise, and sampler.Source: §D.3 · p. 21

D.4 Ablation of the ray-aligned cross-attention

The ablations test whether ray-aligned cross-attention matters at inference and during training, and whether attention aligns with viewing rays. Qualitative comparisons and long-rollout evaluations extend the evidence to generated videos and co-player fidelity.

D.4 Ablation of the ray-aligned cross-attention

  • Removing all cross-attention sites raises POV LPIPS by 0.045 and increases POV FVD; removing pano→POV alone reproduces the POV degradation, while removing POV→pano raises panoramic FVD by 8.0.The inference ablations use identical episodes, conditioning inputs, sampler, seed, and recorded noise; the paired LPIPS interval is [−0.058, −0.032].Source: §D.4 · p. 21, Table 9 · p. 22, Table 10 · p. 23
  • Training with cross-attention widens the POV LPIPS advantage over training without it to 0.056 [0.039, 0.073] after 3000 updates, alongside better FVD and PSNR.The arm without cross-attention increasingly collapses late in the rollout, while the enabled arm retains the reference scene layout.Source: §D.4 · pp. 21–22, Table 11 · p. 24
  • Attention aligns with viewing rays during training: late distilled sites place 43–46% of mass inside the POV frustum, with the most-attended key a median 7°–9° from the query ray.Reinitializing the sites returns alignment to chance, while the causal model reaches a 5° median angle; alignment emerges without an auxiliary loss.Source: §D.4 · pp. 22–23, Table 12 · p. 24

D.5 Qualitative comparisons

  • The qualitative comparisons show evaluated two-player and four-player forecasts, with recorded continuation as the reference and both distilled models using four steps.Figure 6 compares the two-player episode against baselines; Figure 7 shows synchronized four-player forecasts.Source: Figure 6 · p. 22, §D.5 · p. 23, Figure 7 · p. 23

D.6 Co-player fidelity over long rollouts

  • In the last rollout band, both PAWA variants achieve lower co-player LPIPS than the static copy and the other listed models; the advantage persists across forecast bands.For four players, co-player LPIPS rises from 0.476 to 0.618 for PAWA (Distilled) and from 0.547 to 0.719 for Gamma-World (Distilled).Source: §D.6 · pp. 23–24, Table 13 · p. 24
Loading 2610.20240v1…