PaperlayerView exact revision on arXiv

Source-linked AI summary

FlockWorld: Toward Population-Scale Multi-Agent Video World Models through Artificial Life Simulation

Meryem Koksal, Yunzhe Wang, Volkan Ustun

arXiv:2610.34564v1cs.CVcs.MA

TL;DR

Action-conditioned video world models have mostly focused on single players, while multiplayer models support only small, fixed groups. FlockWorld and FlockDiT provide a flocking-based environment and model for jointly generating synchronized views, with joint generation improving detection at density-matched settings and diffusion forcing improving rollout fidelity.

  • Problem

    Action-conditioned video world models remain mostly single-player, and existing multiplayer models support only small, fixed agent groups, leaving population-scale modeling underexplored.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces FlockWorld, a Boids-based environment with synchronized egocentric observations and acceleration actions, and FlockDiT, an action-conditioned latent diffusion transformer for joint multi-agent view generation.

    Source: §1 · p. 2

  • Results

    Joint generation achieves higher detection rates than independent generation, clearest at density-controlled settings with 10 and 30 camera agents, while diffusion forcing most consistently improves rollout fidelity.

    Source: §1 · p. 2, §5.3 · p. 6

  • Takeaways & Limitations

    FlockWorld and FlockDiT provide a controlled testbed for multi-agent video generation, while consistency metrics should be judged against density-matched references.

    Source: §6 · p. 6

    The study uses a single, simple two-dimensional flocking environment with deterministic dynamics, so its consistency findings may not transfer directly to visually rich, three-dimensional, or stochastic settings.

    Source: O Limitations · p. 25

Abstract

from arXiv · show

Most action-conditioned video world models remain limited to a single player, while existing multiplayer models support only small, fixed groups of agents. We introduce FlockWorld, an artificial-life environment built on Reynolds's Boids algorithm for studying population-scale multi-agent video world models. Specifically, FlockWorld renders synchronized egocentric crops, assigns distinct colors to preserve camera-agent identity across views, exposes rule-derived accelerations as actions, and lets the number of camera agents be configured independently of the size of the surrounding population. We further introduce FlockDiT, an action-conditioned latent diffusion transformer trained with flow matching on FlockWorld data to jointly generate synchronized egocentric videos for multiple agents. Our results suggest that joint generation benefits agent detection at density-matched settings and that diffusion forcing improves rollout fidelity. Together, FlockWorld and FlockDiT provide a controlled testbed for multi-agent video generation.

1 Introduction

Interactive video world models have extended to multiplayer settings, but scaling them to large populations and socially aware environments remains underexplored. The paper introduces FlockWorld and FlockDiT to study this challenge through synchronized, independently controllable views of a flock.

1 Introduction

  • Scaling multi-agent world models to massive populations and socially aware interactive environments remains underexplored despite multiplayer models such as Solaris [37], Gamma-World [26], and MIRA.The introduction identifies potential applications including persistent virtual societies and population-scale training environments for embodied and social agents.Source: §1 · p. 1
  • FlockWorld is a visually grounded artificial-life environment built on Reynolds’s Boids algorithm, where local interactions appear in agents’ spatial configurations and trajectories [33].The paper uses this controlled visual setting to study how individual behavior and collective dynamics relate.Source: §1 · pp. 1–2
  • FlockWorld allows camera-agent count to vary independently of population size and provides synchronized egocentric observations with acceleration actions.The study fixes the population at 100 boids and evaluates 2 to 30 camera agents.Source: §1 · p. 2
  • FlockDiT is an action-conditioned latent diffusion transformer that jointly generates multiple agents’ views using temporally causal cross-agent attention.The paper examines identity representation, action conditioning, diffusion forcing, single-agent pretraining, and position-encoding backgrounds.Source: §1 · p. 2
  • Joint generation achieves higher detection rates than independent single-agent generation, most clearly at density-controlled settings with 10 and 30 camera agents.Diffusion forcing most consistently improves rollout fidelity, reducing over-generation and improving cross-view agreement.Source: §1 · p. 2

2 FlockWorld

FlockWorld is a 2D flocking simulation that renders synchronized egocentric views of selected boids. It exposes each camera agent’s rule-derived acceleration as the action signal.

2 FlockWorld

  • The environment simulates 100 boids in a 720 × 720 pixel arena using alignment, cohesion, and separation rules with a 25-pixel neighbor radius.Reflecting boundaries are used, and the initial 60 simulation steps are discarded as warm-up before rendering at 30 frames per second.Source: §2 · pp. 2–3

2.1 Multi-Ego Video Data Generation

  • Each camera agent receives a synchronized, agent-centered 128×128 crop, with colored agents identifying camera boids and ambient boids rendered white.The default setup uses 10 camera agents, while agent-count experiments vary this number from 2 to 30; a gradient background encodes global position.Source: §2.1 · p. 3

2.2 Acceleration as Action Data

  • Each camera agent’s alignment, cohesion, and separation acceleration is exposed as its action, with four simulation steps causally averaged to match the model’s lower latent frame rate.One simulation step corresponds to one video frame.Source: §2.2 · p. 3, §2 · p. 3

3 FlockDiT

FlockDiT models synchronized multi-agent video in a compressed latent space with an action-conditioned, temporally causal transformer. The paper compares two ways to represent agent identity and route actions, alongside alternative training strategies.

3.1 Action-Conditioned Video World Model

  • FlockDiT uses a frozen causal VAE that compresses 128 × 128 video by 16× spatially and 4× temporally into 8-channel 8 × 8 latent frames.A separate VAE is trained from scratch for each visual condition.Source: §3.1 · p. 3
  • The 27.5M-parameter diffusion transformer uses block-causal temporal attention and full cross-agent attention, allowing information exchange across views without attending to future timesteps.Actions are pooled onto the latent timeline, and the model is trained with an optimal-transport flow-matching objective.Source: §3.1 · pp. 3–4

3.2 Multi-agent Conditioning

  • The agent-embedded configuration adds a learned identity embedding to each view and uses each agent’s acceleration to modulate only that agent’s tokens.Identity and action assignment are explicit in this configuration.Source: §3.2 · p. 4
  • The spatially tiled configuration encodes identity through non-overlapping latent-grid positions and broadcasts a pooled, agent-tagged action vector to all tokens.The model must infer which component of the pooled action signal applies to each tile.Source: §3.2 · p. 4

3.3 Training Strategies

  • Diffusion forcing trains on independently noised context frames as well as targets, while two-stage training initializes the multi-agent model from a single-agent checkpoint.In the tiled configuration, views of a frame share a noise level; subsequent multi-agent training uses clips containing all P agents.Source: §3.3 · p. 4

4 Evaluation

Evaluation measures both fidelity to recorded trajectories and consistency between generated views. The protocol compares per-view detections and positions with cross-view agreement measures, using independent-generation and VAE-based bounds.

4 Evaluation

  • Models are evaluated on 28 held-out episodes by autoregressively generating 10-second rollouts from two ground-truth latent frames and recorded actions.Each prediction window uses a 50-step Euler solver; generated latents are decoded to pixels for evaluation.Source: §4 · p. 4
  • Per-view fidelity measures the fraction of visible ground-truth boids detected, matched-detection position error, and predicted detections per frame against the ground-truth count.Cross-view measures assess reciprocal detections, disagreement in relative position and motion, ambient-boid correspondence in overlapping regions, and SSIM.Source: §4 · p. 4
  • Cross-view consistency metrics compare mutual detection, relative-offset and motion disagreement, and ambient-boid agreement in shared regions.The independent single-agent rollout is the floor, while ground-truth latents decoded through the VAE provide the ceiling; Table 1 covers the mechanism comparison at P = 10.Source: §4 · pp. 4–5, Figure 3 · p. 5, Table 1 · p. 5

5 Experimental Results

The experiments find that diffusion forcing is the most effective tested intervention for rollout fidelity, while density-matched settings provide the strongest evidence that joint generation improves agent detection as camera-agent counts rise. Qualitative rollouts reveal geometry and identity failures that aggregate consistency metrics can miss, and the background comparison is limited by separately trained autoencoders.

5.1 Mechanism Comparison

  • Experiment 3 gives the strongest overall fidelity among learned models at a usable detection rate, while raw reciprocity can reward over-rendering rather than agreement.Experiment 2 records 23.14 detections per frame against 5.04 in ground truth; motion error separates diffusion-forcing runs from teacher-forced runs and the floor.Source: §5.1 · p. 5

5.3 Number of Camera Agents

  • At density-matched P = 10 and P = 30, joint generation detects more agents than the single-agent floor; diffusion forcing is the most effective tested intervention for fidelity.Experiment 3 has the strongest overall fidelity among learned models at a usable detection rate, while Experiment 5 matches pixel and motion measures but detects half as many boids.Source: §5.1 · p. 5, Table 2 · p. 6, §5.3 · p. 6

5.2 Qualitative Analysis of Rollout Behavior

  • Experiments 5 and 7 can turn tiled outputs into a partitioned panorama rather than independently conditioned viewpoints, and Experiment 5 loses agent color.Aggregate consistency metrics can therefore reward coherent outputs with incorrect multi-view geometry; the budget-fixed reimplementation cannot establish that tiling causes the panorama-like failure.Source: §5.2 · pp. 5–6

5.4 Position-encoding Background

  • The gradient-background condition has a higher, more stable reciprocity ceiling, but separately trained autoencoders and differing data and training lengths prevent attributing that difference to background alone.Absolute reciprocity and the fraction of ceiling recovered are not comparable across conditions.Source: §5.4 · p. 6, Table 3 · p. 6

6 Conclusion

FlockWorld and FlockDiT provide a controlled setting for jointly generating many agents’ egocentric views, with detection advantages over independent generation and diffusion forcing the strongest tested training strategy. Unlike prior multi-agent models and datasets with small or fixed view counts, this synthetic environment separates camera-agent count from population size; scaling further will require sparser cross-agent attention.

6 Conclusion

  • FlockWorld and FlockDiT jointly generate egocentric views, and joint generation reproduces more agents than independent generation as camera agents are added.Diffusion forcing is the most effective tested training strategy; consistency metrics require density-matched references.Source: §6 · p. 6

A Related Work

  • Prior multi-agent models demonstrate joint generation but generally use small, largely fixed agent counts and environments whose interactions are difficult to vary or verify against ground truth.FlockWorld independently configures population size and camera-agent count, enabling evaluation from P = 2 to P = 30 observers in a 100-agent population.Source: A Related Work · p. 13
  • Existing multi-egocentric datasets and environments tie view counts to participants, games, or pipelines, while FlockWorld renders synthetic clips on the fly with camera count independent of population size.Its data is bounded by compute rather than collection.Source: A Related Work · p. 13

B FlockWorld Implementation Details

FlockWorld is a fully synthetic, reproducible flocking simulator with 100 boids governed by local steering rules. It renders synchronized egocentric views and generates fresh training clips, with held-out episodes reserved for evaluation.

B FlockWorld Implementation Details

  • The fully synthetic environment contains no human data and uses 100 boids in a 720 × 720 pixel arena.Alignment, cohesion, and separation use a 25-pixel vision radius and weights 1.1, 1.0, and 1.5, respectively.Source: B FlockWorld Implementation Details · p. 13
  • Steering forces are clamped to 0.2 per step, speeds to [1] [4] pixels per step, and velocity drag is 0.005 per step.When candidate neighbors exceed 32, each is retained independently with probability 32/N.Source: B FlockWorld Implementation Details · p. 14
  • Reflecting boundaries make boids bounce off arena walls, and each boid is rendered as a 10-pixel oriented triangle with a white border marking the arena.At a boundary, position is clamped 9 pixels from the canvas edge and the corresponding velocity component is negated.Source: B FlockWorld Implementation Details · p. 14
  • Episodes use deterministic seeds, begin after 60 warm-up steps, and run at 30 frames per second.Initial positions, headings, and speeds are sampled within specified ranges; the simulator follows the reference implementation’s rules and geometry.Source: B FlockWorld Implementation Details · p. 14
  • The first P boids serve as camera agents and each receives a synchronized 128 × 128 crop centered on itself.The camera agents follow the same flocking dynamics as the other boids, and each episode records the full state of every boid.Source: B FlockWorld Implementation Details · p. 14
  • Training uses fresh 21-frame clips generated on the fly, while evaluation uses held-out 30-second episodes with seeds disjoint from training.Multiple adjacent windows from an episode amortize simulation cost but introduce some correlation; the number of unique training clips is primarily limited by compute budget.Source: B FlockWorld Implementation Details · p. 14

C Visual Conditions

FlockWorld couples synchronized agent-centered views with explicit camera identities, while FlockDiT models next-frame latents jointly across agents. The visual, latent, and conditioning choices define how multi-agent observations and actions enter that model.

C Visual Conditions

  • Camera agents receive evenly spaced hues while other boids remain white, supporting identity tracking and ground-truth-free cross-view consistency checks.The black-background condition is used for the mechanism comparison and agent-count sweep; the gradient-background condition is a separate run without a matching single-agent floor.Source: C Visual Conditions · p. 14

D Problem Setup

  • Each of P camera agents supplies an egocentric video stream and per-frame acceleration actions; the model predicts all agents’ next latent frames from past latents and actions.The model is rolled out autoregressively, with predicted latents decoded to video; training uses rendered pixels and actions rather than simulator state.Source: D Problem Setup · p. 15

E Video Autoencoder

  • The condition-specific causal autoencoders compress video 16× spatially and 4× temporally, then remain frozen while the world model trains and rolls out in latent space.The black-background autoencoder’s best validation loss was 0.0042; the gradient-background autoencoder was trained for roughly 887k steps on streamed clips, versus roughly 115k steps for the black-background model on recorded data.Source: E Video Autoencoder · pp. 15–16

F FlockDiT Model Implementation Details

  • FlockDiT treats each latent vector as a token and uses block-causal attention across all agents’ tokens at each latent timestep.Attention blocks span P · S tokens, making attention cost quadratic in the number of agents; action and noise-level conditioning modulate attention and feed-forward layers through AdaLN-zero.Source: F FlockDiT Model Implementation Details · p. 16

G Multi-Agent Conditioning

  • The embedded scheme distinguishes agents with learned embeddings, while the tiled scheme associates actions with view tiles using learned per-agent tags and mean-pooled action embeddings.The tiled scheme follows MIRA’s warm-start curriculum; the table caption states that the runs use 27.5M-parameter embedded or 27.7M-parameter tiled models.Source: G Multi-Agent Conditioning · pp. 16–17, Table 4 · p. 17

H Experiments and Training Protocol

The experiments isolate view conditioning, diffusion forcing, and two-stage training under a shared training protocol. Configurations were selected without hyperparameter search and evaluated under a common fixed setup.

H Experiments and Training Protocol

  • Experiments 2–4 each add one ingredient to the baseline, while Experiments 2, 5, 6, and 7 form a 2 × 2 design testing diffusion forcing and two-stage training with tiled views.Experiment 6 combines both techniques and represents the MIRA-inspired recipe.Source: H Experiments and Training Protocol · p. 17
  • All runs use the same optimizer, learning rate, batch size, and clip structure under a fixed 48-hour wall-clock budget, so more expensive methods complete fewer optimization steps.Experiments are in the black-background condition; differences between conditions are not attributable to per-run hyperparameter tuning, but each uses a common configuration rather than an individually optimized one.Source: H Experiments and Training Protocol · pp. 17–18

I Hyperparameters

  • The study performs no hyperparameter search; model size and batch size were chosen to fit a single-GPU run within the 48-hour allocation rather than by validation performance.The world-model and frozen-autoencoder configurations are listed in Tables 5 and 6; the latter’s training-data and training-length entries describe the black-background checkpoint used by all but the gradient-background run.Source: H Experiments and Training Protocol · pp. 17–18, Table 5 · p. 18, Table 6 · p. 19

J Evaluation Protocol and Metrics

Evaluation separates fidelity to the recorded trajectory from consistency among generated views, using held-out episodes and decoded pixels. The protocol’s episode-level uncertainty and checkpoint and density differences qualify how comparisons should be read.

J Evaluation Protocol and Metrics

  • Models are evaluated on 28 held-out episodes by generating 10-second autoregressive rollouts from two latent context frames while replaying recorded actions.Metrics are computed per episode and averaged; cross-view metrics are intended to distinguish internal view inconsistencies from expected divergence relative to ground truth.Source: J Evaluation Protocol and Metrics · pp. 19–20
  • Error bars capture variability across 28 held-out episodes, not variation across training runs, and no hypothesis tests are performed.Some cross-view metrics exclude episodes without required sightings, so their effective sample size can be below 28.Source: J Evaluation Protocol and Metrics · p. 19
  • Table 2 mixes best-validation checkpoints for models at P ≥5 with a final-snapshot floor, and agent-count comparisons also reflect separate training runs with differing rendered density.The agent-count sweep and Experiment 1 differ at 7.11 versus 13.08 detections per frame; the passage cautions that fixed-configuration density is only partly reproducible across runs.Source: J Evaluation Protocol and Metrics · p. 20

J.1 Per-View Fidelity (versus Ground Truth)

  • Detection rate measures detected ground-truth-visible boids, while position error averages residuals only over matched detections and must be read alongside detection rate.Low position error can reflect a small, relatively easy subset when a model renders very few boids; detections per frame tracks over- and under-rendering.Source: §J.1 · p. 20

J.2 Cross-View Consistency (Ground-Truth-Free)

  • Cross-view reciprocity measures the fraction of sightings that are mutual, and displacement error measures disagreement in reciprocal views’ relative-position estimates.Motion error applies the relative-offset test across consecutive frames only when the pair remains reciprocal; white-boid and pixel-overlap measures provide additional checks.Source: §J.2 · p. 21
  • Each condition has its own decode-based ceiling, while the single-agent floor measures cross-view agreement without joint modeling; the gradient-background run has no floor.Cross-view rates are interpreted with their volume counts because models that rarely render other agents have fewer opportunities for consistency failures.Source: §J.2 · p. 22

K Evaluation Details

The evaluation detector extracts foreground components, assigns camera-agent identities from hue, and filters boundary artifacts before matching detections.

K Evaluation Details

  • Foreground pixels are grouped into connected components, whose positions are brightness-weighted centroids; hue assigns camera-agent identity and low-saturation detections are classified as white boids.Three filters remove elongated border lines, large corners, and small detections touching the image boundary.Source: K Evaluation Details · p. 22
  • Tracks link detections across frames within 8 pixels and use track-wide identity votes to reduce false sightings from brief hue flicker.Ground-truth projections match detections within 6 pixels, while white-boid correspondence uses optimal assignment within 8 pixels and rejects ambiguous matches.Source: K Evaluation Details · p. 22

L Qualitative Rollouts

The qualitative rollouts contrast independent views that hallucinate colored agents and lose focal identity with diffusion forcing, which preserves focal identity and improves cross-view agreement. Reproducibility details specify the evaluation references, run configurations, and compute budget.

L Qualitative Rollouts

  • The independent single-agent floor hallucinates colored boids and loses focal identity, while diffusion forcing preserves identity and makes fewer, more cross-view-consistent claims.Both generated models accumulate softer boids and ring-like artifacts as they rely on generated frames for context.Source: L Qualitative Rollouts · pp. 22–23

M Reproducibility

  • The seven released runs map to Experiments 1–7 in Table 4, although checkpoint identifiers are noncontiguous and differ from the paper’s experiment numbering.Each YAML variant changes only the flags for its added mechanism.Source: M Reproducibility · p. 23
  • Evaluation fixes held-out episodes, rollout length, sampling steps, and references; its dedicated full-budget floor differs from the unused stage-1-pretrain floor in released tables.The VAE-decode ceiling and floor use the same protocol and detector as the tested models.Source: M Reproducibility · pp. 23–24
  • Fixed seed 42 governs the simulator, data splits, and training; episode seeds derive from the base seed and episode index, and each visual condition uses one fixed autoencoder.This makes evaluation episodes and streamed training distributions reproducible from the configuration files.Source: M Reproducibility · p. 24

N Compute Resources

  • World-model experiments used 720 GPU-hours across fifteen runs, with one GPU and a 48-hour slot per run; two-stage runs split that budget between stages.Autoencoder training and evaluation are additional to the 720 GPU-hours.Source: N Compute Resources · p. 24
  • A full evaluation pass over all checkpoints takes about eleven hours on one GPU, while the video autoencoder was trained separately on an RTX 4090.Evaluation ran outside the cluster, and the autoencoder campaign used different hardware from the world-model runs.Source: N Compute Resources · p. 24

O Limitations

The study’s conclusions are bounded by a simple deterministic flocking environment, single runs per condition, metric and model constraints, and dense-attention scaling. The paper also describes accessibility, evaluation contributions, and incorporated code assets.

O Limitations

  • The two-dimensional, deterministic flocking environment isolates cross-view consistency, so findings may not transfer directly to visually rich, three-dimensional, or stochastic settings.The authors state that the study is intended to isolate the consistency problem rather than establish generality.Source: O Limitations · p. 25
  • Each condition was trained once under an approximately 27.5M-parameter, 48-hour single-GPU budget, leaving persistence across repeated runs undetermined.The study therefore cannot establish whether observed differences would persist across training runs.Source: O Limitations · p. 25
  • Replayed accelerations come from the original simulation’s neighboring boids and may become inconsistent with the generated flock as rollout proceeds.The authors suggest closing the action-state loop by recomputing accelerations from the generated multi-agent state.Source: O Limitations · p. 25
  • Hue-based detection limits metrics, reciprocity depends on event volume, and identity-dependent scores cannot be compared directly across agent counts because their measurement ceilings vary with P.The frozen autoencoder also bounds reconstruction quality, while 16× spatial compression limits representable boid detail.Source: O Limitations · p. 25
  • Dense cross-agent attention is the primary agent-count bottleneck: increasing agents from 10 to 30 raises sequence length from 3,840 to 11,520 tokens and per-timestep cost roughly ninefold.The simulator supports hundreds of boids, but scaling FlockDiT beyond the reported P = 30 requires relaxing dense attention; sparse spatial attention is proposed but not evaluated.Source: O Limitations · pp. 25–26
  • The gradient-background comparison has one run and no condition-matched floor, and the study leaves interactive rather than replayed-action generation for future evaluation.A condition-matched single-agent model is needed to complete that comparison.Source: O Limitations · p. 26

P Broader Impacts

  • Procedural data, released code, and a 27.5M-parameter single-GPU setup support accessibility; the evaluation also separates reference-trajectory errors from disagreement between views and reports consistency alongside density.The authors say these evaluation controls can reveal high consistency scores achieved for the wrong reason and can apply beyond flocking.Source: P Broader Impacts · p. 26

Q Existing Assets and Licenses

  • The project adapts Solaris’s flow-matching transformer and Wan 2.2’s causal video VAE, reimplements the Boids update loop, and reimplements MIRA from its published description.The paper specifies that it does not redistribute released Wan weights.Source: Q Existing Assets and Licenses · pp. 26–27

Related papers

Loading 2610.34564v1…