Controllable Crowd Generation through World-Model Planning

Motivation

• A realistic simulator must model how pedestrians change motion when conditions change, such as rerouting around a construction blockage, while responding to user objectives.Crowd simulation is relevant to urban planning, autonomous vehicles, and social robots.Source: §1 · pp. 1–2

Research gap

• Existing learned control is limited to predefined population and agent parameters or to gradient guidance and sample filtering, while hand-designed rules do not directly learn motion from pedestrian videos.TRACE reports 1 ∼3 seconds to sample a single pedestrian trajectory.Source: §1 · p. 2

Approach

• Ctrl-CWM learns a shared representation through trajectory prediction on real-world pedestrian data, freezes its encoder, and uses a shared actor to propose each pedestrian’s next displacement.The state comprises the scene and each pedestrian’s position, recent motion, and local goal; actions are next-step displacements.Source: §1 · p. 2
• The actor’s actions update the crowd state to generate imagined trajectories, which the critic ranks using supervision derived from ground-truth trajectories.At runtime, the planner combines critic rankings with user costs for objectives such as avoiding or approaching a region, then repeats planning to advance the crowd.Source: §1 · p. 2

Results

• Ctrl-CWM outperforms the state-of-the-art method on most crowd realism and collision metrics, achieves the highest average avoidance compliance across tested region configurations, and demonstrates control in reconstructed real-world scenes.Competitive trajectory-prediction results are reported as evidence that the shared representation preserves human motion dynamics.Source: §1 · pp. 2–3

2.1 HUMAN TRAJECTORY PREDICTION

• Trajectory-prediction research represents social interaction, scene context, intent, and future uncertainty using approaches including graphs, semantic maps, goal heatmaps, and diffusion; Ctrl-CWM uses an agent-centric representation for crowd generation and control.Examples include graph-based social structure (Mohamed et al., 2020; Shi et al., 2021), semantic maps (Mangalam et al., 2021), and diffusion (Gu et al., 2022; Mao et al., 2023).Source: §2.1 · p. 3

2.2 CROWD SIMULATION AND GENERATION

• Crowd-generation work includes rule-based, data-driven, and learned approaches, while control methods range from animation editing and diffusion guidance to population controls and text-generated crowd fields.The passage describes GREIL-Crowds, SPDiff (Chen et al., 2024), CEDRL, TRACE (Rempe et al., 2023), and CrowdES.Source: §2.2 · p. 3

2.3 WORLD MODELS AND PLANNING

• World models support online planning through learned dynamics, imagined rollouts, and value estimates; Ctrl-CWM adapts these ideas to crowd motion by freezing a trajectory-trained encoder and applying user costs through CEM.Its actor and critic are learned over imagined rollouts on the frozen representation.Source: §2.3 · p. 3, Figure 2 · p. 4

3.1 PROBLEM FORMULATION

• The crowd state includes the static scene and each agent’s position, velocity history, and local goal; each action is a one-step displacement, and goals remain fixed within an L = 12 prediction and behavior-learning window.The transition updates positions from assigned displacements and then updates velocity histories.Source: §3.1 · p. 4
• Ctrl-CWM uses a feature extractor, shared actor, and shared critic; the actor proposes each agent’s displacement and the critic scores its motion within an imagined crowd-state sequence.The planner evaluates candidate displacements through imagined rollouts to optimize its planning objective.Source: §3.1 · p. 4

3.2 LEARNING THE WORLD MODEL FROM PREDICTION

• Trajectory prediction learns an agent-centric representation by combining scene-map and agent heatmap features with attention-based social context and GRU-encoded velocity history.The interaction embedding modulates the U-Net bottleneck through feature-wise linear modulation.Source: §3.2 · pp. 4–5
• Goal, waypoint, and dynamics heads predict the endpoint of the L-step window, waypoint heatmaps, and a one-step position heatmap, respectively; the dynamics head provides auxiliary supervision but does not advance rollouts.At inference, the waypoint decoder uses the predicted goal, from which a coordinate serves as the local navigation goal.Source: §3.2 · p. 5

3.3 LEARNING BEHAVIOR IN IMAGINATION

• After prediction training, the encoder is frozen; the actor then learns crowd motion in imagined rollouts, re-encoding updated states at each step and training against ground-truth trajectories.Its imitation loss combines pointwise position error with soft-DTW, and it additionally matches speed profiles and successive speed changes.Source: §3.3 · pp. 5–6
• The critic ranks candidate-action rollouts using costs derived from generated trajectories, while the actor supplies the remaining actions throughout each rollout.For each training state and agent, candidate displacements are sampled around the actor proposal; the critic is trained to assign higher scores to lower-cost candidates.Source: §3.3 · p. 6

A GRU followed by an MLP processes agent i’s velocity, speed, nearest-neighbor distance, goal distance, and velocity-goal alignment over the first H steps to estimate Vϕ(ˆs(k)

• The critic estimates an agent’s rollout score from the first H imagined steps, using velocity, speed, nearest-neighbor distance, goal distance, and velocity-goal alignment.At inference, it evaluates imagined rollouts without future ground-truth trajectories.Source: A GRU followed by an MLP processes… · p. 6

3.4 PLANNING AND CONTROL

• CEM samples candidate displacements, evaluates each through an H-step imagined rollout using critic quality and a task-dependent spatial cost, then refines its sampling distribution and executes the final mean.Setting λuser = 0 yields critic-guided crowd generation; planning repeats after the crowd advances.Source: §3.4 · p. 6
• Table 1 compares crowd generation under identical replayed random-surface and diffusion arrivals; AVG aggregates five ETH–UCY folds, SDD, and GCS.Lower is better except for diversity, and collision rates are reported in percent.Source: Table 1 · p. 7

4.1 CROWD BEHAVIOR GENERATION

• Ctrl-CWM ranks first on nine of sixteen average scores and second on seven, with the lowest average density error and DTW under both arrival protocols.Relative to CrowdES, average DTW falls by 22% and 18% under random-surface and diffusion arrivals, respectively.Source: §4.1 · p. 7

4.2 RUN-TIME CONTROL

• For multiple disjoint avoidance zones, Ctrl-CWM reaches 0.950 compliance, versus 0.579 for CrowdES and 0.087 for ORCA.Ctrl-CWM has the highest average compliance for every tested zone configuration; Figure 4 depicts avoidance and attraction rollouts.Source: §4.2 · pp. 7–8, Table 2 · p. 8, Figure 4 · p. 8

4.3 ANALYSIS

• Pretraining reduces kinematic error from 1.205 to 0.465 and increases diversity from 0.157 to 0.285.Trajectory-prediction heads achieve 0.21/0.31 m average ADE/FDE on ETH–UCY and 5.89/10.26 px on GCS under best-of-20 evaluation.Source: §4.3 · p. 8, Table 3 · p. 9
• Critic-guided planning reduces DTW from 0.970 to 0.925 and collisions from 0.931% to 0.756% versus actor-only generation, with a small kinematic-error increase.User-cost-only planning has higher compliance (0.858 vs. 0.824) but performs worse on all eight realism and accuracy metrics.Source: §4.3 · pp. 8–9, Table 3 · p. 9
• Removing soft-DTW raises DTW to 0.989 and kinematic error to 0.846 while lowering diversity to 0.182; removing collision supervision raises collisions to 0.919%.Among tested planning settings, K = 16 maximizes compliance and minimizes DTW and collisions; defaults K = 16, I = 4, H = 4 yield about 10 ms per call.Source: §4.3 · p. 9, Figure 5 · p. 9

• Ctrl-CWM unifies crowd generation and run-time control through a representation learned from real-world pedestrian videos.An actor proposes motion, a critic ranks imagined futures, and user-defined costs steer behavior without retraining or gradient-based guidance.Source: §5 · p. 9
• The framework currently requires user objectives as explicit spatial costs rather than high-level semantic commands.Future work will ground language-based instructions into planning objectives.Source: §5 · p. 9

• Generative AI tools were used only to edit text for readability, and the authors reviewed the edits and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The work uses publicly available datasets and 3D scenes constructed from open-source resources or purchased assets.Source: ETHICS STATEMENT · p. 10
• Generated behaviors may reflect source-data biases and simulation-environment limits, and should not be assumed to represent all populations or settings.Simulation results do not establish real-world safety; additional validation and appropriate safeguards would be required.Source: ETHICS STATEMENT · p. 10

A IMPLEMENTATION DETAILS

• The appendix covers model architecture, training, planning, evaluation protocols and metrics, analyses, visualizations, and per-benchmark results.Source: A IMPLEMENTATION DETAILS · p. 14

A.1 MODEL ARCHITECTURE

• The scene input combines a walkability mask, eight semantic channels, and focal-agent and crowd heatmaps; scene and interaction encoders provide features for goal-conditioned prediction heads.The goal decoder predicts the endpoint of the L-step window, while waypoint and auxiliary dynamics heads produce L waypoint heatmaps and a next-position heatmap; the auxiliary head is not used to advance rollout states.Source: §A.1 · p. 14

A.2 TRAINING PROCEDURE

• Behavior learning rolls out actor-predicted states using imitation, kinematic matching, and adversarial supervision, while the critic learns to rank candidate displacements from trajectory-derived costs.Candidate costs use L rollout steps, but the critic observes geometric features from the first H steps; actor training has no direct collision penalty, critic score, or run-time user cost.Source: §A.2 · p. 16

A.3 PLANNING, GOAL UPDATES, AND EXECUTION

• CEM refits candidate displacements around the actor proposal, while the reported planner optimizes each agent independently and executes the resulting displacements simultaneously.Full-model generation retains CEM with the user-cost weight set to zero; control adds the user cost. U-Net maps are cached during imagined rollouts while interaction embeddings and local sampling locations are updated.Source: §A.3 · pp. 16, 18

A.4 CHECKPOINT SELECTION

• Checkpoint selection pairs actors and critics using held-out rollout position error and critic–cost ranking correlation, choosing the highest-correlation pair among actors within 5% of the lowest selection error.Reported ranking correlations range from 0.60 to 0.76 across folds.Source: §A.4 · p. 18

B.1 EVALUATION SCOPE

• Within each generation protocol, compared methods receive identical replayed entries from the same emitter, and the main average weights seven benchmark entries equally.Generation and control columns use separate evaluations, so their values should not be combined as if they came from the same controlled rollout.Source: §B.1 · p. 18, Table 7 · p. 18
• The main avoidance comparison uses 10 trials per scene over the final two thirds, but Ctrl-CWM activates its objective after the first third while baselines use obstacles from initialization.Thus the baseline comparison does not match online objective-activation timing.Source: §B.1 · p. 19

B.2 GENERATION METRICS

• Generation metrics compare density, frequency, coverage, population, kinematics, DTW, diversity, and collision rate; lower is better except for diversity.Frequency and coverage coincide on single-category ETH–UCY and GCS data but can differ on SDD, so displayed ranks are not independent measurements.Source: §B.2 · p. 19

C.1 INTERPRETATION OF THE MAIN ABLATIONS

• User-cost-only planning achieves higher compliance (0.858 vs. 0.824), while the full model performs better on all eight motion metrics.The user-cost-only planner retains the actor, imagined rollouts, and CEM but removes the critic score; both variants use the same avoidance protocol and post-activation evaluation window.Source: §C.1 · p. 19
• The loss-removal variants retain the planner and critic, and their separately evaluated compliance decreases do not establish a control–realism trade-off within the same controlled trajectories.Collision supervision here is the collision term in the critic target, not a direct actor collision loss.Source: §C.1 · p. 19

C.2 INTERACTION AND AUXILIARY SUPERVISION

• The interaction encoder raises mean compliance from 0.578 to 0.880 and improves compliance on four of five folds, although collision effects vary by fold.The additional component experiments use disc avoidance, diffusion arrivals, and 20 trials per scene, with evaluation in the post-activation window.Source: §C.2 · pp. 19–20
• The auxiliary dynamics head raises mean compliance from 0.584 to 0.880 and improves it on four folds, although ZARA1 improves without the head.Because the auxiliary head is not used for state updates, this comparison concerns training supervision rather than an inference-time transition module.Source: §C.2 · p. 20

C.3 CRITIC TRAINING OBJECTIVE

• The listwise critic has lower mean kinematic error, DTW, and collision rate, and higher compliance than pointwise regression.On ETH, collision rate is 0.458% for the listwise critic and 7.874% for pointwise regression; this evaluates the critic’s training objective, not removal of the critic.Source: §C.3 · p. 20

D.1 SPATIAL COSTS AND COMPLIANCE

• Avoidance cost penalizes the fraction of imagined agent steps inside a zone, while attraction cost penalizes mean distance to the target center.Changing the target or zone changes the planning objective without retraining the model or changing the static scene map.Source: §D.1 · p. 21
• Compliance measures relative occupancy change, not route completion or the percentage of agents obeying, and avoidance requires nonzero uncontrolled occupancy.For avoidance, one means zero controlled occupancy, zero means no reduction, and negative values indicate increased occupancy.Source: §D.1 · p. 21

D.2 EFFECT OF THE USER-COST WEIGHT

• Avoidance compliance rises from 0.607 at weight 5 to 0.863 at 10 and 0.985 at 25, but higher weights can slow or stop pedestrians near the region.Between weights 25 and 100, mean compliance stays between 0.968 and 0.985 without improving monotonically; the largest mean increase in windowed kinematic error is 0.099.Source: §D.2 · p. 21, Table 11 · p. 22
• Attraction compliance rises from 0.381 at λuser = 1.5 to 0.645 at 3.0 and 0.951 at 5.0, while stronger attraction can increase collisions and alter crowd statistics.At λuser = 5.0, changes in kinematic error and collision rate reach +7.76 and +26.5 pp, respectively; attraction and avoidance weights use different cost scales.Source: §D.2 · p. 21, Table 12 · p. 22, Table 13 · p. 22

D.3 SENSITIVITY TO THE EMITTER POPULATION

• At avoidance weight 25, mean compliance ranges from 0.928 to 0.985 across emitter settings, while weight 10 compliance falls from 0.863 at multiplier 1 to 0.701 at multiplier 2.At weight 25, collision changes have magnitude at most 0.6 pp; at weight 10, changes stay within 0.5 pp except for a 9.5 pp increase on ETH at multiplier 0.5.Source: §D.3 · p. 23, Table 14 · p. 23

E TRAJECTORY PREDICTION

• Prediction pretraining yields lower generation kinematic error and higher diversity than random initialization, but does not establish whether freezing beats fine-tuning or whether counterfactual control responses are accurate.The prediction evaluation uses eight observed steps to predict 12 future steps with best-of-20 ADE/FDE; ETH–UCY errors are in meters and SDD and GCS errors are in pixels.Source: E TRAJECTORY PREDICTION · p. 23, Table 15 · p. 24

F ADDITIONAL QUALITATIVE RESULTS

• Figure 6 shows avoidance redirecting pedestrians away from a specified region and attraction drawing them toward a target across three reconstructed scenes per objective.The columns show the initial scene, motion without control, and motion with control; the visualizations do not independently establish quantitative transfer performance or collision-free operation.Source: F ADDITIONAL QUALITATIVE RESULTS · p. 25, Figure 6 · p. 25

G DETAILED BENCHMARK RESULTS

• The detailed benchmark results show that aggregate improvements are not uniform across datasets or metrics, including higher diffusion-arrival kinematic error than CrowdES on several entries and a higher SDD collision rate.Tables 16 and 17 reproduce generation values; Table 18 reports per-benchmark avoidance results, with Ctrl-CWM activating its objective after a shared first third.Source: G DETAILED BENCHMARK RESULTS · p. 26, Table 16 · p. 26, Table 17 · p. 27