PaperlayerView exact revision on arXiv

Source-linked AI summary

ReactiveGWM: Steering NPC in Reactive Game World Models

Zeqing Wang, Danze Chen, Zhaohu Xing, Zizhao Tong, Yinhan Zhang, Xingyi Yang, Yeying Jin

arXiv:2605.15256v1cs.CV

TL;DR

Existing game world models often treat NPCs as background rather than autonomous agents, limiting explicit player–NPC interaction modeling. ReactiveGWM decouples player control from NPC strategy, achieving prompt-aligned NPC behavior while preserving fine-grained player controllability and enabling zero-shot transfer across two Street Fighter games.

  • Problem

    Most game world models model NPCs as player-conditioned background elements, lacking explicit channels for autonomous, high-level tactical behavior.

  • Method

    ReactiveGWM uses strategy-aligned NPC prompts, additive player-action injection, and cross-attention strategy grounding to separate player control from NPC autonomy.

  • Results

    ∼43% to over 75% on SF2 and ∼41% to ∼79% on SF3, ReactiveGWM’s VLM-judged NPC instruction accuracy improves over the vanilla model while preserving player control.

  • Takeaways & Limitations

    ReactiveGWM’s NPC autonomy modules transfer to off-the-shelf world models across games without additional annotation or domain-specific retraining.

  • Takeaways & Limitations

    Evaluation is limited to 2D fighting games, and the diffusion backbone has high inference latency that prevents truly real-time interaction.

Abstract

from arXiv · show

Current game world models simulate environments from a subjective, player-centric perspective. However, by treating the Non-Player Character (NPC) merely as background pixels, these models cannot capture interactions between the player and NPC. In that sense, they act as passive video renderers rather than real simulation engines, lacking the physical understanding needed to model action-induced NPC reactivities. We introduce ReactiveGWM, a reactive game world model that synthesizes dynamic interactions between the player and NPC. Instead of entangling all interaction dynamics, ReactiveGWM explicitly decouples player controls from NPC behaviors. Player actions are injected into the diffusion backbone via a lightweight additive bias, while high-level NPC responses (e.g., Offense, Control, Defense) are grounded through cross-attention modules. Crucially, these modules learn a game-agnostic representation of interactive logic. This enables zero-shot strategy transfer: our learned modules can be plugged directly into off-the-shelf, unannotated world models of different games. This instantly unlocks steerable NPC interactions without any domain-specific retraining. Evaluated on two Street Fighter games, ReactiveGWM maintains fine-grain player controllability while achieving robust, prompt-aligned NPC strategy adherence, paving the way for scalable, strategy-rich interaction with the NPC.

1 Introduction

ReactiveGWM addresses the limitations of player-centric game world models by explicitly decoupling player control from autonomous, strategy-driven NPC behavior. It enables dynamic player–NPC interactions and zero-shot strategy transfer across different games without domain-specific retraining.

  • Motivation: Existing game world models treat NPCs as background pixels, relying on player-centric prompts and implicitly deterministic player–background relations.This prevents them from explicitly modeling NPCs as dynamic and autonomous agents.
  • ReactiveGWM: ReactiveGWM synthesizes dynamic interactions by using datasets that decouple NPC autonomy from player control and provide structured NPC-only strategy prompts.Each sample combines gameplay videos and player action labels with a structured prompt guiding only the NPC.
  • Architecture: Player actions enter the video diffusion backbone through a lightweight additive bias, while high-level NPC strategies are grounded through cross-attention modules.The cross-attention modules learn a player-agnostic representation of interaction logic from strategy-aligned data.
  • Evaluation: Across two distinct Street Fighter games, ReactiveGWM preserves fine-grained player controllability while producing autonomous, strategy-aligned NPC behavior and realistic dynamic interactions.The experiments also show strong zero-shot strategy transferability.
  • Zero-shot Transfer: Its learned NPC autonomy modules plug directly into off-the-shelf vanilla world models for different games without additional domain-specific retraining.This supports strategy transfer while retaining fine-grained player control and strategy-driven NPC autonomy.

2 Related works

Related work progresses from controllable video generation toward world models that predict future states under interventions, including game world models conditioned on player inputs. However, most game world models treat NPCs as background dynamics, lacking explicit channels for tactical intent or strategy following.

  • Controllable video generation: Video generation frameworks add motion priors and trajectory inputs to improve spatial and temporal alignment beyond text-prompt control.Text prompts enable customized generation but can produce spatiotemporal ambiguities.
  • World models: World models predict future states and environmental transitions from current observations and external interventions, supporting agents’ predictive mental models and strategic planning.This work frames world models as an evolution from localized controllability toward causal physical-mechanics simulation.
  • Game world models: Game world models simulate environments by predicting future visual frames conditioned on player inputs, with prior systems targeting real-time operation, visual fidelity, and broader game coverage.Examples include GameNGen, DIAMOND, Matrix-Game 2.0/3.0, LingBot-World, GameFactory, and Oasis.
  • Game world models: Most game world models restrict conditioning to the primary player’s action stream, leaving NPC behavior as passive background dynamics without explicit tactical intent or strategy following.This conditioning design compromises NPC autonomy and neglects high-level interaction control.

3 Method

ReactiveGWM decouples fine-grained player control from high-level, strategy-driven NPC behavior by replacing entangled scene prompts with NPC-specific strategy prompts. It injects actions through lightweight additive biases and enables zero-shot strategy transfer by transplanting learned Cross-Attention modules into another game’s vanilla backbone.

  • Method Overview: ReactiveGWM replaces the entangled vanilla prompt with an NPC-specific strategy prompt that guides tactical intent and behavior mode while preserving fine-grained player control.The NPC prompt governs decision-making and interaction patterns, so generated videos reflect both player-action consequences and autonomous strategy-following NPC behavior.
  • Data Construction: The dataset uses SF2 and SF3 gameplay clips with aligned video and frame-level action streams collected through stable-retro.A random agent samples 10 discrete action buttons; episodes are segmented into 5-second clips at 20 fps.
  • Data Construction: Gemini annotates active behaviors, passive behaviors, and one of three mutually exclusive strategies: Offense, Control, or Defense.Each clip forms a triplet (x0:T, aT, PNPC), and the pipeline curates ∼10,000 training triplets per game.
  • Action Conditioning: Discrete actions are temporally aligned by adaptive max-pooling and injected into each DiT block through independent, bias-free linear projections and spatial broadcasting.For both games, K=10 action channels produce a tensor of shape [B, f, K], which becomes an action bias added to the video latent residual stream.
  • Training: ReactiveGWMbase is obtained by full-parameter fine-tuning on a fully annotated source-game dataset, with Cross-Attention grounding PNPC into visual-temporal latents.The jointly optimized DiT components include the Action Module, Self-Attention, Cross-Attention, and FFN.
  • Autonomous NPC Transfer: ReactiveGWMtransfer preserves a target game’s vanilla backbone and directly injects learned source-game Cross-Attention layers for zero-shot strategy conditioning.The retained target modules include the pre-trained Action Module, Self-Attention layers, and FFN, avoiding dense frame-aligned strategy annotations for the new game.

4 Experiments

Experiments evaluate ReactiveGWM on Action Control, NPC Strategy Following, and Visual Quality, showing strategic NPC autonomy alongside high visual fidelity and player controllability. Transfer experiments further show that strategy compliance can be added to vanilla models without exhaustive full-parameter retraining.

  • Evaluation Setup: ReactiveGWM is evaluated across Action Control, NPC Strategy Following, and Visual Quality, with baselines restricted to strategy following and image quality.The evaluation uses player action adherence, VLM-judged tactical behavior, and long-term visual fidelity against ground-truth game-engine outputs.
  • NPC Strategy Following: Over 75% instruction accuracy on SF2 and ∼79% on SF3, versus ∼43% and ∼41% for the vanilla model, respectively.The strategy prompt PNPC provides an explicit tactical-intent signal, enabling behaviors beyond passive environmental dynamics.
  • NPC Strategy Following: ReactiveGWM follows Offense, Defense, and Control directives, producing aggressive engagement, evasive distancing, and projectile-based zoning behaviors.Under Offense, the NPC approaches and fights; under Defense, it maintains distance and evades; under Control, it uses ranged projectiles.
  • Action Control and Visual Quality: 100.0% Move-Acc and Att-Acc in SF3 demonstrate near-perfect single-action control, while visual quality remains strictly on par with the vanilla baseline.The model also qualitatively adheres to diverse, fine-grained sequence commands.
  • Transferability: 97.5% Move-Acc in SF2 and up to 73.7% NPC Strategy Following in SF3 show that ReactiveGWMtransfer retains controllability and fidelity while transferring strategy compliance to a vanilla model.The transfer avoids exhaustive full-parameter retraining.
  • Transferability: 0.71% ρcross for ReactiveGWMTransfer, versus 0.70% for Vanilla, indicates that cross-attention is a low-bandwidth channel while still steering NPC behaviors.The remaining ∼99.3% of energy governing main visual components is left undisturbed, and the accumulated directional signal steers NPC trajectories without interfering with dominant visual dynamics.

5 Conclusion

ReactiveGWM separates player control from autonomous NPC behavior through additive action biases and cross-attention strategy modules. Experiments on two Street Fighter games show prompt-consistent NPC behavior and zero-shot transfer across games without annotation or retraining.

  • 5 Conclusion: ReactiveGWM separates NPC autonomy from player control, injecting player actions through an additive diffusion-backbone bias and grounding strategies through cross-attention.Strategies provide high-level tactical guidance such as Offense, Control, and Defense rather than describing all scene dynamics in one prompt.
  • 5 Conclusion: Experiments on two Street Fighter games show more autonomous and prompt-consistent NPC behavior than vanilla game world models.
  • 5 Conclusion: Learned NPC behavior modules transfer to off-the-shelf world models for different games without additional annotation or retraining.This transfer demonstrates that the modules capture a game-agnostic representation of interaction logic.

A Data construction … A.4 Prompt assembly

The data pipeline records controllable Street Fighter interactions, segments them into labeled clips, and constructs reproducible NPC-strategy guidance from factual observations and deterministic rules. It supports shared behavior IDs across both games while preserving game-specific button semantics.

  • A Data construction: The appendix presents the data construction pipeline used for the paper’s interactive game-world modeling experiments.It covers action recording, clip segmentation, NPC strategy annotation, and prompt assembly.
  • A.1 Action space and recording setup: SF2 and SF3 use stable-retro with fixed Player-versus-NPC matchups, 10 physical buttons, and 13 shared discrete behaviors spanning both games.The emulator exposes a 12-bit vector, comprising four directional and six attack buttons used here; behavior IDs 0–12 are shared.
  • A.1 Action space and recording setup: Game-specific attack-button mappings preserve distinct punch/kick semantics, while directional keys remain shared across the two titles.The six attack buttons map differently to light, medium, and heavy punches or kicks in each game.
  • A.1 Action space and recording setup: Each behavior uses EDGE/HOLD serialization over a 10-frame decision block, holding directional keys while emitting attack and UP edges at block start.Thus, one action decision corresponds to 10 video frames.
  • A.2 Episode recording and clip segmentation: A uniform random agent samples behaviors 0–12 until knockout, discards the first 5 s, and segments the remainder into 5-second, 100-frame clips at 20 FPS.Each clip includes native-resolution video.mp4 and a frame-level actions.parquet file recording the 12-bit button vector.
  • A.3 Two-stage NPC strategy annotation: NPC strategy annotation separates VLM factual observation from deterministic classification, preventing strategy naming during observation and dropping clips matching no rule.Gemini answers 12 closed-value factual questions and tags NPC moves as active or passive before rule-based labeling.
  • A.3 Two-stage NPC strategy annotation: The mutually exclusive rules assign each surviving clip a unique Offense, Control, or Defense label solely from unambiguous observed facts.The provided rules define category-specific conditions, including melee activity, movement, guarding, and passive reactions.
  • A.4 Prompt assembly: Final NPC guidance combines active and passive vocabulary tags with the deterministic strategy label and a hashed natural-language paraphrase selected from its category pool.MD5(video_path) modulo the category-pool size guarantees bit-wise reproducibility while varying equivalent strategy wording.

B Benchmark details … B.3 Visual quality

The benchmark evaluates models on player action following, NPC strategy following, and visual quality under shared frozen splits, rollout settings, and preprocessing. It uses dedicated action, strategy, and perceptual metrics across Street Fighter II and III, with SF2 shown as the procedural example.

  • B Benchmark details: All compared methods use identical frozen data splits, rollout settings, and preprocessing, while SF2 illustrates the evaluation procedure also applied to SF3.The protocol covers Player Action Following, NPC Strategy Following, and Visual Quality.
  • B.1 Player action following: The player-action benchmark contains 100 runs from 10 initial frames × 10 single-key actions, with each rollout generating 41 frames.Actions are LEFT, RIGHT, UP, DOWN, Y, X, Z, A, B, and C, sparsely encoded with repeated key presses.
  • B.1 Player action following: Player trajectories are extracted from SAM2.1 and Grounding DINO segmentation masks using normalized bbox centers, dimensions, mask area, and aspect ratio.The trajectory tuple is (x, y, w, h, area, aspect), with aspect = h/w.
  • B.1 Player action following: Movement Accuracy scores normalized-coordinate displacement with thresholded rules, applies a 5-frame rolling median, and averages performance over LEFT, RIGHT, UP, and DOWN.The UP criterion is peak-only because repeated sparse UP inputs may leave the character airborne near the clip end.
  • B.1 Player action following: Attack Accuracy uses ClipAttackNet, a 6-way classifier trained on ∼5k labeled clips, with clip-level top-1 accuracy over attack keys.Attack-active frames require max pk > 0.7; clips with no such frame are classified as “noop”.
  • B.2 NPC strategy following: NPC strategy following evaluates a frozen 99-clip subset spanning Control, Defense, and Offense, with each sample represented as a 101-frame generated video.Predictions come from Gemini and Qwen3-VL-8B, and Categorical Accuracy is 3-way top-1 agreement with ground-truth categories.
  • B.2 NPC strategy following: The referee decision rules prioritize Control for Sonic Boom or distance-based zoning, classify sustained approach or close-range attacks as Offense, and otherwise assign Defense.The prompts use strict JSON-only output with NPC visibility, side, category, reason, and scene description fields.
  • B.3 Visual quality: Visual quality measures long-horizon structural and perceptual fidelity on the same 99 clips, comparing 100 aligned frames after matched center-crop and bicubic-resize preprocessing.Metrics are SSIM, where higher is better, and full-frame LPIPS with an AlexNet backbone, where lower is better.

C Visualization

Figure 9 visualizes LingBot-World and Matrix-Game-3.0 in the SF2 scenario, where neither model is well-suited because they were not designed for this game. They are therefore treated as reference baselines, while analysis focuses on the vanilla model versus ReactiveGWM.

  • C Visualization: In SF2, LingBot-World and Matrix-Game-3.0 are included only as reference baselines because neither model is designed for this game.The analysis instead focuses on the vanilla model and ReactiveGWM.

D User study · D.1 Player action following

A human study with 19 participants evaluated player-action and NPC-strategy following in Street Fighter II and III. For player-action following, all three generators achieved high, statistically similar Likert scores across both games.

  • D User study: The user study covered Street Fighter II and Street Fighter III and evaluated Player Action Following alongside NPC Strategy Following.Nineteen participants familiar with 2D fighting games completed the questionnaire for both games.
  • D User study: The study used mean Likert scores together with the standard error of the mean to report participant evaluations.Each participant completed the full questionnaire for both games.
  • D.1 Player action following: Participants rated generated clips using a 1–5 Likert scale based on alignment between the key-input overlay and the on-screen player character.A score of 1 indicated poor alignment, while 5 indicated full alignment.
  • D.1 Player action following: All three generators obtained player-action-following scores between 4.32 and 4.60 across Street Fighter II and Street Fighter III.The reported values are mean Likert scores with standard error of the mean.
  • D.1 Player action following: The difference between any two models remained within one standard error of the mean in all conditions.This indicates no reported separation beyond one SEM among the generators for player-action following.
  • D.1 Player action following: The results indicate reliable player-action control across all evaluated models.The paper presents this finding as further support for the model architecture described in Section 3.3.

D.2 NPC strategy following

The study measures whether prompted strategies steer NPC behavior through human classification of generated clips. ReactiveGWM substantially improves strategy adherence across both Street Fighter games, though transferred Control behavior remains weak on SF3.

  • Evaluation: Participants classified each NPC clip as following Control, Defense, or Offense, making accuracy a direct measure of prompted strategy adherence.Higher classification accuracy indicates that the strategy condition successfully controls the NPC’s behavior.
  • Overall results: 86.0% overall accuracy for ReactiveGWMbase and 84.2% for ReactiveGWMtransfer on SF2 contrasted with 43.9% for Vanilla.ReactiveGWM’s SF2 accuracies were roughly double the unconditioned Vanilla baseline.
  • Overall results: 77.2% accuracy for ReactiveGWMbase and 61.4% for ReactiveGWMtransfer on SF3 contrasted with 17.5% for Vanilla.The corresponding gains over Vanilla were 59.7 and 43.9 percentage points.
  • Conclusion: Overall, the strategy prompt provides an effective and human-perceivable handle on the generated NPC.This conclusion follows from the consistent gains in human strategy-classification accuracy.
  • Per-class results: ReactiveGWMtransfer reached 100% on Offense but only 16% on Control in SF3, identifying Control or zoning behavior as its hardest cross-game transfer axis.The passage describes this contrast as the remaining weak spot of ReactiveGWMtransfer.

E Limitations and future work

ReactiveGWM demonstrates robust strategy following and zero-shot transferability, but its evaluation is limited to 2D fighting games. Future work should test the learned game-agnostic representations across broader game categories, including 2D FPS and multi-agent strategy games.

  • Evaluation scope: ReactiveGWM’s evaluation is limited to 2D fighting games, which provide a testbed for fine-grained action control and high-level tactics.The paper identifies this domain restriction as a main limitation.
  • Future work: Extending ReactiveGWM to 2D FPS and multi-agent strategy games is needed to assess the generality of its learned game-agnostic representations.These categories are proposed as future evaluation domains.
Loading 2605.15256v1…