Source-linked AI summary
WorldMind: Decoupled Game World Model for State-Aware NPC Behavior
Zhiyang Deng, Boran Zhang, Danze Chen, Yeying Jin
TL;DR
Existing game world models lack an explicit interface for grounding NPC responses in evolving game state. WorldMind separates state reconstruction, NPC planning, action control, and video generation in a closed loop, supported by BOSS-140K. Experiments show coherent state-responsive behavior, with WorldMind preferred in approximately 70% of pairwise comparisons.
Problem
Existing models model NPC behavior implicitly through video generation or explicitly through external control, leaving no explicit interface for state-grounded NPC decisions.
Method
WorldMind uses four decoupled layers to reconstruct compact state, plan NPC actions from state and mechanics, translate actions into conditions, and synthesize visual outcomes in a closed loop.
Results
WorldMind is preferred over both baselines in approximately 70% of pairwise comparisons and ranks first in action validity and sequence fit under both judges.
Takeaways & Limitations
WorldMind makes NPC behavior an explicit component of interactive world modeling that responds coherently to evolving game state.
Takeaways & Limitations
The language model is not fine-tuned on BOSS-140K, avoiding direct imitation of the dataset’s action distribution.
Abstract
from arXiv · showhide
Game world models have recently demonstrated promising capabilities in generating visually coherent and action-controllable gameplay videos. However, non-player character (NPC) behavior in existing models is either implicitly entangled with video generation or explicitly prescribed through external control signals. Consequently, a game world model has to jointly understand the state, plan the NPC's response and render its visual outcome, limiting its ability to produce responsive and state-aware NPC behavior. The challenge lies in the lack of an explicit interface for state-grounded decision-making. To this end, we introduce WorldMind, to our knowledge the first decoupled framework for state-aware NPC behavior in game world models. WorldMind separates interactive world modeling into four layers: an Understanding Layer that constructs a compact state from generated frames; a Decision Layer that reasons over the compact state to plan the NPC's next action; a Control Layer that translates the actions into temporally aligned conditions; and a Generation Layer that synthesizes their visual outcomes. By reconnecting layers in a closed interaction loop, WorldMind grounds NPC behavior in the evolving game state. We further introduce BOSS-140K, a dataset of gameplay videos paired with rich internal game states, together with an agent that automates the collection at scale. Experiments on BOSS-140K demonstrate reliable compact state reconstruction and mechanics-grounded planning, with WorldMind preferred over the baselines in approximately 70% of pairwise comparisons for its more tactically appropriate and coherent NPC behavior. Project page: https://teawhite.cn/worldmind_projectpage/
1 Introduction
Existing game world models either leave NPC behavior implicit in video generation or prescribe it externally, without an explicit interface grounding responses in evolving game state. WorldMind decouples state understanding, NPC decision-making, action conditioning, and visual synthesis, while BOSS-140K supplies rich aligned supervision for this setting.
- NPCs shape gameplay dynamics by reacting to the current game state and player behavior, but remain underexplored in interactive game world modeling.
- Existing models either learn NPC behavior implicitly through generated visual dynamics or require an externally specified NPC action.
- WorldMind addresses this gap with four decoupled layers for state understanding, NPC decision-making, action conditioning, and visual synthesis.
- BOSS-140K pairs gameplay video with frame-aligned player controls, NPC skills, and rich internal state annotations, collected through an automated state-conditioned gameplay agent.
- WorldMind enables state-aware NPC decisions as the game state evolves instead of behavior emerging from video generation or being externally specified.
2 Related Work
Prior game world-model datasets and NPC-control approaches provide limited supervision for state-dependent NPC decisions. BOSS-140K extends this landscape with aligned gameplay, internal state, skills, and captions collected by an adaptive agent.
- Interactive World Models: Game world models generate controllable gameplay rollouts from action inputs and increasingly support longer-horizon, real-time interaction.
- NPC Behavior Modeling: Conventional NPC systems map structured engine state to scripted actions, while LLM agents provide more open-ended reasoning and planning.
- Datasets for Game World Models: Existing game-dynamics datasets commonly pair gameplay video with player controls, demonstrations, or action annotations but provide limited supervision for state-dependent NPC decisions.
- Datasets for Game World Models: BOSS-140K pairs gameplay video with frame-aligned player controls, boss skills, animation states, rich engine variables, and natural-language captions.
- Datasets for Game World Models: Its state-conditioned collection agent adapts strategies to the evolving game state to elicit diverse boss responses.
3 Method Decoupled GWM Architecture
WorldMind decomposes interactive world modeling into four layers that expose compact state and NPC actions while preserving a closed interaction loop. The system plans from state and skill mechanics, temporally aligns actions, generates video, and replans from resulting frames.
- 3 Method Decoupled GWM Architecture: WorldMind addresses the absence of an explicit NPC-response interface by asking what is happening, what the NPC should do, and how the components can form a coherent closed loop.
- L1: Understanding Layer: L1 constructs a compact state from geometry variables predicted from generated frames and skill-history variables derived from boss action history.
- L2: Decision Layer: L2 uses a language model to reason jointly over the compact state and skill mechanics, forming a short-horizon NPC action plan.
- L2: Decision Layer: L2 does not fine-tune the language model on BOSS-140K, avoiding direct imitation of the dataset’s action distribution.
- L3: Control Layer: L3 converts complete player and boss actions into temporally aligned compositional prompts, supporting both Direct and Director Control before L4 synthesizes gameplay video.
- Closed Interaction Loop: After L4 generates a segment, its frames and executed boss action update L1 and the skill history, enabling L2 to replan from the revised interaction context.
- BOSS-140K Dataset: BOSS-140K contains 144,631 clips totaling over 200 hours from 14 bosses, pairing video with aligned controls, boss actions, engine states, and captions.
4 Experiments
Experiments evaluate WorldMind’s compact-state reconstruction, mechanics-grounded planning, closed-loop NPC behavior, generalization, and key design choices. Across these tests, the system shows reliable state reconstruction, mechanics-sensitive skill selection, and stronger closed-loop behavior than the baselines.
- Experimental Design: The experiments assess compact-state reconstruction, mechanics-grounded planning, closed-loop NPC behavior, cross-game generalization, and ablations of visual encoder and input modality.The full-system evaluation focuses on Game A, while L1 and L2 are evaluated across all three BOSS-140K games.
- L1: Compact-State Reconstruction (Q1): WorldMind’s compact state is evaluated for boss–player geometry reconstruction and retention of information relevant to subsequent decisions.Distance and angle errors, normalized distance, and distance- and angle-bin accuracy measure reconstruction properties.
- L2: Reasoning over the Compact State (Q2): Removing or swapping skill descriptions substantially changes the selected skill, with swapped mechanics followed more often than the original anonymized identifier.These interventions test whether L2 responds to skill mechanics rather than fixed skill names.
- L2: Reasoning over the Compact State (Q2): L2’s planned skills are rated more appropriate than random legal choices from the same cooldown-ready menu, indicating that the compact state informs tactical selection beyond legality.The evaluation separates mechanics grounding from tactical appropriateness.
- Full Closed-Loop System (Q3): Approximately 70% of pairwise comparisons prefer WorldMind over both baselines, and it ranks first in action validity and sequence fit under both judges.The closed-loop evaluation uses one-minute Game A rollouts with 12 decision points, evaluated by two judges across the reported metrics.
- Cross-Game Generalization (Q4): The Decision Layer shows partial cross-game generalization on WildWorld, while compact-state reconstruction requires target-domain adaptation.This result defines distinct generalization behavior for decision-making and state reconstruction.
- Ablation Studies: The task-trained L1 encoder outperforms frozen DINOv2-S and VideoMAE-B on every reconstruction metric, while adding rendered frames to compact-state text yields no consistent improvement.The ablations compare matched visual encoders and text-only versus frame-augmented inputs.
5 Conclusion
WorldMind makes NPC behavior an explicit component of interactive world modeling by separating state reconstruction, reasoning, control, and video generation, then reconnecting them in a closed loop.
- WorldMind separates compact-state reconstruction, NPC reasoning and planning, action control, and video generation within interactive world modeling.
- The closed loop uses each generated outcome to inform the NPC’s next action.
- Experiments show NPC behavior responds coherently to the evolving game state rather than remaining implicit in video generation or being prescribed externally.