PaperlayerView exact revision on arXiv

Source-linked AI summary

World Models for Policy Refinement in StarCraft II

Yixin Zhang, Ziyi Wang, Yiming Rong, Haoxi Wang, Jinling Jiang, Shuang Xu, Haoran Wu, Shiyu Zhou, Bo Xu

arXiv:2602.14857v1cs.AI

TL;DR

Existing LLM-based StarCraft II agents largely improve policies without integrating learnable action-conditioned dynamics models, limiting exploration of foresight under partial observability. StarWM adds such a world model and a Generate–Simulate–Refine agent, achieving consistent online gains against built-in AI across LV5–LV7.

  • Problem

    Existing LLM-based StarCraft II methods primarily improve policies without systematically integrating a learnable dynamics model into the decision loop.

  • Method

    StarWM predicts short-horizon observations from observations and candidate actions, using structured semantic representations and a Generate–Simulate–Refine policy-refinement loop.

  • Results

    StarWM shows nearly 60% gains in resource prediction accuracy and self-side macro-situation consistency, while StarWM-Agent yields consistent online gains against built-in AI across LV5–LV7.

  • Takeaways & Limitations

    The results demonstrate the value of action-conditioned world-model predictions for foresight-driven policy refinement in partially observable StarCraft II.

  • Takeaways & Limitations

    Without temporal history or explicit opponent-intent modeling, opponent forecasting is under-determined and depends heavily on the training-data distribution.

Abstract

from arXiv · show

Large Language Models (LLMs) have recently shown strong reasoning and generalization capabilities, motivating their use as decision-making policies in complex environments. StarCraft II (SC2), with its massive state-action space and partial observability, is a challenging testbed. However, existing LLM-based SC2 agents primarily focus on improving the policy itself and overlook integrating a learnable, action-conditioned transition model into the decision loop. To bridge this gap, we propose StarWM, the first world model for SC2 that predicts future observations under partial observability. To facilitate learning SC2's hybrid dynamics, we introduce a structured textual representation that factorizes observations into five semantic modules, and construct SC2-Dynamics-50k, the first instruction-tuning dataset for SC2 dynamics prediction. We further develop a multi-dimensional offline evaluation framework for predicted structured observations. Offline results show StarWM's substantial gains over zero-shot baselines, including nearly 60% improvements in resource prediction accuracy and self-side macro-situation consistency. Finally, we propose StarWM-Agent, a world-model-augmented decision system that integrates StarWM into a Generate--Simulate--Refine decision loop for foresight-driven policy refinement. Online evaluation against SC2's built-in AI demonstrates consistent improvements, yielding win-rate gains of 30%, 15%, and 30% against Hard (LV5), Harder (LV6), and VeryHard (LV7), respectively, alongside improved macro-management stability and tactical risk assessment.

1 Introduction

StarWM addresses the challenges of learning and integrating action-conditioned world models in partially observable StarCraft II by combining structured dynamics prediction with inference-time policy refinement. The paper introduces a dataset, evaluation framework, and StarWM-Agent’s Generate–Simulate–Refine loop for world-model-augmented decision-making.

  • Motivation and Challenges: SC2 poses major challenges for LLM decision-making because of its enormous state-action space, imperfect information, and strongly coupled hybrid dynamics.These dynamics include resource flows, task progression, unit kinematics, and combat evolution under partial observability.
  • StarWM: StarWM is introduced as the first action-conditioned SC2 world model, predicting short-horizon future observations under partial observability.Its structured textual representation factorizes observations into five semantic modules to capture hybrid dynamics.
  • Dataset and Evaluation: SC2-Dynamics-50k and a multi-dimensional offline evaluation framework assess dynamics prediction across economy, development, micro-entities, and macro-situation.The dataset is presented as the first instruction-tuning dataset for SC2 dynamics prediction.
  • StarWM-Agent: StarWM-Agent integrates StarWM into a Generate–Simulate–Refine loop that uses action-conditioned predictions for inference-time policy refinement and consistent online improvements.The system is designed to integrate predicted futures into an LLM policy’s decision process.

2 Related Work

Prior world-model research supports policy learning through imagined trajectories or online planning, while SC2 research has largely emphasized end-to-end or LLM-based policy design. This work addresses the gap by introducing an LLM-based SC2 world model for short-horizon lookahead and policy refinement.

  • World Models: World models in decision making learn internal environment representations for imagination-based policy optimization or online planning.DreamerV3 uses latent dynamics for imagination-based policy optimization, while MuZero supports online Monte Carlo Tree Search.
  • SC2 World Models: Prior work had not studied world models for SC2, despite its partial observability and coupled hybrid dynamics.The paper presents the first LLM-based world model for SC2 and examines its decision-making effectiveness.
  • LLM-based SC2 Agents: SC2 agents have progressed from AlphaStar’s end-to-end neural approach to LLM-based agents that use external knowledge, hierarchical planning, collaboration, or strategic memory.AlphaStar requires massive human data and long-term league-based self-play, whereas recent LLM methods target generalized and interpretable agents with lower training-resource consumption.
  • Policy Refinement: Most existing LLM-based SC2 methods enhance the policy itself rather than integrating a learnable dynamics model into the decision loop.The proposed world model performs short-horizon lookahead over observations and candidate actions to enable policy refinement for more reliable decision making.

3 Method

This method models StarCraft II as a partially observable environment and learns an action-conditioned world model to predict future player-view observations. It factorizes observations into five semantic modules and integrates StarWM into a Generate–Simulate–Refine loop for foresight-driven policy refinement.

  • Problem Formulation: SC2 is modeled as a POMDP, distinguishing the inaccessible perfect-information Environment Simulator from the player-view World Model under Fog of War.The simulator transitions global states using joint actions, whereas the world model operates from a single player’s restricted perspective.
  • World-Model Prediction: Given the current observation and intended action sequence, StarWM predicts the future observation after a specified number of steps.The prediction must capture both intrinsic evolution, such as resource and task changes, and extrinsic interaction, such as combat outcomes.
  • Structured Representation: The structured textual representation factorizes observations into five semantic modules: Info, Queue, My Units, My Structures, and Visible Hostiles.These modules separately represent economy and status, ongoing tasks, self units, self structures, and partially observed hostiles.
  • Structured Representation: The modular structure invokes different sub-dynamics for different tasks, reducing learning burden and accelerating convergence.SC2 combines heterogeneous dynamics, including resource accumulation, temporal progress, spatial movement, and combat interactions.
  • StarWM-Agent: StarWM-Agent feeds predicted future observations back to the policy in a Generate–Simulate–Refine pipeline for foresight-driven action refinement.Forecasting supports macro-management through resource, supply, and task planning and micro-tactical assessment of combat and feasibility risks.

4 Experiment

Experiments show that StarWM learns action-conditioned SC2 dynamics and improves both prediction quality and decision-making. Integrating it into Generate–Simulate–Refine yields stronger macro-management, economic efficiency, tactical outcomes, and win rates, while opponent forecasting remains limited under partial observability.

  • Dynamics Prediction: 60% and 65% reductions in SMAPE errors for minerals and gas, respectively, improve over the zero-shot 32B baseline.StarWM records SMAPE errors of 0.19 / 0.09 versus 0.48 / 0.26 for the zero-shot 32B baseline; Queue F1 reaches 0.92 and Progress MAE drops to 0.43%.
  • Limitations: Opponent forecasting remains under-determined without temporal history or explicit opponent-intent modeling, making predictions dependent on the training-data distribution.Under Fog of War, a Static Bias strategy can achieve better AWD over short horizons because enemy displacement is often limited.
  • Dynamics Prediction: StarWM maintains extremely low self-side macro-situation prediction error early and remains robust during high-intensity mid-game battles.Predicted self-unit spatial distributions preserve army formation structure more closely than scattered zero-shot LLM predictions.
  • Online Evaluation: 30% / 15% / 30% win-rate gains for StarWM-Agent (32B) against LV5 / LV6 / LV7 show consistent improvements across difficulty levels.StarWM-Agent (8B) improves win rates by 10% / 5% / 20% against LV5 / LV6 / LV7, despite training only on LV6 and LV7 trajectories.
  • Online Evaluation: 53% and 15% reductions in Supply Block Rate accompany 60% and 31% increases in Valid Action Rate for StarWM-Agent (8B) and StarWM-Agent (32B), respectively.Predictive lookahead anticipates supply bottlenecks, while implicit action verification filters invalid commands through simulation discrepancies.
  • Ablation and Analysis: Win rate increases to 20% with trajectory-trained StarWM, exceeding Generate + Refine and zero-shot world-model simulation through more accurate action-conditioned simulation.Generate + Refine raises win rate from 0% to 5%, zero-shot simulation raises it to 10%, and StarWM simulation reaches 20%; action revisions occur at 32.74% for 8B and 19.45% for 32B.

5 Conclusion · A Introduction to StarCraft II

The paper presents StarWM as an action-conditioned world model for StarCraft II policy refinement under partial observability, supported by structured dynamics representations, SC2-Dynamics-50k, and multidimensional offline evaluation. It frames SC2 as a complex two-player, partially observable real-time strategy environment with asymmetric races and distinct strategic capabilities.

  • 5 Conclusion: StarWM is introduced as the first action-conditioned world model for StarCraft II.It is designed to support policy refinement under partial observability.
  • 5 Conclusion: The structured textual observation representation factorizes SC2 dynamics into semantic modules for learning in a hybrid, large-scale environment.The representation is intended to enable dynamics learning in SC2.
  • 5 Conclusion: SC2-Dynamics-50k is introduced as the first instruction-tuning dataset for SC2 dynamics prediction.The dataset supports the proposed dynamics-learning approach.
  • 5 Conclusion: The proposed offline evaluation framework assesses predicted dynamics across economy, development, micro-entities, and macro-situation.The framework is described as multi-dimensional and is used to assess StarWM’s ability to capture key dynamics.
  • A Introduction to StarCraft II: StarCraft II is a real-time strategy game widely regarded as a benchmark for complex sequential decision-making.The game combines macro-management and other forms of strategic complexity.
  • A Introduction to StarCraft II: The standard competitive 1v1 setting can be modeled as a two-player zero-sum partially observable stochastic game.This formulation captures the game’s sequential decision-making and partial observability.
  • A Introduction to StarCraft II: Each player chooses one of three asymmetric races: Terran, Protoss, or Zerg.Each race has race-specific units, technologies, and strategic styles.
  • A Introduction to StarCraft II: SC2’s complexity integrates macro-management with race-specific units, technologies, and strategic styles.These asymmetric capabilities contribute to the environment’s strategic complexity.

B Detailed Metrics for Online Testing

The section defines metrics for online testing that characterize decision quality from multiple aspects.

  • B Detailed Metrics for Online Testing: Online testing is evaluated using metrics designed to characterize decision quality.The passage frames these metrics as an evaluation method for online testing.
  • B Detailed Metrics for Online Testing: The metrics assess decision quality from multiple aspects rather than a single criterion.The evaluation is explicitly described as multidimensional.
  • B Detailed Metrics for Online Testing: The section introduces the metric set as a basis for analyzing decision quality during online testing.No individual metric definitions or quantitative results are provided in the passage.

B.1 Primary Metric

The primary metric is Win Rate, which measures the proportion of games won and reflects overall agent performance. It is computed as the percentage of games won among all games played.

  • B.1 Primary Metric: Win Rate measures the proportion of games won and reflects the agent’s overall performance.
  • B.1 Primary Metric: Win Rate is calculated as the number of games won divided by the total number of games played, multiplied by 100%.

B.2 Macro-Management Metrics

This section defines two macro-management metrics: Supply Block Rate captures time spent supply-blocked, while Resource Conversion Rate captures how effectively collected resources are spent.

  • Macro-Management Metrics: Supply Block Rate measures the fraction of in-game time an agent is supply-blocked; lower rates indicate better macro management.It reflects the ability to balance production and supply expansion.
  • Macro-Management Metrics: Resource Conversion Rate measures the proportion of collected resources that are effectively spent; higher rates indicate more efficient resource utilization.It reflects the ability to translate economic growth into actions.
  • Macro-Management Metrics: Resource Conversion Rate = Total Resources Spent / Total Resources Collected × 100%.This formula is given as Equation (8).

B.3 Combat Metric

The Kill–Loss Ratio measures combat efficiency through the economic value of army units. Higher ratios indicate more favorable resource exchanges in combat.

  • Combat Metric: Kill–Loss Ratio evaluates combat efficiency based on the economic value of army units.
  • Combat Metric: Higher scores indicate more favorable resource exchanges, comparing enemy units killed with own units lost by total resource value.

B.4 System Stability Metric

The system stability metric is Valid Action Rate, which measures the proportion of issued actions that are valid and executable under diverse situations.

  • Valid Action Rate measures the proportion of valid actions issued by the agent, reflecting action-generation robustness under diverse situations.
  • Valid Action Rate = Number of Valid Actions / Total Number of Issued Actions × 100%.
  • An action is invalid when it cannot be executed by the game engine at the corresponding timestep.

B.5 Mechanism Analysis Metric

This section introduces Action Revision Rate to quantify how often the agent changes its initial action proposal during decision refinement. The metric indicates the influence of world model predictions on final decisions.

  • B.5 Mechanism Analysis Metric: Action Revision Rate measures how often the agent modifies its initial action proposal during decision refinement.It is defined over the total number of decision steps and compares initial with final actions at each step.
  • B.5 Mechanism Analysis Metric: The metric reflects the extent to which world model predictions influence final decisions.Greater action revision indicates more changes between the initial and final actions during the refinement process.

C Detailed Quantitative Results of Offline Evaluation … E.2 Example of StarWM-Agent Online Decision Making

The appendix reports detailed offline evaluation results and documents how StarWM predicts structured future observations for both training and online simulation. It also illustrates the Generate–Simulate–Refine loop, its self-reflection ablation, and representative SC2-Dynamics-50k and online decision examples.

  • C Detailed Quantitative Results of Offline Evaluation: Table 4 reports StarWM’s offline quantitative evaluation across multiple dimensions, supporting the conclusions presented in Section 4.2.The supplied passage identifies the table as a detailed multi-dimensional offline evaluation but does not include its numerical entries.
  • D.1 World Model Prompt: The world-model prompt defines StarWM as a simulator that predicts a player’s future partial observation from the current observation, action sequence, and game mechanics.The prompt is used for offline training and inference and for online simulation in the Generate–Simulate–Refine loop.
  • World Model Prompt: Its structured observation schema separates player information, production queues, self units, self structures, and visible hostiles.The reasoning rules update time, resources, supply, queues, movement, status, combat, visibility, and unchanged entities.
  • D.2 Online Refinement Prompt with World Model Predictions: The online refinement prompt presents a predicted five-second game state and instructs the policy to compare it with the current state before producing optimized JSON actions.The policy generally keeps original commands, adds beneficial actions, and changes or removes commands only when predictions indicate harmful trades.
  • Online Refinement Prompt with World Model Predictions: Refinement checks predicted resources, supply, production queues, idle structures, unit health, combat activity, and vision to add, adjust, or avoid commands.The prompt also requires validating actor abilities and selecting a valid SCV for TERRANBUILD commands.
  • D.3 Online Refinement Prompt for Self-Reflection Ablation: The self-reflection ablation removes external future prediction and refines the initial action using only the policy’s internal reasoning over the current game state.Its guidance mirrors the prediction-based refinement process, including resource, queue, idle-structure, army, damage, scouting, and actor-validation checks.
  • E.1 Example from SC2-Dynamics-50k: The SC2-Dynamics-50k example pairs a structured partial observation with a five-second action sequence and a target future observation.The example applies explicit transition rules for resources, supply, movement, combat, visibility, and consistency, then outputs the predicted structured state.
  • E.2 Example of StarWM-Agent Online Decision Making: The StarWM-Agent example shows an initial action proposal, a world-model simulated future observation, and the policy’s refined action within the Generate–Simulate–Refine loop.This example demonstrates the online decision-making process in practice.

Example of StarWM-Agent Online Decision Making

StarWM-Agent uses a Generate–Simulate–Refine loop in which a proposed action is evaluated through a predicted five-second game state before producing optimized commands. The refinement compares predicted and current states, preserving commands by default while adding, changing, or removing actions under explicit tactical checks.

  • Simulation and refinement: The agent receives a predicted game state five seconds after its proposed action and uses it for tactical analysis before execution.The simulation report is explicitly hypothetical: the original action has not yet been executed.
  • Simulation and refinement: The refinement procedure keeps original commands by default, adds commands for idle units or extra resources, and changes or removes commands only after detecting bad trades.This policy is stated as the required comparison between the simulation report and the initial current game state.
  • Tactical checks: The agent uses predicted visibility, unit health, and unit survival to decide whether to attack or search, retreat or reposition, kite, or heal and repair.An empty visible-enemy-structures list triggers attack or move commands to search for enemy bases.
Loading 2602.14857v1…