Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach

1 Introduction

• A benchmark for assessing LLMs’ real-time strategy decision-making and long-term planning in environments like StarCraft II is absent.The authors select StarCraft II to evaluate these capabilities.Source: §1 · p. 2
• TextStarCraft II translates StarCraft II gameplay into an interactive text format, while CoS uses single- and multi-frame summarization to support strategic decisions.The environment uses python-sc2, and macro actions are issued through language commands.Source: §1 · p. 2
• The study evaluates CoS, proprietary and fine-tuned open-source LLMs, prompts, strategic preferences, interpretability, expert-reviewed game knowledge, and human-AI interaction.The experimental framework also tests gameplay performance and real-time interaction.Source: §1 · p. 2
• TextStarCraft II supports both strategic-game evaluation and real-time human-computer interaction.The authors present it as a new environment for research in strategic gaming contexts.Source: §1 · p. 2
• The authors introduce CoS and release an open-source agent as a high-level interface for community development and interaction with TextStarCraft II.Source: §1 · p. 2

StarCraft II Full Game AI

• AlphaStar reached Grandmaster level and defeated top players, while later StarCraft II work explored reinforcement learning, offline learning, and federated learning approaches.Examples include AlphaStar Unplugged, TStarBotsX [7], and SCC.Source: §2 · p. 2

LLM Agent and Benchmark

• LLM-agent work spans open-world environments and text-based benchmarks, including Voyager [25], TextWorld, and ALFWorld [20].Related work also describes multi-agent systems and benchmarking platforms such as AGENTBENCH.Source: §2 · p. 3
• Despite advances in AI agents, evaluating LLMs for real-time strategy and long-term planning in StarCraft II remains a gap.TextStarCraft II was created to provide an environment with natural-language interaction for evaluating these capabilities.Source: §2 · p. 3

Text Interface

• TextStarCraft II uses python-sc2 to translate gameplay into text through Observation-to-Text and Text-to-Action adapters.Its dynamic gameplay requires real-time resource allocation, base building, and military strategy.Source: §3 · p. 3

Observation

• The observation space covers resources, player units and buildings, ongoing construction and production, visible enemy units and buildings, and research progress.These categories provide game-state information for strategic decisions.Source: §3 · p. 3

Action

• Macro actions handle broad decisions such as training units, building structures, and researching technologies, while scripts manage precise micro-actions.Micro-actions include placements and targeting and are not directly controlled by the agent.Source: §3 · p. 3

Reward and Game Modes

• The reward function assigns −1, 0, or 1 for losing, drawing, or winning, and the environment offers built-in AI, agent AI, and human modes.Built-in AI mode includes 10 difficulty levels and 6 strategic styles.Source: §3 · pp. 3–4

Interaction Cycle

• The gameplay cycle converts initial game data to text, summarizes observations, schedules actions, and executes them in the game.New game data is then converted to text as the interaction continues.Source: Figure 1 · p. 4

4 Chain of Summarization

• CoS is presented as a plug-in inspired by hardware caching and reinforcement-learning frame skipping, with single-frame and multi-frame summarization plus action extraction.It can also operate as a direct interface for users interacting with TextStarCraft II.Source: §4 · p. 4
• CoS is described as compressing information, accelerating inference, and supporting a broader understanding of game state for decisions.Source: §4 · p. 4

Single-Frame Summarization

• Single-frame summarization compresses dense observations into a condensed form, and the experiments primarily use a faster rule-based approach.A language-model-based few-shot approach is also described for better alignment with game rules.Source: §4 · pp. 4–5

Multi-Frame Summarization

• Multi-frame summarization aggregates condensed observations across K steps into a period summary instead of querying the LLM at every time step.The method is motivated by the computation costs and slower inference associated with repeated queries during long games.Source: §4 · p. 5
• CoS reasons over the summaries through situation overview, analysis, planning, opponent-strategy analysis, suggestions, and decision-making.The method describes this reasoning sequence as supporting comprehensive strategic planning.Source: §4 · p. 5

Action Extraction for Action Queue

• The action extractor uses regular-expression matching and similarity search to turn reasoning output into K actions ready for execution.The resulting actions populate the action queue that connects CoS to the TextStarCraft II environment.Source: §4 · p. 5

Interaction Cycle

• The interaction cycle initializes observations, summarizes queued frames, reasons over the period summary, extracts actions, and executes them through the environment.The algorithm adds K actions to the action queue before stepping the environment.Source: §4 · pp. 5–6

5.1 Performance Evaluation of Various LLMs

• All evaluated closed-source LLMs and fine-tuned open-source models defeated the level-5 built-in AI.Un-fine-tuned Llama2 70B could not understand the task requirements or generate commands from the prompts.Source: §5.1 · p. 6
• 54/100 was the win rate for fine-tuning on wins from the top 25% APU games, versus 28/100 with the full dataset.The study used Average Population Utilization to partition the training data; Table 2 reports win rates across dataset subsets.Source: §5.1 · p. 6, Table 2 · p. 7

5.2 Assessing LLMs’ Mastery of StarCraft II Concepts

• LLMs showed basic StarCraft II knowledge but struggled with tech trees and supply constraints; GPT-4 and ChatGPT performed best on complex questions.The knowledge evaluation covered five areas and used Grandmaster-level experts and GPT-4 as double-blind assessors; Figure 2 presents double-blind assessment results.Source: §5.2 · p. 7, Figure 2 · p. 7

5.3 Human-AI Interaction

• The fine-tuned Qwen1.8B agent achieved performance on par with a Gold-level player in real-time matches against human players.The tests ran on home PCs requiring 4GB of GPU memory; Table 3 reports Qwen1.8B versus human match results.Source: §5.3 · p. 8, Table 3 · p. 8

6.1 Impact of Different Prompts

• More complex prompts improved LLM-agent performance against Zerg when playing Protoss, as reported in Table 4.The table reports win rates against TextStarCraft II AI at varied difficulty levels; the supplied passages do not provide the values.Source: §6.1 · p. 8, Table 4 · p. 8
• The complex prompt supported research upgrades, tech-tree exploration, and complex military maneuvers, and was particularly effective against Harder (Lv 5).It guided the agent through situation overview, analysis, strategic planning, and decision-making.Source: §6.1 · p. 8

6.2 Policy Interpretability

• AlphaStar overlooks an Oracle threat, while the LLM agent infers potential Roach aggression from enemy structures and recommends Photon Cannons and Shield Batteries.The LLM agent also recommends scouting to gather more information.Source: §6.2 · pp. 8–9, Figure 3 · p. 9
• Against a Hydra-Roach ground army, the LLM agent recommends expanding its composition and technology, including researching Psionic Storm; the authors contrast this adaptability with AlphaStar’s fixed compositions.The authors attribute the LLM agent’s threat anticipation and unit-composition adaptability to its structured, transparent decision-making process, which they say supports interpretability and strategic adaptation.Source: §6.2 · pp. 9–10

7 Discussion

• TextStarCraft II is limited to non-visual data, a subset of game races, and rule-based scripts for micro policies.The authors say these constraints may restrict strategy diversity and applicability.Source: §7 · p. 10
• Resource limitations have constrained the system’s performance capabilities.The authors identify resource optimization as a target for future improvement.Source: §7 · p. 10
• The authors plan to integrate visual inputs, expand race support, and optimize resource use to enhance strategic complexity and performance against established AI models.Source: §7 · p. 10

• The work introduces a method for using LLMs on real-time strategy tasks and strategic missions beyond traditional application domains.Source: Impact Statements · p. 13
• The approach aims to make LLM decisions more interpretable by offering insight into the language policies governing behavior in complex scenarios.Source: Impact Statements · p. 13
• The authors describe potential implications for sectors that require nuanced decision-making and strategic planning.Source: Impact Statements · p. 13

• Language-driven strategies may improve understanding and transparency in AI decision-making for settings requiring complex strategic thinking.Source: Broader Impact · p. 13
• Misleading AI-generated strategies or recommendations could contribute to suboptimal or harmful decisions when users rely on them without sufficient oversight or understanding.Source: Broader Impact · p. 13
• The authors recommend rigorous validation, user education about system capabilities and limits, and fail-safes for potential inaccuracies or biases.Source: Broader Impact · p. 13

Code Of Ethics

• The authors report ethical safeguards including controlled-environment use, which they say mitigates harm risk, and anonymity for training and testing data.They state that experiments, data handling, and dissemination adhere to the NeurIPS Code of Ethics.Source: Code Of Ethics · p. 13

A.1 Gameplay Overview

• StarCraft II requires managing resources, constructing bases, composing and controlling armies, and adapting strategy to opponent moves.Players choose among Terran, Protoss, and Zerg, each with distinct units and strategies.Source: §A.1 · p. 14

A.2 E-Sports and Competitive Play

• StarCraft II has been a major esports title since its 2010 launch, with tournaments including the WCS and IEM drawing competitors and viewers worldwide.The paper also describes a community of players, fans, and content creators, alongside extensive strategy discussions and guides.Source: §A.2 · p. 14
• StarCraft II’s professional scene has international participation, with South Korean players traditionally dominating global tournaments.Competitors from Europe, North America, and other parts of Asia have also made significant impacts.Source: §A.2 · p. 14

B.1 Experiment Setup

• The experiments pit LLM agents playing Protoss against Zerg AI across difficulty levels, using Altitude LE and Ancient Cistern LE maps and three game patches.The temperature is fixed at 0.1; the selected maps are from the 2023 1v1 ladder.Source: §B.1 · p. 15

B.2 Evaluation Metrics:

• Win rate is the share of games won; APU averages used population relative to its cap until maximum supply, with higher APU indicating more effective macro-management.The evaluation also includes the technology rate, which measures completed technologies and buildings as a proportion of those available; it is not necessarily a performance measure.Source: §B.2 · pp. 15–16
• PBR measures time spent at population capacity until 200/200 supply, while RUR assesses resources used relative to game duration until maximum supply.Higher PBR indicates poorer macro-strategic decision-making; higher RUR signifies resource underutilization and weaker macro-strategic capabilities.Source: §B.2 · p. 15

B.3 Compute Resource

• Fine-tuning used two NVIDIA A100 40GB GPUs for approximately 70 hours; testing used those GPUs alongside an RTX 3060 and an i5-13400F for game simulations.The listed CPU operates at 2.50 GHz.Source: §B.3 · p. 16

B.4 Human Experts

• The study recruited 30 StarCraft II masters and grandmasters, including professional and semi-professional players, and paid each participant 15 USD.The paper does not disclose participants’ personal information.Source: §B.4 · p. 16

C Experiment Figure

• A unit-preference figure compares the LLM agent’s 30 games with professional tournament players from 2015, 2020, and 2023.The passage identifies the comparison but does not report its results.Source: Figure 5 · p. 17
• The figures show early and mid-game defense and scouting, later mixed-unit assembly and troop transitions, and a defense that recovers from losses by expanding and switching to air units.In the Zerg assault example, the Protoss loses its second base, then uses Void Rays and Phoenixes to repel a subsequent attack.Source: Figure 4 · p. 17, Figure 6 · p. 18, Figure 7 · p. 19

D Prompt and data Example

• The Chain of Summarization examples pair raw text observations and L1 summaries with multi-frame prompts that review game state, strategy, and key information across rounds.The prompts request race-specific analysis and strategic recommendations concerning strategy, units, economy, and technology.Source: D Prompt and data Example · pp. 20, 22, Figure 8 · p. 20, Figure 9 · p. 21
• The LLM’s multi-stage analysis produces strategic recommendations and decisions, from which an Action-Extractor can retrieve executable actions such as training units and building structures.The example decisions include training Stalkers and Immortals, building a Gateway and Shield Battery, and expanding the base.Source: Figure 10 · p. 21, Figure 11 · p. 22, Figure 12 · p. 23

E Alphastar and llm agent

• Against varied enemy tactics, the LLM agent adapts unit selection, while AlphaStar often adheres to fixed compositions; one example responds to Zerg Hydra-Roach forces with more Immortals.The example describes producing Immortals as effective against Ultralisks.Source: E Alphastar and llm agent · p. 24, Figure 13 · p. 24

F Policy interpretability examples

• The interpretability examples present an L1 input, the LLM’s analysis, suggestions, and decisions, and a figure showing their implementation in the game engine.The section introduces four illustrative examples of strategies and decisions in TextStarCraft II.Source: F Policy interpretability examples · p. 25

F.1 Example 1

• At 00:46, the example describes an early-game Protoss with 15 workers, 105 minerals, one base, and no known enemy units, prompting recommendations to expand, build infrastructure, and scout.The listed decisions include building a Nexus, Pylon, and Gateway, training a Probe, and sending a scouting Probe.Source: §F.1 · pp. 25–26, Figure 15 · p. 26

F.2 Example 2

• At 03:44, the agent identified an early-game shortage of workers, units, and resources, then recommended continuing Warpgate research while producing Probes and Zealots.It also suggested expanding Pylon supply, Chrono Boosting the Cybernetics Core, and preparing for possible Zergling attacks.Source: §F.2 · pp. 27–28

F.3 Example 3

• At 04:25, the early-game position was challenging: two bases, 215 minerals, 128 gas, supply cap 54 with 16 supply left, army supply 8, and four Zealots; resource collection needed urgent expansion.Source: §F.3 · p. 29

F.4 Example 4

• At 13:28, the agent described a six-base mid-to-late-game position with 54 workers and 87 army supply, recommending more anti-air units and upgrades against an enemy composition including Broodlords.Its proposed actions included training Phoenixes, Void Rays, Stalkers, and Colossi, and building a Stargate.Source: §F.4 · pp. 31–32

G Questions and LLM Responses

• The paper documents responses from GPT3.5, GPT4, Claude2, and Bard across five distinct question categories, including StarCraft2 familiarity, racial mechanics, matchup strategies, and strategy or counterplay questions.Source: G Questions and LLM Responses · p. 33, §1 · p. 33, §3 · p. 33

1. Q1:Basic Knowledge of StarCraft2

• The prompts ask whether models know StarCraft II and request an overview of its races and esports scene; GPT-3.5, GPT-4, and Bard describe a three-race RTS with competitive play.Source: §1 · p. 33, Q1: Are you familiar with StarCraft II? · pp. 33–34

3. Q3:Typical Strategies in StarCraft2 for Different matchups?

• The matchup-strategy questions span Terran, Zerg, and Protoss pairings, and also ask for standard build orders for each race.The listed pairings include TVZ, TVT, TVP, ZVZ, ZVP, ZVT, PVZ, PVP, and PVT.Source: §3 · p. 33

5. Q5: Classic Strategies and Counterplays in StarCraft2

• The classic-strategy questions ask about counters to a Roach-Hydra attack, a Mutalisk-Zergling-Baneling composition, and an Archon-Zealot push.Source: §5 · p. 33

Q1: Are you familiar with StarCraft II?

• The familiarity question elicits affirmative responses describing StarCraft II as a Blizzard-developed RTS with Terran, Zerg, and Protoss factions and a competitive esports scene.The answers also mention its campaigns and multiplayer modes.Source: Q1: Are you familiar with StarCraft II? · pp. 33–34
• The overview describes three asymmetrically balanced races, with games involving resource gathering, base expansion, and army command.Terrans, Zerg, and Protoss have distinct units, strengths, and weaknesses that call for different strategies.Source: Q1: Are you familiar with StarCraft II? · p. 36

G.2 Racial Mechanics in StarCraft II

• The racial-mechanics overview highlights Zerg larva-based production, Protoss warp-in and Chrono Boost, and Terran building mobility, MULEs, and siege warfare.These features are presented as requiring distinct strategies for each race.Source: §G.2 · p. 39
• The racial-mechanics material describes Protoss shields and Psionic Storm, alongside Terran MULEs, Supply Drops, Siege Tanks, and Medivacs.It characterizes shields as regenerating protection and Storm as area damage; Terran mechanics support faster mineral gathering, additional supply, ranged siege attacks, transport, and healing.Source: §G.2 · p. 41

Q1: TvZ: What’s a common strategy for Terran vs. Zerg?

• Terran-versus-Zerg strategies combine bio or mech armies with harassment to disrupt Zerg economy and expansion, while building toward a stronger army or decisive push.Suggestions include Medivac drops, Hellion attacks on workers, Widow Mines, and Reaper-expand openings into Hellions or Marine/Medivac.Source: Q1: TvZ: What’s a common strategy for… · pp. 41–42

Q2: TvT: What’s a typical strategy in Terran vs. Terran matchups?

• Terran-versus-Terran play centers on Siege Tank positioning and Viking air control, with scouting, adaptation, and bio or mech choices shaping engagements.The answers also mention Reaper-expand openings, securing bases, and countering a detected bio build with Tanks and Hellions.Source: Q2: TvT: What’s a typical strategy in… · p. 42

Q3: TvP: How do Terran players generally approach a Protoss opponent?

• Against Protoss, Terran answers recommend mobile bio and harassment or early pressure, while using Ghosts and other high-tech units to counter Protoss threats.The suggestions include Medivac drops, Ghost EMP or snipes, Liberator zones, and expanding while slowing Protoss economy and technology.Source: Q3: TvP: How do Terran players… · p. 42

Q4–Q6: Zerg matchup strategies

• Zerg matchup answers emphasize early defense and map control, then use matchup-specific tech and composition switches against Zerg, Protoss, or Terran armies.Examples include Zergling-Baneling skirmishes and Roach or Mutalisk transitions in ZvZ; creep spread and Hydralisk-Lurker or tech switches in ZvP; and Banelings, Infestors, Brood Lords, or Ultralisks in ZvT.Source: Q4: ZvZ: What’s a standard strategy in… · p. 43, Q5: ZvP: What are Zerg strategies when… · p. 43, Q6: ZvT: How do Zerg players counter… · pp. 43–44

Q7–Q9: Protoss matchup strategies

• Protoss matchup answers pair economic expansion and early defense with scouting and tech choices, using powerful area-damage or specialized units against Zerg, Protoss, and Terran.Examples include High Templars and Colossi against Zerg, Blink Stalkers and Disruptors in PvP, and Colossi or Storm against Terran bio.Source: Q7: PvZ: How should Protoss approach a… · p. 44, Q8: PvP: What’s the strategy for… · p. 44, Q9: PvT: How do Protoss players… · p. 45

Q:Can you provide a standard build order for each race?

• The Terran, Protoss, and Zerg build-order answers outline race-specific openings, early production and expansion, and later technology or upgrade choices.Terran’s sequence includes a Reaper, Factory, and Medivac; Protoss’s includes a Gateway, natural expansion, and Warp Gate research; Zerg’s includes a natural Hatchery, Spawning Pool, Queen, and Zerglings.Source: Q:Can you provide a standard build… · pp. 45–46
• The build-order responses outline Terran Reaper-and-expansion, Zerg Hatchery-and-Queen, and Protoss Gateway-expand openings, while emphasizing scouting and adaptation.Examples include a Terran second Command Center and Barracks, Zerg Queens and Zergling speed, and a Protoss Nexus followed by a Cybernetics Core and Adept or Stalker.Source: Q:Can you provide a standard build… · pp. 47–48

Q1: PvZ: What are effective strategies and counterplays against a roach-hydra attack?

• Zerg counterplay to Roach-Hydra includes Hive tech for Vipers, Hydralisk speed, Burrow, harassment drops, and creep spread for mobility and vision.The suggested Viper abilities are Abduct and Blinding Cloud.Source: Q1: PvZ: What are effective strategies… · p. 51
• Against Roach-Hydra, Protoss responses recommend Immortals, Archons, Colossi, Sentries, and High Templars, alongside favorable positioning and harassment.Sentries can split the Zerg force with Force Fields, while High Templars use Psionic Storm against biological units.Source: Q1: PvZ: What are effective strategies… · p. 51

Q2: TvZ: How should Terran approach a mutalisk-zergling-baneling composition? Are there gambits or counter-strategies to consider?

• Against Mutalisk-Zergling-Baneling, Terran advice combines Marines, Medivacs, and Widow Mines with Thors or Liberators, base defenses, and map vision.The responses also recommend splitting Marines against Banelings and using Missile Turrets and Bunkers to defend against harassment.Source: Q2: TvZ: How should Terran approach a… · p. 52
• Against Archon–Zealot forces, the responses recommend EMP, bio kiting, Liberator and Siege Tank zones, and building layouts that funnel melee units.The advice concerns a Protoss melee composition, despite the subsection’s TvZ/mutalisk-zergling-baneling question.Source: Q2: TvZ: How should Terran approach a… · pp. 53–54
• Other suggestions include bio, Siege Tanks, Medivac healing and drops, defensive structures, scans, and ranged picks against Protoss units.The responses also mention mines, aggressive scanning, and adapting tactics to control, positioning, and changing circumstances.Source: Q2: TvZ: How should Terran approach a… · p. 54

H.1 Claims

• The claims summary says the abstract and introduction describe LLM integration into RTS gaming, especially StarCraft II, interpretability, and applicability to other complex decision-making environments.It characterizes those sections as accurately reflecting the research’s scope and contributions.Source: §H.1 · p. 55

H.2 Limitations

• TextStarCraft II relies on non-visual inputs and scripted micro policies, with performance also constrained by resource limitations.These are identified as limitations of the environment.Source: §H.2 · p. 55

H.3 Theory Assumptions and Proofs

• The paper focuses on experimental results and method development rather than theoretical proofs.Source: §H.3 · p. 55

H.4 Experimental Result Reproducibility; H.5 Open access to data and code; H.6 Experimental Setting/Details

• The authors describe the environment and experimental configurations, and provide code, data, and a demo to support replication.The setup includes agent and opponent selection, map choice, and parameter settings.Source: §H.4 · p. 55, §H.5 · p. 55, §H.6 · p. 55

H.7 Experiment Statistical Significance

• Reported error bars are standard deviations across experiment runs and capture variability due to initialization.Source: §H.7 · p. 55

H.8 Experiments Compute Resources

• The paper points to Appendix B.3 for details of its experimental compute resources.Source: §H.8 · p. 56

H.9 Code Of Ethics; H.10 Broader Impacts; H.11 Safeguards

• The authors state that the work follows the NeurIPS Code of Ethics, discusses positive and negative societal impacts, and identifies no high-misuse-risk assets requiring special safeguards.The safeguards statement concerns strategic AI development in the controlled StarCraft II environment.Source: §H.9 · p. 56, §H.10 · p. 56, §H.11 · p. 56

H.12 Licenses for existing assets; H.13 New Assets

• The authors say existing assets are credited with license information, while TextStarCraft II is documented with setup, usage, and licensing details.Source: §H.12 · p. 56, §H.13 · p. 56

H.14 Crowdsourcing and Research with Human Subjects; H.15 Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

• The study involved 30 StarCraft II masters and grandmasters, and describes participation as voluntary, game-related QA and gameplay testing.The authors state that these activities generally do not require IRB approval because they involve no risks beyond normal gameplay.Source: §H.14 · p. 56, §H.15 · p. 56