PaperlayerView exact revision on arXiv

Source-linked AI summary

DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments

Wenjie Tang, Yuan Zhou, Erqiang Xu, Keyan Cheng, Minne Li, Liquan Xiao

arXiv:2503.06047v2cs.AIcs.CL

TL;DR

Complex strategic environments require long-horizon reasoning and multi-agent decision-making under uncertainty, while existing benchmarks can isolate skills, lack diversity, or rely on broad metrics. DSGBench evaluates agents across six strategic games with fine-grained capability scoring and decision tracking. The evaluation finds distinct strengths and limitations across tasks rather than consistent general competence.

  • Problem

    Existing benchmarks may isolate skills, lack environmental diversity, or rely on broad outcome metrics that poorly distinguish agent strengths and weaknesses.

    Source: §1 · p. 2

  • Method

    DSGBench combines six strategic games, customizable tasks, five capability dimensions, and decision tracking to evaluate LLM-based agents.

    Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 6

  • Results

    Evaluations show varying performance and systemic limitations across task types rather than consistent general competence.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Fine-grained evaluation and trajectory analysis reveal strengths and weaknesses that can help select agents for specific tasks or capabilities.

    Source: §5 · pp. 9–10

    Agents often struggle to infer hidden pieces and predict future moves from incomplete information, and can fail to revise beliefs when new information appears.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Large language model (LLM)-based agents are increasingly applied to complex strategic environments that demand long-horizon reasoning, multi-agent interaction, and decision-making under uncertainty. However, common existing benchmarks either assess isolated skills, lack environmental diversity, or rely on broad overall metrics. To address these issues, we introduce DSGBench, a more rigorous evaluation platform for strategic decision-making tasks. Firstly, it incorporates six complex strategic games which serve as ideal testbeds due to their long-term and multi-dimensional decision-making demands and flexibility in customizing tasks with various difficulty levels and targets. Secondly, DSGBench employs a fine-grained evaluation scoring system which examines the decision-making capabilities by looking into the performance in five specific dimensions, offering a comprehensive assessment in a better-designed fashion. Furthermore, DSGBench also incorporates an automated decision-tracking mechanism which enables in-depth analysis of agent behaviour patterns and the turning points in their strategies. We evaluate six popular LLM agents, including open-source and closed-source models, and observe distinct strengths and limitations among various tasks. Through decision trajectory analysis, we further identify systemic limitations in different LLMs. These findings offer valuable insights for model selection and future LLM-based agent development.

1 Introduction

Complex strategic settings require long-horizon planning under uncertainty, but existing benchmarks often isolate skills, use limited environments, or rely on broad outcome metrics. DSGBench combines six strategic games, five capability dimensions, and decision tracking to evaluate LLM agents and analyze their behavior.

1 Introduction

  • Existing benchmarks often isolate skills or lack structural diversity, while broad success and win-rate metrics can obscure strengths and weaknesses when models rarely succeed in challenging settings.Table 1 compares representative benchmarks across five evaluation dimensions.Source: §1 · p. 2, Table 1 · p. 3
  • DSGBench evaluates six strategic games through five capability dimensions and decision tracking, addressing limitations in skill coverage, environmental diversity, and broad outcome metrics.The games include StarCraft II, Civilization, Street Fighter III, Diplomacy, Werewolf, and Stratego.Source: §1 · p. 2
  • The games involve multi-agent interaction, long-context decisions, and varied sub-goals, while tasks can be customized across difficulty levels and targets.The framework supports asynchronous and synchronous interaction and uses observation-to-prompt and response-to-action loops.Source: §1 · p. 2, Figure 1 · p. 2
  • Evaluation of six representative agents finds varying performance and systemic limitations across task types rather than consistent general competence.Decision tracking supports analysis of agents’ decision-making processes.Source: §1 · p. 2

2 Related Work

Prior work has studied LLM agents’ reasoning and interaction with tools and environments, and has expanded evaluation into dynamic settings. Existing benchmarks still vary in their coverage of integrated cognition, long-horizon planning, and decision-process insight.

2.1 LLM-based agents

  • Research on LLM-based agents spans tool and environment interaction, perception, memory, decision-making, and reasoning capabilities such as world modeling and theory of mind.These capabilities are discussed in relation to agents’ responses to adversary changes, uncertain environments, and successive decisions.Source: §2.1 · p. 3

2.2 Existing Benchmarks for LLM-based agents

  • Benchmarks have moved from static datasets toward executable environments and multi-round interaction across areas including software development, embodied agents, simulations, gaming, and policy making.Static datasets are described as insufficient to reflect LLM performance in dynamic, complex environments.Source: §2.2 · p. 3
  • DSGBench’s evaluation suite covers strategic planning, real-time decision-making, social reasoning, team collaboration, and adaptive learning with tailored metrics across games and scenarios.Table 2 summarizes the capability coverage.Source: Table 2 · p. 4
  • Existing benchmarks may focus on single dimensions or eventual success, limiting evaluation of integrated cognitive skills and insight into complex decision processes.AgentBench and GameBench use eventual success as a core metric; GTBench and SmartPlay are described as unidimensional.Source: §2.2 · p. 4

3 DSGBench - Overview

DSGBench evaluates LLM agents through diverse strategic games, five cognitive dimensions, fine-grained metrics, and decision tracking. Its configurable scenarios and trajectory analysis support assessment of both performance and agent behavior.

3 DSGBench - Overview

  • DSGBench combines diverse strategy games, fine-grained capability metrics, and decision tracking in a unified Gym-based interface with configurable tasks.Its scoring process can also be customized, and the interface supports integrating new games and extending existing ones.Source: §3 · p. 4

3.1 Cognitive Decision-Making Capabilities

  • The benchmark evaluates strategic planning, real-time decision-making, social reasoning, team collaboration, and adaptive learning as five complementary cognitive dimensions.These dimensions draw on planning, rapid response, distributed information processing, multiagent coordination, and feedback-driven adaptation.Source: §3.1 · pp. 4–5

3.2 Diverse Strategic Environments and Scenarios

  • The benchmark covers real-time and turn-based strategy, fighting, diplomatic negotiation, and incomplete-information board play, while its customizable tasks vary opponent behavior, interaction paradigms, prompts, and reasoning strategies.Trajectories range from 24 iterations in Street Fighter III to 1270 in Stratego.Source: §3.2 · pp. 5–6

3.3 Fine-Grained Capability Metrics

  • Fine-grained metrics map capability dimensions to games and game-specific measures, addressing the limited detail of outcome metrics and the need for interpretable benchmarks.For StarCraft II, RPM measures resource gathering, while SUR measures used supply relative to maximum supply.Source: §3.3 · p. 6

3.4 Decision Trajectory Tracking

  • Decision tracking links key decisions to game states and objectives, recording actions, contexts, and outcomes to analyze agents’ strategic reasoning over time.The records include unit configurations, resource reserves, historical events, opponent actions, and intermediate goals inferred from chain-of-thought outputs; Figure 2 tracks EER, EPM, and GA across sessions.Source: §3.4 · pp. 6–8, Figure 2 · p. 7, Table 4 · p. 7

4 Experiments

The experiments assess six LLMs across six strategic games, using fine-grained capability scores and decision-trajectory analysis to examine performance and recurring limitations. Results show differentiated strengths across models and tasks, alongside difficulties integrating long-term plans, incomplete information, and memory.

4.1 Evaluation Setup

  • Six models were tested without finetuning using standardized prompts across six games, with 10 matches per scenario and temperature fixed at 0.2.Games used a unified text-based interaction loop; Stratego received custom prompt structures and reasoning frameworks.Source: §4.1 · p. 8

4.2 Quantitative Results

  • Gemini 1.5 Flash scores 72.88 in strategic planning and 64.23 in adaptive learning, but 48.45 in real-time decisions; GPT-4o scores 83.27 in social reasoning and 34.31 in team collaboration.GPT-4o also scores above 40 in its other dimensions.Source: §4.2 · pp. 8–9
  • Gemini 1.5 Flash and GPT-4o score 56.16 and 54.10 overall, versus 53.75 for DeepSeek-V2.5 and 45.11 for Llama-3.1-70B-Instruct.The largest stated strategic-planning gap is Gemini 1.5 Flash (72.88) versus Llama-3.1-70B-Instruct (51.47); Llama-3.1-70B-Instruct is competitive in StarCraft real-time decision-making.Source: §4.2 · p. 9
  • In Civilization, Gemini 1.5 Flash scores 72.88 in strategic planning while GPT-3.5 Turbo scores 32.94; in real-time gaming, Llama-3.1-70B-Instruct scores 66.35 in real-time decision-making.The authors suggest the variation across game types may reflect agents optimized for particular cognitive tasks at the expense of others.Source: §4.2 · p. 9

4.3 Qualitative Analysis

  • Agents can set long-term goals in StarCraft II yet fail to adapt to immediate tactical needs, missing emerging threats or fleeting advantages.The trajectory analysis identifies this as difficulty balancing long-term planning with immediate decision-making.Source: §4.3 · p. 9
  • In Stratego-like settings, agents may infer hidden pieces and future moves incompletely, then fail to revise beliefs when new information appears.This constrains strategic decisions based on partial data.Source: §4.3 · p. 9
  • In Diplomacy, agents may lose track of past interactions and other agents’ intentions, producing inconsistent or repetitive strategies that adaptive opponents can exploit.Source: §4.3 · p. 9

5 Conclusion

The paper presents DSGBench as a benchmark for evaluating LLM agents’ strategic decision-making across diverse, dynamic games. Its evaluations reveal differences across ability dimensions and scenarios, and the framework supports selecting agents for particular tasks and extending the benchmark with new games and scenarios.

5 Conclusion

  • DSGBench evaluates LLM agents’ strategic decision-making in diverse, dynamic gaming environments using cognitive dimensions and an integrated evaluation approach.Source: §5 · p. 9
  • Experiments show significant differences across ability dimensions, while fine-grained metrics and trajectory analysis reveal agents’ strengths and weaknesses across scenarios.Source: §5 · pp. 9–10
  • The unified framework supports adding games and customizing scenarios, and DSGBench can help select agents suited to particular tasks or capabilities.Source: §5 · p. 10

Supplementary Material

The supplementary material describes cognitive decision-making capabilities and how their scores are computed. It defines five capability dimensions and combines game-linked, fine-grained performance metrics into weighted capability scores.

A Cognitive Decision-Making Capabilities

A.1 Key Dimensions of Cognitive Decision-Making Capabilities

  • The five dimensions are strategic planning, real-time decision-making, social reasoning, team collaboration, and adaptive learning.They concern long-term goals, time pressure, understanding other agents, coordinated action, and learning from experience, respectively.Source: §A.1 · p. 17

A.2 Capability Score Computation

  • Capability scores weight fine-grained metrics from associated games; aggregate scores combine capability dimensions using weights, with scenario performance also weighted and normalized against theoretical bounds.Source: §A.2 · p. 17

B.1 Overall Architecture

DSGBench uses a shared evaluation architecture to configure games and agents, manage interactions, and track decision histories. Its interaction formulation distinguishes single-level reasoning from macro-and-micro inference for complex strategy games.

B.1 Overall Architecture

  • The GameManager coordinates setup and execution, while DataCollector configures evaluations, GameEnv standardizes actions and observations, and HistoryTracker records game and agent information.During execution, the environment advances through state transitions and the tracker records state information.Source: §B.1 · pp. 17–18

B.2 Game Interaction Formulation

  • DSGBench models agent-environment interaction as a POMDP, with the LLM policy mapping observation history and the current observation to the next action.The formalization includes victory conditions, states, actions, observations, and transitions.Source: §B.2 · p. 18
  • Centralized-state games such as Street Fighter III, Diplomacy, Werewolf, and Stratego use single-level reasoning that maps states and feedback to actions.The passage describes this as one layer of abstraction for complex tasks.Source: §B.2 · pp. 18–19
  • StarCraft II and Civilization use two-level inference: macro strategic planning, such as resource management, and micro tactical decisions, such as local combat.The two levels address large observation spaces and multidimensional tasks.Source: §B.2 · p. 19

C.1 Automated Evaluation in DSGBench

  • The automated evaluation platform combines configuration, environment interaction, and decision-history tracking to support uniform evaluation and analysis of agents’ behavior and strategy evolution.GameManager coordinates initialization and execution across these components.Source: §C.1 · pp. 19–20

C.2 Customizable Evaluation Scenarios

DSGBench supports targeted scenario configuration and uses decision trajectories to identify recurring agent difficulties. The analyses describe problems balancing long-term plans against immediate needs and adapting to incomplete or newly revealed information.

C.2 Customizable Evaluation Scenarios

  • Custom scenarios let evaluators control opponent behavior, interaction paradigms, prompt approaches, and reasoning strategies; game trajectories average 24 iterations in Street Fighter III and 1270 in Stratego.These extended sequences challenge contextual learning, long-term planning, and decision consistency.Source: §C.2 · p. 20

C.3 Model Description

  • The study summarizes the evaluated models’ names and API codes in Table 6.The supplied passage does not specify the model names or codes.Source: §C.3 · p. 20, Table 6 · p. 20

C.4 Runtime Estimation

  • Runtime estimates use each game’s total rounds and average inference time per round, measured with API-accessed models in a single-threaded setup on a 24 GB NVIDIA GeForce RTX 4090 server.The benchmark also supports multithreaded evaluation, which can reduce total evaluation time.Source: §C.4 · p. 20

D Decision Trajectory Analysis: Examples and Insights

  • Trajectory analysis identifies three systemic limitations: balancing long-term planning with immediate decisions, handling incomplete information and multi-hop inference, and overcoming memory limits in long-horizon tasks.The analysis examines decision points, contexts, and outcomes.Source: D Decision Trajectory Analysis:… · p. 20

D.1 Case 1: Difficulty balancing long-term planning with immediate decision-making.

  • In StarCraft II, agents can persist with economic growth or technology upgrades despite immediate threats, leaving them vulnerable or missing expansion opportunities.The examples describe insufficient defensive allocation against an early attack and failure to redirect resources when an opponent pushes or expansion is available.Source: §D.1 · pp. 21–22

D.2 Case 2: Challenges handling incomplete information and multi-hop inference.

  • In Stratego, limited visibility can prompt aggressive moves based on incorrect assumptions, while failures to revise plans as information emerges can lead to increasingly divergent responses and poor predictions of opponents’ moves.Examples include advancing a Scout without accounting for hidden defenses and moving a Miner without considering possible bomb placements.Source: §D.2 · pp. 22, 24

D.3 Case 3: Difficulty overcoming memory limitations in long horizon tasks.

The Diplomacy case traces Turkey’s repeated unit disbandments for Welfare Points, continuing after the game’s broader shift toward building forces. The following StarCraft II material describes the game’s decision demands, settings reference, and simplified action space.

D.3 Case 3: Difficulty overcoming memory limitations in long horizon tasks.

  • Turkey repeatedly disbanded units for Welfare Points, continuing the strategy as other powers shifted toward building forces.The trace links the early pattern to initial beliefs that relations were stable and early WP gains; later, reports described other powers building units, yet Turkey still considered disbanding.Source: §D.3 · pp. 24–26

E StarCraft II

  • The StarCraft II setting demands resource management, strategic planning, and real-time adaptation amid a large action space, imperfect information, and time pressure.Its complexity also involves interactions between micromanagement and macromanagement.Source: E StarCraft II · p. 26

E.1 Game Detailed Description

  • Table 10 provides a manual covering StarCraft II settings, unit configurations and policy, fixed opening strategies, and gameplay goals.Source: §E.1 · p. 26, Table 10 · p. 27

E.2 Action Space

  • The agent uses five predefined macro-level actions that abstract away micro-level tactics.The listed action types include training units, building structures, researching techniques, scouting, and battling.Source: §E.2 · pp. 26–27

E.3 Fine-Grained Metrics

This block describes StarCraft II metrics, example trajectories, and prompting, then outlines Civilization and its action space. The metrics target distinct aspects of decision-making, while the later material shows example inputs and actions and summarizes the game’s strategic demands.

E.3 Fine-Grained Metrics

  • RPM, EER, SUR, and TCR assess resource gathering, resource use, supply use, and technology completion, respectively.The metric descriptions frame these measures as components of evaluating decision-making in StarCraft II.Source: §E.3 · p. 28
  • APM, EPM, WR, and GA measure action speed, effective-action efficiency, match wins, and grounding accuracy, respectively.The supplied metric descriptions define the underlying action counts, game time, wins, games, and grounding attempts.Source: §E.3 · p. 28

E.4 Example Trajectory

  • The example trajectory combines game-state summaries and strategic suggestions with five selected actions at each illustrated decision point.The examples describe economic, army, and technology status, alongside recommendations such as increasing mineral income or diversifying units.Source: §E.4 · pp. 29–30

E.5 Prompt Example

  • The prompt example supplies game-state information and requests structured analysis, strategic recommendations, and five feasible actions from an action dictionary.Its requested analysis covers the game overview, stage, player and enemy situations, and key information across rounds.Source: §E.5 · pp. 30–31

F Civilization

  • Civilization requires long-term decisions about resources, development, diplomacy, and competition across hundreds of turns.The game spans city-building, technological and cultural development, alliances, and warfare.Source: F Civilization · p. 33

F.1 Game Detailed Description

  • Table 11 is a game manual detailing Civilization settings, unit configurations and policy, fixed opening strategies, and gameplay goals.Source: §F.1 · p. 34, Table 11 · p. 34

F.2 Action Space

  • Civilization actions span engineering, movement, and combat, while gameplay also includes city, government, technology, and diplomacy actions.These cover activities such as city construction, unit production, tax adjustment, research goals, and negotiations.Source: §F.2 · p. 35

F.3 Fine-Grained Metrics

This section defines fine-grained Civilization metrics and shows how agents receive observations, advice, and legal actions while planning and acting. It then introduces Street Fighter III as a fast-paced setting for testing combat decisions.

F.3 Fine-Grained Metrics

  • The Civilization metrics track economy, city growth, technology, land use, military growth, wins, and grounding accuracy.They are designed around expert insights into the game's mechanics, with each metric targeting an aspect of decision-making.Source: §F.3 · p. 35

F.4 Example Trajectory

  • The example trajectory has the agent prioritize founding a city and scouting, while an Explorer moves west toward a less-explored region.The agent reasons about safety, resources, and the available actions when choosing movement.Source: §F.4 · pp. 36–37

F.5 Prompt Example

  • The prompt supplies observations, available actions, and advisor messages, and requests structured thought, review, reasoning, planning, and a legal final action.The Explorer example selects a northeast move toward terrain considered more suitable for a city.Source: §F.5 · pp. 37–38

G Street Fighter III

  • Street Fighter III's parry system rewards precisely timed blocks followed by counters, alongside character-specific movement, combos, and attack timing.Agents are evaluated in a fast-paced, high-pressure setting that demands quick decisions and anticipation of opponents' tactics.Source: G Street Fighter III · pp. 39–40

G.1 Game Detailed Description

  • Table 12 is a game manual covering Street Fighter III settings, character mechanisms, strategies, and gameplay goals.The section points readers to the table for further game details.Source: Table 12 · p. 40, §G.1 · p. 40

G.2 Action Space

  • The Street Fighter III action space comprises movement, normal attacks, and special attacks.Normal attacks include punch, kick, and directional variants; special attacks include moves such as Fireball and super attacks.Source: §G.2 · p. 41

G.3 Fine-Grained Metrics

The section defines Street Fighter III metrics and illustrates the agent's combat prompt and trajectory. It then describes Diplomacy as a setting for negotiation, alliance management, and simultaneous unit orders.

G.3 Fine-Grained Metrics

  • Street Fighter III metrics assess attack and super-move hit rates, health change, win rate, and grounding accuracy.The metrics are designed around expert insights into the game's mechanics and target distinct aspects of agent decision-making.Source: §G.3 · p. 41

G.4 Example Trajectory

  • In the example trajectory, the agent moves closer and uses Fireball, medium punch, and low kick; scores are +17.0 at step k+3 and -45.0 at step k+5.Source: §G.4 · pp. 41–42

G.5 Prompt Example

  • The prompt frames the agent as Ken, supplies health, score, opponent state, and available moves, and asks for a bullet list of actions.Its instructions encourage attacking while avoiding hits.Source: §G.5 · pp. 42–43

H Diplomacy

  • Diplomacy centers on negotiation, alliances, and simultaneous player orders, making it a testbed for strategic foresight and anticipating opponents.Agents are expected to manage long-term plans and adapt to changing alliances.Source: H Diplomacy · p. 43

H.1 Game Detailed Description

  • Table 13 is a game manual covering Diplomacy settings, unit configurations, mechanics, strategies, and gameplay goals.The section directs readers to the table for further game details.Source: Table 13 · p. 44, §H.1 · p. 44

H.2 Action Space

  • Diplomacy orders include hold, move, support, and convoy, and each Great Power may order all, some, or none of its units each turn.Support increases attacking or defending strength; convoys move armies between coastal provinces by fleet.Source: §H.2 · pp. 44–45

H.3 Fine-Grained Metrics

This section lays out Diplomacy metrics and examples of agents balancing welfare goals with military strength and coordination. It then presents Werewolf as a social-deduction setting involving limited information, discussion, and role-specific actions.

H.3 Fine-Grained Metrics

  • Diplomacy metrics cover control centers, welfare, betrayal, alliance stability, and alliance duration.The metrics are designed around expert insights into the game's mechanics and target specific aspects of agent decision-making.Source: §H.3 · p. 45

H.4 Example Trajectory

  • The trajectory examples show agents coordinating with allies while revising plans when aggression shifts the balance between welfare maximization and defense.One example calls for rebuilding defenses and preparing a counter-offensive after Russia's aggression; another considers coordinated disbandments to increase welfare points.Source: §H.4 · pp. 45–47

H.5 Prompt Example

  • The Welfare Diplomacy prompt sets a fixed end date and makes accumulated welfare points the objective, with unit strength traded off against welfare and up to two message rounds per phase.The response format includes private reasoning, orders, and optional messages to other powers.Source: §H.5 · pp. 47–48

A BUL - CON, GRE, RUM, SER Current Supply, Unit, and WP Count (Supply Centers/Units/Welfare Points)

  • In the Winter 1903 example, Germany builds two armies while maintaining welfare goals and communicating with allies about balancing military presence and welfare points.The example's reasoning cites other powers' disbandments and Germany's vulnerability as considerations.Source: A BUL - CON, GRE, RUM, SER Current… · p. 51

I Werewolf

  • Werewolf requires players to infer hidden roles from limited information, debate and vote, while special roles can eliminate, protect, or investigate players.The game's shifting social dynamics call for trust assessment, deception detection, and adaptation.Source: I Werewolf · pp. 51–53, §I.2 · p. 53

I.1 Game Detailed Description

  • Table 14 is a game manual covering Werewolf settings, roles, abilities, strategies, and victory conditions.The section points to the table for further game details.Source: Table 14 · p. 52, §I.1 · p. 53

I.2 Action Space

  • Werewolf actions include debate and voting, plus nighttime actions tied to special roles such as Werewolves, Doctors, and Seers.Those roles respectively eliminate a villager, protect a player, or investigate a player's role.Source: §I.2 · p. 53

I.3 Fine-Grained Metrics

The section defines fine-grained Werewolf measures and illustrates role-conditioned play through trajectories and prompts, then outlines Stratego’s mechanics and available actions.

I.3 Fine-Grained Metrics

  • Werewolf’s IRP measures role-identification precision, while KSR measures key-role survival through the end of a game.Both are defined using counts of correct identifications or surviving roles relative to their respective attempts or games played.Source: §I.3 · p. 53
  • VSS assesses voting efficacy at critical junctures, including votes that eliminate opponents or save allies.It is calculated from successful critical votes relative to all critical votes cast.Source: §I.3 · p. 53

I.4 Example Trajectory

  • The Werewolf example shows a Werewolf using debate and voting to redirect suspicion, while a Villager tracks possible alliances and threats.The sample includes role-specific reasoning, a debate, a vote, and a subsequent summary.Source: §I.4 · p. 54

I.5 Prompt Example

  • The Werewolf prompt assigns role-specific choices such as protection, investigation, debate, and voting within the game’s night and day phases.The described setup has eight players, including two Werewolves, a Seer, a Doctor, and four Villagers.Source: §I.5 · pp. 55–56

J Stratego

  • Stratego centers on capturing a hidden-rank opponent’s flag through planning, tactical movement, and inference under incomplete information.Its pieces have different ranks and special abilities, and combat outcomes depend on rank.Source: J Stratego · p. 57

J.1 Game Detailed Description

  • The Stratego manual collects game settings, piece configurations, movement rules, strategies, and goals.It is provided as a detailed game manual in Table 15.Source: Table 15 · p. 57, §J.1 · p. 58

J.2 Action Space

  • DSGBench exposes two Stratego actions: moving a piece and attacking by moving onto an opponent’s occupied square.Bombs and flags cannot move; Scouts can travel any distance, while other movable pieces move one square orthogonally.Source: §J.2 · p. 58

J.3 Fine-Grained Metrics

The section operationalizes Stratego performance with piece-value and outcome metrics, illustrates move reasoning in example trajectories, and gives a prompt format for choosing valid moves from concealed-rank board states.

J.3 Fine-Grained Metrics

  • CPR is the proportion of a player’s critical pieces alive relative to their initial number; TPCV sums piece values at a given point, typically game start, using rank-specific values.For the initial setup, TPCVinitial = 10NMarshal + 9NGeneral + 8NColonel + 7NMajor + 6NCaptain.Source: §J.3 · p. 58
  • WR is the percentage of matches won, and GA measures the precision and success rate of grounding actions.GA compares successful grounding actions with total grounding attempts.Source: §J.3 · p. 58

J.4 Example Trajectory

  • Example trajectories show agents probing concealed defenses with moves intended to reveal opponent ranks and locate the flag.One example advances a Scout to reveal more of the opponent’s setup.Source: §J.4 · pp. 59–60

J.5 Prompt Example

  • The prompt represents unknown opponent pieces as B(#), supplies the board state and valid moves, and requests a strategic move.The response format asks for reasoning about the macro strategy and a move selected from the listed valid options.Source: §J.5 · pp. 61–62

K.1 Limitations

DSGBench’s scope is constrained by lengthy evaluations, limited scenario diversity, and the absence of a unified cross-game trajectory dataset; the authors describe potential uses for standardized assessment and agent development.

K.1 Limitations

  • A trial may take hours or days, creating computational demands that can limit the scale of experimentation.The authors link this runtime burden to the games’ complex, extended decision processes.Source: §K.1 · pp. 63–64
  • The curated scenario set remains restricted in number and diversity, leaving generalization to broader strategic challenges or new games open.The authors identify expansion of the benchmark suite as a topic for further investigation.Source: §K.1 · p. 64
  • The benchmark lacks a large, standardized trajectory dataset spanning games, which the authors say hinders rapid, standardized development of generalizable strategic reasoning.The missing resource would support cross-game training and pre-training of LLM-based agents.Source: §K.1 · p. 64

K.2 Broad Impacts

  • The authors suggest DSGBench may support standardized evaluation and targeted improvements, with potential application to gaming AI and multi-robot coordination.They also mention some autonomous systems operating in interactive environments.Source: §K.2 · p. 64

Related papers

Loading 2503.06047v2…