PaperlayerView exact revision on arXiv

Source-linked AI summary

Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games

Nick Leenders, Roy Lindelauf, Joost van Oijen, Boris Cule

arXiv:2610.11663v1cs.AI

TL;DR

RTS agents can fail against strategies outside their recent training distribution, and separating strategy choice from unit control requires an executor whose command-following can be measured. This paper combines a constrained command-conditioned executor with an observation-based bandit strategist. Against three of the four strongest opponents, the system wins significantly more often than flat PPO, including against two held-out opponents.

  • Problem

    RTS agents can be brittle against strategies they have not encountered recently, while separating strategic choice from unit control requires a way to measure whether commands are followed.

    Source: §1 · p. 1

  • Method

    A constrained command-conditioned executor fulfills state-measurable commands, while a Thompson-sampling bandit selects command tuples using observed opponent behavior.

    Source: §1 · p. 2

  • Results

    The strategist–executor system achieved significantly higher win rates than flat PPO against three of the four strongest opponents, including two held-out opponents, with no significant difference against the weaker ones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported gain stems from command-conditioned training rather than adaptive behavior switching, and intervention tests found that only economy and composition commands changed executor behavior.

    Source: §5 · p. 10

    Intervention tests show that commands can meet their success-rate constraints without changing behavior, so adherence does not ensure controllability.

    Source: §4.1 · p. 8, §5 · p. 10

Abstract

from arXiv · show

Deep reinforcement learning agents reach strong performance in real-time strategy games but can be brittle against opponents outside their training distribution. Separating strategic command selection from learned unit control allows different strategies to be selected for different opponents while reusing the same execution policy. This requires an executor that can follow different commands and measurable criteria for assessing whether it does so. We introduce a constrained command-conditioned Proximal Policy Optimization (PPO) policy, the executor, for MicroRTS, a real-time strategy environment. Discrete commands specify strategic objectives and behavioral requirements for economy, army composition, military posture, and worker policy over multiple environment steps; the executor determines the unit-level actions used to fulfill them. A Thompson-sampling bandit acts as the strategist, selecting command tuples from an estimate of the opponent's strategy built from in-game observations rather than opponent identity. In a controlled comparison with a flat PPO baseline trained with the same architecture, budget, curriculum and self-play league, the strategist-executor system wins significantly more often against three of the four strongest opponents on a training map, including the two strongest held-out ones (0.55 to 0.97 and 0.01 to 0.34), with no significant difference against the others.

1 INTRODUCTION

RTS agents can be brittle against unfamiliar strategies, while existing ways to separate strategic choice from unit control sacrifice either alignment with winning or learned execution. The paper introduces a measurable command system, a constrained executor, and an observation-based strategist, then compares the system with flat PPO.

1 INTRODUCTION

  • Prior separations either reward goal-following with a bonus that trades off against winning and lack an adherence criterion, or rely on scripted execution.Scripted execution cannot match learned policies in unit-level control.Source: §1 · pp. 1–2
  • A four-slot command tuple specifies economy, unit composition, military posture, and worker policy, with state-based success criteria for each command.Commands define when they may be issued and what the executor must achieve within an execution window.Source: §1 · p. 2
  • The executor maximizes game performance while meeting a separate success-rate constraint for each command, using adaptive per-command adherence pressure.Lagrange multipliers respond to measured adherence shortfalls, with a heuristic guard for persistently stalled constraints.Source: §1 · p. 2
  • A Thompson-sampling bandit selects command tuples using observed opponent behavior and learned estimates, without opponent identity or executor retraining.During training, it explores promising and uncertain combinations; during evaluation, the strategist is frozen.Source: §1 · p. 2
  • Significantly higher win rates against three of the four strongest opponents, including two held-out opponents, with no significant difference against the weaker ones.The comparison uses the same base architecture, environment-step budget, curriculum, and self-play league.Source: §1 · p. 2

2 RELATED WORK

Related work makes strategic choices explicit through scripts, goals, or reusable behavior selection, but differs in how it represents goals, executes them, and adapts to opponents.

Scripted portfolios and action abstraction

  • Scripted portfolios and programmatic strategies make strategic choices explicit, but specify low-level behavior with scripts or programs rather than a learned executor.This paper retains a fixed strategic vocabulary while pairing each item with a state-measurable goal for a learned executor.Source: §2 · p. 2

Goal-conditioned and hierarchical RL

  • FeUdal-style methods measure alignment with learned goals, whereas this paper’s related-work review notes they lack predefined, human-readable success criteria for each goal.FeUdal Networks use task reward and an intrinsic directional-alignment reward; Bugeja et al. (2019) extend the approach to an N-Layered FeUdal Network evaluated on StarCraft II minigames.Source: §2 · pp. 2–3

Explicit goals and interpretable abstractions

  • h-DQN, policy sketches, and language-instruction controllers use explicit goals or subtasks, with different reward or training interfaces.Jiang et al. (2019) train a reusable controller with instruction-achievement rewards and hindsight relabeling.Source: §2 · p. 3

Goal-conditioned and hierarchical RL

  • Grounded RL measures command adherence by bounding behavior-cloning loss on human demonstrations; this paper instead uses explicit game-state criteria and per-command targets.The executor learns to meet those targets through interaction, without human demonstrations.Source: §2 · p. 3

Strategy selection and opponent adaptation

  • Opponent-adaptation methods select among policies, build orders, or unit actions; this strategist selects compositional command tuples for one shared executor and shares payoff evidence across tuples.Bayesian Policy Reuse and Deep BPR+ infer responses from observed signals or behavior, while Gehring et al. (2018) switch among rule-based build orders.Source: §2 · p. 3

3 METHOD

The method separates strategic command selection from unit-level control: a shared executor follows state-measurable commands, while a bandit selects tuples using observed opponent behavior. The executor is trained to maximize game performance subject to per-command success-rate constraints.

3.2 COMMAND SYSTEM

  • The shared executor uses a four-slot command tuple whose commands have temporal success criteria, deadlines, and issue preconditions.An unchanged command keeps its window; windows without valid opportunities or interrupted before a terminal outcome are excluded from success-rate calculations.Source: §3.1 · p. 4, §3.2 · p. 4

3.1 STRATEGIST–EXECUTOR INTERFACE

  • The strategist supplies command tuples to a shared executor, which translates them into unit-level actions in MicroRTS.Command fulfillment is defined by a state-measurable success criterion.Source: Figure 1 · p. 3, §3.1 · p. 4

3.3 EXECUTOR

  • A GridNet PPO executor conditions its policy on commands and optimizes game performance under a separate success-rate constraint for each command.Separate normalized advantage streams keep reward-scale differences from setting adherence influence; measured below-target success rates increase the relevant multiplier, with a heuristic guard for stalled constraints near the upper bound.Source: §3.3 · pp. 4–5

3.4 STRATEGIST

  • A scouting-based bandit selects command tuples using observed opponent behavior and payoff estimates, then uses frozen estimates and current evidence at evaluation.It represents an unseen opponent as a mixture of known classes; evaluation changes with context and scouting evidence, not opponent identity or retraining.Source: §3.4 · p. 6

3.5 TRAINING PROCEDURE

  • The executor and strategist are trained together from scratch, using scheduled tuples early and retaining schedule episodes as strategist use grows.Game outcomes are split among active tuples by time, which does not identify each tuple’s causal contribution; the flat PPO comparison uses the same curriculum, pool, and interaction budget.Source: §3.5 · pp. 6–7

4 RESULTS

The executor met its training adherence targets, but intervention tests showed that some commands changed behavior while others did not. On map A, the strategist-executor system significantly improved win rates against three strong opponents, while its evaluation commands varied mainly in economy and composition.

4.1 COMMAND ADHERENCE AND CONTROLLABILITY

  • All sixteen smoothed command success rates ended between 0.75 and 1.00 against a target of 0.8, but meeting that criterion did not establish behavioral control.Rates can reflect easy issue states or default behavior, and commands without effects earn the default payoff; the authors recommend paired intervention estimates instead.Source: §4.1 · pp. 7–8
  • HEAVY produced the requested unit type and turned 8 wins in 8 games against coacAI into 8 losses; RANGED reduced obiBotKenobi wins from 7 in 7 to 1 in 8.GROW added one to two workers, whereas OFFENSE and HARASS did not shift army or worker position despite meeting their training constraint.Source: §4.1 · p. 7

4.2 EVALUATION AGAINST SOTA OPPONENTS

  • The bandit won at least as often as flat PPO against eight of twelve opponents, while differences against four other opponents were not significant.Both agents reached win rates of at least 0.96 against the four rush scripts; the evaluation used frozen agents and one training run per method on map A.Source: §4.2 · p. 8
  • Win rates rose from 0.61 to 0.93 against agent_sota, 0.55 to 0.97 against held-out obiBotKenobi, and 0.01 to 0.34 against held-out RAISocketAI; all three differences were significant.RAISocketAI still won most games against both agents; the comparison used one training run per method on map A.Source: §4.2 · p. 8, Table 3 · p. 9

4.3 WHAT THE STRATEGIST DECIDES DURING EVALUATION

  • The strategist usually changed the economy order to GROW, kept posture and worker policy at their defaults, and selected RANGED in all 100 coacAI games.Economy changes typically occurred at median ticks 360–460; the RANGED switch against coacAI occurred at median tick 242.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • With the same executor, LIGHT matched or came within 0.02 of the highest win rate against every opponent except coacAI, where RANGED won 1.00 and HEAVY 0.00.RANGED won only 0.11 against obiBotKenobi and 0.03 against RAISocketAI.Source: §4.3 · p. 9

5 CONCLUSIONS

The strategist–executor system wins more often than flat PPO against three of four strongest opponents on a training map, but its gain comes from command-conditioned training rather than adaptive switching.

5 CONCLUSIONS

  • The system wins more often than flat PPO against three of four strongest opponents, including the two strongest held-out opponents, but less often against the 2024 competition winner.The reported gain stems from command-conditioned training rather than adaptive behavior switching; fixed commands selected with opponent-identity knowledge perform on par with the strategist.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

  • Reported intervals do not include training-seed variance because each result comes from a single training run per arm.The paper states that evaluation protocols are in Appendices G–K and that opponents are publicly available.Source: REPRODUCIBILITY STATEMENT · p. 10

B.1 TERMINAL OUTCOMES

  • Command adherence is measured independently of game outcome in temporal windows that end in success, failure, or timeout.Interrupted windows are not terminal failures or timeouts and do not enter the success-rate calculation.Source: §B.1 · pp. 15, 17, Table 5 · p. 16
  • Composition success uses accumulated positive changes in living unit counts, a proxy that can miss production offset by losses or include units started before a command switch.Composition commands also time out if no barracks is present at or after tick 350.Source: §B.1 · p. 15
  • Posture criteria classify military units by proximity to targets and bases, while HARASS measures forward presence rather than attacks.Posture success requires at least 50 evaluable ticks; ECON_ONLY permits local defense.Source: §B.1 · p. 15

B.2 ISSUE PRECONDITIONS

  • Issue preconditions gate strategist commands, not measurement; infeasible commands are temporarily replaced by feasible alternatives and reissued when possible.At episode start, only WORKER_ECON_ONLY and POSTURE_DEFENSE are issuable in their respective slots.Source: §B.2 · p. 17

C REWARD STRUCTURE

  • The executor separates game-performance and command-slot reward streams, combining their normalized advantages with command-specific multipliers.Intermediate construction, production, and attack rewards provide denser feedback than terminal outcomes, but need not rank policies identically to win rate; attack rewards do not measure damage or kills.Source: C REWARD STRUCTURE · p. 17, §C.1 · p. 17

C.2 ADHERENCE REWARDS

  • Command rewards combine within-window shaping, terminal payments, and interrupted-progress credit, which is a learning signal rather than terminal success.Shaping terms are task-specific; the paper does not assume they preserve the optimal policy for either the terminal game objective or terminal command-success objective.Source: §C.2 · pp. 17–18

D ARCHITECTURE

The architecture conditions a spatial actor and critics on command embeddings, with a dedicated command-only head for military-unit production.

D ARCHITECTURE

  • A convolutional stem, conditioned residual blocks, and an upsampling decoder produce per-cell action logits, while critics pool spatial features and concatenate command embeddings.On a 16 × 16 map, the bottleneck is 8 × 8; commands use 16-dimensional embeddings concatenated in slot order.Source: D ARCHITECTURE · p. 18
  • The implementation passes a fixed ω = (0.5, 0.5) through a learned projection, so it is not a requested objective trade-off.The flat control instead uses the constant (1, 0).Source: D ARCHITECTURE · p. 18
  • A command-only production head replaces spatial-actor outputs for military unit types and receives auxiliary cross-entropy supervision from composition targets.The head's auxiliary coefficient decreases from 0.10 to 0.06 between five and ten million learner steps; the flat control disables this head and loss.Source: D ARCHITECTURE · pp. 18–19

E COMMAND SAMPLING FOR EXECUTOR TRAINING

Executor training uses generated command schedules alongside bandit-selected commands, with contextual observations and episode outcomes informing command-tuple estimates.

E COMMAND SAMPLING FOR EXECUTOR TRAINING

  • The generated pool contains 256 random schedules with one to five phases, sampled command tuples, and time- or state-based transitions.Coverage sampling balances schedule selections and favors underexposed commands; infeasible commands are temporarily replaced and later reissued.Source: E COMMAND SAMPLING FOR EXECUTOR TRAINING · p. 19
  • Bandit-selected episodes rise from coverage-only table learning to a 0.6 episode probability over the next 10% of training, while generated schedules retain a 0.4 share.The bandit table learns throughout the initial 40% coverage period.Source: E COMMAND SAMPLING FOR EXECUTOR TRAINING · p. 19
  • Opponent context combines game stage and military momentum with observed barracks, unit types, worker aggression, worker-count bucket, and time bucket.Opponent identities supply training classes, while league snapshots share one class.Source: E COMMAND SAMPLING FOR EXECUTOR TRAINING · p. 19
  • Outcome credit is assigned to commands used during a game, but the update does not estimate their causal effect on winning.Records shorter than 40 ticks are excluded unless all records are shorter, in which case the longest is retained; decisions are generally due every 150 ticks or after persistent evidence changes.Source: E COMMAND SAMPLING FOR EXECUTOR TRAINING · pp. 19–20

F TRAINING DETAILS

Training uses a 240M-step PPO setup with multiplier updates, a scripted-bot curriculum, and a self-play league; final command success rates meet or approach their target.

F.1 HYPERPARAMETERS

  • 240M learner steps across 72 environments and three 16 × 16 maps yield approximately 8,370 PPO updates.Each rollout collects 28,672 transitions from 56 learner environments; the flat comparison uses the same curriculum, maps, league rules, and budget.Source: §F.1 · p. 20, §F.3 · p. 20

F.2 MULTIPLIER SETTINGS

  • Command multipliers target a success rate of 0.8 and update only when a rollout contains at least five terminal events for that command.A stall guard can hold a multiplier at its lower bound after 100 eligible updates without sufficient rate improvement.Source: §F.2 · p. 20

F.3 CURRICULUM AND SELF-PLAY LEAGUE

  • The scripted curriculum introduces lightRushAI and rojo in phase B, mayari in C, and coacAI and TMA in D; six held-out opponents are never encountered in training.The final phase assigns 30 of 40 scripted environments to coacAI, mayari, and TMA; self-play draws opponents from frozen policies and learner snapshots.Source: §F.3 · pp. 20–21

G.1 CONSTRAINT SATISFACTION

  • Table 7 reports each command’s multiplier and smoothed success rate at the strategist–executor run’s final checkpoint, with target 0.8 and bounds [0.05, 1.0]; no constraint is flagged as stalled.Source: §G.1 · p. 21, Table 7 · p. 21

G.2 INTERVENTION AUDIT

The intervention audit tests command adherence by installing commands after their preconditions hold and measuring both contract outcomes and behavior scores. Composition commands change the requested behavior, whereas posture and worker-policy commands show little observed effect; early game endings can prevent composition windows from resolving.

Protocol

  • Interventions begin only once the executor owns a base, barracks, four workers, and a military unit; commands are then pinned at every step.The audit uses the final strategist–executor checkpoint without further training, with up to 900-tick windows and a 2,000-step game cap.Source: §G.2 · p. 21
  • Same-seed reference and intervention games match state at the intervention for coacAI, but droplet and obiBotKenobi comparisons are seed-blocked because their trajectories diverge from the start.Ten games ended or stalled before the intervention gate and were excluded from every table, including win counts.Source: §G.2 · p. 21

Measurement

  • Unit births are counted by new game-assigned identifiers, including production already underway at intervention; physical metrics use the shared first 300 ticks when possible, otherwise the observed window.Composition shares exclude pairs without military production, leaving 19–23 of 24 pairs per command; intervals omit training-seed uncertainty and multiple-comparison correction.Source: §G.2 · pp. 21–22
  • Composition contract success is lower against droplet and obiBotKenobi than coacAI because games end before the 700- to 800-tick windows resolve, although produced units match the request in every case.Table 8 reports contract outcomes for the intervention games.Source: §G.2 · p. 22, Table 8 · p. 22

Behavior scores

  • 0.78 and 0.80: HEAVY and RANGED raise requested-type behavior scores; OFFENSE changes no posture score, while HARASS shifts neither worker-policy score by more than 0.04.These Figure 2 scores are heuristic behavior measures, not contract outcomes.Source: §G.2 · p. 22, Figure 2 · p. 22

H STRENGTH OF THE OPPONENTS ON THE TRAINING MAPS

Opponent strength is estimated from pairwise games using a Bradley–Terry model, with tiers clarifying where the model comparison is meaningful. The rankings have map- and seat-dependent qualifications, and the Flat–Bandit differences occur among upper- and middle-tier opponents.

Strength model

  • The Bradley–Terry model assigns each player a scalar strength so that the probability of beating another player is the logistic function of their strength difference plus a shared first-player seat term.A one-unit strength difference corresponds to a 0.73 win probability; the absolute strength level is arbitrary.Source: H STRENGTH OF THE OPPONENTS ON THE… · p. 23
  • The pool includes 12 Table 3 opponents plus rojo and randomBiasedAI, with both seats played across the three training maps except for agent_sota, which plays only as player 0 on map A.Deterministic duplicate cells are collapsed before fitting, leaving 2,993 effective games; draws count as half-wins, and bootstrap intervals use 1,000 resamples.Source: H STRENGTH OF THE OPPONENTS ON THE… · p. 23

Opponent tiers

  • Flat–Bandit differences appear against upper- and middle-tier opponents—agent_sota, obiBotKenobi, droplet, and aggrobot—while perfect scores against several rush bots are against the weak half of the pool.RAISocketAI is strongest on every map except I; tma and agent_sota (on map A) follow, then obiBotKenobi.Source: H STRENGTH OF THE OPPONENTS ON THE… · pp. 23–24

Ranking qualifications

  • Opponent tiers vary by map and seat: coacAI ranks sixth pooled, fourth as player 1, and tenth on map D, while the executor faces it as player 0 on map A, the less-favored seat.Map-specific strengths are centered separately and are comparable within, but not across, maps.Source: H STRENGTH OF THE OPPONENTS ON THE… · pp. 23–24
  • Mirror games mainly inform the shared seat term: 20 of 36 end in identical draws, while other deterministic mirrors award every game to the same seat.Only obiBotKenobi, droplet, and RAISocketAI vary between mirror games.Source: H STRENGTH OF THE OPPONENTS ON THE… · p. 24

I MODEL COMPARISON: GAME COUNTS AND FIXED TUPLES

The model comparison reports wins, losses, and draws for each opponent and evaluates a Static executor that holds a selected command tuple for the entire game. Static tuples are selected on separate screening games and tested on fresh games.

Game-count analysis

  • Flat–Bandit win-rate comparisons use two-sided Fisher exact tests with Holm correction across 12 opponents, assessing decisive games only.Bandit–Static comparisons use the same test with a separate Holm correction family; the Flat–Bandit head-to-head uses an uncorrected exact binomial test.Source: I MODEL COMPARISON: GAME COUNTS AND… · p. 24

Fixed tuples

  • Static holds one fixed tuple throughout a game, with the best economy-and-composition tuple chosen from 18 candidates using a separate 20-game-per-tuple screen.The screen maximizes W/(W + L), breaking ties by fewer draws and then shorter games; HOLD is excluded because it halts worker production at about four workers.Source: I MODEL COMPARISON: GAME COUNTS AND… · p. 24
  • The selected Static tuple is evaluated in 100 fresh games against every opponent; Table 10 reports wins, losses, and draws in Table 3's opponent order.The table's Static tuple labels abbreviate economy and composition choices, including DOUBLE for DOUBLE_BARRACKS and L, H, and R for unit types.Source: I MODEL COMPARISON: GAME COUNTS AND… · p. 24, Table 10 · p. 24

J PLAYSTYLE OF FLAT, BANDIT AND STATIC

The bandit generally uses a Light army and trains fewer workers than Flat, while its command choices are reflected in its builds. Flat's opponent-dependent builds can succeed against one opponent and fail sharply against another.

Build patterns

  • Bandit's builds follow its commands: where its composition matches Static's against agent_sota, obiBotKenobi, and coacAI, their barracks timing, worker counts, and army sizes are nearly identical.Against coacAI, the strategist selects RANGED in all 100 games before the first barracks exists; RANGED scores 1.00 there versus 0.73 for LIGHT.Source: J PLAYSTYLE OF FLAT, BANDIT AND STATIC · p. 25
  • The bandit first places a barracks at median tick 300 against each of the listed opponents; it first has at least two military units in the enemy half at median ticks 550–700, or 1,030 against RAISocketAI.It trains fewer workers than Flat against every opponent except RAISocketAI.Source: J PLAYSTYLE OF FLAT, BANDIT AND STATIC · p. 25, Figure 3 · p. 25

Flat's builds

  • Against obiBotKenobi and agent_sota, Flat places its first barracks at tick 360, trains about three more workers than Bandit, and builds mostly Ranged armies.For the command-conditioned executor, RANGED scores 0.11 and 0.25 against these opponents, respectively.Source: J PLAYSTYLE OF FLAT, BANDIT AND STATIC · p. 25
  • Against RAISocketAI, TMA, and lightRushAI, Flat trains fewer than two military units per game and never has two in the enemy half in at least 94 of 100 games.The same worker rush wins 99 of 100 games against TMA but loses 98 of 100 against RAISocketAI.Source: J PLAYSTYLE OF FLAT, BANDIT AND STATIC · p. 25

K PAYOFF OF FIXED COMMAND TUPLES

The fixed-command screen finds composition-dependent scores against several strong opponents, but little out-of-sample gain from choosing an opponent-specific tuple over the best single tuple. Training and evaluation times are reported as indicative or approximate.

K PAYOFF OF FIXED COMMAND TUPLES

  • Across 5,760 games, the screen tested 24 tuples against 12 opponents on map A (20 games per cell), selecting and scoring on separate halves over 2,000 random splits.Tuple choices were restricted to the 18 combinations available in the Static screen for the single-tuple versus oracle comparison.Source: K PAYOFF OF FIXED COMMAND TUPLES · pp. 25–26
  • Out of sample, the oracle’s difference was −0.011 (95% of splits between −0.062 and +0.029), versus a same-game advantage of 0.033 (0.954 vs 0.921).The same-game comparison favors the oracle; the out-of-sample estimate does not show a gain.Source: K PAYOFF OF FIXED COMMAND TUPLES · p. 26
  • LIGHT was best or within 0.02 of best against every opponent except coacAI, where switching to RANGED was the only payoff from opponent knowledge.RANGED scores 1.00 against coacAI and 0.11 against obiBotKenobi; HEAVY scores 0.00 against coacAI and above 0.96 against six other opponents.Source: K PAYOFF OF FIXED COMMAND TUPLES · p. 26, Table 11 · p. 26
  • HOLD scores low because it halts worker production: the executor had at most four workers in 95% of games, versus a median peak of six under GROW.The table averages HOLD across all six compositions.Source: Table 11 · p. 26

L COMPUTATION TIME

  • Training to 240M learner steps took about 120 h for Flat and 180 h for Bandit; shared, concurrent hardware makes these times indicative rather than controlled.The runs used one GPU each and shared the server with other jobs.Source: L COMPUTATION TIME · p. 26
  • Evaluation took about 3.5 h for model comparison, 1.6 h for the tournament, 0.8 h for the payoff matrix, and under 1 h for the intervention audit.These times exclude development runs and discarded variants.Source: L COMPUTATION TIME · p. 26
Loading 2610.11663v1…