PaperlayerView exact revision on arXiv

Source-linked AI summary

STRATA: Self-Learning Through Role-Aligned Tiered Agents for Real-Time Strategy Games

Xinhe Tian, Xiaoyue Zhang, Ziyou Zhang, Jiacheng Li, Xiaoqiang Jin, Qianchuan Zhao, Gaochen Cui

arXiv:2609.38881v1cs.AI

TL;DR

RTS language-model systems can lag behind changing game states and remain heavily dependent on manually written experience prompts. STRATA divides in-game strategic, logistical, and tactical decisions among role-aligned agents and uses a Review Agent to validate and revise experience across matches for card retrieval. When the SA uses these cards, the observed win rate increases from 30% to 100%.

  • Problem

    RTS language-model systems can lag behind changing game states and remain heavily dependent on manually written experience prompts.

    Source: §1 · p. 1

  • Method

    STRATA divides in-game strategic, logistical, and tactical decisions among role-aligned agents and uses a Review Agent to validate and revise experience across matches for card retrieval.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    When the SA uses these cards, the observed win rate increases from 30% to 100%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sequential learning against Rush and Turtle opponents produces distinct long-term strategic preferences.

    Source: §5 · p. 7

    Experiments use one map, one fixed seed, one decision model, and a limited number of matches.

    Source: §5 · p. 7

Abstract

from arXiv · show

Real-time strategy (RTS) games require agents to coordinate economic development, production and construction, base defense, unit organization, and attack timing over long matches. Existing studies have applied large language models to command decision-making in RTS games, enabling agents to read textual game states and generate high-level plans. However, long inference latency can cause them to miss critical tactical events. The complexity and tactical diversity of full RTS matches also leave existing systems heavily dependent on manually written experience-based prompts, with limited ability to learn continuously from past games. We present STRATA, a role-aligned hierarchical system with cross-game self-learning for Red Alert. STRATA assigns in-game strategic, logistical, and tactical decisions to a Strategic Agent (SA), Logistics Agent (LA), and Tactical Agent (TA), respectively. The SA generates high-level directives based on the global game state and relevant experience cards, while the LA and TA handle logistics and tactical execution. After each match, a Review Agent (RA) derives candidate experience from game traces, validates and revises it using evidence from subsequent matches, and compresses strategic experience supported across multiple games into concise experience cards for SA retrieval. We evaluate STRATA through the formation of experience cards, full-match comparisons before and after learning, and experience learning against AI opponents with different play styles. Under a fixed scenario, using the learned experience cards increases the observed win rate from 30% to 100%. Sequential learning against AI opponents with different play styles also produces distinct long-term strategic experience.

1 Introduction

STRATA separates strategic, logistical, and tactical decisions during Red Alert matches and uses post-game review to build experience for later matches. Its fixed-scenario evaluation reports a higher observed win rate with learned cards and distinct strategic experience against different opponent styles.

1 Introduction

  • RTS agents must balance economy, technology, and military production while decisions have delayed effects and opponents change the conditions for earlier plans.Existing RTS systems can still lag behind game changes and rely heavily on manually written experience prompts; stored experience in full matches may need frequent revision.Source: §1 · pp. 1–2
  • The Strategic Agent sets priorities and issues directives, while the Logistics and Tactical Agents execute role-specific actions at different information scopes and update frequencies.This separation prevents low-frequency strategic inference from serially blocking logistics and tactical execution.Source: §1 · p. 2
  • After each match, the Review Agent proposes or revises experience, and guidance supported across multiple matches is compressed into cards for later retrieval.The evaluation examines card formation, before-and-after full-match performance, and sequential learning against Rush and Turtle opponents.Source: §1 · p. 2
  • Using cards from a formal experience library raises the observed win rate from 30% to 100%, and the system develops distinct long-term experience for AI opponents with different play styles.The formal experience library contains 16 cards.Source: §1 · p. 2

2 Related Work

Related work covers interactive language-model planning, game-agent hierarchies, RTS interfaces, and experience reuse. The paper identifies decision time-scale separation and continued correction of experience as open needs in full RTS matches.

LLM agents in interactive environments

  • Interactive language-model planning methods support decomposition and multi-step reasoning, but are mainly evaluated where environments wait for inference to finish.The cited methods include Chain-of-Thought, ReAct, Tree of Thoughts, and LATS.Source: §2 · pp. 2–3

Research on games

  • Game-agent hierarchies and role specialization demonstrate task allocation, but the cited work mainly addresses exploration, cooperation, or relatively stable interactions.Examples include MineDojo, GITM, ProAgent, and multi-agent role organizations.Source: §2 · p. 3

LLM agents for RTS games

  • RTS interfaces improve state summaries, action spaces, and rapid responses, yet input coverage and reasoning burden remain trade-offs, while many approaches retain manual rules or a core control loop.The paper frames separating strategy, logistics, and tactics across different information needs and rates of change as underexplored.Source: §2 · p. 3

Reflection, memory, and experience learning

  • Reflection and memory methods reuse experience, but limited trajectories and changing RTS conditions can make single-match conclusions require correction through later evidence.Full RTS strategic effects vary with opponent behavior, economy, game phase, and force composition.Source: §2 · p. 3

3 STRATA

STRATA separates strategic, logistical, and tactical decisions across agents with distinct roles and operating timescales. After matches, a Review Agent uses trace evidence to build and revise experience cards for later strategic decisions.

3.1 System Overview

  • The SA sets priorities and directives, while the LA handles logistics and the TA handles tactical execution using role-specific observations.The SA considers economy, technology, force accumulation, and attack timing; the LA manages construction, production, repair, and resource allocation.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 Role-Aligned Hierarchical In-Game Decision-Making

  • The three agents receive role-specific observations and have bounded decision authority, separating global strategy, logistics, and local combat.The SA uses global strategic information; the LA sees economic and production details; the TA receives assigned combat units and local opponent information.Source: §3.2 · pp. 3–4
  • Agents operate asynchronously at different timescales, so logistics and tactical execution can continue while the SA's lower-frequency strategic inference is pending.Previous valid outputs remain active until updated results arrive.Source: §3.2 · p. 4

3.3 Review-Agent-Driven Cross-Game Experience Learning

  • Experience cards encode the situation, possible strategic directions and their preference strengths, plus a directive example, weight, and confidence.The SA combines retrieved preferences with the current game state to select priorities.Source: §3.3 · pp. 4–5
  • The RA adds new themes to a candidate pool, accumulates support or counterexamples across matches, promotes sufficiently supported candidates, and revises existing cards when evidence warrants.Later outcomes and observed side effects can change cards' applicability, preferences, weight, and confidence.Source: §3.3 · p. 5

3.1 System Overview

  • The Review Agent turns evidence from completed matches into experience cards that the SA can retrieve in subsequent matches.The cards transfer strategic findings from sequential matches into later decisions.Source: §3.1 · p. 3, §3.3 · p. 5

4 Experiments

In Red Alert, STRATA formed experience cards through sequential matches, then compared fixed-scenario performance with and without those cards and examined learning against Rush and Turtle opponents. The observed win rate rose from 30% to 100%, while learning under different play styles produced distinct strategic preferences.

4.1 Experimental Setup

  • The experiments used GLM-5.1 in OpenRA’s Red Alert mod, with 80 learning matches and separate before-and-after evaluations against Normal AI.Rush and Turtle each ran ten sequential matches from the same 16-card starting library, with separate memory spaces and reviews after every match.Source: §4.1 · pp. 5–6

4.2 Experience-Card Formation

  • Early reviews identified recurring production, economy, technology, and defense problems; seven candidates later gained cross-game evidence and entered the formal library.The initial ten cards were tested in more complex situations; overly broad rules were removed or replaced as candidate experience accumulated.Source: §4.2 · p. 6
  • By the end of the 80-match sequence, the library held 16 cards and had undergone 14 revisions across nine cards.Revisions adjusted applicability and preferences, including production batches, cash and queue conditions, low-power technology delays, and rally or attack conditions.Source: §4.2 · p. 6
  • After 80 matches against Normal AI, STRATA formed 16 formal cards covering economy, production, technology, defense, and offense.Later updates mainly refined card applicability conditions and strategic preferences.Source: §4.2 · p. 6

4.3 Full-Match Performance Before and After Learning

  • 30% to 100%: against Normal AI, the observed win rate increased after learning with experience cards.The comparison used the same environment configuration and fixed experience state; card access was the only difference.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Experience Learning under Different Play Styles

  • Rush exposed cash shortages amid production and technology demands; Turtle exposed queue congestion, idle production, missed expansion, and late-game force gaps.Each pattern occurred in multiple matches, including cash shortage in 7 of 10 Rush matches and vehicle-queue congestion with idle infantry production in 7 of 10 Turtle matches.Source: §4.4 · p. 6
  • Three of the 16 formal cards developed different versions under Rush and Turtle conditions, and single-observation candidates remained unpromoted.The candidate pool retains single-match proposals until cross-game evidence supports promotion.Source: §4.4 · pp. 6–7, Table 3 · p. 7
  • Rush more often retrieved recovery, short-batch defense, and basic-unit cards; Turtle more often retrieved post-technology advances and large-force attack cards.The learned preferences align with continued production and recovery under a thin economy against Rush, and expansion, resource conversion, and proactive attacks against Turtle.Source: §4.4 · p. 7, Figure 2 · p. 7

5 Conclusion and Limitations

STRATA combines role-aligned in-game agents with a review agent that validates and revises experience cards from match traces. In Red Alert, learned cards raised the observed win rate from 30% to 100% in the fixed scenario, while experiments were limited in scope.

5 Conclusion and Limitations

  • In Red Alert, STRATA formed 16 cards over 80 Normal AI matches; using them raised the observed win rate from 30% to 100% in the fixed scenario.Sequential learning against Rush and Turtle opponents produced distinct long-term strategic preferences.Source: §5 · p. 7
  • Experiments used one map, one fixed seed, one decision model, and a limited number of matches.Future work will evaluate real-time performance and generalization across alternative role organizations, decision frequencies, and experience-update strategies.Source: §5 · p. 7

Related papers

Loading 2609.38881v1…