PaperlayerView exact revision on arXiv

Source-linked AI summary

StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning

Kun Shao, Yuanheng Zhu, Dongbin Zhao

arXiv:1804.00810v1cs.AIcs.LGcs.MA

TL;DR

StarCraft micromanagement requires efficient multi-agent control in large, complex state spaces and varied combat scenarios. The paper combines compact state representation, shared-policy PS-MAGDS with reward shaping, and curriculum transfer learning. The method defeats built-in AI in small scenarios and outperforms some baselines in large target scenarios.

  • Problem

    StarCraft micromanagement poses large-state-space, multi-agent coordination, and training-efficiency challenges across scenarios.

  • Method

    The paper combines efficient state representation, parameter-sharing multi-agent gradient-descent Sarsa(λ), reward shaping, and curriculum transfer learning.

  • Results

    100% win rates are achieved in small-scale combats, while the method shows superior performance over some baselines in two large-scale target scenarios.

  • Takeaways & Limitations

    The approach learns effective coordination strategies and defeats built-in AI across the reported StarCraft micromanagement scenarios.

Abstract

from arXiv · show

Real-time strategy games have been an important field of game artificial intelligence in recent years. This paper presents a reinforcement learning and curriculum transfer learning method to control multiple units in StarCraft micromanagement. We define an efficient state representation, which breaks down the complexity caused by the large state space in the game environment. Then a parameter sharing multi-agent gradientdescent Sarsa(λ) (PS-MAGDS) algorithm is proposed to train the units. The learning policy is shared among our units to encourage cooperative behaviors. We use a neural network as a function approximator to estimate the action-value function, and propose a reward function to help units balance their move and attack. In addition, a transfer learning method is used to extend our model to more difficult scenarios, which accelerates the training process and improves the learning performance. In small scale scenarios, our units successfully learn to combat and defeat the built-in AI with 100% win rates. In large scale scenarios, curriculum transfer learning method is used to progressively train a group of units, and shows superior performance over some baseline methods in target scenarios. With reinforcement learning and curriculum transfer learning, our units are able to learn appropriate strategies in StarCraft micromanagement scenarios.

I. INTRODUCTION

StarCraft micromanagement is a demanding multi-agent control setting with complex state spaces, cooperation requirements, and long model-free RL training times. The paper addresses these challenges with efficient representation, shared-policy learning, reward shaping, and curriculum transfer.

  • Motivation: StarCraft offers a real-time environment for studying control of multiple units across varying difficulty levels.The game involves dynamic tactics, reactive control, and challenges including multi-agent collaboration and adversarial planning.
  • Challenges: Traditional methods struggle with complicated state and action spaces and cooperative tactics in StarCraft micromanagement.Model-free RL methods also require substantial training time, especially in large-scale scenarios.
  • Contributions: The proposed state representation uses unit attributes and distances while allowing an arbitrary number of units on both sides.The representation is intended to reduce the complexity of the large state space and remain concise and efficient.
  • Contributions: PS-MAGDS shares a centralized policy among homogeneous agents, which update it simultaneously from their own experiences.A neural network approximates action values, while intermediate rewards address sparse and delayed feedback.
  • Contributions: Curriculum transfer learning extends the model to different scenarios and accelerates training compared with learning from scratch.The method progressively transfers knowledge across increasingly difficult tasks.

II. PROBLEM FORMULATION AND BACKGROUNDS

The paper formulates micromanagement as a multi-agent reinforcement-learning problem in which units act from individual observations while sharing a cooperative policy. It reviews RL, temporal-difference learning, deep function approximation, multi-agent RL, and curriculum transfer learning.

  • A. Problem Formulation: StarCraft combat is modeled as a Markov game with shared states, agent-specific actions and observations, transitions, and rewards.Each unit interacts using its own observation and action while the environment reflects the joint actions of all units.
  • A. Problem Formulation: Units share a policy to promote cooperation while each unit maximizes its expected cumulative return.The shared policy supports an arbitrary number of units while preserving individual observations and actions.
  • B. Reinforcement Learning: Reinforcement learning models an agent selecting actions from states, receiving rewards, and transitioning to new states through environmental interaction.The objective is to learn a policy that maximizes discounted future rewards.
  • B. Reinforcement Learning: Temporal-difference learning updates value estimates from partial experience without requiring an environment model or waiting for a final outcome.Q-learning and Sarsa are identified as the main TD algorithms discussed.
  • B. Reinforcement Learning: Q-learning is off-policy, whereas Sarsa is on-policy because the action-selection policy also determines updates.The distinction concerns whether the learning policy differs from the behavior policy.
  • B. Reinforcement Learning: Deep neural networks provide function approximation for end-to-end learning in high-dimensional state spaces.Deep Q-networks use experience replay and target networks to reduce sample correlations and stabilize training.
  • B. Reinforcement Learning: Curriculum transfer learning trains on progressively harder tasks, transferring knowledge from a source scenario toward target scenarios.The approach is motivated by the high sample requirements of model-free RL across varied micromanagement settings.
  • B. Reinforcement Learning: Multi-agent reinforcement learning studies multiple agents interacting in one environment, including cooperative and competitive behaviors.The paper uses policy sharing among agents to learn cooperative behaviors.

C. Curriculum Transfer Learning

The paper combines transfer learning with a compact, unit-count-independent state representation for StarCraft micromanagement. Curriculum tasks progressively increase difficulty, while the model encodes current and previous combat information and action history.

  • Curriculum Transfer Learning: Transfer learning initializes target-scenario training with a model learned in a source scenario.This transfers knowledge across related tasks using the same model architecture.
  • Curriculum Transfer Learning: Curriculum learning organizes tasks by increasing difficulty so earlier tasks guide performance on the final task.In micromanagement, difficulty can be changed through the number and type of units.
  • A. Representation of High-Dimension State: The proposed state representation is efficient and independent of the number of units in combat.It combines current-step information, previous-step information, and the previous action.
  • A. Representation of High-Dimension State: Current and previous state information include weapon cooldown, hitpoints, own-unit distances, enemy-unit distances, and terrain distances.Distance features are computed across eight sector areas using aggregate own- and enemy-unit statistics.
  • A. Representation of High-Dimension State: Out-of-sight unit distances are assigned a fixed value, while visible distances vary linearly with distance.Terrain distances use a separate out-of-sight value and likewise depend linearly on distance when visible.
  • A. Representation of High-Dimension State: The learning model uses the three-part state representation as neural-network input and outputs probabilities for eight movement directions and attack.The previous action is included as part of the representation, and the output layer has nine action neurons.

B. Action Definition

The paper simplifies StarCraft’s large action space by restricting each unit to fixed-distance movement in eight directions or attacking the weakest enemy. This produces a discrete set of practical actions for learning.

  • B. Action Definition: Attack actions target the weakest enemy rather than allowing arbitrary enemy selection within weapon range.This restriction simplifies the original action space, which permits arbitrary movement distances and attack targets.
  • B. Action Definition: Each unit can move in eight compass and diagonal directions by a fixed distance.The directions are Up, Down, Left, Right, Upper-left, Upper-right, Lower-left, and Lower-right.

C. Network Architecture

The model uses a neural network to approximate state-action values from a compact state representation, while PS-MAGDS shares policy parameters across units for multi-agent learning.

  • A neural network parameterized by θ approximates state-action values to improve generalization across the large StarCraft state space.
  • The network receives a 93-dimensional state tensor, uses 100 hidden neurons with ReLU activation, and outputs nine action probabilities.The nine actions represent movement in eight directions and attack.
  • PS-MAGDS extends Sarsa(λ) to multiple units by sharing the policy network parameters among agents.Shared parameters support a single learned policy while units can still behave differently from their individual observations and actions.
  • Eligibility traces assign credit across previously experienced transitions, helping address delayed rewards during multi-unit combat.The trace factor λ controls the weighting of backups made after multiple steps.
  • The gradient-descent update trains the Sarsa(λ) policy network, while ε-greedy action selection balances exploitation with random exploration.The implementation chooses the current best action with probability 1 − ε and a random action with probability ε.

B. Reward Function

The reward function addresses sparse and delayed combat feedback by combining attack outcomes with intermediate penalties and movement rewards that encourage cooperation.

  • The final combat objective alone produces sparse, delayed rewards, making it difficult to associate actions with later outcomes.
  • Intermediate attack rewards equal enemy damage received minus hitpoint loss suffered by the agent’s units.All agents receive this main attack reward at each time step.
  • The reward is normalized using a factor ρ to balance the total hitpoints of friendly and enemy units across differing scenarios.The paper states that normalization is necessary when unit numbers and types vary.
  • Destroyed units receive an extra negative reward of -10 to discourage losses that harm combat results.
  • Additional movement rewards are introduced to encourage units to work as a team and make cooperative actions.

C. Frame Skip

Frame skip makes action selection practical for real-time micromanagement by executing training steps less frequently than every game frame.

  • Frame skip executes a training step every fixed number of frames, avoiding the impracticality of acting on every game frame.
  • Small frame skips can create strong training-data correlations, whereas large frame skips reduce the number of effective training samples.
  • The experiments tested frame skips of 8, 10, and 12 in a small-scale scenario before selecting 10.
  • With frame skip set to 10, each unit takes an action every 10 frames.

V. EXPERIMENT SETTINGS

The experiments evaluate StarCraft micromanagement across two small-scale and one large-scale combat scenario against the built-in AI, using curriculum transfer for scaling.

  • The experiments cover Goliaths versus Zealots, Goliaths versus Zerglings, and Marines versus Zerglings.These scenarios are represented in Fig. 5.
  • The first scenario controls 3 Goliaths against 6 Zealots, with friendly units disadvantaged in count, hitpoints, and damage factor but advantaged in fire range.
  • The second scenario pits Goliaths against 20 Zerglings, while the third controls up to 20 Marines against 30 Zerglings.The scenarios differ in unit advantages involving hitpoints, damage, range, speed, quantity, and cooldown time.
  • The first two scenarios are small-scale and the last is large-scale; enemies are controlled by the built-in AI and episodes end when either side is destroyed.
  • Training uses γ = 0.9, α = 0.001, λ = 0.8, and a maximum of 1000 steps per episode across all scenarios.
  • The first scenario is used as the starting point, while transfer learning scales training to the remaining scenarios, including the large-scale setting.

A. Small Scale Micromanagement

Small-scale training begins with Goliaths facing different enemy configurations and develops combat competence through reinforcement learning. The units ultimately achieve 100% wins while learning to balance movement and attack.

  • The small-scale experiments train Goliaths against enemy units with different amounts and types.
  • Goliaths vs. Zealots: 100% win rates are reached after 3000 episodes in the 3 Goliaths versus 6 Zealots scenario.Win rates are evaluated every 200 training episodes using 100 combats.
  • Goliaths vs. Zealots: Episode steps initially remain low, increase as Goliaths learn to avoid damaging attacks, and later decline as combat behavior improves.
  • Goliaths vs. Zealots: The trained Goliaths destroy enemies in almost 300 steps after learning an appropriate movement-and-attack policy.
  • Goliaths vs. Zealots: Average rewards increase during training and become smooth after almost 3000 episodes.Average reward is computed by dividing total combat reward by episode steps.

2) Goliaths vs. Zerglings:

Transfer learning initializes Goliaths against Zerglings from a model trained against Zealots, while the broader evaluation extends to curriculum-trained Marines. Transfer accelerates learning, and PS-MAGDS performs strongly against large-scale baselines.

  • Goliaths vs. Zerglings: The Goliaths-versus-Zerglings model reuses the well-trained first-scenario policy network rather than learning entirely from scratch.
  • Goliaths vs. Zerglings: 100% win rates are reached with transfer learning, whereas scratch training does not produce a win until 1800 episodes and remains below 60% after 4000 episodes.
  • Goliaths vs. Zerglings: With transfer learning, average episode steps remain between 200 and 400 throughout training.The authors suggest previously learned movement and attack skills explain this stable range.
  • Goliaths vs. Zerglings: Average rewards with transfer learning are higher from the beginning and perform better throughout training than rewards from scratch training.
  • Marines vs. Zerglings: 97% win rate is achieved by PS-MAGDS in M10 vs. Z13, exceeding the listed rule-based and DRL baselines.In M20 vs. Z30, PS-MAGDS has the second-best performance and is close to the best method.
  • Marines vs. Zerglings: PS-MAGDS performs outstandingly in curricular scenarios and acceptably in unseen scenarios with more units.

C. Strategies Analysis

The learned strategies adapt to the scale and composition of combat. Goliaths disperse and focus fire against numerically stronger opponents, while Marines coordinate as a unified team against Zerglings.

  • Rule-based control must handle many conditions, whereas reinforcement learning and curriculum transfer learning produce useful strategies in these complex combats.
  • Disperse Enemies: Against larger Zealot forces, Goliaths disperse enemies and destroy them one group at a time instead of fighting face-to-face.
  • Disperse Enemies: After isolating Zealots, winning Goliaths regroup and the units focus fire on the remaining enemies.
  • Disperse Enemies: Against Zergling Rush, Goliaths separate enemies into groups, maintain suitable distance, and attack when their weapons are off cooldown.
  • Team Coordination: Because Marines have low hitpoints and cannot resist enemies in small groups, they move and attack together as a coordinated team.
  • Team Coordination: The Marines learn to move forward and retreat in a queue while targeting the same enemy.

3) Hit and Run:

Hit and Run emerges as the paper’s most widely used learned local tactic, appearing across small- and large-scale micromanagement scenarios. However, learned tactics coexist with coordination and positioning problems during combat.

  • 3) Hit and Run:: Hit and Run is the most widely used local tactic, learned rapidly across all evaluated scenarios.The paper reports both single-unit and group-level uses of the tactic.
  • 3) Hit and Run:: The tactic appears in single-unit replays for 3 Goliaths versus 6 Zealots and 20 Zerglings.These examples correspond to Figs. 12 and 13.
  • 3) Hit and Run:: A group of units uses Hit and Run in the 20 Marines versus 30 Zerglings scenario.The paper identifies this as the group-level example in Fig. 14.
  • 4) Existing Problems:: Despite these learned tactics, Goliaths sometimes fail to join combats, while units move toward map boundaries to avoid enemies.These behaviors are described as remaining combat problems after training.
Loading 1804.00810v1…