Source-linked AI summary
AlphaStar: An Evolutionary Computation Perspective
Kai Arulkumaran, Antoine Cully, Julian Togelius
TL;DR
The paper addresses how AlphaStar’s complex AI system can be understood through evolutionary computation. It analyzes the system using Lamarckian evolution, competitive co-evolution, and quality diversity, arguing that this perspective connects evolutionary computation with deep reinforcement learning and supports further work on AlphaStar.
Problem
AlphaStar draws on multiple AI fields, but its evolutionary-computation perspective is described as hitherto undersold.
Method
The paper analyzes AlphaStar through evolutionary-computation concepts including memetic algorithms, Lamarckian evolution, competitive co-evolution, and quality diversity.
Results
The analysis identifies AlphaStar as combining Lamarckian evolution, competitive co-evolution, and quality diversity within a system that also uses deep learning, reinforcement learning, and game theory.
Takeaways & Limitations
This perspective may help evolutionary-computation and deep reinforcement-learning communities better appreciate and build upon AlphaStar.
Takeaways & Limitations
The paper presents a high-level overview and leaves aside many deep links between evolutionary computation and game theory.
Abstract
from arXiv · showhide
In January 2019, DeepMind revealed AlphaStar to the world-the first artificial intelligence (AI) system to beat a professional player at the game of StarCraft II-representing a milestone in the progress of AI. AlphaStar draws on many areas of AI research, including deep learning, reinforcement learning, game theory, and evolutionary computation (EC). In this paper we analyze AlphaStar primarily through the lens of EC, presenting a new look at the system and relating it to many concepts in the field. We highlight some of its most interesting aspects-the use of Lamarckian evolution, competitive co-evolution, and quality diversity. In doing so, we hope to provide a bridge between the wider EC community and one of the most significant AI systems developed in recent times.
1 BACKGROUND
Games provide human-relevant benchmarks for AI, and StarCraft II presents an especially difficult challenge because it combines real-time play, partial observability, complex rules, and a large action space. AlphaStar addressed this challenge by combining imitation learning, reinforcement learning, self-play, and population-based training.
- Games serve as AI benchmarks because they are created by humans for humans, giving them external validity.
- StarCraft II is unusually challenging because it requires real-time play under partial observability, lacks a single dominant strategy, and has complex rules and a large action space.
- AlphaStar beat a professional StarCraft II player in December 2018 after imitation learning, reinforcement learning, self-play, and population-based training.
2 COMPONENTS
The paper interprets AlphaStar through three evolutionary-computation components: Lamarckian population-based training, competitive co-evolution, and quality diversity. Together, these mechanisms combine neural-network optimization with populations of diverse agents evaluated against one another.
- 2.1 Lamarckian evolution: Population-based training is a memetic algorithm using Lamarckian evolution: backpropagation trains networks, while selection replaces losers’ parameters and mutates their hyperparameters.
- 2.1 Lamarckian evolution: PBT is asynchronous, distributed, and steady-state, allowing individual optimization and evaluation to proceed continuously while preserving elite and non-elite diversity.
- 2.2 Co-evolution: Competitive co-evolution extends self-play by retaining and evaluating an entire population, providing a natural curriculum and robustness against varied opponents.
- 2.2 Co-evolution: AlphaStar’s co-evolutionary training combined population-based training, backpropagation-based deep reinforcement learning, and evolved reward functions.
- 2.3 Quality diversity: Quality diversity optimizes solution quality while encouraging varied solution types through behavior descriptors, and AlphaStar uses diverse strategies from a Nash distribution because StarCraft has no best strategy.
- 2.3 Quality diversity: AlphaStar adapts game-specific diversity criteria online, including criteria based on beating particular agents or sets of agents.
3 DISCUSSION
The paper argues that AlphaStar is best understood partly as an evolutionary algorithm, combining Lamarckian evolution, competitive co-evolution, and quality diversity. This perspective connects evolutionary computation with deep reinforcement learning and may help both communities build on AlphaStar.
- AlphaStar combines Lamarckian evolution, competitive co-evolution, and quality diversity as an evolutionary-computation perspective on the system.