Source-linked AI summary
Grounding Natural Language Commands to StarCraft II Game States for Narration-Guided Reinforcement Learning
Nicholas Waytowich, Sean L. Barton, Vernon Lawhern, Ethan Stump, Garrett Warnell
TL;DR
Sparse rewards make long-horizon reinforcement-learning tasks such as StarCraft II challenging, while many reward-shaping approaches require specialized human knowledge. This paper trains a mutual-embedding model to ground natural-language commands in corresponding game states, and reports that the learned representations distinguish goal states and align commands with them.
Problem
Sparse reward in long-horizon tasks such as StarCraft II provides agents with little meaningful feedback, while many reward-shaping approaches require specialized human knowledge.
Source: §1 · pp. 1–2Method
The authors train a multi-input mutual-embedding model to project natural-language commands and corresponding goal states into a shared representation space.
Source: §1 · p. 2, §3.2.1 · p. 4Results
The model learned distinct representations for goal states and projected corresponding natural-language commands onto their manifolds.
Source: §4.1 · p. 8, §5.1 · p. 8Takeaways & Limitations
The results support grounding natural-language goals in StarCraft II game states, while the paper discusses this as a potential mechanism for reward shaping.
Source: §5.1 · p. 8, §5.2 · p. 9
Abstract
from arXiv · showhide
While deep reinforcement learning techniques have led to agents that are successfully able to learn to perform a number of tasks that had been previously unlearnable, these techniques are still susceptible to the longstanding problem of {\em reward sparsity}. This is especially true for tasks such as training an agent to play StarCraft II, a real-time strategy game where reward is only given at the end of a game which is usually very long. While this problem can be addressed through reward shaping, such approaches typically require a human expert with specialized knowledge. Inspired by the vision of enabling reward shaping through the more-accessible paradigm of natural-language narration, we investigate to what extent we can contextualize these narrations by grounding them to the goal-specific states. We present a mutual-embedding model using a multi-input deep-neural network that projects a sequence of natural language commands into the same high-dimensional representation space as corresponding goal states. We show that using this model we can learn an embedding space with separable and distinct clusters that accurately maps natural-language commands to corresponding game states . We also discuss how this model can allow for the use of narrations as a robust form of reward shaping to improve RL performance and efficiency.
1. INTRODUCTION
Sparse rewards leave agents with little meaningful feedback, a challenge exemplified by long-horizon StarCraft II. The paper explores grounding natural-language guidance in goal-specific game states as a more accessible route to reward shaping.
1. INTRODUCTION
- Sparse reward functions provide meaningful feedback in few states, making it harder for an agent to learn, as illustrated by StarCraft II’s long time horizons.A win-only reward can leave an agent taking random actions for considerable time before it receives feedback.Source: §1 · p. 1
- Many reward-shaping approaches require people to understand the agent’s programming and have the knowledge and access needed to modify it.The paper motivates shaping methods accessible to people without that specialized knowledge.Source: §1 · p. 2
- The paper proposes narration as a more accessible way to shape reward: people provide natural-language commands suggesting how the agent should accomplish its task.The commands are intended to convey suggestions without modifying the agent’s source code.Source: §1 · p. 2
- The authors train a multi-input mutual-embedding model to project natural-language commands and corresponding goal-specific StarCraft II states into the same representation space.They investigate whether commands can be grounded in game states and discuss how this approach could support reward shaping for reinforcement learning.Source: §1 · p. 2
2. RELATED WORK
The related work frames sparse reward as a challenge for reinforcement learning and identifies reward shaping as one possible response. It reviews natural-language guidance across domains, while positioning this work in the StarCraft II setting.
2.1 Challenges for Reinforcement Learning
- Sparse reward can make policy learning difficult in long-horizon tasks, and reward shaping modifies environmental rewards to provide more frequent signals.Prior approaches include demonstrations, binary feedback, and natural-language guidance.Source: §2.1 · p. 2
2.2 Narration-Guided Reinforcement Learning
- Prior narration-guided work spans text games, robotics, language-conditioned rewards, narrated demonstrations, and Atari; the cited Atari framework outperformed RL approaches without reward shaping.The paper focuses on natural-language guidance for StarCraft II.Source: §2.2 · p. 2
3. METHODS AND APPROACH
The approach studies BuildMarines, a StarCraft II mini-game with sequential construction and training goals, and proposes grounding natural-language commands in its game states. A mutual-embedding model learns shared representations from labeled state-command pairs so commands can indicate interim goals.
3.1 StarCraft II: BuildMarines Mini-game
- BuildMarines requires an agent to build workers, gather resources, construct supply depots and barracks, and train marines; rewards arrive when marines are built.The authors identify the mini-game’s sparse reward as a challenge and reduce the action space to actions sufficient for the task.Source: §3.1 · p. 3
3.1.1 State and Action Spaces for BuildMarines Task
- The SC2LE state combines 7 64x64 mini-map layers, 13 64x64 screen layers, and 13 non-spatial features; compound actions include an action identifier and two spatial vectors.The state also includes information such as resources and build queues.Source: §3.1.1 · p. 3
3.2 Grounding Natural Language Commands to Task States and Goals
- The study proposes grounding commands in task states to support narration-guided interim rewards when environmental rewards are sparse.The stated goal is to help agents avoid taking many random actions before task completion.Source: §3.2 · p. 3
3.2.1 The Mutual-embedding Model
- The mutual-embedding model learns a shared representation of game states and commands that correspond to interim goals, giving current states contextual meaning.The command set is identified in Figure 1, whose caption describes separate state and language inputs projected into a shared space.Source: §3.2.1 · p. 3, Figure 1 · p. 4
- The model is trained to bring matching state-command pairs closer and mismatched pairs farther apart in the shared space.The dataset uses hand-crafted rules to label states generated by a random agent, with matched and mismatched pairs and null states; training uses validation to select when to stop.Source: §3.2.1 · pp. 4, 6
- Language commands are encoded from word-level word2vec embeddings through an LSTM, while game-state features are processed from mini-map, screen, and non-spatial inputs into a 256-length embedding.The state pathway uses convolutional layers for screen and mini-map inputs and a fully connected layer for non-spatial features; training uses a two-frame temporal stack.Source: §3.2.1 · pp. 4–5
4. EVALUATION OF THE MUTUAL EMBEDDING MODEL
The mutual-embedding model achieved 80.40% test accuracy on command-state association. Its learned representation separated goal-state examples and projected corresponding commands onto the same manifolds, though some similar goal states remained difficult to distinguish.
4. EVALUATION OF THE MUTUAL EMBEDDING MODEL
- 80.40% test accuracy was achieved using an embedding-distance threshold of 0.5 to associate command and game-state embeddings.Training accuracy was 95.61% and validation accuracy was 82.35%; the model was evaluated at minimum validation loss.Source: §4 · p. 6
4.1 Representation
- t-SNE showed distinct representation groups for different goal states, with command embeddings projecting onto manifolds corresponding to their intended goals.Some classification errors remained near the boundary between building a barracks and building a supply depot, which are similar game events.Source: §4.1 · pp. 7–8, Figure 3 · p. 7
5. DISCUSSION
The mutual-embedding model places natural-language commands in the shared representation space of StarCraft II states, aligning commands with their corresponding game situations. The discussion presents this grounding as a possible basis for flexible, language-guided reward shaping in reinforcement learning.
5.1 Natural Language as Representation
- Commands for desirable sub-goals occupy the same representation space as game states and correspond to their paired situations.The discussion describes this alignment as grounding language goals in the agent’s state space.Source: §5.1 · p. 8, §5 · p. 8
5.2 Implications for Reinforcement Learning
- Pang et al. [26] combined hierarchical policies, hand-crafted reward, and curriculum learning, reporting a 93% win rate against the expert game AI.Their reported setup used 48 cores and 8 Tesla K40 GPUs for two days.Source: §5.2 · p. 8
- The authors present the model as a promising mechanism for natural-language reward in StarCraft II that could avoid expert-dependent hand-crafted sub-tasks or checkpoints.They describe this as their first attempt to deploy a natural-language-based reward system in StarCraft II, to their knowledge.Source: §5.2 · p. 9
- Semantically similar commands, such as “build” and “construct,” are expected to indicate similar sub-goals even when a command has not previously been observed or learned.The authors suggest this flexibility may make RL easier to guide without domain-specific expertise.Source: §5.2 · p. 9
- State-conditioned goals let an agent compare its current state directly with a desired state, unlike hand-tuned sub-task features that may not be represented in its neural network.The authors describe this as a more reliable means of communicating sequential tasks, and suggest it could improve RL performance and efficiency.Source: §5.2 · p. 9