Source-linked AI summary
Beating Atari with Natural Language Guided Reinforcement Learning
Russell Kaplan, Christopher Sauer, Alexander Sosa
TL;DR
Sparse rewards make Montezuma’s Revenge difficult for reinforcement-learning agents, and natural-language guidance had not previously been tried for this problem. The paper trains an agent to match instructions with observations and reward instruction completion. The authors report that it wins comparisons against other algorithms and OpenAI Gym leaderboard agents.
Problem
Sparse-reward environments such as Montezuma’s Revenge remain difficult for reinforcement-learning agents, and natural-language guidance had not previously been tried for this problem.
Source: §2 · p. 3, §3.5 · p. 7Method
The agent learns instruction–observation matches, then uses a pretrained embedding to detect completed commands, provide additional reward, and supply features to reinforcement learning.
Source: §1 · p. 2, §3.8 · p. 8Results
The authors report their agent as the clear winner in comparisons with other reinforcement-learning algorithms and OpenAI Gym leaderboard agents.
Source: §3.9 · p. 9Takeaways & Limitations
The authors present natural-language instruction as a promising start to cooperation between reinforcement-learning agents and human trainers.
Source: §4 · p. 9The comparisons are not apples-to-apples because this agent receives natural-language instructions in addition to visible frames.
Source: §3.9 · p. 9
Abstract
from arXiv · showhide
We introduce the first deep reinforcement learning agent that learns to beat Atari games with the aid of natural language instructions. The agent uses a multimodal embedding between environment observations and natural language to self-monitor progress through a list of English instructions, granting itself reward for completing instructions in addition to increasing the game score. Our agent significantly outperforms Deep Q-Networks (DQNs), Asynchronous Advantage Actor-Critic (A3C) agents, and the best agents posted to OpenAI Gym on what is often considered the hardest Atari 2600 environment: Montezuma's Revenge.
1 Introduction
The introduction frames natural-language guidance as a needed capability for agents acting in stateful worlds, where human instruction can specify goals at a high level. The paper proposes a two-stage agent that learns command meanings and then learns actions for carrying them out.
1 Introduction
- Agents need to follow convenient high-level human instructions while acting in stateful worlds.The authors contrast such instructions with low-level, completely specified program instructions.Source: §1 · p. 2
- The paper aims to let an agent learn from high-level English instruction while acting in Atari games.Source: §1 · p. 2
- A two-stage approach first maps English commands to game observations, then learns actions that satisfy commands.The authors describe the stages as learning command meaning and filling in the implementation of instructions.Source: §1 · p. 2
2 Background
The background identifies sparse rewards as a persistent challenge, especially in Montezuma’s Revenge, and reviews reinforcement-learning approaches that learn action values or action distributions. It motivates language guidance as an untried way to address the challenge.
2 Background
- Montezuma’s Revenge has sparse rewards, and DQN and A3C struggle to learn in it.DQN also scores no points after 200 hours, performing no better than a random agent.Source: Figure 2 · p. 2, §2 · p. 3
- The authors note that natural-language instructions had not previously been tried as reinforcement-learning guidance for sparse-reward environments.Prior efforts discussed in the passage added rewards or model capabilities.Source: §2 · p. 3
2.1 Approaches to Reinforcement Learning
- Reinforcement learning repeatedly maps observations to actions, receives updated states and possible rewards, and seeks greater time-discounted reward.The cycle continues until a termination condition is met.Source: §2.1 · p. 3
2.1.1 Deep Q-Learning: The Action-Value Formulation
- Deep Q-learning estimates discounted value for each possible action and selects the highest-valued action at test time.Its Q function is a convolutional network trained on saved histories of observed values.Source: §2.1.1 · p. 3
2.1.2 Policy Iteration and Asynchronous Advantage Actor-Critic: The Action-Distribution Formulation
- A3C learns an action-distribution policy from parallel games and subtracts estimated state value in updates, supporting faster convergence [9] [13].Policy networks directly map states to action distributions rather than estimating each action’s value.Source: §2.1.2 · p. 4
3 Approach and Experiments
The approach combines a multimodal instruction–frame embedding with reinforcement learning, using instruction completion to provide additional reward and features. For its Breakout baseline, the authors report that A3C converged to a perfect score within 30 million frames.
3.1 Overview of Approach
- The agent combines recent game frames and a current instruction with policy/value networks and an embedding that detects instruction completion.Completing an instruction advances the command list and grants additional reward; frame and sentence embeddings also enter the policy/value networks as features.Source: Figure 4 · p. 4
3.2 Baseline Models
- After 30 hours, the authors’ DQN baseline had not converged to an agent that could beat Breakout.Source: §3.2 · p. 4
- A3C converged to a perfect Breakout score within 30 million frames of overnight training.Source: §3.2 · p. 5
3.3 Viability of Sub-Task Rewards
In Breakout, additional template-matched rewards improved early learning, but the advantage diminished as the game progressed. The authors also tested multimodal embeddings for matching language to frames, finding that several language-processing strategies achieved roughly 4% validation error while simple bag-of-words barely beat priors.
3.3 Viability of Sub-Task Rewards
- More than 120 points after 12,000 iterations—over three times ordinary A3C—followed additional rewards for two hard-coded Breakout instructions.The rewards covered keeping the paddle under the ball and losing a life; each iteration played 128 frames.Source: §3.3 · p. 5
- The added Breakout reward later converged with baseline A3C at 36,000 iterations and performed worse after 100,000.The authors attribute this to the paddle-under-ball strategy becoming less useful in later rounds and dense rewards leaving limited room for improvement.Source: §3.3 · p. 5
3.4 Experiments in Multimodal Embeddings
- The authors generated several thousand Breakout frames with template-matched relational descriptions to train the multimodal embedding.Example descriptions distinguished whether the ball or paddle was to the right of the other.Source: §3.4 · p. 6
- The embedding maps sequential frame pairs and candidate sentences into same-sized vectors, training their dot product to distinguish matching from nonmatching descriptions.Dot products allow multiple sentences to match one frame pair without the normalization penalty associated with cosine similarity.Source: §3.4 · pp. 6–7
- Roughly 4% validation error was achieved by LSTM, GRU, bidirectional variants, and unique-word BOW, while ordinary BOW barely beat priors.The authors explain that ordinary BOW loses word order, which matters for relational instructions such as ball-right-of-paddle versus paddle-right-of-ball.Source: Figure 6 · p. 6
3.5 MONTEZUMA’S REVENGE
Montezuma’s Revenge combines sparse rewards with hazards that can kill the agent before it reaches the key, making learning difficult for standard reinforcement-learning agents. Baseline A3C scored zero, while manually specified location rewards helped the agent escape the first room.
3.5 MONTEZUMA’S REVENGE
- Sparse rewards, lethal hazards, and the need to reach the key make Montezuma’s Revenge difficult for DQN and A3C agents.The agent receives no reward for following the seven listed instructions; only reaching the key yields reward, and falling or touching the skull causes death.Source: §3.5 · p. 7
- After 72 hours and more than 200,000,000 observed frames, baseline A3C consistently scored 0 in Montezuma’s Revenge.Its chosen actions tended to result in near-immediate death.Source: §3.5 · p. 7
- Manual location rewards produced a mean score of 400 and consistently helped the agent escape the first room before exhausting its instructions.The result suggested that informative auxiliary reward helped the agent progress farther.Source: §3.5 · p. 7
3.6 Dataset Generation
Because no dataset mapped natural-language descriptions of game states to pixels, the authors created one from their own playthroughs of the game.
3.6 Dataset Generation
- The authors created mappings between game states and natural-language descriptions because no such dataset existed.They played through the game several times and saved frames from each run.Source: §3.6 · p. 7
- Template matching generated lists of commands satisfied by consecutive frames from the recorded playthroughs.Source: §3.6 · p. 7
- The resulting dataset contained 15,000 training frames and 3,000 validation frames from a separate playthrough.Source: §3.6 · p. 7
3.7 Learning Frame-Command Mappings with Bimodal Network
The authors trained a bimodal embedding to associate commands with game frames, then tested whether it generalized beyond training positions and into an unseen room.
3.7 Learning Frame-Command Mappings with Bimodal Network
- The bimodal model paired command embeddings from a word-level LSTM with frame embeddings from a CNN over stacked frame pairs.Satisfied commands were trained to have positive dot products with frames, and unsatisfied commands negative ones.Source: §3.7 · p. 7
3.7.1 Evidence of Generalization
- To test cross-room generalization, the authors excluded Room 2 from training while retaining its frames in validation.The test examined whether the embedding recognized game elements such as ladders in a new configuration.Source: §3.7.1 · pp. 7–8
- The embedding maintained extremely low error on a room unseen during training, providing some evidence of cross-room generalization.Table 1 compares validation error rates with the full training set and with Room 2 excluded.Source: §3.7.1 · p. 8, Table 1 · p. 8
- After learning to descend a ladder, the agent often completed that command on its first try in later encounters, while learning to climb one took longer.The authors report this behavior qualitatively during training.Source: §3.7.1 · p. 8
3.8 Run-Time Learning From Natural Language Reward Using Multimodal Embedding
At runtime, a pretrained bimodal embedding detects completed commands, gives the agent additional reward, and supplies embeddings as features. The section identifies comparisons with other agents and the OpenAI Gym leaderboard, but the supplied passages do not state their scores.
3.8 Run-Time Learning From Natural Language Reward Using Multimodal Embedding
- At runtime, the pretrained embedding marks a command complete when the frame-command dot product is positive.Source: §3.8 · p. 8
- On command completion, the agent receives additional reward, advances to the next command, and receives the embeddings as additional features.Source: §3.8 · p. 8
- Table 2 covers Montezuma’s Revenge results after 10 million frames, while Table 3 compares the agent with the OpenAI Gym leaderboard without a frame limit.The supplied passages do not provide the table scores.Source: Table 2 · p. 9, Table 3 · p. 9
3.9 Final Results
The agent outperformed the compared reinforcement-learning algorithms and OpenAI Gym leaderboard agents, but the comparisons are not apples-to-apples because it receives natural-language instructions. The authors also compare its exploration with an intrinsic-motivation agent and note that limited compute prevented a direct comparison at 100 million frames.
3.9 Final Results
- The agent was the clear winner against the compared reinforcement-learning algorithms and OpenAI Gym leaderboard agents, though the tables are not apples-to-apples comparisons.Its environment includes natural-language instructions in addition to visible frames, unlike the other agents’ environments.Source: §3.9 · p. 9
- The authors argue that language supervision generalizes to unseen states and resembles instruction that agents could reasonably receive in the real world.They report that the agent navigates frames absent from both its training set and the multimodal embedding’s training set.Source: §3.9 · p. 9
- At 10 million training frames, the intrinsic-motivation agent explored to the same depth of rooms as ours, but its score at that point was not reported.The cited work reports a score of 6600 after 100 million frames; limited computational budget prevented the authors from training their agent for 100 million frames.Source: §3.9 · p. 9
4 Conclusion
The conclusion presents a framework for training reinforcement-learning agents from natural-language instructions and reports strong scores in relatively few frames where traditional agents fail. It points to robotics as a possible application and proposes combining the approach with other sources of auxiliary reward.
4 Conclusion
- The paper presents a framework that lets reinforcement-learning agents learn from natural-language instruction.The authors describe this as a promising start to cooperation between agents and human trainers.Source: §4 · pp. 9–10
- The agent achieves impressive scores in relatively few frames where traditional agents fail.Source: §4 · p. 10
- Robotics is proposed as a possible application because rich labeled image datasets could support more sophisticated image-language embeddings and many robotic tasks have sparse rewards.The approach is specifically designed to address sparse-reward problems.Source: §4 · p. 10
- The authors suggest combining the additional-reward mechanism with orthogonal sources such as intrinsic motivation as future work.They state that such combinations could feasibly achieve state-of-the-art results on many challenging environments.Source: §4 · p. 10
Appendix A: Full List of Commands Usable for MONTEZUMA’S REVENGE
The appendix lists English commands for navigating rooms and completing tasks in Montezuma’s Revenge, alongside the final network architectures. Its command sequences include movement, ladder climbing, item collection, and room transitions.
Appendix A: Full List of Commands Usable for MONTEZUMA’S REVENGE
- The appendix gives final architectures for the multimodal embedding and reinforcement-learning policy and value network.The frame and sentence embedding heads each have output dimension 10; the sentence head uses word vectors of size 12 and an LSTM with hidden state size 10.Source: Appendix A · p. 11
- The command list covers room-by-room navigation, including movement, climbing ladders, traversing lasers, and moving between rooms.The sequences also include actions such as using a key, collecting a sword or torch, and jumping to a rope.Source: Appendix A · p. 12
- The listed commands extend through later rooms and include collecting a coin and using ropes, keys, and torches.Source: Appendix A · pp. 10–13