Source-linked AI summary
Language Understanding for Text-based Games Using Deep Reinforcement Learning
Karthik Narasimhan, Tejas Kulkarni, Regina Barzilay
TL;DR
Text-based games require control from textual observations even though their underlying states are hidden, creating a language-understanding challenge for game players. The paper jointly learns text representations and action policies with deep reinforcement learning, and LSTM-DQN outperforms bag-of-words, bag-of-bigrams, and random baselines across two MUD worlds. The results support learning expressive representations for text-based game control, although the evaluation excludes a complex quest requiring memorization and high-level planning.
Problem
Text-based games hide the underlying state and expose players to variable textual descriptions, making language understanding necessary for action selection.
Method
A deep reinforcement-learning framework jointly learns LSTM-based text state representations and action policies from game rewards.
Results
LSTM-DQN significantly outperformed random, bag-of-words, and bag-of-bigrams baselines on two MUD games; on a fantasy MUD, it completed 96% of quests versus 82% and 5% for bag-of-words and random baselines.
Takeaways & Limitations
The experiments demonstrate the importance of learning expressive text representations for understanding game states and choosing intelligent actions.
Takeaways & Limitations
The evaluation focuses on one quest because a second secret-tomb quest requires memorizing game events and high-level planning beyond the work’s scope.
Abstract
from arXiv · showhide
In this paper, we consider the task of learning control policies for text-based games. In these games, all interactions in the virtual world are through text and the underlying state is not observed. The resulting language barrier makes such environments challenging for automatic game players. We employ a deep reinforcement learning framework to jointly learn state representations and action policies using game rewards as feedback. This framework enables us to map text descriptions into vector representations that capture the semantics of the game states. We evaluate our approach on two game worlds, comparing against baselines using bag-of-words and bag-of-bigrams for state representations. Our algorithm outperforms the baselines on both worlds demonstrating the importance of learning expressive representations.
1 Introduction
Text-based games require players to infer hidden game states from rich textual descriptions before choosing natural-language actions. The paper learns representations jointly with control policies through reinforcement learning and outperforms bag-based and random baselines.
- Task and challenge: Text-based games expose players to textual descriptions and natural-language commands while hiding the underlying game state.This makes understanding the text necessary for acting in the game.
- Task and challenge: Bag-of-words representations ignore word order and compositional meaning needed to distinguish state-dependent actions.In the example, understanding different directions requires interpreting the sentence-level relation between actions and destinations.
- Approach: The proposed approach learns state representations together with control policies in a reinforcement-learning formulation of game sequences.The policy is represented by an action-value function Q(s, a) learned from rewards produced by game events.
- Approach: The action-value function uses a deep recurrent network whose LSTM module maps text to state vectors and whose second module scores actions.Both modules support learning from game feedback.
- Evaluation: 96% of quests were completed by LSTM-DQN on a fantasy MUD, compared with 82% for bag-of-words and 5% for a random baseline.The evaluation used two MUD games, including one controlled game and one publicly available game with variable human-generated descriptions.
2 Related Work
Prior work learned game control from annotated text, documents, or fully observable states, whereas this paper infers the state representation directly from textual descriptions. It also differs from pixel-based game playing by processing sequential text with recurrent representations.
- Research context: Learning control policies from text has also been studied for software documentation, navigation, and computer games.These applications establish a broader context for grounded language analysis.
- Language-based game control: Prior language-grounding approaches learned command mappings from manually annotated transcripts or game-related documents and precompiled traces.These methods addressed related game-control or rule-learning settings with additional supervision or resources.
- Language-based game control: Earlier work on Civilization jointly learned text analysis and control strategies from simulation feedback but assumed fully observable game states.The present setting requires inferring state representations from textual descriptions instead.
- Pixel-based game control: Pixel-based game players also use deep reinforcement learning, but they infer representations from raw pixels rather than sequential text.The paper uses LSTMs to handle the sequential nature of textual observations.
3 Background
The game is modeled as a partially observed environment in which hidden states generate textual observations, while reinforcement learning estimates action values from transitions and rewards. A deep Q-network approximates these values with a neural network that predicts all actions from the current state representation.
- Game representation: A game is represented by hidden states, action-object commands, stochastic transitions, rewards, and a stochastic function producing textual descriptions.The agent lacks access to the hidden state space and observation function during training and testing.
- Game representation: Rewards are generated by the game’s reward function and provided only when in-game quests are completed.This defines the feedback signal available to the learning agent.
- Q-learning: Q-learning updates action-value estimates over MDP transitions using discounted future rewards.The discount factor γ weights future rewards, and the expectation covers stochastic game transitions.
- Q-learning: The agent selects actions with the highest estimated Q-value, while ε-greedy exploration randomly acts with probability ε.This balances exploitation of current estimates with exploration.
- Deep Q-network: A DQN uses a deep neural network to predict Q(s, a) for all possible actions simultaneously from the current state.This avoids maintaining a separate Q-value for every state-action pair and reduces reliance on manual feature engineering.
4 Learning Representations and Control Policies
The model jointly learns textual state representations and control policies with deep reinforcement learning, using an LSTM representation generator and an action-scoring network trained from game feedback.
- Model architecture: The DQN architecture jointly learns a representation generator and action scorer from in-game reward feedback.The representation generator converts textual state descriptions into vectors, which the action scorer uses to estimate action values.
- Representation generator: LSTMs replace bag-of-words representations by modeling long-range word patterns and capturing aspects of sentence semantics.Mean pooling over LSTM output vectors produces the final state representation.
- Action scorer: The action scorer predicts scores simultaneously for all available actions from the textual state vector.This multi-layer network forms the action-value approximation together with the representation generator.
- Action space: Commands are restricted to one action and one object argument, despite some game commands requiring two arguments.The model predicts actions and objects using the same network, covering the majority of commands in the authors’ worlds.
- Parameter learning: Experience replay samples transitions from memory rather than the current episode to reduce correlation between updates.Mini-batch updates improve experience reuse and computational efficiency; prioritized sampling increases use of positive-reward transitions.
- Parameter learning: The Q-learning target combines the observed reward with the discounted value of the next state, except when the next state is terminal.Parameters are updated by minimizing the squared difference between predicted and target Q-values.
5 Experimental Setup
The experiments use two MUD worlds with different complexity and language variability, evaluate reward and quest completion, and compare LSTM-DQN against random, bag-of-words, and bag-of-bigrams baselines.
- Game worlds: The evaluation uses a smaller controlled Home world and a larger Fantasy world with stochastic transitions and highly variable descriptions.Home descriptions are structured but adversarial, while Fantasy includes human-generated descriptions and up to 100 descriptions for a room.
- Game worlds: Both worlds reward quest completion and penalize harmful situations and nonterminating steps.The step penalty encourages policies that solve quests in fewer steps.
- Game worlds: Home-world quests test language understanding through negation and conjunction, such as distinguishing hunger from sleepiness.The example requires navigating to the kitchen and issuing the command “eat apple,” rather than taking a nap.
- Evaluation: Agents are evaluated using average cumulative reward per episode and fraction of quests completed.Training and testing are conducted each epoch, with M = 50 and T = 20 for Home and M = 20 and T = 250 for Fantasy.
- Baselines: The baselines are a uniformly random player, BOW-DQN, and BI-DQN using bag-of-words and bag-of-bigrams state representations.These baselines isolate the contribution of the representation layer.
- Action space: The command space contains 222 combinations of 6 actions and 37 object arguments, with game cues narrowing object choices in some states.When no cues are available, the agent considers all objects in the game.
- Evaluation: Figure 3 tracks reward and quest completion over training in both worlds and separately shows transfer learning and prioritized sampling on Home.Fantasy-world reward is displayed on a log scale.
6 Results
Across Home and Fantasy worlds, LSTM-DQN performs best by learning expressive textual state representations and action policies from game feedback. Transfer learning and prioritized sampling further accelerate learning, while deeper action scorers outperform linear alternatives.
- Home World: 100% quest completion after around 50 epochs shows LSTM-DQN outperforming BOW-DQN and BI-DQN on Home world.LSTM-DQN achieves close to the optimal reward possible, whereas BOW-DQN and BI-DQN remain confused by differing quest descriptions.
- Action Scoring: DQN action scorers clearly outperform linear counterparts, indicating better control-policy modeling.The comparison includes BOW-DQN and BI-DQN versus BOW-LIN and BI-LIN.
- Fantasy World: LSTM-DQN achieves an average reward of −11.33 and 96% quest completion on Fantasy world.It completes the quest in fewer steps and remains resilient to variations in state descriptions.
- Transfer Learning: Transferred representations reach the optimal policy almost 20 epochs earlier than training from scratch on a new Home world.The new world preserves the rooms but changes the map and pathways, requiring high-level planning.
- Prioritized Sampling: Prioritized sampling achieves the optimal policy around 50 epochs faster than uniform sampling.The result suggests that alternative transition-prioritization schemes merit further research.
- Representation Analysis: LSTM-DQN embeddings place semantically similar words and descriptions of similar underlying states near one another.Examples include pizza, kitchen, and hungry forming meaningful associations across room, object, and quest semantics.
7 Conclusions
The paper addresses end-to-end learning of control policies for text-based games, where interactions are textual and the underlying state is unobserved. It concludes that learning strong text representations is important for playing these games well, while identifying high-level planning and strategy learning as future directions.
- Text-based games challenge automatic players because interactions occur through language while the underlying game state remains unobserved.
- The proposed deep reinforcement learning framework jointly learns state representations and action policies from game rewards.It maps text descriptions into vector representations intended to capture game-state semantics.
- The experiments demonstrate that learning good text representations is important for playing text-based games well.
- Future work includes high-level planning and strategy learning to improve intelligent-agent performance.