Source-linked AI summary
Deep Reinforcement Learning with a Natural Language Action Space
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, Mari Ostendorf
TL;DR
The paper tackles reinforcement learning when both states and actions are represented in natural language, including action spaces that may be unbounded and vary over time. It introduces DRRN, which separately embeds state and action texts and combines them through an interaction function to estimate Q-values. DRRN converges faster and to a better solution than alternative architectures, while generalizing to paraphrased action descriptions; its experiments use simple bag-of-words representations and experienced human players can outperform it in one game.
Problem
Reinforcement learning with natural-language actions is difficult because the action space can be unbounded and the feasible action subset varies over time, making fixed-output DQN architectures impractical.
Method
DRRN uses separate deep neural networks for state and action text embeddings, combines them with a pairwise interaction function, and uses the result to approximate Q-values.
Results
DRRN converges faster and to a better solution for Q-learning than alternative architectures without separate state and action embeddings, and generalizes to unseen paraphrased actions.
Takeaways & Limitations
The results support using separate representations for natural-language states and actions in sequential decision-making tasks such as text games.
Takeaways & Limitations
The experiments use only simple bag-of-words representations, and experienced human players outperform the algorithm in Game 2.
Abstract
from arXiv · showhide
This paper introduces a novel architecture for reinforcement learning with deep neural networks designed to handle state and action spaces characterized by natural language, as found in text-based games. Termed a deep reinforcement relevance network (DRRN), the architecture represents action and state spaces with separate embedding vectors, which are combined with an interaction function to approximate the Q-function in reinforcement learning. We evaluate the DRRN on two popular text games, showing superior performance over other deep Q-learning architectures. Experiments with paraphrased action descriptions show that the model is extracting meaning rather than simply memorizing strings of text.
1 Introduction
The paper addresses sequential decision-making when both states and actions are expressed in natural language. It proposes DRRN, which separately embeds state and action texts and measures their relevance for Q-learning, including generalization to paraphrased actions.
- Problem: Natural-language decision tasks require selecting the most relevant action from state and action text to maximize long-term reward.The setting includes text-based games, dialog systems, and tutoring systems.
- Problem: Reinforcement learning models these tasks because actions change the environment and create long-term dependencies.
- DRRN: DRRN uses separate deep neural networks to embed state and action texts, then combines their vectors with an interaction function to approximate Q-values.The Q-function approximation is learned end-to-end with Q-learning.
- Challenge: Natural-language action spaces can be discrete and potentially unbounded, unlike the bounded action spaces commonly handled by earlier approaches.The paper attributes this unboundedness to the exponential complexity of language with sentence length.
- DRRN: DRRN learns distinct meaning representations for scene-describing states and user-action-describing texts, supporting generalization to unseen paraphrased actions.
2 Deep Reinforcement Relevance Network
The DRRN handles natural-language action spaces by separately embedding state and action texts, then combining them to estimate Q-values for feasible actions. Its training uses Q-learning with experience replay, while related figures illustrate architectural differences and learned embedding alignment.
- Natural language action space: A vanilla DQN is impractical when natural-language actions form an unbounded space and the feasible action set varies over time.A fixed output for every action would depend explicitly on the large number of possible language actions.
- Baselines: Max-action DQN uses concatenated state and action vectors with outputs sized to the maximum number of possible actions, whereas Per-action DQN outputs one Q-value per state-action input.Max-action DQN requires the maximum feasible action count to be known; Per-action DQN handles varying action counts.
- DRRN architecture: DRRN uses separate deep neural networks for state-text and action-text embeddings, combined through a pairwise interaction function to estimate Q-values.Separate structures accommodate differences between potentially long state descriptions and concise action phrases.
- Forward activation: For each state-action pair, DRRN maps both texts into embedding vectors and applies an interaction function, such as an inner product, before selecting the highest-valued feasible action.The interaction function can also be bilinear or nonlinear; experiments primarily use the inner product.
- Relevance learning: End-to-end learning aligns state and action embeddings so relevant actions receive higher interaction outputs and therefore higher Q-function values.The architecture represents good and bad action texts in separate finite-dimensional embedding spaces whose alignment is learned during training.
- Back propagation: Experience replay randomly samples transition tuples for temporal-difference learning, while shared action-side parameters update models associated with all actions.Random sampling scrambles trajectories into a bag of transitions and is described as avoiding oscillations or divergence and accelerating convergence.
3 Experimental Results
The DRRN is evaluated on two text games against neural and linear baselines, with experiments covering convergence, interaction functions, and paraphrased actions. It converges faster and achieves higher rewards than the baselines, while retaining performance on unseen action descriptions.
- Experimental setup: The study evaluates DRRN and multiple baselines on the deterministic game “Saving John” and stochastic game “Machine of Death”.The baselines include a linear model, max-action DQNs, and per-action DQNs.
- Performance: After around 4000 episodes, all methods converge, but DRRN converges much faster and achieves a higher average reward than the three baselines.Both games begin near random-policy performance, at roughly -7 and -8 average rewards respectively.
- Performance: DRRN performs consistently better than all baselines at convergence, often with lower variance across hidden-layer sizes.Final performance is evaluated with hidden dimensions of 20, 50, and 100.
- Interaction functions: The bilinear interaction function gives similar results to the inner product, whereas concatenating embeddings for a nonlinear DNN degrades performance.These experiments fix the state embedding at 100 dimensions and vary the action embedding dimensions.
- Paraphrased actions: 0.95 predictive R-squared indicates a strong positive correlation between Q-values for paraphrased and original actions.The DRRN also achieves high reward when tested on paraphrased action descriptions unseen during training.
- Paraphrased actions: 10.5 vs. 11.2 average reward shows a modest decrease under paraphrased actions, while DRRN still significantly outperforms the other methods.The game’s state-transition mechanism remains unchanged; only the action descriptions are paraphrased at test time.
4 Related Work
Prior work applied deep reinforcement learning to language-related decision tasks, but often used constrained, fixed command structures. The DRRN work uses simpler bag-of-words representations while relating performance and generalization to action-language structure.
- Deep reinforcement learning has been applied to dialogue management and text-game control, but relatively few studies address natural-language state or action spaces.
- Narasimhan et al. used an LSTM DQN for parser-based text games and outperformed random and Bag-of-Words DQN baselines.
- That prior setup constrained actions to known command structures containing one action and one argument object, with a limited action-side vocabulary.
- The DRRN experiments use simple bag-of-words representations, while the paper notes that more complex sentence models may improve performance.Preliminary Machine of Death experiments did not find improved performance from LSTMs, though the authors conjecture benefits for larger tasks or pretrained embeddings.
- The DRRN learns a continuous representation for an inherently discrete action space, and paraphrasing experiments suggest representation-level generalization.
5 Conclusion
The paper presents DRRN as a deep architecture for natural-language actions in sequential decision tasks. It reports faster convergence and better Q-learning solutions than alternative architectures without separate state and action embeddings, while identifying broader applications as future work.
- DRRN is a deep neural architecture for handling natural-language actions in decision-making tasks such as text games.
- The DRRN converges faster and reaches a better Q-learning solution than alternative architectures that do not use separate state and action embeddings.
- Future work includes adding attention to analyze strategically relevant text and applying the methods to more complex text games or other natural-language-action tasks.
A Percentage of Choice-based and Hypertext-based Text Games
The supplied passages describe DRRN’s separate state and action networks and their forward and backward computations, but do not provide the percentage results requested by this section title.
- The supplied table references identify a percentage table for choice-based and hypertext-based games, but no percentages are stated in the passages.
- The DRRN uses separate state-side and action-side neural networks, each with L hidden layers.
- Action representations are computed for each action in the available action set using a nonlinear hidden-layer activation, specified as tanh.
- The state and action representations are updated through backpropagation using derivatives of the Q-function and element-wise Hadamard products.
C Final Rewards in the Two Text Games
The supplied passage indicates that final rewards for the two text games are presented in Tables 2 and 3, without reporting their values.
- Final rewards for the text-game experiments are presented in Tables 2 and 3.
- The supplied passage does not state the final-reward values for either text game.
- The referenced results therefore cannot be compared quantitatively from the supplied passage alone.
D Game 2 Learning curve with shared state and action embedding
Parameter tying initially accelerates convergence in Game 2 but produces high variance and instability.
- Parameter tying gives faster convergence during the first 1000 episodes, but the learning curve has high variance and is unstable.
E Examples of State-Action Pairs in the Two Text Games
The supplied passage provides no substantive examples of state-action pairs from the two text games.
- The passage only indicates that Tables 4 and 5 present examples, without describing their state-action pairs.
F Examples of State-Action Pairs that do not exist in the feasible set
The supplied passages identify figures and tables involving state-action values, including actions outside the feasible set, but provide little accompanying detail.
- Figure 1 compares learning curves for shared state-action embeddings and the proposed DRRN in Game 2.
- Table 4 reports Q values for state-action pairs in “Saving John,” where higher values indicate more cooperative actions.
- Table 5 reports Q values for state-action pairs in “Machine of Death” using a trained DRRN.
- Table 6 reports Q values for “Machine of Death” state-action pairs with made-up actions outside the feasible set.