A Survey of Text Games for Reinforcement Learning informed by Natural Language

• Text Games provide safe, data-efficient environments that mimic language found in real-world scenarios for RL research.Source: §1 · p. 1
• Natural-language RL remains limited, motivating improved evaluation settings where language is a first-class component.Source: §1 · p. 1
• Text Games are partially observable and offer reward-relevant outcomes, making them suitable RL problems.Source: §1 · p. 1, Figure 1 · p. 1
• Author-generated environments lack uniformity, making comparisons between proposed architectures challenging.Source: §1 · pp. 1–2
• The survey systematically reviews Text Game generation tools, generated environments, and RL agent architectures.Source: §1 · p. 2

2 Text Game Generation

• TextWorld generates game worlds, objects, quests, and goal states that define subsequent reward signals.Source: §2 · p. 2
• Jericho supports human-made interactive fiction across genres, including Zork and Hitchhiker’s Guide to the Galaxy.Source: §2 · p. 2

2.1 Environment Model

• Reinforcement Learning formulates sequential decision making as optimization over an MDP containing states, actions, transitions, rewards, and discounting.Source: §2.1 · p. 2
• An RL agent seeks a policy that selects actions in states to maximize expected discounted cumulative return.Source: §2.1 · p. 2
• Text games are POMDPs because agents never directly observe states and instead receive textual feedback after commands.Source: §2.1 · p. 2

• Handicaps limit challenge scope to support simpler and more rigorous comparisons across environments.Source: §2.2 · p. 2
• TextWorld controls state-space size, command requirements, and training–testing variation to target specific generalization measures.Source: §2.2 · p. 2
• Partial observability can be reduced by exposing complete game-state information or augmenting observations with bonus commands such as look and inventory.Source: §2.2 · pp. 2–3
• Language complexity can be reduced through restricted vocabularies, simplified grammar, symbolic object tokens, or predefined choice-based commands.Source: §2.2 · p. 2
• Other handicaps include fixed random seeds, save functionality, game-specific vocabularies, auxiliary world representations, valid-action detection, and more immediate rewards.Source: §2.2 · pp. 2–3
• Action spaces range from template-based and parser-based commands to ranking predefined choices, with templates separating verb selection from entity extraction.Source: §2.2 · p. 3

• TextWorld bounds complexity through generated objects and rules, whereas Jericho includes 57 human-designed games with stochasticity, unnatural interactions, and unknown objectives.Source: §2.3 · p. 3
• Text Game design and partial observability create natural RL challenges alongside language-understanding challenges identified by TextWorld and Jericho.Source: §2.3 · p. 3
• The design and partially observed representation of text games create natural reinforcement-learning challenges, alongside language-understanding challenges identified by Côté et al. [2018] and Hausknecht et al. [2019a].Source: §2.3 · p. 3

• Partial observability forces agents to infer hidden world structure from limited feedback, often requiring extensive exploration and failures.Source: Partial Observability · p. 3
• Causal dependencies can require agents to backtrack when they leave a state without completing prerequisites for future states.The difficulty grows with trajectory length because missed prerequisites become harder to recover from.Source: Partial Observability · p. 3
• Handcrafted rewards work in easier settings, but more nuanced exploration methods are needed for real-world text games.Source: Partial Observability · p. 3
• Go-Explore prioritizes advanced or under-explored states through weighted random exploration to find high-reward trajectories.Source: Partial Observability · p. 4
• A modular policy approach combines patience and intrinsic motivation to detect bottlenecks, chain policies, and backtrack when stuck.Source: Partial Observability · p. 4
• Heuristics and dynamically learned knowledge graphs reduce uncertainty by restricting navigation or filling action templates with known entities.Source: Partial Observability · p. 4

• Exploration bonuses encourage agents to visit unseen states, but can reduce their use of high-level contextual knowledge about the environment.Source: Large State Space · p. 4
• Treasure Hunter enlarges the challenge with locked doors and color-matching keys that require object affordances and generalization.Source: Large State Space · p. 4
• QAit location and existence settings require agents to traverse states while determining an object’s location or whether it exists.Source: Large State Space · p. 4
• These environments instantiate the standard reinforcement-learning trade-off between exploration and exploitation.Source: Large State Space · p. 4

Large, Combinatorial and Sparse Action Spaces

• Unbounded action spaces arise from unrestricted command length and semantics, so agents commonly select templates and fill them with extracted entities.Source: Large, Combinatorial and Sparse Action… · p. 4
• Action-elimination networks approximate whether commands change the game state, while generative models produce commands from observations and vocabularies.Source: Large, Combinatorial and Sparse Action… · p. 4
• CALM, a fine-tuned GPT-2 generative model, is competitive with models that use valid-action handicaps.Source: Large, Combinatorial and Sparse Action… · p. 4

Long-Term Credit Assignment

• Sparse rewards make learning difficult when positive feedback appears only at game completion, prompting subtask rewards and handcrafted signals.Source: Long-Term Credit Assignment · p. 4
• Subtask rewards align with hierarchical approaches by providing feedback for intermediate accomplishments in environments with difficult end-game exploration.Source: Long-Term Credit Assignment · p. 4

Understanding Parser Feedback & Language Acquisition

• LIGHT requires agents to speak and act with humans, embodied agents, and the world to complete quests in grounded fantasy dialogue.Source: Understanding Parser Feedback &… · p. 4
• A related system combines commonsense language-model pre-training with a factorized action space for commands and dialogue, evaluated using held-out human demonstrations.Source: Understanding Parser Feedback &… · pp. 4–5

Commonsense Reasoning & Affordance Extraction

• Text games require commonsense associations between actions and objects, such as opening doors rather than cutting them and retaining usable items.Source: Commonsense Reasoning & Affordance… · p. 5
• First TextWorld Problems generates thousands of themed instances requiring planning and memory for hierarchical cooking tasks.Cooking decomposes abstract goals into sequences of high-level actions and object-level subproblems.Source: Commonsense Reasoning & Affordance… · p. 5
• TW-Commonsense explicitly tests object affordances and detection of out-of-place objects using prior commonsense knowledge.Source: Commonsense Reasoning & Affordance… · p. 5
• ClubFloyd and Jericho-QA provide playthroughs, timestep questions, and knowledge-graph supervision for pre-training action generators and question-answering modules.Source: Commonsense Reasoning & Affordance… · p. 5
• ALFWorld transfers policies between textual and embodied environments with a shared latent structure, while QAit tests attribute affordances such as cutting, eating, and cooking.Source: Commonsense Reasoning & Affordance… · p. 5

• Agents can track a belief graph of observed entities and relationships that converges toward the full game state as exploration proceeds.Source: Knowledge Representation · p. 5
• Subgraph-based representations support multi-step reasoning by separating knowledge according to different semantic meanings rather than compressing the entire graph into one vector.Source: Knowledge Representation · p. 5
• Object-centric retrieval augments current observations with relevant past information, after which attention focuses action-value prediction on correlations.Source: Knowledge Representation · p. 5

3 Benchmark Environments and Agents

• Benchmark environments largely use collection objectives, while their complexity varies through rooms, objects, vocabulary, quest structure, and obstacles.TextWorld’s object logic enables these tasks but can constrain environmental scope.Source: §3 · p. 6
• Single-game, joint, and zero-shot settings evaluate agents respectively on matched games, shared game sets, and games unseen during training.Source: §3 · p. 6
• CookingWorld varies challenge variables independently or jointly rather than providing clear discrete difficulty levels.Source: §3 · p. 6
• The benchmark table records evaluation regime, generation support, difficulty, rooms, objects, action-verb space, observation length, vocabulary, and quest length.Source: Table 1 · p. 6

3.1 Agent Architectures

• Deep-RL agents encode observations, feedback, and knowledge-graph information before an action selector estimates action values.Source: §3.1 · p. 6
• Architectures shifted from early LSTM or Bag-of-Words encoders toward common GRUs, while DQN and A2C remained typical action selectors and knowledge graphs or pretrained Transformers stayed limited.Source: §3.1 · p. 6
• Most agents target TextWorld/CookingWorld or Jericho, representing either generated games with simple rules or human-designed pre-existing games.Source: §3.1 · p. 6
• Task-specific pretraining, parser-based actions, and varied encoder and action-selection modules extend the standard architecture design space.Only two cited papers considered parser-based action spaces.Source: §3.1 · pp. 7–8

• Text Game environments and benchmark comparisons support systematic study of agents’ learning efficiency, generalization, and overfitting.Most methodologies focus on either learning efficiently or generalizing knowledge, with the broader motivation of addressing limited and expensive real-world data.Source: §4 · p. 10
• Pretrained transformer models have produced the most recent advances in agents’ performance on linguistically challenging goals.The survey identifies BERT, GPT-2, and ALBERT as examples supporting these advances.Source: §4 · p. 10
• Dynamically learned knowledge graphs are increasingly common, with pretrained knowledge graphs from text corpora identified as a future direction.Source: §4 · p. 10
• No current method treats language-based interpretability as a primary goal, although LIME is identified as a methodology worth considering.Source: §4 · p. 10
• Language grounding in environments remains incompletely addressed, despite dialogue-grounding environments being introduced as an initial step.Source: §4 · p. 10
• Only 5 surveyed papers report the time and resources required for training, limiting reproducibility and applicability in resource-constrained settings.Source: §4 · p. 10