Source-linked AI summary
TextWorld: A Learning Environment for Text-based Games
Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, Adam Trischler
TL;DR
Text-based games combine combinatorial, compositional action and observation spaces with partial observability, while existing algorithms struggle when many challenges appear together. TextWorld addresses this with a reinforcement-learning environment that supports curated, handcrafted, and automatically generated games, and evaluates baseline agents on benchmark and hand-authored games.
Problem
Text-based games pose combined challenges from combinatorial, compositional action and observation spaces, partial observability, and directed exploration requirements.
Method
TextWorld provides a sandbox environment for interactive text-game play, state and reward handling, controllable game generation, and reinforcement-learning formulations using Markov decision processes.
Results
The framework was used to develop benchmark games and evaluate several baseline agents on both benchmark and curated hand-authored games.
Takeaways & Limitations
TextWorld supports controlled testing and debugging of reinforcement-learning agents by isolating selected challenges and scaling from simpler games toward more complex ones.
Takeaways & Limitations
Quest generation is limited to simple linear dependencies on immediately preceding actions, excluding the multi-subquest structures common in text adventure games.
Abstract
from arXiv · showhide
We introduce TextWorld, a sandbox learning environment for the training and evaluation of RL agents on text-based games. TextWorld is a Python library that handles interactive play-through of text games, as well as backend functions like state tracking and reward assignment. It comes with a curated list of games whose features and challenges we have analyzed. More significantly, it enables users to handcraft or automatically generate new games. Its generative mechanisms give precise control over the difficulty, scope, and language of constructed games, and can be used to relax challenges inherent to commercial text games like partial observability and sparse rewards. By generating sets of varied but similar games, TextWorld can also be used to study generalization and transfer learning. We cast text-based games in the Reinforcement Learning formalism, use our framework to develop a set of benchmark games, and evaluate several baseline agents on this set and the curated list.
1 Introduction
Text-based games combine language understanding with planning, memory, exploration, and common sense, while partial observability and combinatorial action and observation spaces challenge current RL algorithms. TextWorld addresses these challenges through controllable game generation, benchmark games, and baseline evaluation.
- Text-based games require language understanding, long-term memory, planning, exploration, and common sense for successful play.
- Language serves as both the action and observation space, making text-based games combinatorial, compositional, and partially observable.Descriptive text may omit or ambiguously convey the underlying game state.
- These challenges make hand-authored games such as Zork beyond the capabilities of current learning algorithms.
- TextWorld provides a sandbox for controlled progression from simpler games toward more complex text games, while supporting handcrafted and automatically constructed games.Its generative mechanisms allow users to control game complexity and construction.
- TextWorld automatically builds worlds, populates them with objects and obstacles, generates quests, and produces game-state descriptions using an extensible vocabulary and context-free grammar.Common-sense rules guide generated worlds and quests toward human-interpretable, consistent behavior.
- Generation enables exact state tracking, optional intermediate rewards, and varied related games for studying curriculum learning, generalization, and transfer learning.Training and test games can vary in quest length, environment size, action and object types, synonyms, and grammar complexity.
- The paper contributes a framework description, benchmark games, baseline evaluations, and analysis of curated hand-authored games.
2 Text Games from a Reinforcement Learning Perspective
Text-based games fit the RL setting as partially observable, sequential decision problems in which agents issue language commands and receive textual observations. Their large state and action spaces, exploration demands, sparse rewards, and language phenomena motivate TextWorld’s controlled generative functionality.
- Text-based games are turn-based sequential decision problems where text describes states, commands change them, and parsers map commands to state transitions.
- Parser-based games require players to discover which text commands the game interpreter accepts, and some command outcomes may be stochastic.
- Formally, text-based games are POMDPs with hidden environment states, textual observations, command actions, transitions, rewards, and discounting.
- RL Challenges in Text-based Games: Their state space grows exponentially with rooms and objects, while the word-string action space is large, sparse, and partly dependent on events or elapsed time.
- RL Challenges in Text-based Games: Agents must deliberately explore to discover layouts, objects, dangers, goals, and puzzle clues because reactive control cannot solve these games.
- RL Challenges in Text-based Games: Sparse rewards create long-term credit-assignment problems, while invented words, modifiers, and coreference create language-acquisition challenges.
- RL with TextWorld: TextWorld’s generative functionality supports simpler games that isolate selected challenges, then scales shared maps, objects, and goals across many related instances.
3 The TextWorld Learning Environment
TextWorld is a Python framework whose engine represents generated games as logical-state MDPs and supports interactive play, validity checking, policy tracking, and reward assignment.
- TextWorld trains and tests RL agents on games generated from distributions controlling map size, objects, quest complexity, and text richness.
- Generated games are validated during generation so that the end goal remains reachable from the initial state.
- The framework converts sampled game definitions into Inform 7 code, compiles them to Glulx executables, and mediates agent interaction through Git-Glulx.
- Game states are multisets of logical atoms representing currently true facts, while rules define state transitions through linear-logic resource consumption and generation.
- Forward chaining enumerates grounded actions available in each state and constructs the MDP by discovering transitions and merging duplicate states.
- TextWorld generally rewards only winning states, but it can also provide intermediate rewards based on whether actions shorten or lengthen a tracked winning policy.
3.2 Game Generation
TextWorld generates interpretable games from controlled world mechanics, quests, and text grammars. Its procedures support varied maps, objects, goals, and linguistic renderings while allowing quest difficulty and structure to be specified.
- World Generation: TextWorld generates a combinatorial set of games from predefined dynamics, with knowledge represented in a human-interpretable form.
- World Generation: The generator accepts parameters for rooms, objects, quest length, winning conditions, and text-generation options, then outputs executable Inform 7 source code.
- World Generation: Maps are generated with a parameterized Random Walk procedure controlling room count, grid size, and door connections.
- World Generation: Portable objects can be distributed across rooms, placed on surfaces or in containers, or moved into the player’s inventory.
- Quest Generation: Quest generation constructs action sequences under dependency constraints, rejects cycles, and derives winning conditions from the final action’s generated resources.
- Quest Generation: Backward quest generation starts from a desired end goal, while fact creation adds missing objects and facts to produce more diverse games.
- Text Generation: Context-free grammars render logical game states into controlled natural-language descriptions, with separate grammars for object names, rooms, and instructions.
- Text Generation: TextWorld supports textual variation through extensible grammars, optional grouping and coreference, and house or simplified basic themes.
3.3 Game Interaction with TextWorld
TextWorld provides an interactive API for launching games, stepping through commands, and exposing state information. The framework can supply feedback, descriptions, inventory, objectives, rewards, and winning policies to agents.
- API: TextWorld launches interpretable text games and provides a Gym-inspired API with reset and step interaction.
- State Information: The game_state object exposes interpreter feedback, room descriptions, inventory, location, score, and other game information.
- State Information: Generated games can provide richer state information than existing games, including an explicit objective describing how to win.
- Commands and Rewards: Admissible commands are guaranteed to be understood by the interpreter and either affect the game state or return state information.
- Commands and Rewards: Intermediate rewards represent how useful the previous command was for solving the game, while a winning policy lists commands that guarantee victory from the current state.
4 Related Work
Prior text-game agents address language-based action selection and large action spaces, but existing environments and models have important scope differences. TextWorld is positioned as a sandbox for more controlled, varied text-game research.
- Challenges: Text-based games require language understanding, planning, and efficient exploration, making them difficult even for humans.
- Parser-Based Agents: LSTM-DQN predicts verb and object Q-values independently and completes 100% of Homeworld quests and 96% of Fantasyworld quests.
- Choice-Based Agents: DRRN computes Q-values for observation-action pairs and reaches optimal cumulative reward on Saving John but a stable suboptimal policy on Machine of Death.
- Model Limitations: Neither LSTM-DQN nor DRRN conditions on previous actions or observations, so neither can handle partial observability.
- Action-Space Reduction: Action Elimination Networks reduce parser-game action spaces by estimating which actions are likely to fail using stored game feedback.
- Related Environments: MazeBase generates grid-based games for related-environment learning, whereas TextWorld provides more complex quests and language.
- Related Environments: CommAI supports curricula through bit-level interaction, while TextWorld uses simplified natural-language interaction.
5 Benchmarks
TextWorld benchmarks include a curated evaluation set and procedurally generated treasure-hunter tasks. These benchmarks vary navigation, quest length, obstacles, and observability while evaluating several baseline agents.
- Benchmark Suite: The preliminary benchmark suite combines 50 hand-authored games with manually checked scores and quests and generated evaluation tasks.
- Curated List: The curated-list evaluation compares BYU, Golovin, and Simple baselines under a 1000-step budget, resetting play after losses.
- Curated List: Normalized scores are low on some curated games and zero on many; performance varies with game structure, such as initial score or navigational solvability.
- Treasure Hunter: The treasure-hunter benchmark tests affordance extraction, efficient navigation, and memory by requiring agents to retrieve a designated object.
- Treasure Hunter: Generated difficulty levels vary room counts from 5 to 20 and quest lengths from 1–5 to 3–20 actions.
- Treasure Hunter: Difficulty increases from empty rooms and open connections to closed or locked doors and containers requiring unlocking.
- Generated Evaluation: The benchmark can be simplified by exposing more game-state information or using a simpler grammar.
- Generated Evaluation: Each difficulty level contains 100 games, with agents receiving +1 for the correct object, -1 for the wrong object, or 0 after 1000 steps.
6 Current Limitations
TextWorld’s current limitations concern quest structure, event dynamics, character interaction, multiplayer support, and generated-text continuity. These constraints bound the complexity and scope of games the framework can currently represent.
- Complex Quest Generation: Quests are generated as linear action sequences, so dependencies among multiple sub-quests are not represented.The paper suggests directed graphs of dependent actions as a way to generate more complex quests.
- Time-based Events: State changes cannot currently occur without corresponding user actions.This prevents mechanisms such as doors locking after several steps or traps triggering periodically.
- Generated Game Scope: The generated Treasure Hunter example places the passkey two rooms away behind two closed doors, illustrating controlled spatial and obstacle structure.The game is labeled difficulty 20, with the passkey on a rack in the washroom and the player in the basement.
- Non-Player Characters (NPCs): The generator does not support NPC interaction, although NPCs may autonomously or reactively alter the game state.NPC interaction is common in several text-based games but remains outside the generator’s current scope.
- Multi-User Dungeon (MUD): TextWorld supports only single-agent games, excluding multi-user interactions involving cooperation or competition.
- Text Generation: CFG-based text generation makes continuity between sentences difficult to ensure.
7 Conclusion
The paper introduces TextWorld as a sandbox for training and evaluating RL agents on text-based games, using generation to approach the complexity of hand-authored games. It presents preliminary benchmarks, evaluates baseline agents, and identifies future work in richer tasks and learning methods.
- Conclusion: TextWorld is introduced as a sandbox learning environment for training and evaluating RL agents on text-based games.
- Conclusion: The framework’s generative mechanisms support progression toward the complexity of hand-authored games and underpin a preliminary benchmark suite.
- Conclusion: The authors evaluate several baseline agents on generated benchmark games and a curated collection of analyzed hand-authored games.
- Future Work: Future work targets more complex benchmark tasks and new approaches to representation learning, language understanding and generation, and RL.
A Typical Text-based Obstacles
Text-based games combine language interpretation with exploration, puzzles, survival, timing, and uncertain or irreversible consequences. Their obstacles span navigation, clue discovery, object manipulation, combat, persistence, and game-specific knowledge.
- Language: Parser complexity includes commands the parser cannot handle, distracting descriptions, sarcasm, and fictional languages.
- Maze: Mazes require navigation through similar or changing room configurations, and some are solvable only through actions such as moving walls or flying.
- Clue Hunting: Clue-hunting games may require finding state-changing hints that reveal destinations, trigger events, or make further map information available.
- Treasure Hunting: Treasure hunts can make items serve as clues or prerequisites for unlocking later treasures, as illustrated by gem clusters opening a treasure chest in Infidel.
- Other Obstacles: Other recurring obstacles include trivia knowledge, self-maintenance, combat, time limits, mechanical puzzles, and repeated interaction with NPCs.Examples include hunger and fatigue, equipment-dependent combat, timed access, moving objects, and repeating talk to commands.
- Difficulty and Length: Stochasticity and cruelty can make outcomes uncertain or games unwinnable, while game length varies with commands, rooms, descriptions, side quests, endings, and difficulty.
B Curated List of text-based Games
The curated list catalogs text-based games by structural and challenge-related features, including room counts, maximum length, puzzles, stochasticity, and forgiveness. The entries span simple merciful games to large, difficult, and often cruel or nasty games.
- Catalog Dimensions: The catalog records each game’s room count, maximum moves, maze, trivia, self-maintenance, combat, time, mechanical, persistence, stochasticity, and forgiveness properties.
- Range of Games: The list spans small games such as The Acorn Court and All Quiet on the Library Front, each with 1 room and 30 maximum moves.
- Forgiveness: Forgiveness labels range from Merciful and Polite to Tough, Nasty, and Cruel across the curated games.
- Range of Games: It also includes larger entries such as Spellbreaker with 30+ rooms and 600 maximum moves, and Zork I with 30+ rooms and 350 maximum moves.
B.1 Game Notes
The analyzed games span introductory and simple navigation experiences to long, difficult adventures involving spatial reasoning, spells, world knowledge, mazes, and stochastic obstacles. Their notes also document varied reward structures, language styles, maps, and mechanisms for helping or hindering players.
- Scope and game structure: The collection ranges from short beginner-oriented games to long adventures with complex systems, massive maps, surreal settings, or multiple sub-worlds.Dragon Adventure introduces children to text-based games; other games feature complex spell-casting, massive maps, surreal imagery, or extensive puzzle structures.
- Navigation and spatial challenges: Navigation and spatial reasoning are recurring challenges, including illogical or one-way paths, mazes, vertical room relationships, and multi-part spatial puzzles.Several games require maze navigation or spatial inference, while others include confusing layouts, one-way re-entry conditions, or disconnected rooms with identical names.
- Knowledge and language demands: Many games require specialized knowledge or memory, such as spells, invented words, hieroglyphics, cultural references, historical context, or information from in-game texts.The notes describe reliance on trivia, books, British history and culture, world knowledge, and memorized spells or rare fantasy vocabulary.
- Presentation and themes: Some entries impose distinctive linguistic or thematic conditions, including cave-person speech, pop-cultural references, in-jokes, made-up words, and literary or genre familiarity.These presentation choices can shape how players interpret commands, clues, and narrative content.
- Difficulty factors: Difficulty can arise from stochasticity, misleading or sarcastic language, unwinnable states, and puzzles whose solutions depend on persistence, clue hunting, or unusual command choices.Examples include randomly selected crucial objects, mazes without fixed solutions, deliberately misleading prose, accidental unwinnable states, and endings triggered by command variants.
- Objectives and assistance: The games use varied objectives and support mechanisms, including treasure hunts, object retrieval, innovative solutions, hint systems, wishes, and consultable in-game libraries.Adventure rewards innovative solutions rather than only puzzle completion, while Wishbringer and Deephome provide explicit assistance for bypassing or solving puzzles.
C Inform 7
Inform 7 is presented as a natural language-like programming language and toolchain for interactive fiction. The authors choose it to leverage its extensively refined parser, with Inform 7 source compiling to Inform 6 source.
- C Inform 7: Inform 7 is a domain-specific programming language and tooling for interactive fiction, regarded as especially natural language-like.It was originally created in 1993 by Graham Nelson and later released as Inform 7.
- C Inform 7: The authors use Inform 7 to leverage Inform’s parser, which benefited from more than two decades of tweaks and fixes.Inform 7 source compiles to Inform 6 source.