Source-linked AI summary
Interactive Fiction Games: A Colossal Adventure
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, Xingdi Yuan
TL;DR
Interactive Fiction games expose agents to language understanding, commonsense reasoning, and combinatorial action spaces in sequential text-based environments. The paper introduces Jericho, a learning environment with template-based actions, and evaluates agents across human-made games. Results show progress for multiple agents, with DRRN outperforming TDQN while TDQN exceeds NAIL and random performance.
Problem
IF games combine language understanding, commonsense reasoning, sequential decision making, and combinatorial action spaces, creating a challenging testbed for language-based agents.
Method
The paper introduces Jericho and a template-based action space, then evaluates DRRN and Template-DQN agents across human-made IF games.
Results
DRRN outperformed TDQN, while TDQN outperformed NAIL and the random agent across the reported benchmark results.
Takeaways & Limitations
Jericho provides benchmark environments and scores intended to support future research on language-understanding agents.
Takeaways & Limitations
The evaluation trained and tested agents on individual games, falling short of demonstrating truly general IF game playing.
Abstract
from arXiv · showhide
A hallmark of human intelligence is the ability to understand and communicate with language. Interactive Fiction games are fully text-based simulation environments where a player issues text commands to effect change in the environment and progress through the story. We argue that IF games are an excellent testbed for studying language-based autonomous agents. In particular, IF games combine challenges of combinatorial action spaces, language understanding, and commonsense reasoning. To facilitate rapid development of language-based agents, we introduce Jericho, a learning environment for man-made IF games and conduct a comprehensive study of text-agents across a rich set of games, highlighting directions in which agents can improve.
1 Introduction
Interactive Fiction games are text-based environments that combine language understanding, commonsense reasoning, and sequential decision making. The paper introduces Jericho as a learning environment for studying agents across human-made IF games.
- IF games present textual world descriptions, accept text actions, and provide score as players progress through stories.
- Unlike most NLP tasks, IF games require sequential decisions whose actions alter later world states and can affect eventual endings.
- Jericho is introduced as a learning environment for human-made IF games.
- The paper also proposes a template-based action space and evaluates learning agents across a large set of human-made games.
2 Research Challenges
IF games require agents to act under partial observability while understanding language, reasoning with commonsense, and exploring enormous combinatorial action spaces. Their worlds also require constructing textual maps during navigation.
- IF games can be modeled as POMDPs in which text observations reveal latent world states and text actions change those states.
- Combinatorial Action Space: A four-word command from a 700-word vocabulary yields 700^4 = 240 billion possible actions.
- Combinatorial Action Space: Only a small subset of generated commands is grammatical, parseable, and contextually valid enough to change the world state.
- Commonsense Reasoning: Commonsense reasoning supports affordance extraction, planning with limited inventory, and delayed return to useful objects such as locked chests.
- Knowledge Representation: Textual-SLAM requires agents to construct maps while navigating locations whose connectivity is not necessarily Euclidean.
3 Related Work
Prior work addresses IF through affordance-based action prioritization, reduced parser action spaces, procedural game generation, knowledge graphs, transfer, and dialogue modeling.
- Word2vec affordance vectors prioritize verbs that can plausibly pair with in-game objects for Q-learning agents.
- Action-elimination methods reduce parser-based games to predefined minimal actions but require walkthroughs or expert demonstrations for new games.
- TextWorld procedurally generates parser-based games with scalable complexity and supports agents evaluated on unseen environments.
- KG-DQN uses an exploration-built knowledge graph and question answering to address partially observable states and combinatorial actions.
- Light is a crowdsourced text-adventure dialogue dataset used to train models that generate contextually relevant dialogue, actions, and emotes.
4 Jericho Environment
Jericho is an OpenAI-Gym-like environment for human-made IF games that supports state management, game-specific representations, and action validation. Its template-based action space reduces language-generation complexity, while unsupported games lack several detection and scoring features.
- Jericho provides a Python-based, open-source environment with an OpenAI-Gym-like interface for connecting learning agents to IF games.
- The environment supports loading and saving states, random seeding, and games spanning dungeon crawl, science fiction, mystery, comedy, and horror.
- Unsupported games can run through Jericho but lack score detection, move counts, and world-change detection.
- Template-Based Action Generation: A template-based action first selects a template and then fills up to two blanks with parser-vocabulary words, yielding O(T V^2) actions.
- World Object Tree: The world object tree represents relationships among locations, objects, the player, and inventory, supporting state interpretation and world-change detection.
- Identifying Valid Actions: Jericho identifies valid actions by executing candidates and detecting changes in the world-object tree.
5 Algorithms
The paper evaluates three interactive-fiction agents spanning choice-based, parser-based single-game, and parser-based general-game play. DRRN estimates values for valid actions, TDQN generates template-based actions, and NAIL uses hand-crafted heuristics without handicaps.
- DRRN and TDQN are single-game agents, whereas NAIL is designed to play unseen games.Single-game agents train and evaluate on the same game; general-game agents target unseen games.
- DRRN: DRRN encodes observations and valid actions, then estimates a Q-value for each action before softmax action selection.It batches valid actions in a single forward pass and trains from prioritized replay.
- DRRN: DRRN uses world-change detection to identify valid actions and Load/Save to obtain inventory and location observations.These are Jericho handicaps used during training.
- Template-DQN: TDQN extends LSTM-DQN with three output heads for templates and the two vocabulary slots that fill template blanks.It addresses parser-based action generation through template-based actions, while exploration remains challenging.
- NAIL: NAIL plays unseen games without handicaps, using heuristics for maps, action validity, and object interaction alongside a web-based language model.It serves as a reference point for general interactive-fiction game playing.
6 Experiments
The experiments benchmark DRRN, TDQN, NAIL, and a random agent across thirty-two Jericho games. Reinforcement learning makes progress across many games, with DRRN achieving the highest average normalized completion, while all agents remain far from solving many difficult games.
- Thirty-two Jericho-supported games evaluate reinforcement learning feasibility, reproducibility, action-space choices, and general versus single-game agents.The comparison includes DRRN, TDQN, NAIL, and RAND.
- Reinforcement learning is viable across many games, based on Table 1 results and Figure 3 learning curves.Five separate DRRN and TDQN agents were trained for each game, with scores averaged over the last hundred learning episodes.
- 10.7% completion is achieved by DRRN, compared with 6.1% for TDQN, 4.9% for NAIL, and 1.8% for RANDOM.Progress is normalized by maximum possible score and averaged across all games.
- DRRN outperforms TDQN overall because it estimates values over pre-identified valid actions rather than the larger template-and-vocabulary space.The larger max operation in TDQN’s Q-learning update made it more prone to Q-value overestimation.
- NAIL performs surprisingly well despite no handicaps, no training period, and only one interaction episode, but reinforcement learning agents outperform it.This comparison highlights the difficulty of engineering a general-purpose agent and the promise of learning policies from data.
- None of the agents obtain any score on five games, and all remain below the level needed to solve even average-difficulty games.Some games combine complex structure with difficult exploration, including sparse terminal rewards.
7 Notable Games
Jericho’s games span difficulty tiers shaped by rewards, puzzles, action spaces, and interaction constraints. Easier games are near-term targets, while extreme games require capabilities beyond current agents, including long-range planning and nonstandard action learning.
- Possible Games: Possible Games offer frequent rewards and basic navigation or interaction, making them initial testbeds for learning agents.Detective is among the easiest, while Acorncourt requires a high proportion of complex actions.
- Difficult Games: Difficult Games combine sparse rewards, complex puzzles, and longer solutions; agents can make progress but need new insights and algorithms to solve them.Zork1 adds darkness, mazes, fighting, and limited inventory space as obstacles.
- Jericho organizes games into Possible, Difficult, and Extreme tiers reflecting increasing challenges for reinforcement learning agents.Table 2 defines these tiers by expected solvability and required advances.
- Figure 3 plots episode score against training steps for DRRN and TDQN, with shaded standard deviations across five independent runs.The figure provides learning curves for comparing training progress by game.
- Extreme Games: Extreme Games involve extensive puzzles, long solutions, and nonstandard actions that challenge skilled humans and current agents.Anchorhead requires physical reasoning and long-range dependencies, while the Enchanter Trilogy requires learning spell names and effects.
8 Future Work
Future work focuses on evaluating agents across unseen games, improving exploration without game-provided rewards, and addressing limitations in template-based action generation. Transformer-based NLP advances are another possible direction, requiring adaptation to reinforcement learning.
- Unsupervised learning: Evaluating agents trained on individual games does not establish truly general Interactive Fiction game playing.The proposed paradigm trains on unsupported games and tests on Jericho-supported games.
- Unsupervised learning: Surrogate rewards may guide exploration in unsupported games that lack rewards or scores.Novelty-based rewards are hypothesized to help discover locations, successful interactions, and objects.
- Better Template-based Agents: Conditional word generation could improve TDQN by avoiding invalid actions caused by its independence assumption.Selecting a template and vocabulary word independently can combine individually likely choices into an invalid command.
- Other directions: Transformer-based NLP advances may transfer to human-made IF games if adapted from supervised learning to reinforcement learning.The passage identifies this adaptation as necessary for applying the advances in this setting.
9 Conclusion
The paper presents Jericho as a platform for studying language-based agents in challenging text-based adventures and introduces a template-based action space with TDQN. Benchmark results show progress over NAIL and random agents, while DRRN performs better than TDQN.
- Platform and action space: Jericho extracts game-specific action templates and vocabulary to reduce the complexity of full-scale language generation.The platform supports a template-based action space for human-made Interactive Fiction games.
- Platform and action space: TDQN generates actions by selecting a template and then filling its blanks with vocabulary words.This is the paper’s Template-DQN action-generation procedure.
- Results: TDQN outperformed NAIL and the random agent, while DRRN outperformed TDQN in the reported benchmark comparisons.The authors present the scores as benchmark results rather than direct comparisons between agents with identical training paradigms.
A Experiment Details
Episodes end after 100 valid steps or game termination, and the agents are trained independently on each game. DRRN and TDQN learning curves are provided for their respective experiments.
- Evaluation protocol: Episodes terminate after 100 valid steps or when the game ends in game over or victory.Only valid actions count toward the 100-step limit.
- Training: DRRN and TDQN are trained individually on each game.DRRN uses 100,000 updates and data from 16 parallel environments, totaling 1.6 million interactions; TDQN uses 1 million environment steps.
- Learning curves: Figure 4 presents learning curves for DRRN.The supplied caption identifies the algorithm shown but does not specify axes or outcomes.
- Learning curves: Figure 5 presents learning curves for TDQN.The supplied caption identifies the algorithm shown but does not specify axes or outcomes.
B Select Transcripts
The transcript follows an agent exploring Zork by opening the mailbox, collecting a leaflet, entering the house, obtaining a sword and lantern, and descending into the cellar. Its displayed Q-values prioritize contextually relevant actions across changing observations.
- Opening sequence: The agent begins west of the house, opens the mailbox, and takes the leaflet revealed inside.Both actions receive positive reward-free transitions while the score remains 0.
- Entering the house: After moving around the house, the agent opens a window and enters the kitchen.The kitchen observation includes a water-filled bottle, a sack, and routes to other areas.
- Living room: In the living room, the agent reveals a trap door and takes the elvish sword and brass lantern.The rug exposes the trap door, and the later inventory lists both the sword and lantern as carried.
- Descent: The agent descends through the trap door, turns on the lantern in darkness, and reaches the cellar.Descending yields Reward10 of 25 and Score 35; turning on the lantern precedes the cellar observation.