Source-linked AI summary
Learning to Speak and Act in a Fantasy Text Adventure Game
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, Jason Weston
TL;DR
The paper addresses how dialogue agents can use explicit world knowledge rather than dialogue history alone while speaking and acting in a shared environment. It introduces LIGHT, a large crowdsourced fantasy text-adventure platform and evaluates generative and retrieval models with environmental grounding. Grounding improves predictions of behavior and dialogue, although the strongest models remain below human performance.
Problem
Existing dialogue models generally use language regularities without explicit understanding or perception of the world that language describes.
Method
The paper builds LIGHT, a crowdsourced multiplayer fantasy world with human-human episodes, then evaluates retrieval and generative models conditioned on dialogue and local environmental state.
Results
Grounding models on locations, objects, characters, and previous actions improves predictions of agent behavior and dialogue, while BERT ranking models outperform existing methods but remain below human performance.
Takeaways & Limitations
LIGHT provides a challenging platform for studying grounded language learning in agents that can both speak and act.
Takeaways & Limitations
The environment considers actions and dialogues within a single location rather than requiring a full world map.
Abstract
from arXiv · showhide
We introduce a large scale crowdsourced text adventure game as a research platform for studying grounded dialogue. In it, agents can perceive, emote, and act whilst conducting dialogue with other agents. Models and humans can both act as characters within the game. We describe the results of training state-of-the-art generative and retrieval models in this setting. We show that in addition to using past dialogue, these models are able to effectively use the state of the underlying world to condition their predictions. In particular, we show that grounding on the details of the local environment, including location descriptions, and the objects (and their affordances) and characters (and their previous actions) present within it allows better predictions of agent behavior and dialogue. We analyze the ingredients necessary for successful grounding in this setting, and how each of these factors relate to agents that can talk and act successfully.
1 Introduction
The paper introduces LIGHT, a crowdsourced fantasy text-adventure platform for studying dialogue grounded in a shared world. Models use dialogue and environmental information to predict speech and actions, though they remain below human performance.
- LIGHT is a multiplayer fantasy text-adventure world designed for studying situated dialogue among humans, models, and embodied agents.
- The world contains 663 locations, 3,462 objects, and 1,755 characters described entirely in natural language.
- The dataset includes 11k character-driven human-human episodes combining actions, emotes, and dialogue.
- BERT-based bi-ranker and cross-ranker models outperform existing methods on dialogue and action prediction tasks.
- Grounding in locations, objects, characters, and agents’ actions improves behavior and language prediction, but the best models remain below human level.
2 Related Work
Prior grounded-language environments typically support only part of the interaction problem: dialogue without situated perception, action without two-way dialogue, or perception without action. LIGHT combines grounded perception, action, and two-way human dialogue in a larger fantasy world.
- Most dialogue models rely on recent utterances and lack explicit grounding in the objects and actions of the external world.
- Single-player text adventures and restricted instruction-following environments support actions but do not provide comparable human dialogue or open-ended interaction.
- Visual platforms commonly support perception and action but use templated language or omit natural two-way dialogue.
- Talk the Walk combines action, perception, and two-way dialogue, but is limited to small grids.
- LIGHT learns from both actions and two-way dialogue, with human-produced utterances that retain ambiguity and coreference.
3 LIGHT Environment and Task Setup
LIGHT is a configurable multiplayer fantasy environment in which embodied characters perceive natural-language settings, act or emote, and converse. Its crowdsourced locations, entities, affordances, and episodes support training and in-situ evaluation, with separate seen and unseen tests.
- LIGHT is a large-scale configurable text adventure where humans and models act as embodied agents in a multiplayer fantasy environment.
- The environment is crowdsourced from locations, objects, affordances, characters, personalities, dialogues, and actions, enabling training and in-situ evaluation.
- The unseen test set uses distinct location categories and evaluates generalization to unseen topics within a similar domain.
- Physical actions have explicit game-state effects and validity constraints, whereas emotes notify nearby characters without changing the game state.
- Each episode places two characters and their available objects in a location, with each turn allowing one physical action or emote and one utterance.
- The dataset contains 10,777 crowdsourced dialogues collected from alternating two-character interactions.
4 Learning Methods
The paper evaluates ranking and generative models that condition speech, actions, and emotes on dialogue and structured world context. Its ranking systems include baselines, memory networks, and BERT bi- and cross-rankers, with constrained action candidates and standard automatic and human evaluations.
- Models represent persona, setting, dialogue, objects, characters, and actions as typed text context for predicting actions, emotes, or dialogue.
- Ranking baselines include random selection, TF/IDF information retrieval, Starspace, and fastText emote classification.
- The Transformer Memory Network encodes grounding information into memory slots and attends over them using dialogue context to score candidates.
- The BERT Bi-Ranker separately embeds context and candidates, while the Cross-Ranker concatenates them to build context-dependent candidate representations.
- Action ranking and generation are constrained to valid actions available in the current game state.
- Evaluation uses action and emote accuracy, dialogue Recall@1/20 for ranking, perplexity and unigram F1 for generation, and human ranking performance.
5 Results
Transformer-based models outperform simpler baselines across LIGHT’s prediction tasks, while human performance remains higher. Grounding with dialogue and environmental context improves predictions, although the most useful features vary by task and setting.
- 5.1 Comparison of Models and Baselines: Transformer architectures are significantly stronger across all tasks, with BERT pretraining important for the best Bi-Ranker and Cross-Ranker results.The Cross-Ranker performs better on action and emote prediction, while the generative Transformer performs less well under these metrics.
- 5.1 Comparison of Models and Baselines: Human performance remains above all evaluated models, leaving room for improvement.The ranking comparison reports human performance on a subset of the data.
- 5.2 Generalization Capability on Unseen Test: A 21-point human–model gap on unseen dialogue exceeds the 11-point gap on seen dialogue, highlighting a transfer challenge.The unseen settings are slightly easier in absolute numbers, but the relative gap to humans increases.
- 5.4 Importance of Grounding: All environmental information gives the best dialogue performance for retrieval and generative models, while adding past actions, persona, or setting generally improves scores.Training on dialogue alone substantially decreases performance; object descriptions provide the least improvement.
- 5.4 Importance of Grounding: Dialogue history substantially improves action and emote prediction over action history alone, whereas including every feature can challenge the model.The full input may be difficult to attend over, producing improved results for some ablations rather than uniformly better results.
- 5.4 Importance of Grounding: Changing environmental settings, dialogue partners, or dialogue input changes predicted dialogue, emotes, and actions in context-appropriate ways.The model retrieves setting-appropriate responses and predicts actions related to requested items and the identity of the asker.
6 Conclusion
LIGHT provides a large-scale fantasy text adventure platform in which humans and models act and speak within a rich environment. The work analyzes how models use environmental grounding to support grounded language and action.
- LIGHT enables both human and model agents to act and speak within a rich environment of locations, objects, and other characters.
- The platform supports analysis of how models leverage grounding information present in the environment.
- The authors position LIGHT as a platform for future research in grounded language learning and holistic world modeling.
Supplementary Material
The model input can include multiple grounding features, but the chosen representation is not exhaustive. Additional representations and environment information may improve performance in future work.
- The full input representation includes setting, objects, characters and personas, actions, emotes, and dialogue.These grounding features are presented as special-token-delimited input types.
- The authors did not explore other ways to represent the same grounding information, which could improve performance.
- Character possessions and object affordances are additional information that could potentially be encoded explicitly in the input.Object affordances still constrain available actions and are partially visible through action history.
B Bi-Ranker and Cross-Ranker Speeds
The speed trade-off between BERT bi-rankers and cross-rankers depends on task size and whether candidate representations can be cached. The appendix also documents input examples and limited name overlap in the unseen test set.
- B Bi-Ranker and Cross-Ranker Speeds: 4.6x slower: the Cross-Ranker is slower than the Bi-Ranker on the emote task when 22 candidate representations are cached.The Cross-Ranker remains feasible because the emote task has only 22 labels.
- B Bi-Ranker and Cross-Ranker Speeds: Caching makes the Bi-Ranker feasible for dialogue retrieval over more than 100,000 candidates, whereas the Cross-Ranker is infeasible to compute.Cross-Ranker label representations depend on the input and therefore cannot be cached.
- B Bi-Ranker and Cross-Ranker Speeds: The example model input combines special-token-labeled grounding information with a target label and a fantasy-game setting description.
- C Unseen Test Set Overlap: The unseen set contains exact name matches with training elements for 3 locations, 96 characters, and 203 objects.The associated chats use full location descriptions that are unseen in training, reducing overlap.
D Crowdsourcing Methodology
LIGHT’s crowdsourcing methodology used staged, task-specific collection and quality controls to produce consistent locations, characters, objects, actions, emotes, and dialogues. The authors also document residual ambiguities and annotation errors that constrain grounding quality.
- Quality control: Staged pilots and task revisions reduced discarded content from nearly half of each initial pilot task to less than 1 in 30 tasks.The authors used iterative filtering rather than collecting all content in one pass.
- Locations: Location collection was split into core content and room annotation tasks because the original description task was too disjoint.Tasks 1 and 2 were used sequentially to produce locations.
- Limitations: Residual issues included context-insensitive room annotations, vague personas for collective terms, incompatible object descriptions, and spelling or grammar that could reflect either mistakes or character voice.Some problematic annotations remained after explanatory examples, spot checks, filtering, and staged quality control.
- Characters: Character collection likewise used separate cleanup and description tasks, with sentence validation to flag potentially bad personas and descriptions.Tasks 3 and 4 handled span cleanup, singularization, personas, descriptions, and carried or worn objects.
- Dialogues: Dialogue collection required onboarding tests, mutual worker feedback, response timeouts, and manual example validation because pairing workers made quality highly interdependent.The onboarding task tested whether workers could integrate personas, settings, characters, and available actions.
E Descriptions of Actions and Emotes
LIGHT represents actions through a state-based graph whose allowable operations depend on node attributes and relations, while emotes were deliberately reduced to a compact set. These choices support tractable grounded interaction at some cost to expressive coverage.
- Actions: LIGHT represents game state as graph nodes and edges, with actions constrained by relations and object affordances such as pickup, containment, eating, drinking, wearing, and wielding.Crowdsourced affordances determine which attributes nodes receive and therefore which actions are available.
- Emotes: The emote inventory was reduced from related MUD options to 22 emotes to limit redundancy and crowdsourcing complexity.The authors explicitly accepted reduced expressiveness, selecting one among scream, shout, and yell.
F Descriptions of Human Evaluations
The evaluation examines worker consistency and dataset structure through onboarding filters and embedding visualizations. These analyses probe both the interconnectedness of grounded elements and the dataset’s coverage.
- Human evaluation: Worker onboarding used scenario-based correctness checks for dialogue, emote, and action inputs, followed by training-set performance as a secondary filter.The filters were intended to identify workers capable of representing human task performance.
- Embedding analysis: Starspace embeddings trained without pretrained vectors visualize interconnectedness among locations, objects, characters, and actions learned from LIGHT.Because inputs and outputs share an embedding space, nearest neighbors can expose relationships among element types.
- Embedding analysis: Pretrained GLoVe embeddings are averaged across words in each dataset element to visualize coverage independently of LIGHT-trained relationships.The comparison uses Common Crawl-trained vectors and t-SNE plots of fixed element representations.
H Action and Emote Relationships
LIGHT records physical actions and emotional responses as linked interaction events, enabling analysis of how partners respond across one timestep. The observed behavior is diverse but includes recognizable reciprocal patterns.
- Action–emote analysis: Figure 19 heatmaps count immediate partner responses to actions and emotes, clustering physical actions by root word for visualization.The analysis limits action relationships to a single timestep, although responses may unfold over multiple turns.
- Dataset structure: The available action set is documented with application constraints, while the environment’s location categories support separate seen and unseen settings.The supplied captions identify the action constraints table and the location-category table but do not provide their contents.
- Action–emote relationships: Hit actions are most likely to be answered by another hit, while hug actions are similarly reciprocated.These are highlighted as notable trends amid many possible physical and emotional responses.