PaperlayerView exact revision on arXiv

Source-linked AI summary

Keep CALM and Explore: Language Models for Action Generation in Text-based Games

Shunyu Yao, Rohan Rao, Matthew Hausknecht, Karthik Narasimhan

arXiv:2010.02903v1cs.CL

TL;DR

Text-based games pose enormous action spaces and require agents to understand language while identifying admissible and promising commands. CALM learns context-conditioned action generation from human gameplay and combines it with reinforcement learning to re-rank candidates. On unseen Jericho games, the approach improved average normalized score by 69% over the previous state of the art and competed with models given admissible-action information.

  • Problem

    Combinatorially large action spaces and the need for syntactic, semantic, and game-related knowledge make text-based-game action selection difficult for reinforcement-learning agents.

  • Method

    CALM trains language models on human gameplay to generate context-conditioned action candidates, which a DRRN reinforcement-learning agent evaluates using learned reward values.

  • Results

    69% relative improvement in average normalized score over the previous state-of-the-art method was achieved across 28 Jericho games, with performance exceeding handicap-assisted models on 8 games.

  • Takeaways & Limitations

    Human gameplay can provide linguistic priors and game sense that support high-quality action generation on games unseen during CALM training.

  • Takeaways & Limitations

    Text-based games remain far from solved because sparse rewards and partial observability challenge current agents, even with ground-truth admissible actions.

Abstract

from arXiv · show

Text-based games present a unique challenge for autonomous agents to operate in natural language and handle enormous action spaces. In this paper, we propose the Contextual Action Language Model (CALM) to generate a compact set of action candidates at each game state. Our key insight is to train language models on human gameplay, where people demonstrate linguistic priors and a general game sense for promising actions conditioned on game history. We combine CALM with a reinforcement learning agent which re-ranks the generated action candidates to maximize in-game rewards. We evaluate our approach using the Jericho benchmark, on games unseen by CALM during training. Our method obtains a 69% relative improvement in average game score over the previous state-of-the-art model. Surprisingly, on half of these games, CALM is competitive with or better than other models that have access to ground truth admissible actions. Code and data are available at https://github.com/princeton-nlp/calm-textgame.

1 Introduction

Text-based games challenge language-operating agents with enormous action spaces and the need to identify promising actions using linguistic and game-related knowledge. CALM addresses this by generating context-conditioned candidates from human gameplay and pairing them with reinforcement learning, achieving strong generalization and benchmark performance.

  • Combinatorially large action spaces continue to challenge reinforcement-learning agents in text-based games.
  • Only a tiny fraction of commands are admissible in each state, requiring both syntactic and semantic knowledge to filter actions.
  • Human game sense helps prioritize promising admissible actions, such as “move rug” over “knock on door” when the door is nailed shut.
  • CALM generates action candidates from game context, while DRRN uses rewards to learn values and select actions suited to the game.
  • 426 human gameplay transcripts spanning 590 games train one CALM instance for deployment across downstream games.Evaluation-game transcripts are excluded from language-model training to demonstrate generalization.
  • 69% relative improvement in average normalized score over the previous state-of-the-art method was achieved across 28 Jericho games.CALM also outperformed handicap-assisted methods on 8 games, including INHUMANE, where it scored 25.7 versus 3 for KG-A2C.
  • The work contributes a learning-based approach for reducing text-game action spaces and a human-gameplay dataset with an action-generation evaluation scheme.

2 Related Work

Prior work used reinforcement learning, action-space handicaps, and linguistic priors to address text-game control, while related interactive-language systems commonly generate and re-rank candidates for different objectives.

  • Early reinforcement-learning agents operated in synthetic text games with small predefined action spaces.Action filtering methods improved learning convergence even in settings with fewer than 200 actions.
  • Jericho introduced challenging man-made games with greater linguistic variation and larger action spaces than earlier frameworks.
  • Jericho’s admissible-action handicap supplies reduced action spaces or auxiliary training signals to agents such as DRRN, TDQN, and KGA2C.
  • Linguistic-prior approaches generated commands using affordance embeddings, n-gram language models, and related sequence models.
  • CALM’s figure-level pipeline trains on human transcripts, generates context-conditioned candidates, and lets DRRN select among them using Q-values.
  • Candidate generation and re-ranking also appear in dialogue and communication games, where re-ranking often targets fluency or accuracy rather than future task rewards.

3 Method

CALM reduces text-game action spaces by generating context-conditioned candidates from human gameplay, then DRRN uses reinforcement learning to rank them for gameplay.

  • Background: Text-based games expose agents to large action spaces where RL policies become intractable to explore directly.Only a tiny fraction of commands are admissible, and admissibility requires syntactic and semantic knowledge.
  • Contextual Action Language Model: CALM trains language models on human gameplay to generate compact action candidates conditioned on game context.The default context uses recent observations and actions, although the n-gram model conditions only on the current observation.
  • Language Models: The n-gram model scores candidate combinations of training-derived verb phrases and context-detected nouns, while GPT-2 models flexible context-action dependencies.GPT-2 generates likely actions with beam search; the n-gram model selects high-probability actions from a restricted action space.
  • Reinforcement Learning Integration: DRRN estimates Q-values over CALM’s top-k candidates and replaces the full action space in its policy and learning equations.This combines generic linguistic action priors with reinforcement learning optimized for gameplay.
  • Method Summary: CALM provides a reduced action space that can be combined with DRRN or other reinforcement learning agents.The approach is presented as simple and generic rather than specific to one RL agent.

4 Experimental Setup

The experiments train CALM on human gameplay transcripts and evaluate it on 28 unseen Jericho games, measuring generated-action quality and RL gameplay performance against established baselines.

  • Data and Environment: The ClubFloyd dataset contains 426 human gameplay transcripts covering 590 games and 223,527 context-action pairs.Samples with meta-actions or observations over 256 tokens are removed, and common actions occupy a large portion of the data.
  • Data and Environment: Evaluation uses 28 man-made Jericho text games, with observations augmented by location and inventory descriptions.The games are distinct from the transcripts used to train both CALM models.
  • Data and Environment: The Jericho admissible-action handicap enumerates verb-object combinations and tests them against simulator states, but may be unavailable outside Jericho.Jericho also supplies optimal walkthrough trajectories used for evaluation.
  • CALM Setup: CALM generates the top k = 30 actions for each unique game state, using restricted enumeration for n-gram and beam search for GPT-2.The n-gram setup adds directional actions and limits generated actions per object to encourage diversity.
  • RL Agent Setup: The RL evaluation trains DRRN asynchronously on eight parallel environments for 10^6 steps and averages scores across five random-seed runs.Baselines include NAIL, DRRN, and KG-A2C; DRRN and KG-A2C use ground-truth admissible actions, unlike CALM.
  • CALM Evaluation: Action-generation quality is evaluated with precision and recall on gold and admissible actions across walkthrough trajectories of 28 games.Figure 4 reports metrics as a function of k, with shaded areas showing standard deviation.

5 Results

CALM generates compact action sets and combines them with reinforcement learning to balance admissibility, action quality, and exploration. Across Jericho, GPT-2 CALM improves over baselines, while analyses show gains from complex-action generation, pretraining, and better exploitation of explored trajectories.

  • 5.1 Evaluating CALM on walkthroughs: CALM evaluates generated actions using admissible-action precision and recall, plus recall of the walkthrough’s gold action across 28 Jericho games.The evaluation uses walkthroughs because human actions may be non-optimal or inadmissible, and multiple actions can be admissible.
  • 5.1 Evaluating CALM on walkthroughs: At k = 15, CALM’s GPT-2 and n-gram variants contain around 30% of admissible actions; increasing GPT-2’s set to k = 30 adds 10% more admissible actions.The larger action set improves recall, but action quality declines as k increases.
  • 5.2 Evaluating gameplay on Jericho: CALM (GPT-2) outperforms CALM (n-gram) and NAIL by 4.4% and 3.8% on absolute normalized scores, respectively, representing almost a 69% improvement over NAIL.The comparison uses average normalized scores across Jericho games.
  • 5.2 Evaluating gameplay on Jericho: CALM (GPT-2) outperforms handicap-assisted DRRN and KG-A2C on 8 of 28 games, with normalized scores more than 20% higher on DETECTIVE, IN-HUMANE, and SNACKTIME.The paper hypothesizes that CALM excludes unhelpful admissible actions and may discover actions missed by the handicap.
  • 5.3 Analysis: Pretraining and ClubFloyd training are both important: pretraining supplies linguistic priors, while gameplay data conditions actions toward text-based games.Adding Jericho transcripts improves average normalized score, whereas a randomly initialized model without pretraining performs substantially worse.
  • 5.3 Analysis: Complex actions captured when k > 10 make GPT-2 much better than n-gram, while CALM with a random agent scores only 1.8%.The results indicate that action generation alone is insufficient without reinforcement learning to select actions adaptively.
  • 5.3 Analysis: If reinforcement learning fully exploited the trajectories explored under CALM’s action space, average normalized score would reach 14.7%, exceeding every model in Table 2.The current DRRN agent cannot stably converge to the maximum score observed during exploration on some games.

6 Conclusion

CALM uses language models to generate contextually relevant action candidates for reinforcement learning agents, improving performance on unseen text-based games. The benchmark results also show that these games remain unsolved despite access to admissible actions.

  • CALM generates action candidates for reinforcement learning agents using linguistic priors and human gameplay experience.The approach targets diverse games and contexts, including games unseen during training.
  • 69% improvement in average normalized score over previous approaches was achieved on the Jericho benchmark.The reported improvement is measured as average normalized score.
  • CALM was competitive with models using ground-truth admissible actions on many games.This result was reported alongside the benchmark improvement.
  • Text-based games remain far from solved because sparse rewards and partial observability challenge agents even with ground-truth admissible actions.The conclusion identifies these difficulties as continuing challenges for current agents.

A ClubFloyd Dataset

The ClubFloyd dataset consists of cleaned human gameplay transcripts converted into observation-action training examples. After preprocessing, it contains 223,527 examples spanning hundreds of games.

  • 426 ClubFloyd transcripts span over 500 text-based games and were collected from experienced players who sometimes made mistakes.The logs come from people who regularly played interactive fiction games, but they were not necessarily familiar with each game.
  • The transcripts were cleaned to extract observations and actions for model training.The preprocessing pipeline converts raw gameplay logs into structured examples.
  • Each training example groups consecutive observations and actions, padding the first example with an initial observation and action.The initial padding uses “You are at the start of your journey” and “begin journey.”
  • 223,527 examples remained after preprocessing.

B CALM Training

The training section provides implementation details for CALM's GPT-2, n-gram, and variant models.

  • CALM training includes details for GPT-2, n-gram, and variant models.

B.1 CALM (GPT-2)

CALM (GPT-2) is trained on ClubFloyd transcripts with held-out validation data, using pretrained or randomly initialized GPT-2 variants. Training minimizes cross-entropy over human gameplay actions, with multiple data-scale and dataset variants.

  • CALM (GPT-2) uses a pretrained 12-layer, 768-hidden, 12-head, 117M-parameter GPT-2 model.
  • 193,588 samples from ClubFloyd transcripts excluding Jericho games are split into 90% training and 10% validation data.
  • The 50% and 20% variants use 94,609 and 38,334 samples, respectively, from transcripts excluding Jericho games.Both variants use 90% of their data for training and 10% for validation.
  • The Jericho-inclusive variant uses 206,286 samples after including and randomly ordering every ClubFloyd transcript.The data are split into 90% training and 10% validation sets.
  • The randomly initialized variant uses only ClubFloyd transcripts unrelated to the tested Jericho games.Its samples are randomly ordered before the 90%-10% training-validation split.
  • GPT-2 training minimizes cross-entropy between the model's action distribution and the human action, optimized with Adam.The setup uses learning rate 2e-5, Adam epsilon 1e-8, warmup, and gradient clipping at norm 1.
  • Training metrics include train and validation loss and the percentage of examples where GPT-2 assigns highest probability to the ClubFloyd action.

B.2 CALM (n-gram)

CALM’s n-gram model is trained on action sequences from ClubFloyd transcripts, using smoothed conditional probabilities to model action generation.

  • The n-gram model trains on actions from ClubFloyd transcripts, excluding Jericho games.Actions are ordered by transcript, with the first 70% used for training and the remaining 30% for validation.
  • Laplace α smoothing is used, with α selected to minimize validation perplexity per word for each n.
  • The model estimates each action from preceding action contexts, combining four smoothed n-gram estimates whose weights sum to 1.The interpolation uses contexts of up to three preceding actions, along with shorter contexts and the unigram estimate.

C Walkthrough Evaluation

Walkthrough evaluation compares generated actions with game-specific trajectories, showing that GPT-2 has a larger advantage over n-gram on harder games than on easier ones.

  • Walkthrough evaluation presents GPT-2 and n-gram generated actions at each state in Zork1.The evaluation uses walkthrough trajectories to assess generated-action quality beyond standard validation metrics.
  • GPT-2 generates more complex actions such as “put sword in case,” while n-gram actions are mostly limited to two tokens or fewer.
  • On harder games like Zork1, the gap between GPT-2 and n-gram is significant, whereas on easier games like Detective it is very small.

D Gameplay Evaluation

Gameplay evaluation examines CALM learning curves, action-candidate settings, trajectories, and per-game variants across Jericho games. Results include variant-specific differences and examples of CALM candidates being ranked with Q-values during play.

  • Learning curves: The evaluation reports learning curves for CALM (GPT-2) and CALM (n-gram) on Zork1, plus CALM (GPT-2) curves on Zork3.Zork3 is identified as a game where the method is behind NAIL.
  • Action candidates: On Zork1, CALM (GPT-2) is evaluated with top-k action sets of k = 10, 20, 30, and 40.
  • Per-game results: Per-game results show CALM (w/ Jericho) is significantly better than CALM (GPT-2) on Temple and Deephome.These games achieved non-trivial scores despite not having ClubFloyd scripts added.
  • Per-game results: CALM (w/ Jericho) does not improve games such as 905 and moonlit, even though those games have scripts added.
  • Example trajectory: A DRRN + CALM (GPT-2) trajectory shows generated action candidates, their Q-values, observations, actions, and scores.
  • Evaluation reporting: Table 5 reports raw scores for CALM variants on each game, with some scores based on one trial and selected variants based on five trials with standard deviations.
Loading 2010.02903v1…