PaperlayerView exact revision on arXiv

Source-linked AI summary

LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models

Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, Sergey Levine

arXiv:2311.18232v1cs.CLcs.AIcs.LG

TL;DR

Multi-turn LLM applications need goal-directed planning, but existing RL research lacks accessible, reproducible benchmarks for comparing algorithms in interactive settings. The paper introduces LMRL-Gym, an eight-task benchmark with offline datasets, simulators, and an RL toolkit; RL methods outperform behavioral cloning across tasks, although no RL algorithm is a clear overall winner.

  • Problem

    Existing LLM methods are not explicitly goal-directed, while multi-turn RL research lacks accessible evaluations and head-to-head algorithm comparisons.

    Source: §1 · p. 2

  • Method

    LMRL-Gym combines 8 multi-turn language tasks, offline datasets, online-capable simulators, and a toolkit implementing PPO, ILQL, and baselines.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    RL algorithms outperform BC methods across all tasks, but no RL algorithm is a clear winner across the benchmark.

    Source: §6 · p. 7

  • Takeaways & Limitations

    LMRL-Gym provides an accessible framework for iterating on RL algorithms for multi-turn language interaction and goal-directed decision-making.

    Source: §1 · p. 2, §7 · p. 9

    The benchmark uses synthetic simulators, sacrificing realism, and its RL capability tests are not primarily designed for complex language.

    Source: §4.2.1 · p. 5, §8 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) provide excellent text-generation capabilities, but standard prompting and generation methods generally do not lead to intentional or goal-directed agents and might necessitate considerable prompt tuning. This becomes particularly apparent in multi-turn conversations: even the best current LLMs rarely ask clarifying questions, engage in explicit information gathering, or take actions now that lead to better decisions after multiple turns. Reinforcement learning has the potential to leverage the powerful modeling capabilities of LLMs, as well as their internal representation of textual interactions, to create capable goal-directed language agents. This can enable intentional and temporally extended interactions, such as with humans, through coordinated persuasion and carefully crafted questions, or in goal-directed play through text games to bring about desired final outcomes. However, enabling this requires the community to develop stable and reliable reinforcement learning algorithms that can effectively train LLMs. Developing such algorithms requires tasks that can gauge progress on algorithm design, provide accessible and reproducible evaluations for multi-turn interactions, and cover a range of task properties and challenges in improving reinforcement learning algorithms. Our paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for LLMs, together with an open-source research framework containing a basic toolkit for getting started on multi-turn RL with offline value-based and policy-based RL methods. Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.

1. Introduction

LLMs can generate fluent text but are not explicitly optimized for goal-directed, multi-turn planning. LMRL-Gym addresses the resulting evaluation gap with accessible tasks, simulators, datasets, and an RL research toolkit.

  • LLMs are not explicitly goal-directed, so multi-turn dialogue, games, and tool use require planning and optimization beyond text imitation.Source: §1 · p. 1
  • Multi-turn RL research lacks stable algorithm comparisons because interactive evaluation is costly and lacks an established protocol.Source: §1 · p. 2
  • The benchmark uses LLM simulators to make multi-turn RL evaluation accessible, while targeting algorithm performance rather than human conversational ability.Source: §1 · p. 2
  • LMRL-Gym provides 8 tasks with offline datasets, simulators for evaluation and online training, and implementations of PPO, ILQL, and baseline methods.Source: §1 · p. 2

2. Related Works

Prior language-model benchmarks cover many supervised or single-step capabilities, but generally do not evaluate long-horizon objectives through multi-turn interaction. LMRL-Gym targets this gap with synthetic tasks, rewards, and online simulators.

  • Many NLP benchmarks lack multi-turn interaction and rewards, while human evaluations of dialogue agents are costly and time-consuming.Source: §2 · p. 2
  • Prior RL-for-language work includes online and offline methods, but many studies remain in single-step bandit settings rather than multi-turn goal-directed tasks.Source: §2 · p. 3
  • LMRL-Gym focuses on multi-turn tasks with clearly defined goal-based rewards and synthetic online simulators plus offline datasets.Source: §2 · pp. 2–3
  • Supervised language-model capability research often produces plausible dialogue without explicitly testing achievement of a long-term objective.Source: §2 · p. 3

3. Multi-Turn Generation with RL and Language Models

The paper formulates language generation as partially observable sequential decision-making over token histories. It covers policy-gradient and value-based RL, emphasizing offline learning while retaining online evaluation and training support.

3.1. Definitions

  • Language generation is modeled as a partially observable Markov decision process whose state can be the concatenated token history and whose action is the next token.Source: §3.1 · p. 3
  • A policy generates action tokens while the environment supplies tokens and rewards based on the conversation history, forming complete token trajectories.Source: §3.1 · p. 3

3.2. RL Algorithms

  • PPO directly optimizes expected reward with policy gradients, whereas value-based methods estimate Q or V functions and derive actions from them.Source: §3.2 · p. 3
  • Offline RL learns from existing potentially suboptimal datasets, while online RL collects additional environment data; the paper primarily benchmarks offline methods and includes online PPO.Source: §3.2 · p. 3

4. The LMRL-Gym: Synthetic Benchmarks for RL with Language

LMRL-Gym combines five RL capability tests with three interactive dialogue tasks to evaluate multi-turn language RL across planning, language understanding, credit assignment, partial observability, and trajectory stitching. Its synthetic datasets and simulators prioritize accessible iteration while accepting reduced realism.

4. The LMRL-Gym: Synthetic Benchmarks for RL with Language

  • LMRL-Gym contains 8 tasks split into 5 RL capability tests and 3 interactive dialogue tasks spanning games, information gathering, and negotiation.Source: §4 · p. 3, §4.2 · p. 4

4.1. Evaluating Capabilities Enabled by RL

  • The benchmark targets strategic decision-making, complex language, credit assignment, partial observability, and trajectory stitching.Source: §4.1 · p. 4

4.2.1. The RL Capability Tests

  • The capability tests use text-based games designed to isolate RL properties and include suboptimal data requiring offline methods to assemble optimal trajectories.Source: §4.2.1 · p. 4
  • Maze and Text-Nav include fully observed and partially observed variants, while Chess, Endgames, Maze, and Text-Nav test credit assignment under outcome-dependent conditions.Source: §4.2.1 · pp. 4–5

4.2.2. Interactive Dialogue Tasks

  • The dialogue tasks cover yes-or-no information gathering in 20 Questions, open-ended strategic questioning in Guess My City, and strategy-dependent negotiation in Car Dealer.Source: §4.2.2 · p. 5
  • LLM simulators generate conversational data, provide evaluation environments, support online training, and compute rewards.Source: Figure 3 · p. 5, §4.3 · p. 6

4.3. An Overview of Data Collection for LMRL-Gym Tasks

  • All tasks use synthetic simulators or text-game environments to improve accessibility and rapid iteration, sacrificing some realism.Source: §4.3 · pp. 5–6

5. LMRL Research Framework for Algorithm Development

The LMRL research framework evaluates offline and online RL methods for multi-turn language tasks using GPT2-based policies and value networks. Training starts from behavioral cloning, and performance is measured by average reward in simulated environments.

5. LMRL Research Framework for Algorithm Development

  • The framework evaluates BC, filtered BC, MC Returns, ILQL, PPO, Online Filtered BC, and few-shot GPT4 across the benchmark tasks.GPT2 and its size variants serve as the policy and value networks, with human evaluation used to assess simulator conversation naturalness.Source: §5 · p. 7
  • Filtered BC fine-tunes only the most successful dataset examples, while Online Filtered BC repeatedly collects and filters trajectories from the current policy.Ordinary BC tests whether language models can represent dataset behaviors.Source: §5 · p. 7
  • MC Returns predicts token-level reward-to-go, whereas ILQL learns Q and V functions and uses them to perturb the behavioral-cloning model’s logits.ILQL trains two Q functions with Bellman backups before modifying the original BC policy.Source: §5 · p. 7
  • PPO directly learns a policy through online reinforcement learning, allowing inference-time sampling from the learned policy.Unlike MC Returns and ILQL, PPO is not based on separately learned value functions for policy selection at inference.Source: §5 · p. 7

5.1. Training and Evaluation Protocol

  • Models are initialized from pretrained GPT2 for BC methods and from BC checkpoints for subsequent RL fine-tuning, with policies evaluated by average simulated-environment reward.PPO training uses fewer than 100k samples, and task-specific hyperparameters are reported in Appendix D.Source: §5.1 · p. 7

6. Benchmarking Baseline RL Methods

Across LMRL-Gym tasks, RL methods outperform BC baselines, but algorithm rankings vary by task and important challenges remain. PPO benefits from online exploration on some tasks, while ILQL, MC Returns, partial observability, and prompting show distinct strengths and limitations.

6. Benchmarking Baseline RL Methods

  • RL algorithms outperform BC methods across the benchmark, although no RL algorithm is a clear overall winner.Table 9 reports results for each method across the benchmark tasks.Source: §6 · p. 7
  • Online PPO exceeds offline methods on PO Text-Nav, Chess, and Endgames, while ILQL outperforms PPO on Maze and Wordle.PPO performs comparably to ILQL and better than MC on FO Text-Nav, and leads the other methods by a large margin on Endgames.Source: §6 · pp. 7–8
  • Normalized rewards use 0 for the minimum possible return, 50 for the dataset average, and 100 for the maximum return on each task.This normalization makes average-return results more comparable across tasks; raw scores and evaluation details appear in Appendix E.Source: Table 2 · p. 8
  • Online PPO outperforms Online Filtered BC on every task except Wordle, but its hyperparameter sensitivity and training instability increase practical difficulty.The comparison is attributed to PPO’s exploration and credit-assignment abilities, alongside additional training complexity and resource requirements.Source: §6 · p. 8
  • ILQL outperforms MC Returns on Maze, Text-Nav, and Wordle, whereas MC performs better on Endgames, 20Qs, Guess My City, and Car Dealer.The authors suggest that simpler MC may perform better with more complex text because full TD-learning is harder to scale in those settings.Source: §6 · p. 8
  • Every algorithm shows lower normalized performance on partially observed than fully observed Maze and Text-Nav tasks.Partial observability requires inferring state and taking information-seeking actions, exposing an additional challenge for language-model RL.Source: §6 · p. 8
  • Prompted GPT4 fails to outperform RL fine-tuning baselines on most tasks, with 20Qs and Guess My City as notable exceptions.The exceptions may reflect that their simulator-training data was originally generated by GPT3.5.Source: §6 · p. 8
  • The benchmark results identify unresolved problems in PPO stability and in scaling TD-learning methods to interactive dialogue.The authors present these gaps as opportunities for improving RL algorithms for language models.Source: §6 · p. 8

7. Discussion

LMRL-Gym provides eight multi-turn tasks and a research toolkit with online simulators and offline datasets for developing RL methods for language agents. The evaluation shows promise for RL while retaining substantial room for improvement.

7. Discussion

  • LMRL-Gym spans eight tasks from Maze and Chess to Car Dealer and provides a toolkit for multi-turn RL research.The benchmark is designed to support complex decision making, conversational interaction, credit assignment, and trajectory stitching.Source: §7 · p. 9
  • Online simulators and offline datasets are intended to help researchers iterate on more effective methods for language-based multi-turn interaction.The evaluation reports promise in several tasks while also showing room for better methods.Source: §7 · p. 9
  • The benchmark prioritizes accessibility by primarily training and evaluating models with at most 1.5B parameters.The authors retain the lower parameter count to support researchers with limited computational resources.Source: §7 · p. 9

8. Ethics Statement

The ethics statement recognizes that interactive RL and LLM simulators can be used for persuasion, manipulation, and addictive engagement. The authors describe fairness- and alignment-oriented dataset and reward design as mitigation efforts.

8. Ethics Statement

  • The benchmark datasets are generated with existing LLMs for dialogue and online engines for text games, with steps intended to exclude personally identifiable or sensitive information.The authors also acknowledge possible dataset biases and describe efforts to collect varied conversational responses.Source: §8 · p. 9
  • Interactive RL has dual-use risks because LLM simulators could support large-scale persuasion, manipulation, or addictive engagement.The concern centers on optimizing system objectives in ways that may prioritize those objectives over user safety and human values.Source: §8 · p. 9
  • The authors designed datasets and reward functions to prioritize fairness and human-aligned outcomes.They present these design choices as efforts to mitigate undue persuasion or manipulation.Source: §8 · p. 9

A. Further Details on Task Design

The benchmark organizes multi-turn RL around capability-specific tests and interactive dialogue tasks, with simulators, clear rewards, and datasets designed to expose learning challenges. Its task suite spans partial observability, strategic decision-making, information gathering, negotiation, credit assignment, and trajectory stitching.

A.1. RL Capability Tests

  • The RL Capability Tests target distinct challenges through paired maze and navigation settings, Wordle, chess, and theoretical endgames.Maze and Text-Nav contrast observability and language complexity, while Wordle tests information seeking and chess tasks test strategic decision-making.Source: §A.1 · p. 14

C.3. Wordle

  • Wordle evaluates information-seeking in a partially observed game by providing letter-level feedback over at most six guesses.Words are represented as space-separated letters so standard tokenizers usually encode individual letters separately.Source: §A.1 · p. 14, §C.3 · p. 16

C.4. Chess

  • Chess and endgames test strategic decision-making and trajectory stitching using symbolic board representations and mixtures of optimal, random, or varied-strength play.The endgame task emphasizes strategy over opening memorization, while its dataset contains 1,086,314 unique states, representing 93% of states.Source: §A.1 · pp. 14–15, §C.4 · p. 16, §C.5 · p. 17

A.1.1. Interactive Dialogue Tasks

  • Interactive dialogue tasks cover information gathering, semantic reasoning, open-ended questioning, and negotiation through 20 Questions, Guess My City, and Car Dealer.20 Questions uses yes-or-no object identification, Guess permits open-ended city questions, and Car Dealer pairs buyer and seller strategies.Source: §A.1.1 · p. 15

B. Further details on desiderata for effective multi-turn RL

  • LLM-based dialogue simulators and scripted text-game environments replace infeasible repeated human evaluation while providing indicators of policy efficacy.The authors acknowledge that simulation may not perfectly reproduce natural social language.Source: §A.1.1 · p. 15, B. Further details on desiderata for… · p. 15
  • The benchmark emphasizes clear success measures, unit-test functionality, task-specific reasoning, and learning from suboptimal data.These desiderata support interpretable rewards, capability isolation, adaptation to unfamiliar environments, and improvement beyond behavior represented in datasets.Source: B. Further details on desiderata for… · pp. 15–16

C.1. Maze

  • Maze and Text-Nav datasets force trajectory stitching and credit assignment by separating successful behavior across submazes or room regions.Maze data restricts trajectories to submazes, whereas Text-Nav uses policies that are effective in different halves of the house.Source: §C.1 · p. 16, §C.2 · p. 16

C.6. Twenty Questions

  • Twenty Questions and Guess My City use 100K-conversation datasets with oracle environments, while question rewards penalize questions made before a correct guess.The 20 Questions corpus covers 158 objects, and Guess My City covers 100 cities with constraints preventing the oracle from revealing the answer directly.Source: §C.6 · pp. 17–18, §C.7 · p. 18

C.8. Car Dealer

Car Dealer models negotiation between buyers and dealers with distinct preferences and strategies. The examples illustrate that aligned strategies can produce agreements, whereas incompatible discount requirements can prevent a sale.

C.8. Car Dealer

  • Car Dealer uses three buyer and three dealer strategies so agents must infer counterpart preferences and learn which interactions lead to successful agreements.Buyer personas prioritize discounts, features, or impatience, while dealer personas differ in discounting, feature discussion, and concision.Source: §C.8 · p. 19
  • Incompatible strategies can prevent agreement when a buyer requires a discount but the dealer refuses to negotiate, resulting in a $0 purchase price.The example ends without a sale despite the buyer’s stated $90k budget because the available vehicle costs $95k and the dealer will not discount.Source: §C.8 · p. 19
  • Aligned preferences can produce a sale within the buyer’s budget, as the feature-focused dealer finds a Lexus priced at $69,500 for a $70,000 buyer.The buyer initially rejects a $72,000 option before accepting the lower-priced vehicle with the requested navigation system.Source: §C.8 · pp. 19–20
  • The task also includes cases where conversational behavior accepts a price above the buyer’s stated budget, exposing potential inconsistencies in negotiation outcomes.One example records a buyer with a $10,000 budget agreeing to a $15,000 price after repeated seller statements that the price is fixed.Source: §C.8 · p. 20

E. Evaluation Details

The evaluation details define task-specific normalization and protocols, compare offline and online RL baselines, and document implementation choices and PPO instability. Across chess endgames, PPO Online achieves the strongest overall reward, while PPO can also collapse during training on maze.

E.1. Maze

  • Maze evaluation averages 32 rollouts from each of 25 starting positions, with a 100-move limit and minimum reward of -101.Source: §E.1 · p. 23

E.2. Chess

  • Chess evaluation uses 1,000 games against Stockfish Elo 1200 and tracks illegal moves and average episode length as positions become increasingly out of distribution.Source: §E.2 · p. 23, Table 6 · p. 24

E.3. Chess Endgames

  • Chess endgame evaluation uses 645 held-out, nontrivial positions selected as mate-in-15-or-greater positions with Stockfish tools.Source: §E.3 · pp. 23–24
  • PPO Online achieves the highest overall reward in chess endgames, with a 0.13 gap over MC Returns, although BC-Engine wins more often and MC Returns and ILQL make fewer illegal moves.Source: Table 7 · p. 24
  • Fine-tuning BC on PPO-collected data does not match PPO Online, and performance shows no substantive difference between using the complete dataset and only its last 10 rounds.Source: Table 8 · p. 24, §E.3 · p. 24

E.4. Wordle

  • Wordle performance is measured by average reward over 4,096 trajectories.Source: §E.4 · p. 24

F.1. MC Details

  • MC fits a Q head to discounted return-to-go using mean-squared error, then perturbs the behavior-cloned policy with learned value functions during rollout.Source: §F.1 · pp. 24–25

F.2. PPO Details

  • PPO uses GAE value estimates, clipping, KL penalties, and sometimes a behavior-cloning loss, but its maze training can rise briefly before collapsing for uncertain reasons.Source: §F.2 · p. 25, Figure 5 · p. 25

G. Symbolic Maze Environment

The symbolic maze environment evaluates Q-learning separately from language-based tasks, while generated conversations illustrate trained-model behavior and human-evaluation prompts show the interaction format. Q-learning performs optimally in both online and offline symbolic-maze settings.

G. Symbolic Maze Environment

H. Generated Examples from Trained Models

I. Human Evaluation Examples

  • The 20 Questions evaluation asks agents to identify an oracle-selected object through up to 20 yes-or-no questions, favoring questions that narrow possibilities.Source: §I · pp. 26–27
  • An example identifies a football through sequential questions about indoor use, sports, ball status, throwing, bouncing, and candidate types.Source: §I · p. 27

J. User Study on Naturalness of Conversations

A user study compared the naturalness of GPT and MC-model conversations across three interactive tasks. Participants rated the conversations similarly, with no statistically significant difference between the models.

Related papers

Loading 2311.18232v1…