PaperlayerView exact revision on arXiv

Source-linked AI summary

GEM: A Gym for Agentic LLMs

Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Simon Yu, Xiangxin Zhou, Haotian Xu, Shaopan Xiong, Bo Liu, Chenmien Tan, Chuen Yang Beh, Weixun Wang, Hao Zhu, Weiyan Shi, Diyi Yang, Michael Shieh, Yee Whye Teh, Wee Sun Lee, Min Lin

arXiv:2510.01051v2cs.LGcs.AIcs.CL

TL;DR

LLM RL research has largely emphasized single-turn tasks, leaving a need for multi-turn testbeds suited to long-horizon interactions. GEM supplies a framework and the paper evaluates RL baselines, reporting that ReBN improves on vanilla REINFORCE and performs comparably to or better than PPO and GRPO across evaluated environments.

  • Problem

    Much LLM RL research focuses on single-turn tasks, which oversimplify multi-turn interactions despite the goal of training agents for long-horizon planning and iterative refinement.

    Source: §1 · p. 2

  • Method

    GEM provides a multi-turn environment framework, and the paper presents REINFORCE with Return Batch Normalization (ReBN) and compares RL algorithms.

    Source: §1 · p. 2

  • Results

    ReBN consistently improves on vanilla REINFORCE and outperforms or is comparable with PPO and GRPO in all evaluated environments.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    GEM provides a unified framework for agent-environment interaction and standardized environments, with the stated aim of accelerating agentic LLM research.

    Source: §6 · p. 9

    Math results should be interpreted comparatively within a consistent evaluation framework because small grading-logic differences can substantially change reported performance.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

The training paradigm for large language models (LLMs) is moving from static datasets to experience-based learning, where agents acquire skills via interacting with complex environments. To facilitate this transition we introduce GEM (General Experience Maker), an open-source environment simulator designed for the age of LLMs. Analogous to OpenAI-Gym for traditional reinforcement learning (RL), GEM provides a standardized framework for the environment-agent interface, including asynchronous vectorized execution for high throughput, and flexible wrappers for easy extensibility. GEM also features a diverse suite of environments, robust integrated tools, and single-file example scripts demonstrating using GEM with five popular RL training frameworks. Along with this, we also provide a set of baselines across 24 environments using REINFORCE with Return Batch Normalization (ReBN), which -- unlike GRPO -- is compatible with the full RL setting of dense per-turn rewards and offers better credit assignment. We further conduct apple-to-apple benchmarking of PPO, GRPO and REINFORCE in both single- and multi-turn settings using GEM to shed light on the algorithmic designs. Lastly, GEM also functions as a convenient evaluation toolkit besides a training environment. We hope this framework can help accelerate future agentic LLM research.

1 INTRODUCTION

Interactive RL offers a way for LLMs to learn open-ended tasks, but much current research focuses on single-turn settings. GEM addresses the need for multi-turn testbeds while the paper also presents ReBN and comparisons of RL algorithms.

1 INTRODUCTION

  • Interactive RL can support learning open-ended tasks without direct supervision, including software development and scientific discovery examples.The examples involve adapting to integration failures or evolving requirements, and adjusting strategies based on experimental results.Source: §1 · pp. 1–2
  • Current LLM RL research largely focuses on single-turn tasks, which oversimplify multi-turn interaction and do not fit the goal of training agents for long-horizon planning and iterative refinement.The passage names GRPO as an example of an algorithm fundamentally inapplicable to full multi-turn problems.Source: §1 · p. 2
  • GEM is an open-source framework for diverse, multi-turn, long-horizon tasks, with environments, wrappers, asynchronous execution, tools, baselines, and training scripts for five RL frameworks.Its environments include tasks spanning single-turn to over 100 turns.Source: §1 · p. 2
  • The paper presents REINFORCE with Return Batch Normalization (ReBN), reporting consistent improvements and compatibility with dense per-turn rewards and arbitrary discount factors, unlike GRPO.It also compares REINFORCE-based methods with multi-turn PPO and GRPO and examines discount factors and tool-integrated RL.Source: §1 · p. 2

2 GEM ENVIRONMENTS

GEM organizes tasks and tools as environments within a Gym-like interface. Its functionality includes extensible tasks, asynchronous vectorized execution, autoreset, and modular wrappers.

2.1 INTERFACE

  • GEM follows the OpenAI Gym API with reset() and step() for standardized agent-environment interaction.The interaction loop passes observations and actions through the environment, which returns the next observation, reward, termination and truncation flags, and info.Source: §2.1 · p. 2

2.2 TASKS AND TOOLS

  • GEM environments combine tasks with optional tools to test capabilities such as reasoning, planning, and tool use across seven task categories.Tasks include math, image-based math, code, games, QA, ReasoningGym, and terminal use; datasets, Docker files, or custom environment logic support integration, and tools can augment tasks.Source: §2.2 · p. 3
  • Adding tools to single-turn tasks can make them multi-turn, letting agents learn to call tools and adapt to their outputs.GEM supports Python execution, search, and external MCP-compatible tools.Source: §2.2 · p. 3

2.3 ASYNCHRONOUS VECTORIZATION AND AUTORESET

  • Asynchronous vectorized environments collect episodes in batches, while autoreset lets users continue calling .step() after termination without manually resetting each episode.The returned terminated flag can prevent critic value bootstrapping across episode boundaries.Source: §2.3 · p. 3, Figure 2 · p. 3

2.4 WRAPPERS

  • Observation wrappers control how episodes become observations, and tool wrappers let Python or database/web search be added to task environments.Observation options include recent output, concatenated history, history with actions, or parsed/summarized history.Source: §2.4 · pp. 3–4

3 REINFORCEMENT LEARNING WITH GEM

The section motivates an RL formulation that retains multi-turn rewards and discounting, then presents REINFORCE with Return Batch Normalization as a baseline that avoids shared-advantage credit-assignment limits and critic learning.

3.1 PRELIMINARY: LLMS AS AGENTS

  • Token-level actions create long episodes and difficult per-token reward design, while response-level actions simplify single-turn tasks but make sample-based advantage estimation infeasible in multi-turn settings.Multi-turn response-level sampling requires multiple episode completions from each state, leading to exponential complexity.Source: §3.1 · p. 4
  • Treating the whole interaction as one action makes GRPO applicable to multi-turn tasks but fixes γ = 1 and limits rewards to a single trajectory-level signal.These compromises remove incentives to solve problems quickly and lose fine-grained per-turn credit assignment.Source: §3.1 · p. 4

3.2 BASELINE ALGORITHMS

  • REINFORCE with Return Batch Normalization is compatible with per-step dense rewards and arbitrary discount factors γ ≤ 1, unlike GRPO.ReBN normalizes per-transition returns over the whole batch to provide fine-grained, stable advantage estimates without combinatorial sampling or an additional critic.Source: §3.1 · p. 4, §3.2 · p. 5
  • REINFORCE updates weight each state-action log probability by its return, with trajectories and token-sequence states and actions forming the batch.The return is the discounted sum of rewards from the current time step onward.Source: §3.2 · p. 5
  • Multi-turn GRPO gives all turns in a trajectory the same advantage, while critic-based GAE offers another way to estimate advantages using a learned value function.The text identifies poor credit assignment and tree-like sampling's combinatorial explosion as limitations of shared trajectory advantages.Source: §3.2 · p. 5

4 EMPIRICAL STUDIES WITH GEM

The empirical studies benchmark RL algorithms across GEM environments and examine discounting, tool integration, generalization, and framework compatibility. They report broad advantages for REINFORCE with ReBN, alongside findings on how environment and training choices shape outcomes.

4.1 BENCHMARKING RL ALGORITHMS FOR LLMS

  • GRPO performs reasonably on single-turn tasks but falls short in multi-turn settings; PPO leads on long-horizon Sudoku yet performs poorly in Minesweeper.The authors associate GRPO’s weakness with constant advantage estimates across steps and describe robust critic learning as difficult for PPO.Source: §4.1 · p. 6
  • REINFORCE with ReBN outperforms or matches PPO and GRPO across evaluated environments, while avoiding costly critic learning or extensive rollouts.It also consistently improves over vanilla REINFORCE.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 DISCOUNT FACTOR γ MATTERS

  • Smaller γ encourages fewer turns and convergence toward the approximately 5.6-turn optimum for GuessTheNumber, achievable through binary search.The authors note that this efficiency incentive from γ < 1 is incompatible with GRPO.Source: Figure 5 · p. 6, §4.2 · p. 7

4.3 TOOL-INTEGRATION IN MATH AND QUESTION-ANSWERING TASKS

  • Across five math environments, RL training improves performance over the base model, and Python-tool access yields higher final performance in every setting.Scores are comparative rather than absolute because reported performance is sensitive to grader implementation.Source: §4.3 · p. 7, Table 1 · p. 7
  • Across seven QA environments, RL fine-tuning improves performance, and models with Search achieve the highest accuracy in every scenario.The study compares training on HotpotQA alone with a HotpotQA and NaturalQuestions mixture.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 STUDYING GENERALIZATION

  • Training on game:sudoku-v0-easy yields encouraging initial generalization results when periodically evaluated on three other environments.Figure 6 describes generalization to ReasoningGym.Source: §4.4 · p. 8, Figure 6 · p. 8

4.5 INTEGRATION WITH TRAINING FRAMEWORKS

  • GEM integrates with five RL training frameworks, whose training curves show similar trends; asynchronous rollout in RL2 gives a 2× wall-clock efficiency gain.The authors attribute minor curve differences to framework design choices and RL stochasticity.Source: §4.5 · p. 8, Figure 7 · p. 8

5 AGENT EVALUATION WITH GEM

GEM also serves as a unified evaluation interface for tool-using agents, demonstrated on database operations through MCP and terminal tasks in Docker.

5.1 GENERAL TOOL USE VIA MODEL CONTEXT PROTOCOL

  • MCP server compliance enables plug-and-play tool use without custom integration; on 20 PostgreSQL MCPMark tasks, GPT-5 achieves the best success rate with fewest interactions.Gemini-2.5-Pro and Claude-Sonnet-4 have slightly lower and varied performance.Source: §5.1 · p. 9, Figure 8 · p. 9

5.2 TERMINAL ENVIRONMENT VIA DOCKER CONTAINER

  • On 57 Terminal-Bench tasks without scaffolding, GPT-5 achieves the highest success rate and most efficient interaction, followed by Claude-Sonnet-4 and Gemini-2.5-Pro.The evaluation uses the same interaction loop as RL training.Source: §5.2 · p. 9, Figure 8 · p. 9

6 CONCLUSIONS

GEM combines framework-agnostic infrastructure with standardized environments, and the paper illustrates how to register both extended and custom tasks.

6 CONCLUSIONS

  • GEM combines a framework-agnostic library, unified agent-environment interface, standardized environments, vectorized execution, and multi-turn REINFORCE implementations across five training frameworks.The paper also presents an algorithm benchmark and analyses of algorithmic details.Source: §6 · p. 9
  • Evaluation through a basic response-generation API rather than agent frameworks may cause deviations from original benchmark results.Source: §6 · p. 9

A ENVIRONMENT REGISTRATION

  • Environment registration covers adding datasets to existing tasks and defining custom environments through reset/step state, transition, and reward logic.The examples register a math dataset using GEM's existing answer verification and create a custom string-reversal environment.Source: A ENVIRONMENT REGISTRATION · p. 14

B CASE STUDIES OF LANGUAGE GAMES

The case studies illustrate how discounting affects turn efficiency, while the appendix provides the ReBN algorithm and reward-shaping learning curves.

B CASE STUDIES OF LANGUAGE GAMES

C ALGORITHM

  • ReBN collects multi-turn episodes, computes discounted returns, normalizes returns across the batch, then applies a Monte Carlo policy-gradient update.Source: C ALGORITHM · p. 16

D EXTENDED EMPIRICAL STUDIES WITH GEM

  • Figure 9 compares reward-shaping learning curves across two environments, showing average success rates and corresponding average turns to solve tasks.Source: Figure 9 · p. 16

D.1 IMPROVING LEARNING EFFICIENCY VIA RETURN BATCH NORMALIZATION (REBN)

The ablation contrasts return batch normalization with simple reward schemes for learning tool use in multi-turn environments.

D.1 IMPROVING LEARNING EFFICIENCY VIA RETURN BATCH NORMALIZATION (REBN)

  • Vanilla REINFORCE with 0/1 correctness rewards failed to induce tool use in multi-turn settings despite many initial attempts.Source: §D.1 · p. 16
  • The authors test fixed negative rewards and batch-normalized returns, hypothesizing negative gradients support exploration although 0/1 and ±1 rewards share an optimal policy theoretically.Source: §D.1 · pp. 16–17
  • ReBN performs strongly and consistently across math, QA, and other settings; fixed negative rewards fail to improve math tool use or cause QA tool overuse.Source: §D.1 · p. 17

D.2 RL ON VISION-LANGUAGE ENVIRONMENTS

GEM supports visual observations and demonstrates training vision-language agents to solve geometry problems from images. The section reports the models and training method and points to learning curves and a reasoning trace.

D.2 RL ON VISION-LANGUAGE ENVIRONMENTS

  • GEM supports visual elements in agent observations, demonstrated with a Geometry3K environment for solving geometry problems from images.Source: §D.2 · p. 17
  • Qwen2.5-VL-3B/7B-Instruct agents are RL-tuned with Dr. GRPO on Geometry3K, with training rewards and validation scores tracked.Source: §D.2 · p. 17, Figure 10 · p. 17
  • Figure 11 provides an example problem and response from an agent trained with Qwen2.5-VL-7B-Instruct.Source: Figure 11 · p. 17

D.3 MULTI-AGENT ENVIRONMENTS

GEM provides a high-level multi-agent API for LLM reinforcement learning and demonstrates it through a two-agent retail customer-service environment. Its evaluation finds that stronger simulated users improve assistant success across model strengths.

D.3 MULTI-AGENT ENVIRONMENTS

  • GEM provides a high-level multi-agent LLM RL API, intended for scenarios such as adversarial games, social dilemmas, and multi-party control.Source: §D.3 · p. 18
  • GEM’s design emphasizes minimal prescriptions and a single environment abstraction, while supporting collaboration, competition, and simulated interactions with other agents.Source: §D.3 · p. 18
  • MultiAgentEnv extends the Gym API with reset and step operations keyed by agent identifiers, while AgentSelector supports sequential or parallel turns.Custom environments implement observe(agent) and _process_actions(actions); the framework manages agent lifecycles and cumulative rewards.Source: §D.3 · p. 18
  • The TAU-bench retail integration models an assistant using order-lookup and product-search tools and an LLM user simulator issuing task-based queries.Source: §D.3 · p. 19
  • The evaluation varies user and assistant models across weak, medium, and strong levels, producing nine configurations.Source: §D.3 · p. 19
  • Stronger user agents consistently improve success rates across assistant strengths; GPT-4o gains 20% from weak to strong users and reaches 61.7% with a strong user.Across 115 TAU-bench retail tasks, a strong user paired with a weak assistant scores 44.3%, above 41.7% for a weak user paired with a strong assistant.Source: §D.3 · p. 19

E RELATED WORKS

Prior tool-integrated language-model methods commonly train on continuous sequences of agent actions and environment outputs, but this representation constrains observations and reward structure. GEM instead stores individual transitions and separates environment infrastructure from training code.

E RELATED WORKS

  • Tool-integrated methods including SkyRL-v0, VerlTool, ReCall and ReSearch, Search-R1 (Jin et al., 2025), ReTool, and SimpleTIR commonly train on continuous action-output sequences.Training masks environment outputs from the loss calculation.Source: E RELATED WORKS · p. 19
  • The single-sequence approach fixes observations to complete histories and limits rewards to trajectory-level signals, effectively fixing γ = 1.The passage states that γ < 1 is crucial for optimal fastest search behavior; SimpleTIR must instead tune and enforce a strict turn limit for this behavior.Source: E RELATED WORKS · p. 19
  • GEM stores trajectories as individual transitions, allowing alternative observation constructions and per-turn rewards with arbitrary discount factors γ ≤1.The passage notes that verl-agent (Feng et al., 2025b) also uses this approach to implement GiGPO (Feng et al., 2025b).Source: E RELATED WORKS · p. 19
  • Embedding environments directly in training code couples the two, impedes reuse, and leaves codebases with small, inconsistent environment collections; GEM supplies environment infrastructure and baselines to decouple them.Source: E RELATED WORKS · pp. 19–20
  • Earlier text-game work used games for evaluation or fine-tuning but did not focus on standardized RL training suites or compare RL algorithms.The passage cites Abdulhai et al. (2023) and Tajwar et al. (2025).Source: E RELATED WORKS · p. 20

F EXPERIMENTAL SETTINGS

The experiments were run on eight A100 GPUs and took about one day; Table 3 identifies the hyperparameter configurations used.

F EXPERIMENTAL SETTINGS

Loading 2510.01051v2…