PaperlayerView exact revision on arXiv

Source-linked AI summary

A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning

Ruiyi Wang, Prithviraj Ammanabrolu

arXiv:2510.01132v2cs.LGcs.AIcs.CL

TL;DR

Multi-turn LLM reinforcement learning lacks a systematic formulation of which design choices matter, despite fragmented approaches. The paper analyzes environment, reward, and policy choices across interactive textual tasks and derives a training recipe. Its findings support co-design across those pillars, with scope limited to the studied benchmarks and observation-based setup.

  • Problem

    Existing multi-turn RL frameworks and definitions are fragmented, and there is no systematic analysis of which design choices matter across tasks.

    Source: §1 · p. 2

  • Method

    The paper decomposes multi-turn agent RL into environment, reward, and policy pillars and empirically studies them across TextWorld, ALFWorld, and SWE-gym.

    Source: §1 · p. 2

  • Results

    The experiments report that the recipe works across textual reasoning, situated embodied reasoning, and software engineering, with findings on complexity, generalization, policy, and reward choices.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The findings yield a practical recipe for co-designing environment, policy, and reward in multi-turn agentic RL.

    Source: §1 · p. 2, §8 · p. 10

    The agent does not access the true game state and instead uses observations as a simplified substitute.

    Source: §4 · p. 3

Abstract

from arXiv · show

We study what actually works and what doesn't for training large language models as agents via multi-turn reinforcement learning. Despite rapid progress, existing frameworks and definitions are fragmented, and there is no systematic formulation or analysis of which design choices matter across tasks. We address this gap by first breaking down the design space into three inter-related pillars -- environment, reward, and policy -- and empirically derive a recipe for training LLM agents in situated textual domains. In particular, we test TextWorld and ALFWorld, popular domains for testing situated embodied reasoning, as well as SWE-Gym for more software engineering style tasks. (i) For the environment, we analyze the impacts of task complexity in terms of sizes of the state and action spaces as well as optimal solution length, finding that even simple environments within a domain can provide signal on how well an agent can generalize to more complex tasks. (ii) For the reward, we ablate relative reward sparsity, observing that while dense turn-level rewards accelerate training, performance and stability is highly dependent on the choice of RL algorithm. (iii) And for the agent's policy, we explore the interplay between reward sparsity and biased (PPO, GRPO) and unbiased (RLOO) policy gradient methods in addition to showing how to find the optimal Supervised Fine-tuning (SFT) to RL training ratio given a fixed budget. We distill these findings into a training recipe that guides co-design across the three pillars, facilitating research and practical efforts in multi-turn agentic RL. Code: https://github.com/pearls-lab/meow-tea-taro

1 INTRODUCTION

The paper addresses fragmented definitions and methods for multi-turn LLM reinforcement learning by systematically studying environment, reward, and policy choices. Across three task domains, it derives findings and a recipe for training agentic LLMs.

1 INTRODUCTION

  • The authors systematically decompose the design space into interdependent environment, reward, and policy pillars and empirically derive a training recipe for situated textual domains.Their evaluation covers TextWorld, ALFWorld, and SWE-gym.Source: §1 · p. 2
  • The recipe works across textual reasoning, situated embodied reasoning, and software engineering, with findings spanning environment complexity, generalization, task diversity, model priors, SFT-to-RL allocation, algorithms, and reward density.Dense turn-level rewards accelerate training but require algorithm-specific tuning; biased and unbiased algorithms both achieve stable learning.Source: §1 · p. 2
  • Performance scales with environment complexity, while simpler-environment training shows promising generalization to more complex tasks.Multi-task training also enhances multi-turn RL performance.Source: §1 · p. 2
  • The resulting recipe guides co-design across all three pillars, and the authors plan to release the framework, training scripts, and experimental model weights.The paper frames multi-turn RL as requiring a fundamental rethinking rather than a simple extension of single-turn optimization.Source: §1 · p. 2

2 RELATED WORK

Prior multi-turn RL work uses fragmented formulations, from tool-interleaved single-turn tasks to interactive settings with limited feedback or credit assignment. The paper analyzes how environment, policy, and reward jointly affect performance in interactive environments.

2 RELATED WORK

  • Existing approaches include tool-interleaved single-turn QA, sparse terminal rewards, and uniform turn-level advantages across sequence tokens, while comprehensive understanding of the three pillars remains lacking.Single-turn methods assume rewards follow individual actions, whereas interactive environments may reveal outcomes only after extended sequences.Source: §2 · p. 2
  • The paper systematically analyzes how environment, policy, and reward affect multi-turn RL training and draws practical training insights for interactive agentic environments.Related work is also addressed in individual sections throughout the paper.Source: §2 · p. 3

3 MULTI-TURN AGENTIC REINFORCEMENT LEARNING

The paper formulates multi-turn agentic tasks as partially observable decision processes and represents an agent's interaction as a history-conditioned sequence of language actions. Environment interaction supplies observations and rewards, with action tokens—not state tokens—used in the loss.

3 MULTI-TURN AGENTIC REINFORCEMENT LEARNING

  • Multi-turn agentic tasks are formulated as POMDPs with hidden states, actions, transitions, rewards, observations, and a discount factor.In the TextWorld example, the agent sees an observation that partially describes the true state, and the transition function is assumed deterministic.Source: §3 · p. 3
  • At each step, the LLM samples a natural-language action sequence from its policy conditioned on trajectory history containing the task prompt and prior states and actions.The history is denoted h_t.Source: §3 · p. 3
  • Scalar turn rewards are assigned at action-ending <eos> tokens, and the loss masks out state tokens so only action tokens contribute.The passage links command-boundary reward assignment to environments that execute language commands upon completion.Source: §3 · p. 3
  • The environment executes an action to produce the next state, reward, and completion signal, then appends the action and next state to chat history.The action is generated by the policy and the environment handles transition and reward computation.Source: §3 · p. 3

4 BACKGROUND AND EXPERIMENTAL SETUP

The experiments examine environment, policy, and reward choices, using three situated textual benchmarks and held-out evaluation. Agents must generate executable natural-language commands from observations rather than choose from supplied action lists.

4 BACKGROUND AND EXPERIMENTAL SETUP

  • The experiments study environment complexity, generalization from simpler tasks, task diversity, SFT-to-RL allocation, RL algorithms, and reward density.The paper argues that multi-turn RL requires co-design across components rather than a naive extension of single-turn methods.Source: §4 · p. 3
  • The agent has no access to the true game state; the setup substitutes observation o for state s for simplicity.Source: §4 · p. 3
  • Evaluation uses TextWorld and ALFWorld for embodied reasoning and SWE-Gym for software engineering, with agents generating executable language commands without action hints.The training infrastructure extends veRL and integrates environments through standard step and reset interfaces.Source: §4 · p. 4
  • TextWorld tasks vary world size, object count, and quest length; for example, w2-o3-q4 denotes 2 rooms, 3 objects, and a 4-step quest.Tasks use unique seeds for diversity.Source: §4 · p. 4
  • ALFWorld uses a text-only variant spanning six task categories, with training on the train split and evaluation on the valid unseen split.Source: §4 · p. 4
  • SWE-Gym evaluation covers five task types—getmoto, pydantic, mypy, pandas, and iterative dvc—with 90 training and 25 evaluation instances sampled.Source: §4 · p. 4
  • The study trains Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-8B with PPO, GRPO, and RLOO, reporting held-out task success or SWE-Gym test-suite passing ratios.Task success is the percentage of episodes achieving objectives within exploration budgets; iteration steps and token limits scale with task complexity.Source: §4 · p. 4

5 ENVIRONMENT

The section examines how environment complexity, exploration budgets, and task diversity affect multi-turn RL performance and transfer. Across TextWorld and SWE-Gym, it reports scaling challenges as complexity rises alongside generalization from simpler environments and task subsets.

5.1 HOW DOES MULTI-TURN RL PERFORMANCE SCALE WITH ENVIRONMENT COMPLEXITY?

  • 17% to 3% base-model performance as spatial and object complexity increase together, while PPO’s improvement falls from 88% on the base environment to 51% on the most complex setting.Object complexity is more challenging than spatial complexity; in SWE-Gym, base performance also degrades with longer code solutions and larger test suites, while GRPO recovers performance across difficulty tiers.Source: §5.1 · p. 5, Table 1 · p. 5, Table 2 · p. 5
  • 72% success on w4-o6-q8 for the 7B model, while the 1.5B model gains 65% on w2-o3-q4 and 58% on w4-o6-q8.The results indicate that larger models better handle the increased state spaces of complex environments.Source: §5.1 · p. 5, Table 3 · p. 5
  • 73% PPO success with 8 exploration steps versus 55% with 6 on TextWorld; in SWE-Gym, expanding tool calls and context from 10 to 40 raises success from 3% to 22%.TextWorld performance saturates beyond 8 steps, whereas complex coding tasks benefit from extended horizons.Source: §5.1 · p. 5, Table 4 · p. 6

5.2 HOW DOES MULTI-TURN RL GENERALIZE TO ENVIRONMENTS WITH DIFFERENT COMPLEXITIES?

  • Training on simpler environments transfers to more complex ones: spatially complex training improves w8-o12-q4 by 48%, matching the gain from training directly on that setting; Easy SWE-Gym training gains 4.8% on Medium and 3.6% on Hard tasks.The reported transferable behaviors include spatial exploration and object manipulation in TextWorld.Source: §5.2 · p. 6, Table 5 · p. 6, Table 6 · p. 6

5.3 HOW DOES MULTI-TURN RL GENERALIZE TO DIFFERENT TYPES OF TASKS WITHIN DOMAIN?

  • Single-type training alone yields 12% (ALFWorld) and 7% (SWE-Gym) improvements across all task types; clean, heat, cool, and cook mixtures outperform single pick-and-place specialists by 19% on pick-and-place and 21% on all-type evaluation.Source: §5.3 · p. 7

6 POLICY

The policy analysis studies how demonstration priors, SFT-to-RL allocation, and optimization algorithms affect multi-turn RL. Results show that hybrid initialization can use fewer RL episodes, while algorithm performance depends on task complexity and domain.

6 POLICY

  • Token-level credit assignment uses token-level values and GAE; value bootstrapping gives preceding tokens non-zero advantages despite rewards arriving only at episode-ending <eos> tokens.The formulation also applies the clipped surrogate objective and a per-token probability ratio.Source: §6 · pp. 7–8

6.1 HOW DOES PRIOR MODEL POLICY INFLUENCE MULTI-TURN RL TRAINING?

  • With 60 demonstrations and 400 RL episodes, the initialized model achieves 85% success, nearly matching pure RL’s 88% after 5,000 episodes.The paper frames SFT as establishing a prior and online RL as refining it; the reported demonstration data comes from TextWorld w2-o3-q4 tasks.Source: §6.1 · p. 8
  • Under a fixed budget of 1,000 cost units, 100 SFT demonstrations achieve 95% in-domain success but 55% on w4-o6-q8; the 60-demonstration, 400-episode hybrid achieves 85% and 59%, respectively.The budget analysis assumes SFT data costs 10× more than an RL episode.Source: §6.1 · p. 8, Table 9 · p. 8
  • Cross-domain SFT priors from ALFWorld or TextWorld harm multi-turn RL training, with a different source environment associated with rapid policy collapse.The authors attribute the result likely to behavioral biases and action distributions conflicting with target-environment dynamics.Source: §6.1 · p. 8

6.2 HOW DO RL ALGORITHMS IMPACT MULTI-TURN RL TRAINING?

  • At 7B on w4o6-q8, PPO reaches 72% success versus RLOO’s 47% and GRPO’s 36%; at 1.5B on w2-o3-q4, PPO scores 88% versus RLOO’s 51%.Both PPO and RLOO improve over base models, while Reinforce++ and GRPO give negligible gains on the reported 1.5B comparison; PPO maintains 59% success on w4-o6-q8.Source: §6.2 · pp. 8–9, Table 10 · p. 9
  • For SWE-Gym, GRPO is computationally advantageous, and its results alongside PPO’s support the effectiveness of biased methods under sparse, trajectory-final rewards.The paper contrasts this with GRPO’s underperformance on TextWorld tasks.Source: §6.2 · p. 9

7 REWARD

The reward analysis compares feedback density and reward sources in TextWorld and SWE-Gym. Dense signals improve performance in some settings, while the benefit depends on the algorithm and, for coding tasks, verified unit-test feedback substantially outperforms model-based rewards.

7.1 WHAT REWARD SIGNALS ARE NEEDED FOR MULTI-TURN RL TRAINING?

  • On TextWorld tw-simple tasks, PPO success rises from 41% with sparse rewards to 58% with Dense 2, while RLOO remains at 55% across both dense configurations.The authors conclude that reward density should be calibrated to the optimization strategy; poorly designed intermediate rewards can introduce noise.Source: §7.1 · pp. 9–10, Table 11 · p. 9

7.2 VERIFIED VS. MODEL-BASED REWARDS IN COMPLEX TASKS

  • 22% success with dense ratio-based verified rewards exceeds 4.2% with sparse binary verification and the base model’s 4% on SWE-Gym.The ratio reward is proportional to the percentage of unit tests passed; the comparison is reported in Table 12.Source: §7.2 · p. 10, Table 12 · p. 10
  • On SWE-Gym, CodeRM-8B and GPT-4.1 model-based rewards achieve 7.2% and 9.3% success, respectively, below the 22% with verified ratio rewards.The authors recommend execution-based unit tests over model-based proxies for this setting.Source: §7.2 · p. 10, Table 12 · p. 10

8 THE RECIPE AND CONCLUSION

The conclusion distills the three-pillar analysis into a practical recipe for multi-turn agentic RL: curriculum, stable policy learning, and verified dense rewards.

8 THE RECIPE AND CONCLUSION

  • Start with simpler environments or tasks, where agents develop reusable skills that generalize to complex, object-heavy, or long-horizon tasks.The authors describe this as a curriculum across environment complexity and verification complexity.Source: §8 · p. 10
  • Prefer stabilized biased algorithms such as PPO or GRPO and initialize with a strong SFT prior to reduce RL sample complexity.The passage contrasts these algorithms with unbiased estimators and naive gradients.Source: §8 · p. 10
  • For complex reasoning, prioritize granular verified execution feedback, such as unit-test pass rates, over model-based reward surrogates.The paper’s recipe characterizes reward granularity as preferable to approximation.Source: §8 · p. 10

REPRODUCIBILITY STATEMENT

The statement documents planned release resources and experimental details for replication, while the hyperparameter appendix reports a tuning protocol and selected settings for TextWorld.

REPRODUCIBILITY STATEMENT

  • The reproducibility plan promises the framework, environment integrations, training scripts, evaluation pipelines, and data-generation procedures.The framework is built on veRL and includes integrations for TextWorld, ALFWorld, and SWE-Gym.Source: REPRODUCIBILITY STATEMENT · p. 11
  • The statement says appendices document hyperparameters, model versions, training epochs, convergence criteria, task selection, and environment-specific splits.It also describes SFT data generation from TextWorld gold solutions and SWE-Gym task sampling, with key model weights and installation details slated for release.Source: REPRODUCIBILITY STATEMENT · p. 11
  • The reproducibility statement reports H100 GPU use, approximate training times, and computational requirements to help estimate reproduction costs.It says the framework supports distributed training for replication at scale.Source: REPRODUCIBILITY STATEMENT · p. 11

A HYPERPARAMETER TUNING FOR MULTI-TURN RL

  • Experiments use 5,000 training episodes and 100 held-out episodes to compare PPO hyperparameters over 30 epochs, about 575 steps.The reported measure is task success rate throughout training, with the runs intended to assess early stability.Source: A HYPERPARAMETER TUNING FOR MULTI-TURN… · p. 14
  • Higher KL coefficients (> 0.001) yield more stable curves; gamma below 1.0 impairs performance, and rollout temperatures between 0.7 and 1.0 perform best.Higher actor and critic learning rates improve training efficiency and final performance in the reported comparisons.Source: A HYPERPARAMETER TUNING FOR MULTI-TURN… · p. 14, Figure 2 · p. 14
  • KL coefficient 0.01, temperature 0.7, actor learning rate 1e-6, critic learning rate 1e-5, and gamma 1.0 form the selected configuration.Table 13 reports success rates at epoch 30 on the TextWorld w2-o3-q4 test set and shows only top-performing configurations.Source: A HYPERPARAMETER TUNING FOR MULTI-TURN… · p. 14, Table 13 · p. 15

B DETAILS OF TASKS AND ENVIRONMENTS

The section describes task formats and data mixtures across TextWorld, ALFWorld, and SWE-Gym, alongside TextWorld data used for SFT and reward-density experiments.

B DETAILS OF TASKS AND ENVIRONMENTS

  • ALFWorld tasks resemble TextWorld in format but require more embodied understanding, while SWE-Gym examples are drawn from its Hugging Face repository.The section presents examples of TextWorld w2-o3-q4, ALFWorld heat-and-place, and SWE-Gym getmoto tasks.Source: B DETAILS OF TASKS AND ENVIRONMENTS · p. 15

B.1 ADDITIONAL TASK DETAILS FOR §5

  • ALFWorld training mixtures each contain 1,000 tasks: pick-and-place only, 250 each across four task types, or 1,000 randomly sampled across all types.The source training set contains 3,553 examples.Source: §B.1 · p. 15
  • SWE-Gym mixtures each contain 90 tasks, either getmoto only or 18 each across getmoto, pydantic, mypy, pandas, and interactive dvc.The tasks are sampled from the SWE-Gym Hugging Face repository.Source: §B.1 · p. 15

B.2 ADDITIONAL TASK DETAILS FOR §6

  • The SFT data comprise 0, 20, 40, 60, 80, or 100 gold TextWorld trajectories, using seeds distinct from the RL data.Source: §B.2 · p. 15

B.3 ADDITIONAL TASK DETAILS FOR §7

  • TextWorld reward-density experiments use tw-simple and its per-step reward function, with 3,000 episodes collected at each of three density levels.The levels are sparse, dense 1, and dense 2.Source: §B.3 · p. 15

C DETAILS OF TRAINING AND EVALUATION

Training details cover the models, hardware, evaluation schedules, and default optimization settings used across TextWorld, ALFWorld, SWE-Gym, SFT, and reward-density experiments.

C DETAILS OF TRAINING AND EVALUATION

  • The experiments train Qwen-1.5B, Qwen-7B, and Qwen-8B on eight NVIDIA H100 GPUs, with PPO, GRPO, RLOO, and SFT using specified default settings.The figure captions identify prompts, objectives, observations, and actions in example task trajectories from TextWorld, ALFWorld, and SWE-Gym.Source: C DETAILS OF TRAINING AND EVALUATION · pp. 15, 19, Figure 3 · p. 16

C.1 ADDITIONAL TRAINING DETAILS FOR §5

  • TextWorld runs for 150 epochs and is evaluated on 100 held-out test sets; ALFWorld runs for 90 epochs after one SFT epoch on 300 examples.SWE-Gym runs for 15 epochs and is evaluated on 25 held-out test sets.Source: §C.1 · p. 19
  • ALFWorld begins with one SFT epoch on 300 examples because the base model has zero accuracy on the tasks, followed by 90 RL epochs.Evaluation uses 100 held-out test sets at epoch 90.Source: §C.1 · p. 19

C.2 ADDITIONAL TRAINING DETAILS FOR §6

  • SFT uses exactly one epoch before 100 RL epochs to limit overfitting to demonstration data.The RLOO and PPO comparison trains each method for 150 epochs.Source: §C.2 · p. 19

C.3 ADDITIONAL TRAINING DETAILS FOR §7

  • PPO and RLOO dense-reward runs each train for 150 epochs and are tested on 100 held-out examples at epoch 150.Source: §C.3 · p. 19

E USE OF LLMS

  • The authors report using Claude only for manuscript polishing and grammar checking, not for experiments, figures, tables, or technical content.Source: E USE OF LLMS · p. 19
Loading 2510.01132v2…