Source-linked AI summary
Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, Yang Li
TL;DR
Multi-turn RL for agentic LLMs faces instability and credit-assignment problems when GRPO uses trajectory-level advantages, especially for long-horizon interactions. The paper compares PPO with GRPO and introduces turn-PPO, whose turn-level MDP formulation yields effective results across WebShop and Sokoban settings. The study’s scope is limited to two representative datasets and primarily text-simulated embodied environments.
Problem
GRPO’s sampling-based advantage estimation can be unstable in multi-turn settings, while token-level formulations misalign critic learning with turn-level task structure.
Method
The paper evaluates PPO and introduces turn-PPO, which defines each complete interaction turn as a state–action pair and applies turn-level advantage estimation.
Results
PPO is more robust than GRPO, and turn-PPO is effective across WebShop and Sokoban datasets with and without long reasoning components.
Takeaways & Limitations
Turn-level MDP formulation provides a practical PPO-based approach for stabilizing multi-turn agent training and capturing turn-level credit assignment.
Takeaways & Limitations
The evaluation covers only two representative datasets, with embodied experiments primarily using text-simulated environments.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. However, directly applying the widely used Group Relative Policy Optimization (GRPO) algorithm to multi-turn tasks exposes notable limitations, particularly in scenarios requiring long-horizon reasoning. To address these challenges, we investigate more stable and effective advantage estimation strategies, especially for multi-turn settings. We first explore Proximal Policy Optimization (PPO) as an alternative and find it to be more robust than GRPO. To further enhance PPO in multi-turn scenarios, we introduce turn-PPO, a variant that operates on a turn-level MDP formulation, as opposed to the commonly used token-level MDP. Our results on the WebShop and Sokoban datasets demonstrate the effectiveness of turn-PPO, both with and without long reasoning components.
1 Introduction
Multi-turn agentic LLM training is difficult because GRPO’s sampling-based, trajectory-level advantages can be unstable and misaligned with unequal turn contributions. The paper investigates PPO and proposes turn-PPO, which models each complete turn as a state–action pair to improve advantage estimation.
- 1 Introduction: RL is favored for interactive LLM agents because environmental feedback and sparse rewards can replace difficult-to-obtain optimal step-by-step supervision.The approach lets models optimize policies directly from environmental feedback.
- 1 Introduction: GRPO estimates token advantages by normalizing trajectory-level rewards across sampled trajectories, but multi-turn interactions create high variance and unequal turn contributions.Applying one advantage uniformly across all tokens can make the estimate inaccurate, while prior turn-level methods are specialized and potentially biased.
- 1 Introduction: PPO uses a learnable critic and Generalized Advantage Estimation, making it more suitable than GRPO for sequential multi-step MDP training.The paper identifies more accurate advantage estimation and greater training stability and efficiency as PPO’s advantages over GRPO.
- 1 Introduction: Turn-PPO treats an entire turn’s input and output as one state–action pair, producing a more coherent representation for critic learning and turn-level credit assignment.This contrasts with token-level PPO, where each token is treated as a separate MDP step.
- 1 Introduction: The paper reports that PPO is more robust than GRPO and that turn-PPO performs better across diverse multi-turn tasks and settings.These contributions motivate evaluating both the PPO alternative and the turn-level formulation.
- 1 Introduction: The study focuses on tool-using, text-only environments spanning web navigation and embodied-agent tasks, while related work includes several GRPO extensions for multi-turn training.The paper positions its contribution within broader RL methods for single-turn and multi-turn LLMs.
2 Framework
The framework compares GRPO and PPO under token- and turn-level MDP formulations for multi-turn LLM interaction. It introduces turn-PPO, which treats each complete turn as a state–action pair and estimates advantages with a turn-level critic.
- 2 Framework: The framework provides unified objectives for GRPO and PPO across token- and turn-level formulations in single- and multi-turn settings.The comparison is organized around how states, actions, critics, and advantages are defined.
- 2.2 Token-MDP: PPO estimates advantages with a learned critic and GAE, whereas GRPO normalizes final rewards across multiple rollouts and assigns the result across tokens.PPO can therefore use state-dependent advantage estimates instead of a trajectory-level constant.
- 2.2 Token-MDP: Token-level multi-turn MDPs can misalign state representations with environment-driven transitions, adding noise to critic learning and advantage estimation.This mismatch arises because external tokens interrupt otherwise token-by-token transitions.
- 2.3 Turn-MDP: Turn-PPO defines each turn’s complete interaction context as the state and the LLM’s entire response as the action.The state contains the full interaction history plus the current query, rather than token-level prefixes.
- 2.3 Turn-MDP: Turn-PPO applies GAE, value estimation, actor updates, and clipping at the turn level while retaining the same forward and backward computational cost as token-PPO.Its critic is initialized from a pretrained LLM with a separate value head.
3 Experiments
Experiments compare GRPO, token-PPO, and turn-PPO across WebShop and Sokoban, diagnosing GRPO instability and evaluating PPO training choices. Turn-PPO improves stability and performance through turn-level advantage estimation, while ablations identify settings that support effective PPO training.
- 3.2 GRPO failure investigation: GRPO training often collapses abruptly on WebShop and Sokoban, especially for Qwen3 with long reasoning.The reported crashes persist across GRPO variants and are illustrated by the training curves in Figure 2.
- 3.2 GRPO failure investigation: The GRPO failure is attributed to uniform advantages across heterogeneous turns and high-variance sample-based estimates in dynamic or partially observed environments.These factors cause different turns to contribute unequally while the shared estimate amplifies variance.
- 3.3 Token-PPO and turn-PPO: Turn-PPO improves training stability and performance over GRPO and token-PPO across multi-turn tasks and settings.PPO-based methods substantially outperform GRPO, while turn-PPO achieves further improvements and performs best in most comparisons.
- 3.4 Ablation studies: PPO recipe: Turn-PPO’s turn-level clipping and tunable discounting improve robustness, with γ = 0.99 and λ = 0.9 reported as relatively stable choices.Turn-level clipping prevents updates on turns with large policy shifts, while token-level PPO must keep both parameters at 1.0.
- 3.4 Ablation studies: PPO recipe: PPO training benefits from one sample per question, reducing minibatch size before increasing epochs, and a critic learning rate roughly 5–10× higher than the actor’s.These choices target critic generalization, sample efficiency, and stable optimization.
To-Go List
The study identifies practical training guidelines for PPO-based multi-turn agents, including learning-rate choices, sampling strategies, update schedules, and discount-factor tuning.
- A critic learning rate larger than the actor’s is required for effective training.
- GRPO benefits from more rollouts per sample, whereas PPO benefits from greater sample diversity within each batch.
- Using more minibatch updates per rollout is preferable to increasing the number of epochs.
- Turn-PPO allows flexible tuning of γ and λ, consistently yielding improved results.
4 Conclusion
The paper systematically studies RL algorithms for multi-turn agentic LLMs and proposes turn-PPO to improve turn-level credit assignment and training stability. It also provides practical guidance for PPO-based training.
- The paper proposes a turn-MDP PPO variant that stabilizes training and better captures turn-level credit assignment.
- The study analyzes GRPO and PPO limitations theoretically and empirically across multiple tasks and model scales.
- The paper provides practical training guidelines for PPO-based algorithms.
Limitations
The study evaluates its methods on only two representative datasets, with the embodied evaluation focused primarily on text-simulated environments. Future work should examine richer real-world web and physical embodied settings.
- Evaluation is limited to two representative datasets: one web-agent dataset and one embodied-agent dataset.
- The embodied evaluation primarily uses text-simulated environments rather than physical interactions.
- Future work should extend evaluation to richer web-agent tool use and physical embodied interactions.
A Training details
The appendix reports the main training hyperparameters for GRPO and PPO, including learning rates, rollout configurations, clipping, and training steps.
- GRPO uses a 1×10−6 learning rate and configurations of (256, 16, 64, 1) for WebShop and (512, 16, 128, 1) for Sokoban.The tuple denotes total rollout samples, group size, minibatch size, and epochs.
- PPO uses actor and critic learning rates of 1 × 10−6 and 1 × 10−5, respectively.
- Both GRPO and PPO use a 0.2 clip ratio, with 100 or 200 training steps depending on model scale.
B Trajectory examples
The trajectory examples illustrate WebShop agents making sequential product-search and purchase decisions under action limits, with Qwen3 producing substantially longer reasoning chains than Qwen2.5.
- Qwen3 produces much longer reasoning chains than Qwen2.5 on WebShop, increasing the difficulty of training.
- WebShop trajectories require agents to search products, inspect results, select attributes, and complete purchases through multiple actions.
- One trajectory searches for a holiday hoodie, selects the large size and vintage camo color, and then clicks buy now.
- The examples enforce purchasing within 10 actions while allowing approximate matches and limiting searches to at most two.
- Another trajectory reasons about search terms, product suitability, size, color, and action budgeting before choosing a product.