RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning

Zihan Wang; Kangrui Wang; Qineng Wang; Pingyue Zhang; Linjie Li; Zhengyuan Yang; Xing Jin; Kefan Yu; Minh Nhat Nguyen; Licheng Liu; Eli Gottlieb; Yiping Lu; Kyunghyun Cho; Jiajun Wu; Li Fei-Fei; Lijuan Wang; Yejin Choi; Manling Li

Source-linked AI summary

RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning

Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, Manling Li

arXiv:2504.20073v2Apr 24, 2025cs.LGcs.AIcs.CL

TL;DR

Interactive LLM agents must learn sequentially from stochastic feedback, but multi-turn agent RL remains underexplored and unstable. The paper introduces StarPO for trajectory-level optimization and RAGEN for modular training and evaluation, finding recurring Echo Trap instability, rollout-design effects, and weak reasoning emergence without fine-grained rewards.

Problem
Multi-turn LLM-agent RL remains underexplored, especially regarding how agents can reason and adapt stably under stochastic feedback.
Method
The paper introduces StarPO for trajectory-level multi-turn RL and builds RAGEN as a modular system for training and evaluating LLM agents.
Results
Across four environments, training exhibits Echo Trap instability, rollout diversity and frequency shape learning, and reasoning rarely emerges without fine-grained reasoning-aware rewards.
Takeaways & Limitations
Stable agent RL requires trajectory filtering, critic or gradient stabilization, well-shaped rollouts, and rewards that distinguish useful reasoning.
Takeaways & Limitations
The study focuses on relatively small-scale tasks, omits replay buffers, and does not include multimodal tasks.

Abstract

from arXiv · show

Training large language models (LLMs) as interactive agents presents unique challenges including long-horizon decision making and interacting with stochastic environment feedback. While reinforcement learning (RL) has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents. Our study on four stylized environments reveals three core findings. First, our agent RL training shows a recurring mode of Echo Trap where reward variance cliffs and gradient spikes; we address this with StarPO-S, a stabilized variant with trajectory filtering, critic incorporation, and gradient stabilization. Second, we find the shaping of RL rollouts would benefit from diverse initial states, medium interaction granularity and more frequent sampling. Third, we show that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts. Code and environments are available at https://github.com/RAGEN-AI/RAGEN.

1. Introduction

The paper studies how LLM agents can learn through multi-turn interaction with stochastic environments, where sequential decisions, memory, and adaptation make RL training difficult. It introduces StarPO and RAGEN to analyze these challenges across four environments and identifies instability, rollout design, and reward shaping as central factors.

Multi-turn agent RL remains underexplored because agents must make sequential decisions, maintain memory, and adapt to stochastic environmental feedback.
StarPO provides a unified trajectory-level framework, while RAGEN implements rollout generation, reward assignment, trajectory optimization, and evaluation.
The study evaluates Bandit, Sokoban, Frozen Lake, and WebShop to analyze agent learning across stochastic, deterministic, symbolic, and open-domain settings.
Gradient Stability in Multi-turn RL: Echo Trap is a recurring instability pattern involving reward variability collapse, entropy drop, and gradient spikes; StarPO-S addresses it with filtering, critic baselining, and decoupled clipping.
Rollout Frequency and Diversity: Rollout shaping benefits from diverse initial states, multiple responses, medium interaction granularity, and frequent sampling that reflects current policies.
Emerging Agent Reasoning: Without distinct or fine-grained reasoning-aware rewards, agents may select direct actions, use shallow strategies, or produce hallucinated reasoning disconnected from environment states.

2. Framework

StarPO formulates interactive LLM-agent learning as trajectory-level policy optimization in a multi-turn MDP. RAGEN operationalizes this abstraction with structured rollouts, customizable rewards, and multiple token-level optimization strategies.

MDP Formulation: Interactive agent training is modeled as an MDP whose states encode observations or histories, actions are often token sequences, and transitions produce rewards and new states.
MDP Formulation: A multi-turn interaction continues up to horizon K and yields a complete trajectory that becomes the agent’s learning material.
StarPO: StarPO maximizes expected cumulative reward over entire reasoning-augmented trajectories rather than treating actions independently.
StarPO: StarPO decomposes trajectory policy probabilities into token-level likelihoods, making trajectory optimization compatible with autoregressive language models.
Rollout and Optimization: Each rollout generates reasoning-guided structured outputs, environment-executable sub-actions, subsequent states, and rewards that are jointly optimized.
Rollout and Optimization: The framework interleaves rollout and update steps and supports on-policy sampling or replay-buffer trajectories across repeated training loops.
Modular Optimization Strategies: StarPO supports PPO with a critic and GRPO with normalized trajectory rewards under a unified trajectory-level abstraction.
RAGEN: RAGEN provides the execution backend and research platform for structured rollouts, customizable rewards, and multi-turn stochastic environments.

3. Experiment Setup

The experiments span four environments designed to isolate distinct decision-making challenges, with symbolic tasks enabling controlled analysis and WebShop adding realistic language interaction. Evaluation uses fixed prompts, multiple training rollouts, and metrics covering success, exploration, diversity, verbosity, and stability.

Environments: Bandit tests risk-sensitive reasoning under noisy feedback, Sokoban requires irreversible symbolic planning, Frozen Lake adds probabilistic transitions, and WebShop tests language grounding in web interaction.
Environments: The first three environments are minimal and controllable for clean analysis, whereas WebShop introduces realistic task structure and natural-language input.
Training Setup: The main experiments train 0.5B models on symbolic tasks and a 3B model on WebShop for 100–200 rollout-update iterations.
Evaluation Setup: Evaluation uses 256 fixed prompts per environment, temperature T=0.5, and episodes truncated after five turns.
Evaluation Metrics: Reported metrics measure task completion, exploration, behavioral diversity, reasoning verbosity, and training stability.
Baseline Results: Baseline StarPO collapses on symbolic Bandit and Sokoban, while WebShop starts high and improves rapidly; PPO is stronger on Bandit and Sokoban, whereas GRPO is stronger on Frozen Lake and WebShop.

4. Experimental Results and Findings

Multi-turn agent RL often suffers Echo Trap collapse, but trajectory filtering and rollout design improve stability, efficiency, and generalization. Reasoning gains depend on fine-grained supervision and can fade in multi-turn settings.

Instability and Echo Trap: Vanilla PPO and GRPO adaptations achieve early gains but often collapse; PPO’s critic may delay instability without preventing reasoning degradation.GRPO is more stable on Frozen Lake, while PPO is more stable elsewhere in the reported settings.
Instability and Echo Trap: Echo Trap turns diverse early reasoning into repetitive, deterministic templates that reinforce superficial patterns and hinder long-term generalization.The authors associate this shift with over-amplified reasoning shortcuts and suppressed exploration.
Instability and Echo Trap: Reward standard deviation and entropy often destabilize before performance degrades, whereas gradient-norm spikes typically signal irreversible collapse.These indicators motivate stabilization strategies for multi-turn RL.
StarPO-S stabilization: Filtering low-variability trajectories delays or eliminates collapse, with PPO stability in FrozenLake extending from 100 to 140 steps when retaining 75% of rollouts.Retaining 50% avoids collapse entirely in the cited PPO result; filtering also reduces update steps and training time.
StarPO-S stabilization: StarPO-S consistently delays collapse and improves final task performance through selective training data, balanced optimization signals, and reduced narrowing of reasoning modes.The comparison is against vanilla StarPO across the reported tasks.
Rollout quality and reasoning: Higher task diversity, moderate action budgets, and fresher rollouts improve generalization, while reasoning traces help symbolic generalization but often fade in multi-turn tasks.Four responses per prompt, 5–6 actions per turn, and Online-1 rollouts are reported as favorable settings; explicit reasoning can provide limited multi-turn benefit.

5. Related Work

Prior work advances reinforcement learning for reasoning and develops increasingly structured, general-purpose agent frameworks spanning web, coding, GUI, games, embodied tasks, and social interaction.

Reinforcement Learning for Reasoning: RL methods for LLM reasoning include PPO, actor-critic approaches, RLOO, GRPO, DAPO, structured meta tokens, PRMs, and MCTS.
Existing Agent Frameworks: LLM agent architectures have progressed from reactive planning to modular pipelines, multi-agent cooperation, embodied interaction, and broad application-oriented systems.

6. Conclusions and Limitations

The paper presents RAGEN and StarPO as a platform for studying multi-turn, stochastic LLM-agent training, identifies stabilization principles, and states scope limitations.

Conclusions: RAGEN is a general-purpose system for reinforcement learning with language agents in multi-turn, stochastic environments.
Conclusions: The framework identifies gradient collapse, rollout drift, and reasoning degradation as challenges unique to agent training.
Conclusions: The study highlights rollout filtering, gradient shaping, and reward-aware reasoning supervision as principles for stabilizing training.
Limitations: The work focuses on relatively small-scale tasks, omits replay buffers, and does not include multimodal tasks.
Background: RL foundations described include reward-based learning, PPO clipping and advantage estimation, GRPO, actor-critic methods, and structured reasoning techniques.

C.1. Environments and Tasks

The testbed spans symbolic and realistic environments with distinct decision-making demands, while the evaluation tracks task success, exploration, behavioral diversity, verbosity, and stability.

Environments and Tasks: The four-environment testbed comprises Bandit, Sokoban, Frozen Lake, and WebShop, covering stochasticity, planning, language grounding, and web navigation.
Bandit: Bandit tests risk-sensitive reasoning between a low-risk arm and a higher-variance, higher-expected-return arm.
Sokoban: Sokoban requires irreversible box pushing under constrained steps, rewarding target placement, completion, and efficient action use.
Frozen Lake: Frozen Lake combines long-horizon navigation with stochastic transitions, where each action succeeds with probability 1/3 and deviates perpendicularly with probability 2/3.
WebShop: WebShop requires agents to ground natural-language requests, navigate semi-structured interfaces, and retrieve relevant product information.
Evaluation: Evaluation monitors success rate, rollout entropy, in-group reward variance, response length, and gradient norm to characterize policy quality and training dynamics.

D. Results on Larger Models and Various Optimization Algorithms

Larger models help most on language-rich environments, while optimization changes improve multi-turn RL stability and performance. Gains are limited on symbolic grid tasks.

Scaling Effects: 7B WebShop performance is not reported because its extremely long context length can cause out-of-memory errors on 4×H100 hardware.
Scaling Effects: Larger models substantially outperform smaller ones on Bandit and WebShop, but provide only marginal improvements on Sokoban and FrozenLake.Language-rich tasks can exploit pretrained linguistic priors, whereas symbolic grid environments overlap less with pretraining data.
Frontier Model Performance: 19–28% zero-shot success from GPT-4o and Qwen2.5-72B compares with 20.70% and 21.48% from the trained 0.5B model on Sokoban and FrozenLake.The smaller model used task-specific training, while the larger models were evaluated without fine-tuning or in-context trajectory rollouts.
Gradient Shaping: KL removal and asymmetric clipping both improve peak performance and delay collapse in multi-turn RL.Asymmetric clipping uses εhigh = 0.28 and εlow = 0.2, allowing more aggressive learning from high-reward rollouts.
Turn-Aware Optimization: Response masking and bi-level GAE both improve multi-turn RL performance on Sokoban.

E. When Does Uncertainty-Based Filtering Help?

Uncertainty-based filtering is most useful when rollout rewards have low within-group variance, where it improves stability and performance. The analysis also exposes reasoning collapse and spurious reward signals.

Uncertainty-Based Filtering: Low intra-group reward variance occurs when tasks are too easy or difficult, allowing standard StarPO to propagate misleading gradients and motivating low-confidence rollout filtering.
Uncertainty-Based Filtering: StarPO-S consistently outperforms StarPO on Bandit, Sokoban, and FrozenLake as rollout variance drops, while the gap is smaller on WebShop.WebShop maintains high, stable variance, making filtering less critical.
Echo Trap: Echo Trap examples show diverse hypotheses before training converging to near-identical, unjustified phrasing after RL training.The displayed Bandit examples focus on repeatedly choosing Dragon without justification.
SFT Comparison: SFT achieves 74.6% on Sokoban and 23% on Frozen Lake, compared with 20.3% and 21.8% for StarPO-S.The authors conclude that rule-based RL remains promising but needs more scalable and effective algorithms.

H. Efficient Training with Low-Rank Adaptation (LoRA)

LoRA offers a parameter-efficient alternative for RAGEN training, achieving comparable SimpleSokoban policy quality while substantially reducing resource use. FrozenLake also reveals optimizer-specific stability differences.

Performance Parity: LoRA reaches approximately a 0.2% SimpleSokoban validation success rate, comparable to full-parameter fine-tuning.
Resource Savings: LoRA uses approximately 23% of device memory versus 48% for full updates, cutting peak allocation by more than 50%.
Resource Savings: LoRA reduces average GPU utilization from approximately 34% to 14% and mean power draw from approximately 22% to 12%.The reported power reduction is approximately 45%.
Optimizer Stability: PPO tends to collapse earlier or converge less stably than GRPO on FrozenLake, unlike the general trend favoring PPO.The paper links this difference to challenging value estimation under long-horizon stochastic transitions, while GRPO avoids explicit value learning.

K. Generalization Evaluation Environments

The evaluation introduces unseen environments that test symbol changes, larger layouts, and environment shifts. Case studies show that successful outcomes can coexist with flawed reasoning, limiting outcome-only rewards.

Generalization Environments: SokobanDifferentGridVocab changes grid symbols to test whether policies retain spatial semantics under symbol variation.
Generalization Environments: LargerSokoban expands layouts from 6 × 6 to 8 × 8 and boxes from 1 to 2, increasing spatial complexity and planning demands.
Generalization Environments: These unseen environments probe generalization under symbol shift, size scaling, and environment shift.
Spurious Reward: A three-turn Sokoban rollout reaches the goal despite intermediate reasoning containing redundant or directionally incorrect assumptions.Such trajectories make it harder for RL to distinguish genuinely useful plans from coincidentally effective ones.
Spurious Reward: Outcome-based rewards alone may not sufficiently penalize poor reasoning traces in environments with sparse or delayed feedback.

M. Extended Case Studies

The case studies compare reasoning and decision behavior across model scales and environments. Larger models show more elaborate or structured reasoning in some settings, while smaller agents exhibit loops, shallow strategies, or limited adaptation.

Cross-environment scale effects: Larger models generate longer and more coherent reasoning chains, especially in semantic-rich Bandit and WebShop tasks.The reported comparison covers Bandit, Sokoban, FrozenLake, and WebShop at different model scales.
Reasoning quality: A successful final outcome can coexist with inconsistent or factually incorrect reasoning traces, producing noisy and potentially misleading supervision.The paper presents this as spurious reasoning: optimizing final rewards while bypassing coherent reasoning.
Bandit: Both Bandit agents select Dragon, but the 7B model provides a more elaborate comparison of stability and variance than the 0.5B model.The 0.5B model relies on a short symbolic association, whereas the 7B model gives a longer interpretive chain.
Sokoban: The 0.5B Sokoban agent issues locally valid but often suboptimal actions with minimal reasoning, while the 7B agent shows more structured planning but remains inefficient in long-horizon settings.The rollout examples include short action sequences accompanied by brief planning statements.
FrozenLake: In FrozenLake, the 0.5B agent repeats a fixed plan, while the 7B agent can receive high reward after a suboptimal command because transitions are stochastic.This illustrates why observed success can reinforce spurious action patterns in stochastic environments.
WebShop: In WebShop, the 0.5B agent loops among irrelevant choices, whereas the 3B agent completes a successful multi-step search, navigation, attribute selection, and purchase sequence.The smaller-agent rollout repeatedly applies unsuitable filters and choices across turns, while the larger-agent example follows a narrowing workflow.