PaperlayerView exact revision on arXiv

Source-linked AI summary

ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning

Xiaoxuan Wang, Han Zhang, Haixin Wang, Yidan Shi, Ruoyan Li, Kaiqiao Han, Chenyi Tong, Haoran Deng, Renliang Sun, Alexander Taylor, Yanqiao Zhu, Jason Cong, Yizhou Sun, Wei Wang

arXiv:2602.21534v3cs.AI

TL;DR

ARL training is unstable and prone to collapse, limiting reproducibility and scalability in long-horizon interactive environments. ARLArena standardizes the testbed and analyzes four policy-gradient dimensions, leading to SAMPO, which achieves stable training and strong performance across tasks, including a 25.2% average improvement over GRPO.

  • Problem

    ARL remains highly unstable and prone to collapse, limiting reproducibility, longer-horizon scaling, and systematic study of algorithmic choices.

  • Method

    ARLArena builds a standardized testbed, decomposes policy-gradient design into four dimensions, and uses the resulting analysis to develop SAMPO.

  • Results

    SAMPO achieves stable training and strong performance across diverse agentic tasks, with an average 25.2% improvement over the GRPO baseline.

  • Takeaways & Limitations

    The study provides a unifying policy-gradient perspective and practical guidance for stable, reproducible agentic RL training.

Abstract

from arXiv · show

Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we distill a unified perspective on ARL and propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Overall, this study provides a unifying policy gradient perspective for ARL and offers practical guidance for building stable and reproducible LLM-based agent training pipelines.

1. Introduction

ARL enables multi-step agent training but remains unstable and prone to collapse, limiting reproducibility and scaling. ARLArena standardizes evaluation, analyzes policy-gradient design choices, and yields SAMPO, which improves stability and performance.

  • ARL supports complex multi-step tasks requiring planning, tool use, and long-horizon decision-making.
  • ARL training is highly unstable because interactive environments combine invalid actions, sparse rewards, long-horizon credit assignment, and non-stationary dynamics.
  • ARLArena builds a clean standardized testbed and evaluates four orthogonal policy-gradient design dimensions across diverse agentic tasks.
  • Sequence-level clipping avoids the collapse associated with tolerant clipping, while environment-aware advantage design improves stability and performance.
  • SAMPO combines sequence-level clipping, advantage design, and dynamic filtering, achieving an average 25.2% improvement over the GRPO baseline.
  • The framework provides a unifying policy-gradient perspective, a reproducible stability methodology, and a policy-optimization method with reliable training and strong final performance.

2. Problem Formulation

The formulation represents agentic RL as policy optimization over multi-turn trajectories, then decomposes the objective into four design dimensions. These dimensions cover aggregation, importance-sampling clipping, trajectory sampling, and advantage construction.

  • Policy Gradient for Agentic RL: The policy-gradient objective uses an advantage term and token-level importance-sampling weights comparing the current and behavior policies.
  • Policy Gradient for Agentic RL: Agentic RL generates responses over K turns, extracts actions, transitions environment states, and constructs the next prompt from updated state information.
  • Policy Gradient for Agentic RL: The complete interaction trajectory is decomposed into single-turn updates for policy-gradient optimization.
  • Policy Gradient Decomposition Dimensions: ARLArena studies four orthogonal dimensions: loss aggregation, importance-sampling clipping, trajectory filtering and resampling, and advantage design.
  • Policy Gradient Decomposition Dimensions: Loss aggregation choices differ in how they weight tokens and trajectories, with seq-mean-token-mean biasing optimization toward shorter responses and token-mean weighting unmasked tokens equally.
  • Policy Gradient Decomposition Dimensions: Clipping constrains policy deviation from the old policy; studied variants include tolerant, soft, stop-gradient, and sequence-level strategies.
  • Policy Gradient Decomposition Dimensions: Dynamic sampling filters groups with identical rewards and resamples trajectories to increase informative gradient signals.
  • Policy Gradient Decomposition Dimensions: Advantage designs incorporate interaction-state grouping or entropy-dependent terms to account for uncertainty across multi-turn steps.

3. Experimental Setup

The experimental setup constructs a standardized and stable testbed through incremental stabilization, then evaluates policy optimizers with method-specific tuning across agentic tasks.

  • The testbed progressively applies behavior cloning, format-penalty enforcement, KL regularization, and policy-optimizer-specific tuning.
  • Behavior cloning initializes the policy from high-scoring self-generated multi-turn interaction traces.
  • Format penalties enforce explicit <think> and <action> tags, providing dense early-training signals and reducing invalid rollouts.
  • An auxiliary KL penalty regularizes updates toward a reference policy while preserving exploration, using the k3 Bregman-divergence estimator.
  • Each optimizer receives default evaluation followed by hyperparameter grid search until final-training success-rate variance falls below a predefined threshold.
  • Experiments adapt ALFWorld, WebShop, Sokoban, and TIR Math, using an agentic loop and segmenting complete trajectories into single-turn optimization samples.

4. Exploring Gradient Dimensions on ARL

The analysis shows that importance-sampling design strongly affects agentic RL stability: tolerant clipping produces rapid early gains but can collapse, while sequence-level clipping supports stable improvement. Sequence masking targets the negative-advantage, low-ratio samples associated with collapse, and richer environmental advantage and dynamic filtering improve performance and stability.

  • Importance Sampling Clipping: CISPO and SAPO score 34.03 and 32.22 on average, respectively, versus 46.16 for GRPO, while GSPO improves over GRPO by 13.3%.
  • Importance Sampling Clipping: Tolerant clipping in CISPO and SAPO causes rapid early gains but training collapse, whereas sequence-level clipping in GSPO yields stable improvement.The collapse is accompanied by exploding gradient norms and KL divergence, a sharp valid-format-ratio drop, and degraded success rate.
  • Importance Sampling Analysis: Negative-advantage sequences with low importance ratios are the main contributors to collapse, marked by growing lower-bound out-of-bounds tokens and an imbalanced KL contribution.The analysis partitions tokens and rollout groups by advantage sign, importance-ratio magnitude, entropy, and normalized KL divergence.
  • Sequence Masking: Sequence masking improves success from 54.12 to 78.88 for CISPO and from 25.16 to 76.92 for SAPO, producing stability comparable to GSPO.Masking targets sequences with negative advantages and low importance ratios; the stabilized variants also show steady KL divergence and gradient norms.
  • Advantage Design and Dynamic Filtering: GIGPO generally outperforms GRPO, averaging 49.71 versus 48.08, while dynamic filtering with GIGPO benefits training stability and performance.GIGPO improves ALFWorld by 34.4%; EMPG shows task-dependent effects, improving WebShop by 11.5% but degrading ALFWorld by 7.1%.
  • Off-Policy Staleness: Lower off-policy staleness improves task outcomes: TIR Math reaches 87.34% and 50.00% pass@32, while ALFWorld reaches 60.80%.Under high off-policy ratios, the corresponding figures fall to 74.99% and 43.85% for TIR Math and 52.71% for ALFWorld.

5. SAMPO

SAMPO unifies sequence-level clipping, fine-grained advantage estimation, and dynamic filtering to address dominant sources of instability in agentic reinforcement learning. It achieves strong performance across evaluated tasks and outperforms larger closed-source models on ALFWorld.

  • 5.1. Motivation: Sequence-level clipping, rather than token-wise constraints, suppresses harmful trajectories and substantially improves training stability in long-horizon ARL.The analysis identifies sequence-level control as critical for handling off-policy drift across multi-turn interactions.
  • 5.1. Motivation: Fine-grained advantage design combines global and local signals, increasing advantage diversity and improving credit assignment under sparse rewards.Environment-level information and finer-grained signals contribute to both stability and performance.
  • 5.1. Motivation: Dynamic trajectory filtering removes samples with degenerate advantages, stabilizing gradient updates and producing more informative policy gradients.The method combines this filtering with fine-grained advantage design for further training benefits.
  • 5.2. Our Method: SAMPO integrates sequence-level clipping, fine-grained advantage estimation, and dynamic filtering into a unified policy-optimization framework.These components target off-policy drift, sparse-reward credit assignment, and degenerate gradient updates.
  • 5.2. Our Method: SAMPO shows the strongest overall performance in Table 1 and particularly large improvements on long-horizon interactive tasks such as ALFWorld.The results support combining multiple policy-gradient design dimensions rather than modifying only one.
  • 5.3. Benchmarking against Inference Paradigms: 92.72% all-task success on ALFWorld is achieved by Qwen3-4B-RFT with SAMPO, versus 51.56% for GPT-5.2 and 56.25% for o3-based MAS.The benchmark compares single-LLM and multi-agent inference paradigms on ALFWorld and WebShop.

6. Insights for Future Work

The paper identifies clean training recipes, clipping sensitivity, and stability-enabled scaling as priorities for future ARL research. Stable training permits continued improvement over more optimization steps and supports larger horizons and environments.

  • (1) Clean training recipes are foundational for complex reasoning: Clean initialization, format constraints, and conservative KL regularization are foundational components of complex agentic reasoning training.ARLArena treats these recipe choices as defining the feasible region in which reasoning policies can emerge.
  • (2) IS clipping is highly sensitive, while advantage design offers a comparatively stable gain: Importance-sampling clipping is highly sensitive, whereas advantage design provides more stable but comparatively modest improvements across tasks.Small changes to clipping thresholds or ratio parameterization can drastically affect stability.
  • (3) Stable ARL unlocks long-horizon scaling opportunities: Once collapse is mitigated, agentic policies sustain performance improvements over substantially more optimization steps without degradation.The paper connects this stability with opportunities to scale interaction horizons and environment size.

7. Conclusion

ARLArena systematically evaluates policy-gradient design choices for multi-turn agentic LLM training and identifies sequence-level clipping as critical for stability. SAMPO combines the resulting remedies into stable and effective ARL training.

  • 7. Conclusion: ARLArena shows that sequence-level clipping is critical for stability, while advantage design and dynamic filtering provide smaller but consistent gains.Loss aggregation has limited effect in the reported analysis.
  • 7. Conclusion: SAMPO unifies the identified policy-design principles into a framework for stable and effective agentic reinforcement learning.The conclusion emphasizes principled policy design and reproducible evaluation as central to advancing ARL.

Supplementary Materials for

The supplementary material compares trajectory and token loss aggregation schemes, then describes token-level importance-sampling clipping variants and their stability trade-offs. These choices differ in weighting, normalization, gradient flow, and sequence-level control.

  • A.1. Loss Aggregation: Token-mean weights all unmasked tokens equally, but longer trajectories receive greater total weight because they contribute more tokens.This can bias optimization toward long trajectories.
  • A.1. Loss Aggregation: Sequence-mean-token-mean weights each trajectory equally, giving shorter trajectories larger per-token weight and potentially introducing response-level length bias.Longer trajectories are relatively down-weighted under this estimator.
  • A.1. Loss Aggregation: Sequence-mean-token-sum removes per-trajectory token normalization, so longer trajectories receive proportionally larger weight.The broader aggregation discussion frames these estimators as different empirical approximations to trajectory-token expectations.
  • A.1. Loss Aggregation: Length-normalized sequence-mean-token-sum enforces a uniform upper bound on each trajectory’s contribution under a fixed maximum generation length.It also assigns equal weight to tokens across batches within that fixed-length budget.
  • A.2. Importance Sampling Clipping: GRPO clips token-level importance ratios directly, truncating gradient contributions when ratios leave the clipping range.The clipping mechanism constrains policy updates relative to the old policy.
  • A.2.2. CISPO: CISPO clips the importance ratio itself and uses stop-gradient, preserving gradient flow for clipped tokens while retaining token-local control.It does not explicitly enforce sequence-level coherence.
  • A.2.3. SAPO: SAPO replaces hard clipping with smooth temperature-controlled gating, attenuating off-policy updates continuously and asymmetrically suppressing high-variance negative-advantage updates.Despite improved smoothness, SAPO remains token-level and cannot explicitly prevent a few extreme tokens from destabilizing a full trajectory.

A.2.4. GSPO

The paper contrasts token-level and sequence-level clipping, identifying sequence-level clipping as central to stable multi-turn agentic RL. It also presents hierarchical, entropy-aware advantage designs and dynamic hyperparameter selection for finer-grained and more stable optimization.

  • GSPO: GSPO applies importance-ratio clipping once per sequence, making every token in a trajectory share the same clipped update.This aligns importance sampling with sequence-level rewards and enforces sequence-level coherence.
  • GSPO: Sequence-level clipping suppresses high-variance token outliers and substantially stabilizes optimization in long-horizon agentic RL.The paper’s empirical summary identifies sequence-level clipping as a key factor in stabilizing multi-turn training.
  • Advantage design: GiGPO combines trajectory-level and step-level relative advantages, using repeated environment states to construct finer-grained credit-assignment groups.Its episode-level component normalizes total returns within rollout groups, while its step-level component compares discounted returns for actions associated with anchor states.
  • Advantage design: The combined GiGPO advantage is a linear combination of episode-level and step-level components, with ω controlling the contribution of step-level credit.This preserves the critic-free, group-based structure while adding finer-grained credit assignment.
  • Advantage design: EMPG reshapes step-wise learning signals using policy entropy while retaining a trajectory-level optimization objective.It uses self-calibrating entropy scaling and a future-clarity bonus, followed by batch-level advantage normalization.
  • Advantage design: EMPG’s future-clarity bonus encourages transitions toward lower-uncertainty future states, with ζ and k′ controlling its contribution and sensitivity.The method also applies final batch-level normalization before policy-gradient updates.
  • Experimental setup: Method-specific hyperparameters are selected by task-specific grid search while remaining training and optimization settings stay fixed.The final configurations are stable settings chosen from the sweep.

C.1. Performance on 8B Model

Experiments with Qwen3-8B on ALFWorld and WebShop test whether the policy-design findings from smaller models persist at larger scale. The results show consistent trends, with sequence-level clipping remaining essential and SAMPO retaining the best performance.

  • Evaluation setting: Qwen3-8B is evaluated on ALFWorld as a representative complex, multi-turn agentic benchmark to test whether 4B-model design principles scale.The larger-scale evaluation focuses on ALFWorld because large-scale RL training is computationally demanding.
  • Results: The 8B experiments on ALFWorld and WebShop reproduce the relative performance gains and stability trends observed with 4B models.The paper presents this consistency as evidence that the policy-design findings persist under increased model capacity.
  • Results: Sequence-level clipping remains indispensable for preventing training collapse as model capacity increases.Advantage design and dynamic filtering continue to provide consistent but incremental final-performance improvements, while loss aggregation has limited impact.

C.2. Additional Analysis Result

Additional analyses examine the dynamics behind CISPO instability and summarize broader task and framework context. After collapse, harmful negative-advantage, low-ratio trajectories dominate and coincide with rising KL divergence, while sequence masking avoids these pathological updates.

  • Training-dynamics analysis: CISPO and CISPOSM trajectories are partitioned into eight groups using advantage sign, entropy threshold, and importance-sampling-ratio criteria.The groups are used to analyze how KL divergence evolves during training.
  • Training-dynamics analysis: After CISPO collapses, trajectories with negative advantages and low importance-sampling ratios rapidly dominate the distribution.The imbalance strongly correlates with a surge in KL divergence and subsequent training instability.
  • Training-dynamics analysis: Sequence-level masking improves stability by removing harmful negative-advantage, low-ratio trajectories and maintaining more balanced gradient signals.This explains the substantially improved stability of CISPOSM in the analyzed setting.
  • Task context: ALFWorld, WebShop, and Sokoban provide interactive multi-step environments, while TIR Math evaluates tool-assisted mathematical reasoning with pass@k.The listed environments cover household planning, e-commerce decisions, grid-based navigation, and Python-assisted math reasoning.
  • Additional framework context: The broader framework describes multi-agent debate that iterates through candidate solutions and reasoning until majority consensus or a maximum-round fallback.If consensus is not reached by Tmax, the final output is randomly sampled from the remaining candidates.
  • Additional framework context: An aggressive debate variant prioritizes task completion by imposing bounded exploration and requiring commitment within a finite horizon.It seeks the best available partial success rather than exhaustive exploration for a perfect solution.

E.3. Experiment Results on SLA and MAS

Sankey graphs preserve temporal action-transition structure in agent rollouts, exposing repetitive behaviors that underlie long-horizon failures. WebShop failures are dominated by pagination loops and backtracking oscillations.

  • Sankey analysis: Sankey graphs place time steps in columns, encode action frequencies by node height, and show consecutive-step transitions with edges.Unlike action histograms, they preserve temporal structure and reveal loop-like behavior.
  • WebShop results: API-agent WebShop failures frequently contain repetitive next actions, indicating exploration inefficiency and limited progress toward constraints.The API agent is a single-agent GPT-4o baseline without task-specific training.
  • WebShop results: Failed WebShop trajectories show pagination loops and backtracking oscillations between click product and back.These patterns correspond to repeated pagination or revisits to previously viewed product pages with limited progress toward constraint satisfaction.

F.3. WebShop: How RL Post-training Changes Behaviors

RL post-training changes WebShop behavior toward more targeted product inspection and earlier purchase attempts, while leaving backtracking and constraint-tracking failures. The analysis also points to loop-aware control and explicit state memory as possible remedies.

  • RL post-training produces fewer next-dominated failures and more trajectories reaching click product and eventually attempting buy.This pattern is consistent with more targeted product inspection and earlier decision making.
  • Residual RL failures include repeated click other/back browsing and premature buy attempts that violate some constraints.These patterns suggest inefficient navigation and incomplete constraint tracking.
  • WebShop Sankey diagrams distinguish successful from failed trajectories, while failure-only diagrams color nodes by action type.The visualizations support comparing action-transition patterns between API and RL-optimized agents.
  • Loop-aware control and explicit constraint/state memory are proposed to improve robustness beyond RL post-training.Suggested mechanisms include detecting repeated action cycles and maintaining concise records of visited items and verified constraints.
Loading 2602.21534v3…