PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying Reinforcement Learning in Agentic Reasoning

Zhaochen Yu, Ling Yang, Jiaru Zou, Shuicheng Yan, Mengdi Wang

arXiv:2510.11701v1cs.CL

TL;DR

Agentic reinforcement learning lacks clear design principles for data, algorithms, and reasoning modes, despite its promise for improving LLM reasoning. This paper systematically studies those dimensions and finds that real end-to-end and diverse data, exploration-friendly training choices, and deliberate tool use improve agentic reasoning, with strong performance from smaller models. The study also contributes agentic SFT and RL datasets and the DemyAgent-4B baseline.

  • Problem

    Agentic reinforcement learning still lacks clear guidance on effective policy optimization, exploration–exploitation balance, entropy, data curation, and reasoning modes.

  • Method

    The paper conducts a systematic empirical investigation of agentic reinforcement learning across data, algorithm, and reasoning mode, while providing agentic SFT and RL datasets.

  • Results

    Across challenging benchmarks, the study reports improved agentic reasoning from real end-to-end and diverse data, exploration-friendly techniques, and fewer deliberate tool calls, with DemyAgent-4B outperforming larger models.

  • Takeaways & Limitations

    Practical agentic RL recipes can improve training efficiency and reasoning performance while supporting strong results with smaller models.

  • Takeaways & Limitations

    The experiments use small-sized models such as 4B and 7B, so larger models may exhibit different sensitivities and reasoning interactions with RL training.

Abstract

from arXiv · show

Recently, the emergence of agentic RL has showcased that RL could also effectively improve the agentic reasoning ability of LLMs, yet the key design principles and optimal practices remain unclear. In this work, we conduct a comprehensive and systematic investigation to demystify reinforcement learning in agentic reasoning from three key perspectives: data, algorithm, and reasoning mode. We highlight our key insights: (i) Replacing stitched synthetic trajectories with real end-to-end tool-use trajectories yields a far stronger SFT initialization; high-diversity, model-aware datasets sustain exploration and markedly improve RL performance. (ii) Exploration-friendly techniques are crucial for agentic RL, such as clip higher, overlong reward shaping, and maintaining adequate policy entropy could improve the training efficiency. (iii) A deliberative strategy with fewer tool calls outperforms frequent tool calls or verbose self-reasoning, improving tool efficiency and final accuracy. Together, these simple practices consistently enhance agentic reasoning and training efficiency, achieving strong results on challenging benchmarks with smaller models, and establishing a practical baseline for future agentic RL research. Beyond these empirical insights, we further contribute a high-quality, real end-to-end agentic SFT dataset along with a high-quality RL dataset, and demonstrate the effectiveness of our insights in boosting the agentic reasoning ability of LLMs across four challenging benchmarks, including AIME2024/AIME2025, GPQA-Diamond, and LiveCodeBench-v6. With our recipes, 4B-sized models could also achieve superior agentic reasoning performance compared to 32B-sized models. Code and models: https://github.com/Gen-Verse/Open-AgentRL

1 INTRODUCTION

The paper systematically investigates agentic reinforcement learning through data, algorithm, and reasoning-mode design, addressing persistent challenges in training stability, exploration, and tool use. It reports that real end-to-end data, exploration-friendly algorithms, and deliberate tool interaction improve agentic reasoning, including with smaller models.

  • Motivation: These challenges motivate a systematic study of data, algorithm, and reasoning-mode factors that hinder or enhance agentic reasoning.The paper organizes its investigation around the three perspectives introduced in the problem framing.
  • Data: Real end-to-end SFT trajectories and high-diversity, model-aware RL datasets improve training efficiency and agentic reasoning performance.Stitched synthetic trajectories fail to faithfully represent when and why tools should be invoked, while diverse RL data encourages exploration.
  • Algorithm: Conservative clipping and KL penalties overly constrain exploration, whereas sustaining higher entropy improves reinforcement-learning efficiency.The study compares GRPO-based algorithms and examines pass@k and average@k as indicators of exploration–exploitation trade-offs.
  • Reasoning Mode: Fewer, more deliberate tool interactions often outperform frequent calls, making effective and accurate invocation more important than external-call quantity.The analysis links tool-call frequency and response length to performance and identifies over-reliance on external calls as unhelpful.
  • Contributions: DemyAgent-4B achieves SOTA-level performance on challenging benchmarks and outperforms larger models in agentic reasoning.The paper presents DemyAgent-4B as a strong baseline model, with the comparison reported in Table 2.

2 PROBLEM FORMULATION

This section formalizes agentic RL with tool-interleaved trajectories and presents GRPO-based techniques that vary loss aggregation, reward shaping, and clipping. It also defines the paper’s recipe variants for comparing these choices.

  • 2.1 AGENTIC REINFORCEMENT LEARNING: Agentic RL optimizes a policy LLM against rewards and KL divergence while outputs may interleave reasoning with tool-call feedback.The input is sampled from dataset D, and the output can include tool-call feedback.
  • 2.1 AGENTIC REINFORCEMENT LEARNING: Unlike conventional rollout-only RL, agentic RL incorporates tool-call feedback into reasoning trajectories before final answer generation.The trajectory contains reasoning and tool-call feedback, followed by the final answer.
  • 2.2 GRPO-BASED ALGORITHM AND TECHNIQUES: GRPO is the baseline rule-based RL algorithm, with a generalized objective that exposes loss aggregation granularity, importance ratios, clipping, and normalized advantages.The formulation supports comparing different GRPO-based improvement techniques.
  • 2.2 GRPO-BASED ALGORITHM AND TECHNIQUES: The study compares token-level and sequence-level loss aggregation as alternative optimization granularities.Token-level loss aggregates across tokens, whereas sequence-level loss aggregates at the sequence level.
  • 2.2 GRPO-BASED ALGORITHM AND TECHNIQUES: The reward combines solution accuracy with a clipped tool-use bonus based on invocation count to discourage degenerate tool-abuse reward hacking.The number of tool invocations is denoted by n.
  • 2.2 GRPO-BASED ALGORITHM AND TECHNIQUES: Overlong reward shaping gives zero reward within a safe length budget, applies a linear boundary penalty, and assigns -1 beyond the maximum length.This preserves a smooth signal near the boundary while discouraging overlong completions.
  • 2.3 RECIPE DESIGN: The paper studies loss aggregation granularity, reward shaping, and clipping strategy as three GRPO improvement techniques.These techniques form the basis for the recipe comparisons.
  • 2.3 RECIPE DESIGN: GRPO-TCR combines token-level loss, higher clipping, and overlong reward shaping, while GRPO-SCR substitutes sequence-level loss and GRPO-T serves as the baseline.GRPO-T changes the sample-level loss to token-level within the referenced baseline implementation.

3 DATA IN AGENTIC REASONING

The data study finds that real end-to-end trajectories provide stronger SFT initialization than synthetic stitched trajectories, while diverse and model-aware RL datasets improve exploration and training efficiency. These effects appear through stronger benchmark performance, sustained entropy, and more informative reward signals.

  • 3.1 REAL END-TO-END TRAJECTORIES VERSUS SYNTHETIC STITCH-STYLE TRAJECTORIES: Synthetic stitch-style trajectories omit when, why, and what to do next around tool use, motivating evaluation of real end-to-end trajectories for RL initialization.The comparison finetunes Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 under identical settings.
  • 3.1 REAL END-TO-END TRAJECTORIES VERSUS SYNTHETIC STITCH-STYLE TRAJECTORIES: 29.97% average@32, 72.88% pass@32, and 45.22% maj@32 were achieved on AIME2025 by Qwen3-4B trained on real trajectories, versus below 10% average@32 for synthetic data.The real-trajectory initialization was also described as more stable and having a higher ability upper bound.
  • 3.1 REAL END-TO-END TRAJECTORIES VERSUS SYNTHETIC STITCH-STYLE TRAJECTORIES: Real end-to-end trajectories preserve pre-call analysis, guarded execution, error recovery, and self-reflection that synthetic stitching cannot replicate.These trajectories capture complete agentic reasoning behaviors through real end-to-end processes.
  • 3.2 DIVERSE DATA MAINTAINS HIGH ENTROPY IN TRAINING: Diverse data reaches over 50% average@32 on AIME2025 within 150 steps, whereas the DAPO-Math baseline requires 220 steps.The diverse dataset also produces a larger early entropy gain and sustains higher entropy through convergence.
  • 3.2 DIVERSE DATA MAINTAINS HIGH ENTROPY IN TRAINING: Diverse RL datasets sustain higher policy entropy, indicating broader exploration and faster, more stable agentic RL training.The comparison uses average@32 accuracy and policy entropy during training.
  • 3.3 MODEL-AWARE DATASETS FOR MORE EFFECTIVE RL: A model-aware curated dataset yields more effective improvement than an unfiltered 30k dataset and substantially raises average reward during training.The stronger rewards provide more valid advantages and amplify gradient signals.
  • 3.3 MODEL-AWARE DATASETS FOR MORE EFFECTIVE RL: Model-aware data matches problem difficulty to current model ability, helping overcome weak-model performance bottlenecks through stronger learning feedback.After the bottleneck is broken, the dataset can be recollected based on the model’s updated ability.

4 ALGORITHMIC DESIGN AND TRAINING DYNAMICS IN AGENTIC RL

The study compares GRPO-based recipes and examines exploration, entropy, clipping, and loss granularity in agentic RL. It finds that exploration-friendly optimization improves efficiency, but excessive entropy or conservative constraints can harm training, while stronger models benefit from deliberate exploration.

  • 4.1 The Impact of RLVR Techniques on Agentic RL: 70.93%/68.13% average@32 was reached by GRPO-TCR on AIME2024/AIME2025 within 450 steps, while GRPO-T needed 100 steps to reach only 54.7%/40.93%.GRPO-TCR achieved comparable performance using 25% of GRPO-T’s training computation.
  • 4.1 The Impact of RLVR Techniques on Agentic RL: Token-level loss exceeded sequence-level loss by 3.95% on AIME24 and 3.86% on AIME25 for models with stronger initial exploration.For weaker Qwen2.5-7B models, the two loss granularities achieved comparable average@32 performance.
  • 4.2 Exploration–Exploitation Dynamics in Agentic RL: GRPO-TCR and GRPO-SCR jointly improved pass@k and average@k by over 10% on AIME2024/AIME2025, unlike the conservative GRPO-T baseline.The study attributes GRPO-T’s suppressed exploration to restrictive clipping combined with strong KL regularization.
  • 4.2 Exploration–Exploitation Dynamics in Agentic RL: The gap between pass@k and average@k is identified as a critical bottleneck because RL converts exploration into exploitation through this gap.Training gains are bounded by the discrepancy between the model’s potential solutions and its average performance.
  • 4.3 When High Entropy Drives Better Efficiency: Higher policy entropy was associated with more effective agentic RL, but entropy had to remain balanced to avoid premature convergence or instability.GRPO-T exhibited early entropy collapse, whereas GRPO-TCR and GRPO-SCR rose faster and stabilized at higher levels.
  • 4.3 When High Entropy Drives Better Efficiency: Increasing ϵhigh from 0.28 to 0.315 made Qwen2.5-7B reach equivalent performance 40% faster, whereas ϵhigh = 0.35 harmed Qwen3-4B training effectiveness.The relationship between clipping and efficiency was non-monotonic: moderate increases helped, but excessive exploration slowed convergence and destabilized training.

5 REASONING MODES IN AGENTIC RL

The study identifies deliberative reasoning with fewer, more successful tool calls as the strongest agentic mode, while current Long-CoT models often avoid tools on reasoning-intensive tasks. SFT alignment improves Long-CoT tool use, but instruction-based models remain more suitable for scaling agentic reasoning from scratch.

  • 5.1 WHEN FEWER TOOL CALLS LEAD TO BETTER TOOL USE: The analysis distinguishes Reactive Mode, characterized by short thinking and frequent calls, from Deliberative Mode, characterized by deliberate thinking and fewer calls.Stronger models predominantly adopt the deliberative pattern, while weaker models tend toward reactive behavior.
  • 5.1 WHEN FEWER TOOL CALLS LEAD TO BETTER TOOL USE: Deliberative Mode combines longer internal reasoning with fewer tool calls and consistently outperforms Reactive Mode in tool-use efficiency and overall performance.Deliberative agents achieve over 70% tool-use success, whereas frequent reactive calls are often ineffective or erroneous.
  • 5.2 LIMITATIONS OF CURRENT LONG-COT MODELS IN AGENTIC RL: Long-CoT models initially achieve strong average@32 performance but progressively reduce tool use toward zero during agentic RL training.On reasoning-intensive tasks they rely mainly on internal reasoning, whereas they may invoke search tools for knowledge-intensive tasks.
  • 5.3 ALIGNING LONG-COT WITH AGENTIC RL: SFT with multi-turn tool-use trajectories enables Long-CoT models to invoke tools while retaining internal reasoning, improving performance over non-initialized models.Despite this initial advantage, Long-CoT models ultimately reach only comparable performance to instruction-based models.
  • 5.3 ALIGNING LONG-COT WITH AGENTIC RL: Instruction-based models scale agentic reasoning more effectively because Long-CoT models must simultaneously acquire tool-use skills and overcome conflicting internal reasoning priors.The paper therefore favors direct agentic RL from instruction-based models over training from Long-CoT models with incompatible reasoning patterns.

6 CONTRIBUTIONS AND COMPARISON ON CHALLENGING BENCHMARKS

The paper contributes reusable datasets, cold-start models, and DemyAgent-4B as a compact baseline for agentic RL. Across challenging reasoning benchmarks, the 4B model matches or outperforms substantially larger models.

  • CONTRIBUTIONS: The paper introduces a 3k end-to-end agentic SFT dataset and a 30k diverse RL dataset for agentic reasoning research.These datasets are presented as high-quality resources supporting the proposed training recipes.
  • CONTRIBUTIONS: The authors release Qwen2.5-7B-RA-SFT and Qwen3-4B-RA-SFT as two cold-start models for downstream RL research.The models are intended to enable broader experimentation with agentic RL training.
  • TRAINING RECIPE: DemyAgent-4B uses the complete 30k RL dataset with GRPO-TCR and a higher clip upper bound, ϵhigh = 0.315.The recipe applies Qwen3-4B-RA-SFT as its base model and uses the higher bound to balance exploration and constraint satisfaction.
  • OVERALL RESULTS: DemyAgent-4B matches or outperforms 14B/32B models across challenging self-contained and agentic reasoning benchmarks despite having only 4B parameters.The evaluation covers paradigms using internal reasoning alone and external tools such as code interpreters and search engines.
  • CONTRIBUTIONS: The training recipe unlocks strong agentic capabilities in compact models and achieves SOTA performance against significantly larger models.This baseline is presented as validation of the study’s training insights.

7 RELATED WORK

Related work positions agentic RL as an extension of tool-integrated reasoning toward autonomous multi-tool coordination. The literature also identifies entropy collapse and the lack of systematic understanding as important unresolved challenges.

  • TOOL-INTEGRATED REASONING AND AGENT RL: Tool-integrated reasoning extends LLM capabilities by combining internal reasoning with external tools such as code interpreters and search engines.Prior approaches include prompting-based methods and reinforcement-learning frameworks for more capable agents.
  • AGENT REINFORCEMENT LEARNING: Agent RL frameworks increasingly support multi-turn decisions about whether to invoke tools and which tools to select in complex reasoning chains.These efforts aim to generalize beyond single-tool domains toward flexible multi-tool coordination.
  • AGENT REINFORCEMENT LEARNING: Current Agent RL methods remain tied to specific workflows and lack a systematic understanding of how reinforcement learning should be applied across agentic settings.This gap motivates the paper’s investigation of data, algorithm, and reasoning-mode factors.
  • ENTROPY MECHANISM FOR REINFORCEMENT LEARNING: Entropy collapse is described as a central reinforcement-learning obstacle because outcome-driven optimization can erode the exploration ability needed for effective scaling.Prior work analyzes entropy’s role in exploration and the bottleneck created by its collapse.

8 DISCUSSION AND FUTURE WORK

The discussion highlights data scarcity, difficulty scaling deliberate reasoning, and limited evaluation settings as boundaries for current agentic RL. Future work should develop more efficient data curation, agent-oriented planning, and multi-tool evaluation.

  • 8.1 DATA-FUEL SCARCITY: End-to-end SFT trajectories are computationally costly to collect, making small, high-quality distilled datasets a promising way to reduce data scarcity.The authors connect dataset quality to both training effectiveness and the scaling upper bound of agentic reasoning.
  • 8.2 EFFECTIVE SCALING OF AGENTIC REASONING: Scaling deliberate reasoning remains challenging even though it performs better than frequent tool invocation for agentic problem-solving.The paper proposes exploring reasoning frameworks centered on strategic planning and efficient tool orchestration.
  • 8.2 EFFECTIVE SCALING OF AGENTIC REASONING: Future agent-oriented reasoning chains should emphasize decomposing problems into tool-executable subtasks, selecting tools strategically, and synthesizing tool outputs.This direction shifts emphasis from reasoning-centric behavior toward high-level tool planning.
  • 8.3 ADDITIONAL APPLICATION SCENARIOS IN AGENTIC REASONING: The study mainly evaluates code interpreters in a static, single-tool environment, leaving multi-tool and optimizable environments for further investigation.Multi-tool settings may require additional exploration because correct solutions can involve different tool combinations.

9 LIMITATIONS

The study evaluates agentic RL on small models, leaving its applicability to larger models as an important open direction.

  • Experiments use small-sized models such as 4B and 7B, limiting direct conclusions about larger-model agentic RL.Larger models may differ in hyperparameter sensitivity, reward-signal responses, exploration needs, and interactions with RL dynamics.

10 CONCLUSION

The paper systematically studies agentic RL across data, algorithms, and reasoning modes, identifying practical choices that improve training and tool use. Across challenging benchmarks, its recipes improve agentic reasoning and support efficient reasoning with fewer, more deliberate tool calls.

  • Data: Real end-to-end trajectories strengthen agentic SFT, while diverse model-aware RL data sustain exploration and stable training.The study contrasts faithful multi-turn trajectories for SFT with high-diversity data for RL.
  • Algorithm: Clip higher, reward shaping, token-level loss, and appropriate entropy improve agentic RL training effectiveness.The paper identifies entropy maintenance as a key driver of effective agentic RL.
  • Reasoning Mode: Fewer, more deliberate tool calls outperform frequent calls, while Long-CoT priors can hinder tool adoption and slow scaling.The reasoning findings support a quality-over-quantity strategy for tool use.
  • Evaluation: The recipes improve agentic reasoning across AIME2024/2025, GPQA-Diamond, and LiveCodeBench-v6.The evaluation covers challenging mathematics, science, and coding benchmarks.

A.1 TRAINING SETUP

Training uses Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 with curated agentic data, VeRL, and eight Tesla A100 GPUs.

  • The base models are Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507.
  • The training data include 3k actual agentic trajectories for SFT and 30K high-quality RL examples spanning math, science, and code.
  • Experiments use the VeRL framework and run on 8×Tesla-A100-80G GPUs.

A.2 EVALUATION SETUP

Evaluation covers four challenging benchmarks with repeated sampling for major accuracy and diversity metrics, alongside separately constructed agentic training trajectories and task-specific prompts.

  • The evaluation benchmarks are AIME2024, AIME2025, GPQA-Diamond, and LiveCodeBench.
  • For AIME2024/2025 and GPQA-Diamond, 32 samples per problem support average@32, pass@32, and maj@32 evaluation, while LiveCodeBench uses pass@1 and pass@5.
  • Real end-to-end SFT trajectories are generated with Qwen3-Coder-30B-A3B through Qwen-Agent and Sand-BoxFusion.The construction draws from s1-1k, a self-curated 3k LeetCode dataset, and a 2k ReTool multi-turn SFT set.
  • The RL dataset combines DAPO-Math, Skywork-or1 math and code, and MegaScience science samples into a diverse 30k-example collection.
  • Task-specific prompts standardize answer formats while encouraging simultaneous reasoning and tool use across mathematics, science, and programming.Agentic RL and evaluation prompts include high-level planning before tool use and task-specific answer requirements.
Loading 2510.11701v1…