PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Reinforced Policy Optimization

Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, Zhicheng Dou

arXiv:2507.19849v1cs.LGcs.AIcs.CL

TL;DR

Existing RL methods are effective for single-turn reasoning but inadequately balance long-horizon reasoning with multi-turn tool use. ARPO addresses this gap with entropy-adaptive rollouts and advantage attribution, and it outperforms trajectory-level RL across 13 benchmarks while using half the tool-use budget.

  • Problem

    Current RL algorithms inadequately balance LLMs' long-horizon reasoning capabilities with proficiency in multi-turn interactions with external tools.

  • Method

    ARPO combines entropy-based adaptive rollout sampling with advantage attribution to explore and learn from uncertain stepwise tool-use interactions.

  • Results

    ARPO consistently outperforms trajectory-level RL across 13 challenging benchmarks and achieves this with only half the required tool-call budget.

  • Takeaways & Limitations

    ARPO provides a scalable approach for training multi-turn LLM-based agents in computational reasoning, knowledge reasoning, and deep search domains.

  • Takeaways & Limitations

    The study follows a cold-start supervised-fine-tuning-with-RL paradigm to mitigate reward collapse during initial RL training.

Abstract

from arXiv · show

Large-scale reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks. In realistic reasoning scenarios, LLMs can often utilize external tools to assist in task-solving processes. However, current RL algorithms inadequately balance the models' intrinsic long-horizon reasoning capabilities and their proficiency in multi-turn tool interactions. To bridge this gap, we propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents. Through preliminary experiments, we observe that LLMs tend to exhibit highly uncertain behavior, characterized by an increase in the entropy distribution of generated tokens, immediately following interactions with external tools. Motivated by this observation, ARPO incorporates an entropy-based adaptive rollout mechanism, dynamically balancing global trajectory sampling and step-level sampling, thereby promoting exploration at steps with high uncertainty after tool usage. By integrating an advantage attribution estimation, ARPO enables LLMs to internalize advantage differences in stepwise tool-use interactions. Our experiments across 13 challenging benchmarks in computational reasoning, knowledge reasoning, and deep search domains demonstrate ARPO's superiority over trajectory-level RL algorithms. Remarkably, ARPO achieves improved performance using only half of the tool-use budget required by existing methods, offering a scalable solution for aligning LLM-based agents with real-time dynamic environments. Our code and datasets are released at https://github.com/dongguanting/ARPO

1 INTRODUCTION

ARPO targets the gap between long-horizon reasoning and multi-turn tool interaction by exploring uncertain post-tool reasoning steps. It combines adaptive entropy-based rollouts with advantage attribution and outperforms trajectory-level RL across broad evaluations.

  • Research gap: Current agentic RL methods sample complete tool-use trajectories and assign rewards from final outputs, limiting step-level tool-use alignment.These approaches include trajectory-level algorithms such as GRPO and DAPO.
  • Motivation: Tool-call feedback produces high token entropy immediately afterward, exposing uncertainty that trajectory-level methods may overlook.The observation motivates focusing exploration on post-tool reasoning steps.
  • Method: ARPO adaptively branches rollout sampling during high-entropy tool-call rounds by combining global sampling with partial step-level sampling.Entropy variation after tool calls determines when additional branching occurs.
  • Method: Advantage attribution assigns shared advantages along source reasoning paths and distinct advantages to branched paths, encouraging stepwise tool-use learning.ARPO considers both hard and soft advantage settings.
  • Results: ARPO consistently surpasses trajectory-level RL across 13 datasets spanning computational reasoning, knowledge reasoning, and deep search.The reported evaluation also finds improved performance with only half the tool-call budget.
  • Contributions: The paper presents entropy analysis, adaptive rollout, advantage attribution, theoretical justification, and benchmark evidence as its main contributions.The contribution summary emphasizes practical insights for agentic RL under reduced tool-use budgets.

2 PRELIMINARY

Agentic RL optimizes language-model reasoning while incorporating feedback from external tools. Preliminary entropy analyses find sharp post-tool uncertainty, especially after search feedback, motivating entropy-aware exploration.

  • Agentic RL: The agentic RL objective maximizes expected reward while penalizing divergence from a reference language model.The objective samples inputs from a dataset and outputs from the policy, with tool feedback possibly interleaved in the output.
  • Agentic RL: Agentic RL differs from conventional RL by incorporating tool-call feedback during the reasoning process.The agent produces reasoning interleaved with tool feedback before generating a final answer.
  • Token entropy: Token entropy measures uncertainty in the generation distribution at each step, rather than uncertainty associated with one particular token.The distribution is determined from policy logits, vocabulary size, and decoding temperature.
  • Token entropy: Entropy rises sharply within the first 10–50 tokens after each tool call and remains higher after feedback than during early reasoning.Search feedback introduces more uncertainty than Python feedback.
  • Token entropy: The observed uncertainty is attributed to distributional shift between external feedback and internal reasoning, with textual search results causing greater fluctuations than deterministic Python outputs.The reported explanation distinguishes informative textual content from deterministic numerical feedback.
  • Implications: These findings identify a limitation of trajectory-level RL: it emphasizes initial reasoning while overlooking uncertainty introduced by tool-call feedback.ARPO addresses this limitation with entropy-based exploration for agent training.
  • Evaluation tools: The evaluation uses search engines, web-browser agents, and code interpreters as representative tools for testing ARPO.These tools respectively retrieve web information, parse returned links, and execute model-generated code.

3 AGENTIC REINFORCE POLICY OPTIMIZATION

ARPO combines entropy-guided adaptive rollout with advantage attribution to train multi-turn LLM agents for stepwise tool-use behavior. Its rollout branches at high-entropy tool-use steps, while its policy updates distinguish shared and individual reasoning segments.

  • Entropy-based Adaptive Rollout: ARPO extends trajectory-level rollout with branching at high-entropy tool-use steps to broaden exploration.The mechanism balances global trajectory sampling with entropy-based partial sampling and stops branching when the partial-sampling budget is reached.
  • Entropy-based Adaptive Rollout: ARPO monitors normalized entropy changes after tool calls and uses them to determine whether to branch partial reasoning paths.Positive entropy change indicates increased uncertainty; when the sampling probability exceeds threshold τ, the method branches Z paths.
  • Entropy-based Adaptive Rollout: The adaptive rollout reduces per-rollout complexity from trajectory-level RL’s O(n^2) to between O(n log n) and O(n^2).This estimate neglects minor token-level entropy-calculation overhead.
  • Advantage Attribution Estimation: ARPO assigns advantages differently to shared and individual tokens created by partial rollouts.Hard estimation explicitly averages advantages for shared segments, while individual tokens use normalized rewards; shared prefixes receive common importance weights under the soft formulation.
  • Advantage Attribution Estimation: Soft advantage estimation produces consistently higher and more stable rewards than hard estimation, so ARPO uses the soft setting by default.The comparison is reported for ARPO training in Figure 5.
  • Theoretical Foundation: Theoretical analysis frames partial rollout segments as macro actions and generalizes policy-gradient optimization beyond single-token actions.The resulting Generalized Policy Gradient theorem applies to differentiable Transformer-based policies and includes the traditional policy-gradient theorem as a special case.

4 EXPERIMENT

The experiments evaluate ARPO across mathematical, knowledge-intensive, and deep-search tasks against direct reasoning, trajectory-level RL, and search-agent baselines. ARPO consistently improves performance while reducing tool-use costs and benefits from balanced entropy-guided sampling.

  • Mathematical & Knowledge-Intensive Reasoning: ARPO consistently outperforms trajectory-level RL algorithms across the 10 mathematical and knowledge-intensive reasoning datasets.The reported comparison uses the same experimental setup across Qwen and Llama backbones.
  • Main Results: Prompt-only tool integration provides limited gains and can perform below direct reasoning, while trajectory-level RL struggles with multi-turn tool-use learning.DAPO is described as strong on single-turn reasoning but weaker in multi-turn, especially knowledge-intensive, interactions.
  • Mathematical & Knowledge-Intensive Reasoning: 4% average accuracy improvement is reported for ARPO across 10 datasets while maintaining competitive performance on individual domains.The gains extend across both Qwen and Llama model series.
  • Deep Search: ARPO reaches pass@1 scores of 10.0% on HLE and 43.2% on GAIA using Qwen3-8B and Qwen3-14B, respectively, with only 1K RL samples.The comparison notes 2% for GPT-4o and 8.6% for DeepSeek-R1-671B on HLE.
  • Tool-Call Efficiency: ARPO achieves higher overall accuracy than GRPO while using only half the tool calls during training.The paper attributes this efficiency to selectively branching during high-entropy tool-use steps.
  • Scaling Analysis: Moderate entropy weighting peaks at 0.4, initial sampling size peaks at 8, and larger global rollout sizes improve performance.Complete global sampling at initial size 16 causes a performance decline by disrupting the global-to-partial sampling balance.

5 RELATED WORK

The related work situates ARPO within reinforcement learning for LLM reasoning and agentic interaction with dynamic environments. It connects RLVR’s success in single-turn reasoning with broader RL approaches for tool-using and embodied agents.

  • Reinforcement Learning with Verifiable Reward: RLVR has become a leading RLHF approach, particularly improving mathematical and programming reasoning in LLMs.The passage frames RLVR as a recent development in reinforcement learning for language models.
  • Agentic Reinforcement Learning: Agentic reinforcement learning targets LLM agents that must adapt to dynamic and open environments.The discussion places this work alongside foundational self-play, value-based, and policy-based RL research.

6 CONCLUSION

The paper concludes that ARPO is designed for multi-turn LLM agents and uses post-tool-call uncertainty to guide adaptive exploration. Across 13 benchmarks, it outperforms trajectory-level RL while using half the tool-use budget.

  • Conclusion: ARPO is an RL algorithm tailored to training multi-turn LLM-based agents.Its design combines entropy-based adaptive rollout with Advantage Attribution Estimation for stepwise tool-use interactions.
  • Conclusion: ARPO balances global and step-level sampling to encourage diverse exploration during high-entropy tool-use phases.The paper reports that token entropy is high after tool usage, motivating this adaptive rollout design.
  • Conclusion: Across 13 challenging computational, knowledge, and deep-search benchmarks, ARPO consistently outperforms traditional trajectory-level RL algorithms.The conclusion presents this as the paper’s cross-domain experimental finding.
  • Conclusion: ARPO achieves this performance using only half the tool-use budget of other methods.The paper presents this efficiency as supporting scalable alignment in dynamic environments.

A DATASETS

The paper evaluates ARPO across mathematical, knowledge-intensive, and deep-search benchmarks, alongside established model and agent baselines.

  • Mathematical Reasoning Benchmarks: The evaluation covers mathematical reasoning datasets including AIME24, AIME25, and MATH500.These benchmarks contain challenging problems spanning areas such as algebra, geometry, calculus, number theory, and combinatorics.
  • Knowledge-Intensive Reasoning Benchmarks: Knowledge-intensive reasoning is evaluated with HotPotQA, 2WikiMultihopQA, and Musique, all targeting multi-hop question answering.The datasets test multi-step reasoning and information integration across documents.
  • Deep Search Benchmarks: Deep-search evaluation includes GAIA, HLE, and WebWalker, covering real-world tasks, complex reasoning, and web traversal.GAIA contains 466 questions, while WebWalker contains 680 question-answer pairs.
  • Baselines: The compared methods include GRPO, REINFORCE++, RAG, Search-o1, WebThinker, and ReAct, spanning policy optimization, retrieval, search, and tool-using reasoning.These baselines represent reinforcement learning, retrieval-augmented generation, agentic search, preference optimization, and reasoning-action frameworks.

C IMPLEMENTATION DETAILS

Implementation uses supervised fine-tuning followed by ARPO training, with separate configurations for deep reasoning and deep-search tasks and web retrieval through Bing.

  • Training Configuration: Supervised fine-tuning trains Qwen2.5-3B-Instruct for 3 epochs with a 7 × 10^-6 learning rate and BF16 mixed precision.The setup uses Llama Factory, DeepSpeed ZeRO-3, FlashAttention2, batch size 128, weight decay 0.1, and a 4096-token maximum input length.
  • ARPO Training: ARPO is implemented in VERL, excluding tool-invocation results from loss calculation while retaining text-reasoning and tool-request tokens.This design is intended to prevent bias toward tool outputs.
  • Deep Reasoning Tasks: For 7B deep-reasoning models, training uses batch size 128, PPO mini-batch size 16, global rollout size 16, and initial sampling size 8.Interaction responses are capped at 4096 tokens; ARPO uses entropy weight 0.2, a=0.5, and threshold 0.5.
  • Deep Search Tasks: For deep-search tasks, 8B models extend each interaction response to 8192 tokens, while 14B experiments use the same parameters on 16 NVIDIA H800 GPUs.With a 1K-sample dataset, reinforcement learning lasts 5 epochs.
  • Retrieval Configuration: Bing Web Search API retrieves 10 web pages per query using the US-English locale during training and testing.Mathematical and knowledge reasoning use the top 10 snippets, whereas deep search fetches up to 6000 tokens per page for browser-agent refinement.

C.4 SCALING EXPERIMENT SETUP

Scaling experiments retain the main training and rollout configuration while varying selected parameters for targeted studies.

  • Scaling Experiment Setup: Scaling experiments use batch size 128, PPO mini-batch size 16, global rollout size 16, and initial sampling size 8.ARPO rollouts retain entropy weight 0.2, a=0.5, and threshold 0.5 while specific parameters are varied.

D THEORETICAL ANALYSIS AND PROOFS

The theoretical analysis formalizes Soft Advantage Estimation by decomposing GRPO around partial rollouts and representing autoregressive outputs through macro states and actions.

  • Soft Advantage Estimation: The section analyzes Soft Advantage Estimation using the classic GRPO optimization objective as its starting point.The analysis introduces the objective before deriving its soft-advantage form.
  • Importance Sampling: The analysis assumes partial rollout begins at the l-th token and defines separate importance-sampling ratios before and after that token.The notation includes the input question, shared tokens, and the sequence preceding the t-th token.
  • Objective Decomposition: The classical GRPO objective is decomposed into shared-token and partial-rollout objectives associated with a selected position in the reasoning chain.The definitions distinguish the shared-token portion, the objective after partial rollout, and direct GRPO sampling from that position.
  • Soft Advantage Estimation: Under Soft Advantage Estimation, the GRPO objective combines a standard objective from the partial-rollout position with weighted differences between objectives before and after that point.The weights depend on the lengths of the corresponding reasoning chains.
  • Macro-State Factorization: The policy factorization splits complete outputs into macro actions conditioned on generalized macro states across macro timesteps.Input and output tokens form the macro state and macro action, while segmented outputs define later macro states and actions.

D.2.2 DERIVATION OF THE GPG THEOREM

The section presents the Generalized Policy Gradient Theorem for Transformer-based policies and explains why macro-action segmentation supports flexible trajectory splitting. It then describes ARPO’s entropy-guided rollout and advantage-estimation procedure for multi-turn tool-use training.

  • GPG theorem: The Generalized Policy Gradient Theorem is formulated for Transformer-based policies with macro states and macro actions.
  • GPG theorem: Arbitrary-length macro-action segmentation enables trajectory splitting based on special tokens.
  • GPG derivation: The derivation connects token-level Transformer policy factorization with macro-action probabilities and generalizes policy-gradient estimation from advantages toward GAE.
  • ARPO rollout: ARPO initializes a policy and reference model, samples reasoning paths, and computes entropy over the first k generated tokens.
  • ARPO rollout: After each tool call, ARPO measures normalized entropy change and sets partial sampling probability as α + β · ∆H_i,t.
  • ARPO optimization: When partial sampling probability exceeds τ, ARPO branches additional rollouts; it then completes paths, computes rewards, estimates advantages, and updates the policy.

F CASE STUDY

The case study illustrates ARPO outputs across HLE, GAIA, and AIME24 examples, using colored annotations to distinguish reasoning, tool calls, returned results, and final answers. One GAIA-style search example answers that the Second Great War lasted one year, while acknowledging a shorter core-conflict interpretation.

  • Annotated examples: ARPO case-study tables annotate reasoning, search queries, Python code, returned results, and final answers with distinct colored boxes.
  • GAIA example: The Second Great War is generally dated from 2504 to 2505, corresponding to a one-year duration.
  • GAIA example: Some sources instead describe the core conflict as lasting six months until April 2505, creating a narrower interpretation than the broader war period.
  • GAIA example: The worked reasoning concludes that the requested duration, rounded up in years, is 1 year.
Loading 2507.19849v1…