PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Entropy-Balanced Policy Optimization

Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jinghan Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, Zhicheng Dou

arXiv:2510.14545v1cs.LGcs.AIcs.CLcs.IR

TL;DR

Agentic RL can suffer from entropy-driven rollout over-branching and suppression of exploratory gradients, motivating AEPO. AEPO balances entropy across rollout and policy updates, and across 14 datasets it consistently outperforms mainstream RL algorithms while improving sampling diversity and maintaining stable policy entropy.

  • Problem

    Entropy guidance in agentic RL introduces rollout collapse and high-entropy token gradient clipping, making entropy balancing a central challenge for generalized agent training.

  • Method

    AEPO combines dynamic entropy-balanced rollout sampling with branch penalties and policy optimization that preserves high-entropy gradients and uses entropy-aware advantage estimation.

  • Results

    AEPO consistently outperforms mainstream RL algorithms across 14 datasets; with 1K RL samples, Qwen3-14B reaches 47.6% on GAIA at Pass@1.

  • Takeaways & Limitations

    AEPO improves rollout sampling diversity while maintaining stable policy entropy, supporting scalable web-agent training.

Abstract

from arXiv · show

Recently, Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents. While mainstream agentic RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on entropy signals can impose further constraints, leading to the training collapse. In this paper, we delve into the challenges caused by entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases. AEPO comprises two core components: (1) a dynamic entropy-balanced rollout mechanism that adaptively allocate global and branch sampling budget through entropy pre-monitoring, while imposing a branch penalty on consecutive high-entropy tool-call steps to prevent over-branching issues; and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into the high-entropy clipping term to preserve and properly rescale gradients on high-entropy tokens, while incorporating entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens. Results across 14 challenging datasets show that AEPO consistently outperforms 7 mainstream RL algorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressive results: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalker for Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% on WebWalker for Pass@5. Further analysis reveals that AEPO improves rollout sampling diversity while maintaining stable policy entropy, facilitating scalable web agent training.

1 Introduction

AEPO addresses entropy-driven rollout and policy-update failures in agentic RL with entropy balancing, and outperforms mainstream RL methods across broad web-agent evaluations.

  • 1 Introduction: High-entropy rollout collapse concentrates branching on specific trajectories, limiting exploration of other potential correct branches.Consecutive high-entropy tool-call steps can deplete branching resources for other trajectories.
  • 1 Introduction: High-entropy token gradient clipping suppresses exploratory gradients during policy updates and can prematurely terminate useful exploration behaviors.The paper identifies this as a second entropy-driven challenge in agentic RL.
  • 1 Introduction: AEPO balances entropy in rollout and policy updates through dynamic sampling control, high-entropy branch penalties, gradient preservation, and entropy-aware advantage estimation.Its rollout mechanism allocates global and branch sampling budgets, while policy optimization targets high-uncertainty tokens.
  • 1 Introduction: AEPO consistently outperforms mainstream RL algorithms across 14 datasets spanning deep information seeking, knowledge-intensive reasoning, and computational reasoning.The evaluation covers generalized reasoning tasks and reports results across multiple benchmark families.
  • 1 Introduction: 47.6% on GAIA, 11.2% on Humanity’s Last Exam, and 43.0% on WebWalkerQA are achieved at Pass@1 with Qwen3-14B using 1K RL samples.At Pass@5, the same setup reaches 65.0% on GAIA, 26.0% on Humanity’s Last Exam, and 70.0% on WebWalkerQA.

2 Preliminary

The preliminary section defines the agentic RL setting and empirically characterizes two entropy-related training failures: rollout branch collapse and clipping of exploratory token gradients.

  • 2.1 Problem Definition: The agentic RL setting includes a tool set, a policy and reference language model, a reward function, dataset inputs, and outputs containing tool-call results.The paper evaluates representative web search, browser, and code-execution tools.
  • 2.2 Entropy-based Pilot Experiments: The preliminary experiments quantify consecutive high-entropy tool usage and the distribution of high-entropy branches within rollout batches.The setup uses ARPO with default 1K training samples and a branching budget of 8.
  • 2.2 Entropy-based Pilot Experiments: 93.4% of rollout branches concentrate on 1–3 trajectories, while consecutive high-entropy turns account for 56.5% versus 43.5% isolated turns.Observed trajectories contain as many as six consecutive high-entropy tool-call turns.
  • 2.2 Entropy-based Pilot Experiments: High-entropy rollout collapse occurs when consecutive high-entropy rounds cause over-branching on a few trajectories and imbalanced rollout branching resources.The resulting allocation leaves other trajectories with virtually no budget for high-entropy branch sampling.
  • 2.2 Entropy-based Pilot Experiments: Significant gradient clipping appears during the first policy update, leaving high-entropy exploratory tokens without sufficient gradient support.The paper links this pattern to fixed paradigmatic reasoning that hinders exploration of tool-use patterns.

3 Methodology

AEPO balances entropy during rollout and policy updates through adaptive exploration, branch control, gradient preservation, and entropy-aware learning.

  • Overview: AEPO balances entropy across rollout and policy-update phases with two coordinated components.The method targets high-entropy rollout collapse and high-entropy token gradient clipping.
  • Dynamic Entropy-Balanced Rollout: AEPO adaptively allocates global and branch sampling budgets by comparing question and tool-call entropy.The global sample count increases when question uncertainty exceeds tool uncertainty and decreases when tool uncertainty is greater.
  • Dynamic Entropy-Balanced Rollout: A consecutive-branch penalty lowers branching probability as high-entropy branching continues, promoting more diverse rollout coverage.Branching begins when the probability exceeds a threshold, while termination enforces the partial sampling budget.
  • Entropy-Balanced Policy Optimization: Entropy-aware advantage estimation assigns greater rewards to exploratory tokens, prioritizing learning on high-uncertainty behavior.This complements gradient preservation for tokens retained by entropy-balanced rollouts.
  • Entropy-Balanced Policy Optimization: AEPO retains gradients outside the clipping interval for high-entropy tokens by inserting stop-gradient into the clipping term.The forward computation remains unchanged while backward propagation is protected from clipping constraints.

4 Experiment Settings

The experiments evaluate AEPO on long-term web-agent reasoning across deep information seeking, knowledge-intensive reasoning, and computational reasoning tasks.

  • Evaluation Tasks: AEPO is assessed across three long-term reasoning categories: deep information seeking, knowledge-intensive reasoning, and computational reasoning.The evaluation includes 14 datasets spanning these task families.
  • Evaluation Tasks: Deep information-seeking evaluation includes GAIA, Humanity’s Last Exam, WebWalkerQA, XBench, and Frames.These benchmarks target challenging web-agent information-seeking behavior.
  • Evaluation Tasks: Knowledge-intensive reasoning covers 2WikiMultihopQA, Musique, Bamboogle, and WebWalkerQA.The listed tasks involve multi-hop open-domain question answering and web multi-hop reasoning.
  • Evaluation Tasks: Computational reasoning includes GSM8K, MATH, MATH500, AIME2024, and AIME2025.The set ranges from simple mathematical reasoning to competition-level challenges.
  • Baselines: Baselines span vanilla, clipping-optimized, and agentic RL algorithms, alongside backbone models and workflow-based web agents.The RL baselines include GRPO, Reinforce++, DAPO, CISPO, GPPO, GIGPO, and ARPO.
  • Evaluation Protocol: Evaluation uses F1 for four knowledge-reasoning question-answering tasks and Pass@1 by default for other tasks.Answers are assessed with LLM-as-Judge for other tasks, using fixed temperature and top-p settings.

5 Experiment Results

Across deep information seeking and ten challenging reasoning tasks, AEPO generally outperforms competing RL methods while improving Pass@K, rollout diversity, tool-call efficiency, and entropy stability.

  • 5.1 Main Result on Deep Information Seeking: 47.6% on GAIA, 11.2% on HLE, and 43% on WebWalkerQA are achieved at Pass@1 using only 1k open-source training samples.The models use no data synthesis or filtering.
  • 5.3 Pass@K Sampling Analysis: 65% on GAIA, 26% on HLE, and 65% on XBench-DR are achieved at larger sampling scales, with AEPO consistently exceeding ARPO across five datasets.The comparison covers Pass@1 through Pass@5 for Qwen3-14B.
  • 5.4 Rollout Diversity Analysis: AEPO produces 62 distinct rollout cluster centers versus ARPO’s 54, with tighter clusters and larger inter-cluster gaps.The analysis uses approximately 7.6k trajectories sampled from 640 problems.
  • 5.4 Rollout Diversity Analysis: With both sampling budgets set to 8, AEPO explores more diverse branch patterns than ARPO’s typical 2–3 trajectories.AEPO can potentially cover all 8 paths through different branches.
  • 5.5 Does AEPO Achieve Entropy-Balanced and Efficient RL Training?: AEPO requires about half as many tool calls as vanilla and clipping-optimized RL methods while achieving superior performance.Training curves also show AEPO maintains more stable entropy than clipping-optimized methods, which often experience entropy instability and performance collapse.

6 Related Work

Related work develops web agents through RL, tool-use optimization, and trajectory or tree-structured rollouts, while prior approaches leave entropy-related challenges insufficiently addressed.

  • Web Agent Reinforcement Learning: Initial web-agent efforts enabled autonomous interaction with search engines or code interpreters through rule-based RL, followed by multi-tool and other agentic extensions.Tool-star is cited as incorporating multi-tool usage within agentic RL.
  • Web Agent Training: Web-agent research combines data synthesis, RL algorithm optimization, report generation, and model-based search simulation to broaden training approaches and reduce resource use.These directions extend beyond initial supervised distillation and trajectory-level RL methods.
  • Agentic RL and Entropy Optimization: Agentic RL methods use tree-structured rollouts for autonomous branch sampling under high entropy but often overlook high-entropy token challenges.Prior single-turn RL studies emphasize stable entropy training as important for model performance.
  • Reinforcement Learning Foundations: RL research spans foundational self-play methods, value-based control, and applications to language, embodied intelligence, and complex games.These foundations provide broader context for agentic RL methods.
  • Web Agent Reinforcement Learning: Trajectory-level RL alone struggles to capture the multi-turn, long-horizon behavior of LLM agents, motivating step-level tool-use learning in ARPO.This line of work addresses tool invocation through reward shaping and rollout mechanisms.

7 Conclusion

AEPO balances entropy during rollout and policy updates through two complementary mechanisms. Across 14 benchmarks, it consistently outperforms seven mainstream agentic RL algorithms while improving scalability and stability.

  • AEPO combines dynamic entropy-balanced rollout with entropy-balanced policy optimization to regulate entropy across both training phases.The rollout mechanism allocates global and branch sampling budgets using entropy pre-monitoring and penalizes consecutive high-entropy steps; policy optimization preserves and rescales high-entropy-token gradients while using entropy-aware advantages.
  • AEPO consistently outperforms seven mainstream agentic RL algorithms across 14 benchmarks.Quantitative analyses also support the algorithm’s scalability and stability for general web-agent training.

A Proof of the Gradient of AEPO

The gradient derivation explains how AEPO’s stop-gradient-based clipping changes policy updates relative to clipping-optimized RL methods. Its asymmetric rule removes unstable updates for low-confidence negative advantages, yielding smoother optimization under high-entropy exploration.

  • The derivation evaluates the gradient separately for upper-clipped, lower-clipped, and unclipped importance-ratio regions.The lower-clipping case produces a vanishing gradient, while the other cases retain their corresponding clipped or unclipped behavior.
  • AEPO sets the gradient factor to zero when the advantage is negative and the importance ratio falls below the lower clipping bound.This filters out low-confidence negative advantages rather than applying a fixed penalty.
  • Compared with CISPO, AEPO reduces variance from symmetric updates and achieves smoother optimization dynamics and more stable convergence under high-entropy exploration.CISPO retains a fixed nonzero gradient factor in the lower-clipped negative-advantage region, whereas AEPO discards that signal.
  • Compared with GPPO, AEPO discards residual gradients in the lower-clipped negative-advantage region instead of retaining bounded beta-scaled corrections.The paper links this stricter filtering to improved credit assignment for positively rewarded high-entropy tokens.

C Baselines

The baseline section introduces the models used for the paper’s experiments before detailing their algorithmic categories.

  • The experiments use a set of baseline models described in the following subsections.

C.1 RL algorithms

The baseline algorithms include classical and clipping-optimized RL methods, alongside AEPO’s rollout and policy-optimization procedures. They differ in how they compare responses, control variance, decouple clipping, and allocate samples or gradients.

  • RL algorithms: GRPO performs group-based policy optimization by comparing sampled responses and assigning relative rewards.It is presented as a method for more stable and sample-efficient policy updates.
  • RL algorithms: Reinforce++ extends policy gradients with variance reduction and adaptive normalization while retaining a REINFORCE-aligned objective.
  • Clipping-optimized RL Method: DAPO decouples clipping from policy updates and adds dynamic sampling to adaptively select training examples.The supplied description presents both changes as mechanisms for maintaining effective optimization.
  • AEPO: AEPO’s algorithm is parameterized by rollout size, entropy sensitivity, branch-penalty slope, clipping bounds, and an entropy-aware weight.Its procedure initializes a reference model and updates parameters using the derived gradient rule.
  • Clipping-optimized RL Method: GPPO decouples clipping between forward and backward passes, using clipped ratios for bounded updates and unclipped ratios to preserve gradient information.

C.2 Web Search Agent

This section presents related web-agent frameworks and methods for retrieval, autonomous web search, reasoning, and tool use.

  • RAG: retrieval-augmented generation combines information retrieval with generative modeling to enhance output accuracy, reliability, and timeliness.
  • Search-o1 integrates agentic retrieval-augmented generation with reasoning in documents to improve performance on complex reasoning scenarios.
  • WebThinker enables language models to autonomously search and explore web pages while generating research reports.
  • ReAct combines reasoning and action with external tools such as search engines and databases for iterative decision-making.

D The Overall Algorithm Workflow of AEPO

AEPO combines dynamic entropy-balanced rollouts with entropy-balanced policy optimization to enhance multi-turn tool-use capabilities in large language models.

  • AEPO integrates dynamic entropy-balanced rollouts and entropy-balanced policy optimization in one workflow for multi-turn tool-use training.
Loading 2510.14545v1…