PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Agentic Reinforcement Learning with On-Policy Intrinsic Knowledge Boundary Enhancement

Dingwei Chen, Zefang Zong, Zhipeng Ma, Leo Luo, Yang Li, Chengming Li, Peng Chen, Jie Jiang

arXiv:2605.26952v1cs.CL

TL;DR

Agentic RL can produce redundant tool calls and blur when tools are necessary. AKBE probes this boundary with dual-path rollouts, improving accuracy by 1.85 points while reducing tool calls by 18% across seven QA benchmarks.

  • Problem

    Agentic RL increasingly produces redundant tool calls and blurs whether external tools are necessary, wasting resources and potentially degrading answer quality.

  • Method

    AKBE uses on-policy rollouts with and without tools to identify instance-level tool requirements and construct boundary-guided supervisory signals for efficient tool use.

  • Results

    +1.85 average EM and 18% fewer tool calls across seven QA benchmarks, yielding approximately 25% higher tool productivity over the base method.

  • Takeaways & Limitations

    Explicit on-policy knowledge-boundary modeling improves task accuracy and tool-call efficiency across benchmarks and backbone models.

  • Takeaways & Limitations

    Additional no-tool rollouts add computational cost early in training, and the fixed λ may not provide the optimal objective balance across training stages and task difficulties.

Abstract

from arXiv · show

Agentic reinforcement learning (RL) has proven effective for training LLM-based agents with external tool-use capabilities. However, we identify that agentic RL training induces increasing redundant tool calls and blurs the model's intrinsic knowledge boundary, where the model fails to distinguish when tools are needed versus when parametric knowledge suffices. Existing solutions based on reward shaping create coarse-grained optimization targets that tend to incentivize indiscriminate tool-call suppression, leading to reward hacking. In this paper, we propose AKBE (Agentic Knowledge Boundary Enhancement), an on-policy method that dynamically probes the model's intrinsic knowledge boundary through dual-path (with-tool and no-tool) rollouts during training. We define the knowledge boundary as the per-instance determination of whether tools are required and the minimum tool calls necessary. By comparing correctness across paths, AKBE categorizes trajectories and constructs targeted supervisory signals that guide efficient tool-use patterns for each question. These signals are integrated seamlessly into the agentic RL training loop. Experiments on seven QA benchmarks demonstrate that AKBE improves task accuracy by +1.85 on average and reduces tool calls by 18% over standard agentic RL, yielding 25% higher tool productivity without any accuracy-efficiency trade-off. Further analysis suggests its plug-and-play compatibility across different RL algorithms and the mechanism of each signal category. Our code is available at https://github.com/CuSO4-Chen/AKBE.

1 Introduction

Agentic RL can increase redundant tool use, wasting resources and introducing misleading information, while reward shaping may encourage indiscriminate tool suppression. AKBE addresses this by probing each instance’s knowledge boundary with dual-path rollouts and achieves higher accuracy, fewer tool calls, and greater productivity.

  • Motivation: Agentic RL increasingly produces redundant tool calls when parametric knowledge suffices or when fewer calls would suffice.This cognitive offloading wastes computational resources, increases inference latency, and can degrade answer quality through misleading retrieval.
  • Limitations of Existing Methods: Reward shaping creates coarse-grained targets that can incentivize overall tool reduction regardless of necessity, causing reward hacking and degraded task accuracy.
  • AKBE: AKBE dynamically probes the model’s intrinsic knowledge boundary through dual-path rollouts and constructs boundary-guided signals to reinforce efficient tool-use patterns.The boundary specifies whether tools are required for each question and the minimum invocations needed for a correct answer.
  • Results: +1.85 average task-accuracy improvement and 18% tool-call reduction over standard agentic RL produce 25% higher tool productivity.These results come from experiments on seven QA benchmarks across two backbone models.
  • Results: AKBE outperforms baseline methods in most cases without an accuracy-efficiency trade-off.
  • Analysis: AKBE is compatible with diverse agentic RL algorithms, while its signal categories adapt to distinct tool-use failure modes as the knowledge boundary evolves during training.

2 Related Work

Prior work trains LLM agents with external tools and develops agentic-RL algorithms, while SMART uses offline metacognitive SFT data. AKBE instead probes the evolving intrinsic knowledge boundary on-policy within RL training and provides plug-and-play supervisory signals.

  • Agentic reinforcement learning: Recent work applies reinforcement learning to train LLM-based agents with external tool-use capabilities and develops algorithms tailored to agentic settings.Examples include entropy-driven rollout and credit assignment.
  • Offline boundary modeling: SMART constructs metacognitive SFT data offline, but static datasets cannot track the evolving knowledge boundary during RL training.
  • AKBE: AKBE dynamically probes the model’s intrinsic knowledge boundary through on-policy dual-path rollouts and constructs boundary-guided supervisory signals within the RL training loop.The two paths are with-tool and no-tool rollouts.
  • AKBE: AKBE integrates these signals with any agentic RL algorithm as a plug-and-play module.

3 Preliminary

The preliminary section formulates agentic RL as iterative reasoning-and-action interaction with tools and situates AKBE within GRPO-based policy optimization. It also specifies the clipped objective’s KL regularization and masking of tool observations during training.

  • Agentic setting: An agent policy πθ iteratively interacts with an external tool environment E, producing reasoning-and-action turns until invoking a tool or finishing an episode.Each action is conditioned on the current context; tool calls return observations appended to that context.
  • GRPO background: GRPO reduces PPO’s memory and training overhead by using group-relative advantages, sampling G trajectories for each question from the current policy.The paper identifies GRPO as predominant in recent agentic RL research.
  • Policy objective: The policy objective maximizes a clipped surrogate with KL regularization against a reference policy, while masking tool-observation tokens during training.The importance-sampling ratio uses πθ and πθold, with β controlling KL regularization strength and ϵ defining the clipping threshold.

4 Method

AKBE probes each question with parallel tool-enabled and no-tool rollouts to identify whether external tools are necessary and select efficient on-policy supervisory trajectories. It combines these boundary-guided signals with agentic RL, dynamically reevaluating the boundary throughout training.

  • Dual-path rollout: AKBE performs parallel with-tool and no-tool rollouts for each training question, testing external-tool dependence against the model’s parametric knowledge.No-tool rollouts avoid tool interaction and environment latency, making boundary probing computationally efficient.
  • Knowledge boundary: The knowledge boundary marks whether a question lies within intrinsic knowledge, with KB = 1 indicating unnecessary tools and KB = 0 indicating required tools.The boundary is determined from whether either rollout path produces a correct answer.
  • Trajectory supervision: AKBE selects category-specific targets: minimum-call correct trajectories for tool-dependent questions, and correct no-tool trajectories when tools are redundant or harmful.These categories reinforce necessary tool use, bypass unnecessary calls, or avoid tool-induced errors.
  • Trajectory supervision: For both-wrong questions, AKBE constructs no reliable supervisory signal and relies solely on the original RL objective.Signals are constructed only for Tool-dependent, Efficiency, and Hallucination categories.
  • Joint objective: The training objective combines the original agentic RL loss with an on-policy cross-entropy objective over selected trajectories, with λ controlling their relative strength.The RL loss can be replaced by classic agentic RL losses such as DAPO or GSPO.
  • On-policy adaptation: Because both objectives use current-policy rollouts, AKBE reevaluates and adapts the knowledge boundary at every training step as the model improves.This distinguishes AKBE from static offline approaches that cannot track boundary evolution.

5 Experiments

Across seven QA benchmarks and two backbone models, AKBE improves accuracy while reducing tool usage and increasing tool productivity. Ablations and compatibility studies show that its on-policy signals, category composition, and tuning support efficient, general tool use without suppressing necessary calls.

  • Main Results: +1.85 average EM and 18% lower TC on Qwen3-4B yield approximately 25% higher tool productivity than the base method across seven benchmarks.The same effect holds on Qwen2.5-7B, supporting generality across model architectures.
  • Main Results: AKBE achieves larger TC reduction than β-GRPO while simultaneously improving EM, avoiding the accuracy collapse associated with coarse-grained reward shaping.β-GRPO avoids EM collapse through a confidence threshold but provides limited TC reduction.
  • Plug-and-Play Compatibility: AKBE consistently improves average EM and reduces TC when combined with GRPO, DAPO, GSPO, and AEPO.With DAPO, TC decreases from 2.61 to 2.38 while average EM improves by +0.36; GSPO and AEPO receive TC reductions of −0.39 and −0.35.
  • Signal Ablation: Removing Tool-dependent signals drops EM below GRPO despite the lowest TC, because Efficiency and Hallucination signals alone over-suppress necessary tool calls.Full AKBE outperforms all subsets, confirming that Tool-dependent, Efficiency, and Hallucination categories are complementary.
  • Coefficient Analysis: λ = 0.05 provides the best balance, while AKBE outperforms GRPO in EM for λ ∈[0.05, 0.2] and degrades sharply beyond λ = 0.2.An overly strong boundary-guided objective dominates the RL loss and leads to over-suppression of tool calls.
  • Training Efficiency: 15% faster average training steps are achieved by AKBE than GRPO despite Gnt = 8 additional no-tool rollouts per batch.The result is attributed to faster no-tool rollouts and reduced tool calls shortening overall training time.

6 Conclusion

AKBE dynamically models the model’s intrinsic knowledge boundary through on-policy dual-path rollouts, using fine-grained supervisory signals to eliminate redundant tool calls while preserving necessary ones. Across seven QA benchmarks and two backbone models, it improves task accuracy and tool-call efficiency without modifying the RL objective.

  • Method: AKBE probes the intrinsic knowledge boundary with on-policy dual-path rollouts during agentic RL training.The method constructs knowledge boundary-guided supervisory signals.
  • Method: AKBE eliminates redundant tool calls while preserving necessary ones through instance-level guidance.This guidance is more fine-grained than reward shaping approaches that suffer from reward hacking.
  • Effectiveness: AKBE simultaneously improves task accuracy and tool-call efficiency without modifying the RL objective.Its signals are integrated without changing the underlying reinforcement-learning objective.
  • Effectiveness: Experiments across seven QA benchmarks and two backbone models validate AKBE’s effectiveness and generality.The results support explicit on-policy knowledge boundary modeling as a promising strategy for efficient agentic reinforcement learning.

7 Limitations … A.2 Prompt Template

AKBE trades early extra rollout computation for later efficiency gains, uses a strict correctness-and-format reward, and probes the knowledge boundary through with-tool and no-tool prompt templates.

  • 7 Limitations: AKBE incurs extra computational cost early in training because no-tool rollouts are added before tool calls decrease.Adaptive sampling is proposed to restrict no-tool rollouts to questions likely within the knowledge boundary.
  • A.1 Reward Design: The training pipeline uses a binary reward combining Exact Match correctness with a structural format requirement.The correctness signal follows Search-R1’s reward formulation and uses Exact Match as the primary evaluation criterion.
  • A.1 Reward Design: Responses receive no credit when required structural tags are missing, regardless of answer correctness.The required structure includes <think>...</think>, <answer>...</answer>, and a boxed final answer, supporting reliable parsing and extraction.
  • A.1 Reward Design: A trajectory earns maximal reward 1 only when it satisfies the format requirement and produces an exactly correct answer; format violations receive r = −1.The overall reward combines correctness and format validity.
  • A.2 Prompt Template: AKBE requires two prompt templates to support its dual-path rollouts.The with-tool and no-tool templates are presented separately in Figures 6 and 7.
  • A.2 Prompt Template: The with-tool template separates reasoning, retrieval queries, environment observations, and final prediction using dedicated tags.It uses <think></think>, <search></search>, <result></result>, and <answer></answer>, with the canonical answer enclosed in \boxed{} for Exact Match extraction.
  • A.2 Prompt Template: The no-tool template removes search and result tags, retaining reasoning and answer components so AKBE can compare correctness across tool-use paths.This forces answers to rely solely on parametric knowledge and enables knowledge-boundary probing.

A.3 Datasets … A.7 Hardware and Artifacts

The experiments span multi-hop and single-hop QA benchmarks, with AKBE evaluated under specified rollout, optimization, baseline-selection, retrieval, and hardware configurations. The setup uses Wikipedia-based search, Qwen backbones, and VeRL infrastructure across a single 8-GPU node.

  • A.3 Datasets: AKBE is evaluated on two categories of widely used question-answering benchmarks: multi-hop and single-hop QA.The multi-hop category tests multi-turn tool use and compositional reasoning, while the single-hop category tests single-step retrieval.
  • A.3 Datasets: Multi-hop evaluation uses HotpotQA, 2WikiMultiHopQA, and MuSiQue for questions requiring compositional or explicit multi-hop reasoning.HotpotQA is Wikipedia-derived with supporting-fact annotations, while 2WikiMultiHopQA combines Wikipedia passages with Wikidata triples.
  • A.3 Datasets: Single-hop evaluation uses Natural Questions, TriviaQA, and PopQA to assess retrieval, robustness to surface variation, and reliance on external retrieval versus memorized facts.PopQA is designed to diagnose whether the policy genuinely uses the search tool or relies on parametric knowledge.
  • A.4 AKBE Settings: AKBE uses a training batch size of 64, mini-batch size of 8, maximum response length of 6192, and maximum tool usage of 6.With-tool and no-tool rollout sizes are 16 and 8, respectively; the AKBE clipping threshold is 0.2 and β = 0.
  • A.5 Baseline Settings: All RL-based baselines share the hyperparameters summarized in Table 4, while method-specific configurations follow their original papers.Baselines are trained without an additional SFT phase.
  • A.5 Baseline Settings: Reported baseline results use the checkpoint achieving the highest average EM across all evaluation benchmarks.This checkpoint-selection rule is stated for the reported results.
  • A.6 Search Tool Environment: The search environment follows Search-R1, using a Wikipedia snapshot, e5-basev2 as the dense retriever, and approximately 21M Wikipedia entries.When the policy emits a retrieval action, the engine returns the top-3 candidate passages as tool observations.
  • A.7 Hardware and Artifacts: Training and evaluation run on a single node with 8× NVIDIA H20 GPUs, using Qwen3-4B and Qwen2.5-7B backbone policies with VeRL infrastructure.The backbones were selected for strong reasoning capabilities and compatibility with agentic post-training.

B Theoretical Analysis for Coefficient λ

The analysis selects λ by balancing gradient contributions from LGRPO and LAKBE. With Gwt = 16, it estimates λ ≈0.0625, matching the empirical optimum λ = 0.05, and supports λ ∈[0.05, 0.10].

  • Balancing Condition: The theoretical objective is to balance gradient contributions between LGRPO and LAKBE.The balancing condition compares per-question contributions from the two objectives.
  • Practical Estimate: 70–80% of questions typically produce at least one correct trajectory in either path, while LGRPO aggregates Gwt trajectories and LAKBE uses one full-weight trajectory.Under binary rewards, σA ≈1 by construction; the trajectory-count ratio is the key scaling factor.
  • Practical Estimate: With Gwt = 16, the estimate is λ ≈0.0625, closely matching the empirical optimum λ = 0.05.The estimate follows from the relative trajectory weighting of LGRPO and LAKBE.
  • Practical Estimate: λ ∈[0.05, 0.10] is a reasonable range under task-difficulty and signal-proportion variability.This range extends the point estimate to account for dynamic factors in the experimental setting.

C Cross-Entropy vs. DPO for Signal Integration · D Reliability of Knowledge Boundary Estimation

AKBE’s cross-entropy integration is more stable than its DPO variant, whose rejected-trajectory supervision can cause collapse. No-tool rollout statistics indicate that AKBE’s knowledge-boundary estimates are generally reliable and become stronger during training.

  • C Cross-Entropy vs. DPO for Signal Integration: Cross-entropy AKBE retains only positive signals, while DPO additionally selects rejected trajectories, including the incorrect or inefficient rollout with the most tool calls.The cross-entropy objective supervises the selected target without explicitly penalizing alternatives; DPO introduces an explicit negative signal.
  • C Cross-Entropy vs. DPO for Signal Integration: The comparison evaluates standard GRPO, AKBE with cross-entropy, and AKBE with DPO on Qwen3-4B Multi-Hop using EM and training-reward dynamics with λ = 0.05.
  • C Cross-Entropy vs. DPO for Signal Integration: DPO initially surpasses GRPO during steps 80–140 but collapses after step 150, whereas AKBE with cross-entropy improves stably and monotonically.Both EM and training reward drop sharply after the DPO variant’s collapse.
  • C Cross-Entropy vs. DPO for Signal Integration: DPO instability arises because preferred and rejected trajectories share reasoning and tool-call patterns, so suppressing rejected samples can penalize useful shared behavior.The variants differ mainly in tool-call strategy or final correctness, making fine-grained discrimination difficult.
  • D Reliability of Knowledge Boundary Estimation: The reliability analysis tests whether classifying a question as NT=1 from at least one correct no-tool rollout among Gnt = 8 is distorted by chance guessing.
  • D Reliability of Knowledge Boundary Estimation: 30.9% early to 36.7% late of NT=1 questions achieve 8/8 correct no-tool rollouts, while most achieve at least 4/8.Figure 9 therefore shows that NT=1 classifications are typically supported by multiple correct rollouts.
  • D Reliability of Knowledge Boundary Estimation: 5.0/8 early and 5.2/8 late are the average correct no-tool rollout counts for NT=1 questions.Exactly 1/8 correct occurs for 18.1% early and 16.7% late; 63.4% early and 67.1% late achieve ≥4/8.
  • D Reliability of Knowledge Boundary Estimation: The estimates are high-confidence rather than noise-driven, improve from early to late training, and use λ = 0.05 to limit the effect of the minority of 1/8-correct cases.AKBE’s on-policy design progressively strengthens boundary-estimation reliability while preventing weak signals from overriding the main RL objective.

E Supplementary Overhead Analysis

The supplementary analysis attributes AKBE’s computational efficiency to reduced tool-call frequency and shorter responses. Compared with GRPO, AKBE avoids escalating tool use and stabilizes response length, producing compounding efficiency effects during training and inference.

  • Tool Call Count: Tool-call trajectories diverge during training: GRPO rises from approximately 320 to over 370 calls per batch by step 300, while AKBE remains relatively stable.Both methods begin at a similar level of ∼320 calls per batch.
  • Response Length: Response lengths diverge: GRPO grows from ∼2,300 to over 2,500 tokens by step 300, whereas AKBE stabilizes around 1,600–1,700 tokens after steps 1–100.AKBE shows a decreasing trend during the early-to-middle training phase.
  • Efficiency Effect: AKBE’s knowledge-boundary-guided signals create compounding efficiency by avoiding unnecessary tool calls early, making later rollouts shorter and accelerating training and inference.The analysis links fewer tool calls to inherently shorter subsequent rollouts.

F Case Study

AKBE’s case studies show that dual-path rollouts select the trajectory with the appropriate tool-use behavior for each query. The resulting supervision distinguishes redundant, necessary, and harmful retrieval, strengthening per-question knowledge-boundary awareness.

  • Case #1: Efficiency: In the efficiency case, the model answered correctly from parametric knowledge, so AKBE selected the no-tool trajectory to reduce redundant calls.This represents a Level-1 knowledge-boundary violation: tool use was unnecessary despite correct internal recall.
  • Case #2: Tool-dependent: In the tool-dependent case, the no-tool path hallucinated “Warren Buffett,” whereas targeted retrieval identified Seth Klarman and supplied the correct answer.AKBE selects the minimum tool-call with-tool trajectory when external retrieval is genuinely necessary.
  • Case #3: Hallucination: In the hallucination case, noisy retrieval conflated Arline Burks Gant with Barbara Stoddard Burks, while the no-tool path recalled the correct dates.AKBE therefore selects no-tool supervision when retrieval is misleading and parametric knowledge is more reliable.
  • Cross-case analysis: Across the three cases, AKBE does not uniformly favor either path; it selects the trajectory reflecting the appropriate tool-use decision for each query.This fine-grained supervision develops nuanced knowledge-boundary awareness across diverse scenarios.
Loading 2605.26952v1…