Source-linked AI summary
BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search
Shiyu Liu, Yongjing Yin, Jianhao Yan, Yunbo Tang, Qinggang Zhang, Bei Li, Xin Chen, Jingang Wang, Xunliang Cai, Jinsong Su
TL;DR
RL-based agentic search improves accuracy but often fails to recognize reasoning boundaries or admit IDK when evidence is insufficient. BAPO adds boundary-aware rewards and adaptive modulation to train more reliable agents, achieving superior reliability without compromising accuracy. Its evaluation remains focused on knowledge-intensive tasks, models up to 14B parameters, and local RAG.
Problem
RL-based agentic search models improve accuracy but rarely admit IDK when evidence is insufficient or reasoning reaches its limit, undermining reliable answers.
Method
BAPO combines a group-based boundary-aware reward for appropriate IDK responses with an adaptive reward modulator that prevents IDK from becoming a shortcut.
Results
Across four benchmarks, BAPO achieves superior overall reliability while maintaining accuracy and deep exploration capability.
Takeaways & Limitations
BAPO enables agentic search models to maintain awareness of reasoning boundaries without compromising their capability for deep exploration.
Takeaways & Limitations
Evaluation focuses on knowledge-intensive tasks, models up to 14B parameters, and a local RAG setup rather than dynamic web search.
Abstract
from arXiv · showhide
RL-based agentic search enables LLMs to solve complex questions via dynamic planning and external search. While this approach significantly enhances accuracy with agent policies optimized via large-scale reinforcement learning, we identify a critical gap in reliability: these agents fail to recognize their reasoning boundaries and rarely admit ``I DON'T KNOW'' (IDK) even when evidence is insufficient or reasoning reaches its limit. The lack of reliability often leads to plausible but unreliable answers, introducing significant risks in many real-world scenarios. To this end, we propose Boundary-Aware Policy Optimization (BAPO), a novel RL framework designed to cultivate reliable boundary awareness without compromising accuracy. BAPO introduces two key components: (i) a group-based boundary-aware reward that encourages an IDK response only when the reasoning reaches its limit, and (ii) an adaptive reward modulator that strategically suspends this reward during early exploration, preventing the model from exploiting IDK as a shortcut. Extensive experiments on four benchmarks demonstrate that BAPO substantially enhances the overall reliability of agentic search.
1 Introduction
RL-based agentic search improves accuracy on complex questions but often fails to recognize when evidence or reasoning is insufficient. BAPO addresses this reliability gap by rewarding appropriate IDK responses while preserving exploration and accuracy.
- RL-based agentic search improves accuracy on complex, knowledge-intensive questions but almost never admits “I DON’T KNOW.”
- When evidence is insufficient or reasoning reaches its limit, explicit IDK responses can help users seek alternative information and prevent misinformation.
- RL optimization can improve accuracy while significantly degrading boundary awareness, leading models to fabricate plausible answers.
- Boundary awareness is difficult to measure because it dynamically emerges from planning, retrieval quality, and iterative reasoning.
- Across four challenging benchmarks, BAPO improves overall reliability while preserving accuracy and the ability to solve complex problems.
- BAPO uses a group-based boundary-aware reward and adaptive reward modulator to encourage appropriate IDK responses without reward hacking.
2 Background
Agentic search systems use reinforcement learning and ReAct-style interaction to plan searches, reason over returned results, and produce answers. Existing reward design primarily combines format correctness with final-answer correctness.
- RAG supplements LLM parametric knowledge with external search to improve response quality in knowledge-intensive scenarios.
- Learning-based alternatives such as SFT and DPO remain constrained by complex data pipelines and suboptimal generalization.
- RL-based agentic search systems use format and outcome correctness rewards to support autonomous query decomposition and flexible search interaction.
- Agentic Search is formalized as a ReAct sequential decision process in which a trajectory interleaves reasoning, search actions, and observations before producing an answer.
- GRPO trains agentic search by sampling groups of trajectories and optimizing policy performance from normalized within-group rewards.
- Outcome correctness is measured with character-level F1 between the extracted prediction and ground-truth answer, alongside format correctness.
3 Preliminary Study
The preliminary study examines how correctness-based RL affects boundary awareness and finds a trade-off between accuracy and appropriate IDK responses. Directly rewarding IDK can instead cause reward hacking and impede accuracy gains.
- RL with Correctness Reward: Correctness-based RL improves accuracy while significantly degrading the model’s awareness of when to acknowledge IDK.The study evaluates accuracy, precision, and IDK rate before and after RL.
- Setup: The study evaluates HotpotQA, MuSiQue, 2WikiMultiHopQA, and Bamboogle, using complete test sets for reported Section 5 results.An 80-example validation set contains 20 examples from each benchmark.
- Setup: The evaluation compares a base Qwen2.5-7B-Instruct model with ReSearch-Qwen-7B-Instruct, which is optimized using GRPO and correctness-based reward.The setup also elicits explicit IDK admissions through a reliable prompt.
- RL with Correctness Reward: 51.25 accuracy, 3.65% IDK rate, and 53.24 precision are reported after RL, compared with 41.25 accuracy, 18.75% IDK rate, and 50.76 precision before RL.The post-RL model’s IDK rate drops sharply while precision increases only marginally.
- RL with IDK Reward: Adding a 0.5 reward for IDK when no correct rollout exists prioritizes IDK over incorrect answers, but directly applying it causes reward hacking and impedes accuracy gains.The model learns to default to IDK rather than attempting difficult problems.
4 The Framework of BAPO
BAPO is a GRPO-based RL framework that models a question as beyond the boundary when no correct answer appears among multiple rollouts. It combines correctness and boundary-aware rewards with adaptive modulation to balance exploration and boundary awareness.
- Framework Overview: BAPO is an RL algorithm built upon GRPO that encourages IDK responses when a problem lies beyond the model’s reasoning boundary.The framework is designed to mitigate the reward hacking observed when IDK is rewarded without modulation.
- Boundary-Aware Reward: A question is treated as exceeding the boundary when the model fails to obtain any correct answer among multiple rollouts.The boundary-aware reward then adjusts the reward assigned to IDK responses.
- Boundary-Aware Reward: The final reward combines correctness reward RCorrect with boundary-aware reward RIDK.RCorrect reflects format and outcome correctness, while RIDK handles boundary-aware IDK behavior.
- Adaptive Reward Modulation: The adaptive reward modulator disables RIDK during early exploration until the IDK proportion falls below α, then applies it fully at the plateau stage.Difficult queries can be resampled up to k times when groups contain no correct rollouts.
- Adaptive Reward Modulation: During the plateau stage, RIDK is deactivated for high-diversity rollouts to preserve exploration and retained for low-diversity samples.Rollout diversity serves as a proxy for whether the model is still exploring the solution space.
5 Experiments
Experiments show that BAPO improves reliability across benchmarks and model scales while largely preserving accuracy. Its adaptive reward dynamics support problem solving early, then increase appropriate IDK behavior and produce strong rejection success rates.
- Main Results: BAPO improves average reliability by 15.8 scores across all benchmarks versus existing agentic-search and implemented baselines.It uses only 5k training samples while achieving competitive accuracy and higher reliability than Search-R1 and ReSearch, trained on 90k and 19k samples.
- Main Results: Compared with GRPO, BAPO raises average reliability by 9.7% and precision by 11.8%, with only a 2.2% average accuracy decrease.
- Main Results: Reliable RFT gains precision but loses 27 accuracy points versus the TIR Prompt baseline, whereas BAPO combines boundary awareness with problem-solving capability.
- Generalization Across Model Scales: Across 3B and 14B Qwen2.5-Instruct models, BAPO improves reliability over Reliable TIR Prompt by 76.1% and 27.4%, and over GRPO by 13.9% and 11.9%.
- Reward Dynamics of Two Stage: During training, correctness reward rises early while IDK falls from 20% to 5%, then boundary-aware reward increases and IDK returns above 25% during the plateau stage.The boundary-aware reward stabilizes between 0.25 and 0.3, indicating a staged shift from acquiring problem-solving skills to boundary awareness.
- Awareness of When to Refuse: BAPO-trained models achieve average rejection success rates of 74.7% at 3B and 76.7% at both 7B and 14B.The metric evaluates whether problems refused by BAPO are also errors for the GRPO-trained model.
6 Conclusion
The paper identifies correctness-only RL as a reliability challenge and proposes BAPO to encourage appropriate IDK responses while preserving exploration. Experiments show improved reliability and reasoning-boundary awareness without compromising deep exploration.
- BAPO addresses reliability failures from correctness-only RL by combining boundary-aware rewards with an adaptive reward modulator.
- Experiments show that BAPO maintains reasoning-boundary awareness while improving reliability without compromising deep exploration.
Limitations
The study’s evaluation is limited in task type, model scale, and search setting. It focuses on knowledge-intensive tasks, models up to 14B parameters, and a local RAG setup.
- The evaluation primarily covers knowledge-intensive tasks, leaving generalizability to other reasoning problems unexplored.
- Experiments scale only to models with 14B parameters because of computational constraints.
- The study uses local RAG rather than commercial web search because of commercial search API costs.The local setup improves reproducibility but does not fully reproduce web search noise and dynamics.
A.5 Benchmark Details
Evaluation uses four multi-hop question-answering benchmarks, while hyperparameter sensitivity is summarized separately for Qwen2.5-3B-Instruct across the same benchmark suite.
- Benchmark setup: The evaluation covers HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.
- Benchmark setup: The test set contains 200 samples each from HotpotQA, 2WikiMultiHopQA, and MuSiQue, plus 125 from Bamboogle.
- Hyperparameter analysis: Table 6 reports sensitivity to BAPO’s IDK ratio threshold α and resampling times k on Qwen2.5-3B-Instruct.Metrics are averaged across the four benchmarks.
B.1 Hyper-Parameters Sensitivity Analysis
The sensitivity analysis examines BAPO’s IDK threshold and resampling strategy, linking larger rollout groups to boundary estimation and reliability across model scales.
- Hyperparameter settings: BAPO’s sensitivity analysis varies IDK ratio α and resampling times k while keeping other hyperparameters fixed.Experiments use Qwen2.5-3B-Instruct and report results in Table 6.
- Resampling strategy: Increasing rollout size is assumed to produce a more accurate estimate of the model’s reasoning boundary.
- Boundary estimation: Pass@K accuracy improves with K and stabilizes after K = 32.This supports using the plateau as evidence that additional sampling is no longer finding a correct solution.
- Boundary estimation: Dynamic resampling adaptively expands group size to refine boundary estimation when failures reflect problems beyond the reasoning boundary.
- Reliability results: BAPO achieves the highest reliability under Exact Match across the 3B, 7B, and 14B model scales.These results complement the main experiments evaluated with LLM-as-a-Judge Equal.
- Case study: In a Qwen2.5-14B-Instruct case study, BAPO responds IDK when retrieval finds no relevant information, whereas GRPO hallucinates a director identity.
B.5 Detailed Analysis of Uncertainty Estimation Methods
The analysis compares internal confidence, verbalized certainty, and self-reflection as uncertainty signals for agentic search. These methods can improve precision but substantially reduce accuracy because answer validity depends on retrieved content.
- Methods: The study evaluates internal confidence, expression certainty, and self-reflection for uncertainty estimation in agentic search.
- Methods: Internal confidence uses average log-probability as a proxy for the model’s confidence.
- Methods: Expression certainty measures the ratio of confident to uncertain expressions using a manually curated and LLM-verified phrase lexicon.
- Methods: Self-reflection prompts the model to retrospectively audit its reasoning trajectory for potential logical pitfalls.
- IDK classification: Internal confidence and expression certainty classify the bottom x ∈ [0, 1] of dataset confidence scores as IDK, while self-reflection uses the model’s uncertainty judgment.
- Findings: These uncertainty methods improve precision but substantially reduce accuracy, producing negligible or negative reliability gains.The analysis attributes the challenge to dependence on retrieved-content quality, which decouples answer reachability from internal or verbalized uncertainty.