PaperlayerView exact revision on arXiv

Source-linked AI summary

SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search

Yunbo Tang, Chengyi Yang, Shiyu Liu, Zhishang Xiang, Zerui Chen, Qinggang Zhang, Jinsong Su

arXiv:2605.29796v3cs.AIcs.CLcs.LG

TL;DR

Agentic search agents often over-search because they lack awareness of when their internal knowledge or existing evidence is sufficient. SAAS uses reinforcement learning with dynamic search-boundary modeling, boundary-aware rewards, and stage-wise optimization, reducing over-search across seven benchmarks while maintaining accuracy.

  • Problem

    Existing methods rely on static heuristics or fixed penalties that fail to model evolving search boundaries and can trigger reward hacking.

  • Method

    SAAS contrasts search-disabled and search-enabled rollouts to model evolving search boundaries, then applies boundary-aware rewards and stage-wise optimization.

  • Results

    Across seven benchmarks, SAAS consistently reduces over-search without compromising model accuracy.

  • Takeaways & Limitations

    SAAS supports more efficient search-augmented reasoning by aligning retrieval with questions that require external evidence.

  • Takeaways & Limitations

    The evaluation focuses on unimodal textual retrieval, leaving direct multimodal reasoning for future work.

Abstract

from arXiv · show

Agentic search enables LLMs to solve complex multi-hop questions through iterative reasoning and external search. Despite the effectiveness, these systems often suffer from a critical limitation in practice: agents fail to recognize their own knowledge boundaries, blindly triggering searches when internal knowledge suffices and failing to terminate search even when adequate evidence has been collected. The lack of self-awareness leads to severe \textbf{over-search}, incurring substantial inference latency and prohibitive computational cost. To this end, we propose SAAS, a novel RL framework designed to cultivate dynamic self-awareness that precisely regulates search behavior without compromising accuracy. SAAS introduces three key components: (i) a search boundary modeling mechanism, which identifies the search boundary under the evolving policy by contrasting search-disabled and search-enabled rollouts; (ii) a boundary-aware reward module, which translates this boundary awareness into trajectory-level penalties, suppressing unnecessary and redundant searches; and (iii) a stage-wise optimization strategy, which leverages a sequential curriculum to prioritize reasoning over search regularization, thereby avoiding reward hacking. Extensive experiments demonstrate that SAAS substantially reduces over-search, while maintaining accuracy. Our code and implementation details are released at https://github.com/XMUDeepLIT/SAAS.

1 Introduction

Agentic search combines reasoning with iterative retrieval but can over-search by initiating unnecessary searches or continuing after sufficient evidence, increasing cost, latency, and noise. SAAS addresses this through evolving search-boundary modeling, boundary-aware rewards, and stage-wise optimization to reduce over-search without sacrificing accuracy.

  • Over-search includes unnecessary searches despite sufficient parametric knowledge and redundant searches after sufficient external evidence has been collected.Both behaviors increase computation cost and inference latency, while potentially introducing distracting noisy evidence.
  • Agentic search decomposes complex questions, retrieves external evidence, and integrates it into subsequent reasoning.
  • Existing prompt-based and RL-based methods rely on static heuristics or fixed thresholds that miss evolving capability and remain vulnerable to reward hacking.Outcome-only rewards provide no direct signal about when to initiate or terminate search.
  • SAAS models the evolving search boundary by contrasting search-disabled and search-enabled rollouts, then converts this awareness into trajectory-level penalties.The framework is designed to suppress unnecessary and redundant searches without compromising answer accuracy.
  • SAAS combines boundary awareness with stage-wise policy optimization to prevent reward hacking during reinforcement learning.
  • Extensive experiments on seven benchmarks show that SAAS consistently mitigates over-search without compromising model accuracy.

2 Preliminary Analysis

Preliminary analysis shows that outcome-based RL induces both question-level over-search and redundant searches during reasoning. Naive fixed penalties fail because the search boundary evolves with model capability and can suppress necessary searches, destabilizing optimization, and cause reward hacking.

  • 2.1 Outcome-Based RL: Outcome-based reward optimization produces two forms of over-search: unnecessary searches for answerable questions and redundant searches after sufficient evidence is collected.The analysis attributes both behaviors to rewards that favor answer accuracy without signaling whether search is necessary or still useful.
  • 2.1 Outcome-Based RL: The no-search trajectory ratio steadily falls to nearly zero by step 50, making search the model’s default behavior even when internal knowledge suffices.This reflects question-level over-search and failure to learn when search is actually needed.
  • 2.1 Outcome-Based RL: The redundant search ratio rises throughout training and approaches about 50%, indicating that the model increasingly continues searching after sufficient evidence has been gathered.This is the step-level form of over-search, concerning whether the agent can stop searching during reasoning.
  • 2.2 Naive Search Constraints: The search boundary shifts with capability: questions answerable without search increase from 12.7% at step 100 to 24.3% at step 300.Thus, questions initially requiring external search may later become solvable using parametric knowledge alone, making predefined constraints misaligned with the evolving policy.
  • 2.2 Naive Search Constraints: Fixed search penalties can cause reward hacking: they lower accuracy, trigger training collapse around step 250, and suppress search excessively rather than only removing redundant searches.The resulting performance drop is mainly associated with the sharp decline in search calls, showing that uniform penalties distort the optimization objective.
  • 2.3 Empirical Findings: These findings motivate dynamically modeling the search boundary because fixed constraints cannot reliably balance search efficiency with task performance or indicate when search should stop.Uniform penalties may suppress necessary searches and destabilize optimization.

3 Method

SAAS regulates agentic search through dynamic search-boundary modeling, boundary-aware trajectory rewards, and stage-wise optimization. Together, these mechanisms distinguish necessary from unnecessary search while preserving reasoning and tool-use learning.

  • Search boundary modeling: SAAS contrasts search-disabled and search-enabled rollouts under the evolving policy to identify each question’s current search boundary.This avoids static annotations and reflects that questions may become solvable from parametric knowledge as training improves.
  • Search boundary modeling: Questions are categorized as NOSEARCH, NEEDSEARCH, or UNDETERMINED according to correct trajectories in disabled and enabled rollout groups.NOSEARCH requires nd(q) ≥δ; NEEDSEARCH requires nd(q) = 0 and ne(q) > 0; otherwise the boundary is UNDETERMINED.
  • Boundary-aware reward: Boundary-aware rewards use F1-based answer quality and selectively penalize search: all searches for NOSEARCH, only counts above Nmin for NEEDSEARCH, and none for UNDETERMINED.The search reward activates only for fully correct trajectories, and group-wise advantage normalization stabilizes optimization across rollout groups.
  • Stage-wise optimization: Stage-wise optimization first trains reasoning and tool use with outcome-based rewards, then activates boundary-aware rewards after validation performance stops improving.This sequential curriculum mitigates reward hacking by delaying search regularization until the agent has acquired sufficient tool-use competence.

4 Experiments

Experiments across seven single-hop and multi-hop open-domain QA benchmarks show that SAAS improves accuracy while substantially reducing search cost and over-search. Ablations and sensitivity analysis attribute this balance to stage-wise optimization, on-policy boundary modeling, and a reliable boundary threshold.

  • Main Results: SAAS achieves 45.8% average accuracy on Qwen2.5-3B-Instruct, surpassing HiPRAG by 2.2%, and improves over HiPRAG by 8.0% on Bamboogle.On Qwen2.5-7B-Instruct, SAAS remains competitive with 48.7% accuracy.
  • Over-search Analysis: SAAS attains the lowest average QOR at 45.9% and the lowest SOR on every benchmark, including 6.3% overall and 1.7% on 2WikiMultiHopQA.The Nmin-based reward penalizes searches beyond the sufficient-evidence boundary.
  • Training Dynamics: After boundary-aware reward activation, average search count falls from about 2.0 to below 1.0 while F1 briefly decreases and then remains stable.Unlike the naive penalty setting, this transition does not cause training collapse.
  • Ablation Study: Removing stage-wise optimization lowers accuracy from 45.8% to 40.9%, whereas removing on-policy boundary modeling yields 42.8% accuracy with 1.07 search count.Stage-wise optimization first develops tool-use capability before refining search efficiency, while on-policy modeling tracks the evolving policy.
  • Hyperparameter Analysis: With δ = 2, SAAS reaches 45.8% average ACC and 1.13 average SC; lowering the threshold to δ = 1 reduces SC to 1.05 but drops ACC to 43.1%.The lower search count at δ = 1 reflects over-suppressed search behavior.

5 Conclusion

SAAS is a self-aware reinforcement learning framework that models the evolving search boundary and combines boundary-aware rewards with stage-wise optimization to mitigate over-search. Across 7 open-domain benchmarks, it reduces question-level and step-level over-search while maintaining strong performance.

  • Framework: SAAS models the evolving search boundary of the reinforcement learning policy to regulate search behavior.The framework integrates boundary-aware rewards and stage-wise optimization.
  • Framework: SAAS integrates boundary-aware rewards with stage-wise optimization for over-search mitigation.These components are used to regulate search behavior.
  • Experimental findings: 7 open-domain benchmarks show that SAAS reduces both question-level and step-level over-search while maintaining strong performance.The reported experiments support effectiveness across both over-search levels without sacrificing strong performance.

Limitation

SAAS is evaluated only in unimodal textual retrieval, leaving direct use of multimodal evidence as an open limitation. Textual serialization captures essential semantics, but raw multimodal signals may provide richer context.

  • SAAS’s current evaluation focuses on unimodal textual retrieval within text-based agentic systems.
  • Knowledge-intensive tasks may also involve images, tables, and structured databases as multimodal evidence.
  • Textual serialization can capture the essential semantics of multimodal sources, while raw signals may provide richer contextual information.

Ethics Statement · A Frequently Asked Questions (FAQs)

The study states that it follows the Ethics Policy, uses publicly available data and models without private user information, and reports no known systematic dataset biases affecting its conclusions. SAAS is presented as a research framework for reducing redundant search and improving the deployability of search-augmented agents where latency and retrieval cost matter.

  • Ethics Statement: The study follows the Ethics Policy and uses publicly available data and models containing no private user information.This includes the datasets and models used in the experiments.
  • A Frequently Asked Questions (FAQs): The supplied passages for the merged block address the Ethics Statement and do not provide separate FAQ content.No additional claims about frequently asked questions can be grounded in the provided passages.
  • Ethics Statement: Experiments use publicly available open-domain QA benchmarks and a Wikipedia-based corpus.The paper states that no private user data were collected.
  • Ethics Statement: The datasets are established and widely adopted, with no known systematic biases affecting the validity of the conclusions.The bias assessment concerns the benchmarks and corpus used in the experiments.
  • Ethics Statement: The experiments use standard off-the-shelf pre-trained language models.The paper does not describe custom models in this ethics discussion.
  • Ethics Statement: SAAS regulates search behavior according to the model’s evolving knowledge boundary without relying on or amplifying sensitive attributes.The method focuses solely on regulating search behavior.
  • Ethics Statement: SAAS is intended as a research framework for studying efficient search-augmented reasoning.Its stated use is to support researchers and developers.
  • Ethics Statement: By reducing redundant search, SAAS aims to improve the practical deployability of search-augmented agents.The stated practical considerations are inference latency and retrieval cost.

A.1 Code and Data Availability … B.2 Case Study (Q5)

SAAS releases its code and processed data while introducing boundary-aware, on-policy search regulation that reduces unnecessary and redundant searches without sacrificing accuracy. Additional setup and case studies explain how the method is trained and how it differs from correctness-only optimization.

  • A.1 Code and Data Availability: SAAS releases all code, data, rollout procedures, reward calculations, and stage-wise reinforcement-learning configurations through an anonymous repository.The release is intended to support future research and independent verification.
  • A.2 What are the advantages of SAAS?: SAAS models the agent’s evolving search boundary to improve efficiency while preserving external evidence acquisition when search is genuinely needed.Its boundary-aware rewards reduce search count while maintaining competitive accuracy across single-hop, multi-hop, and different backbone settings.
  • A.3 Why not simply penalize every search action?: Uniformly penalizing every search is too coarse because search may be unnecessary for parametric questions but useful until sufficient evidence is collected for others.SAAS therefore distinguishes question-level unnecessary search from extra calls after adequate evidence has been gathered.
  • A.4 Why is the search boundary modeled on-policy?: SAAS models the search boundary on-policy by contrasting search-disabled and search-enabled rollouts from the current policy.This tracks boundary changes during reinforcement-learning training and supports suppression of both unnecessary initiation and redundant continuation; removing it worsens performance.
  • A.5 How does SAAS differ from prior efficient agentic search methods?: Unlike heuristic triggers, confidence signals, depth constraints, and process rewards, SAAS dynamically aligns search decisions with the policy’s evolving search boundary.The boundary concerns both whether parametric knowledge suffices and whether collected evidence is already sufficient.
  • B.1 SAAS Setup: SAAS estimates search labels and rewards from N = 8 grouped trajectories, split evenly between four search-disabled and four search-enabled rollouts.The two groups estimate parametric solvability and the usefulness of search for difficult questions, respectively.
  • B.1 SAAS Setup: For NOSEARCH questions, SAAS penalizes every search call, whereas for NEEDSEARCH questions it penalizes only calls beyond Nmin, the minimum among correct search-enabled trajectories.Training and evaluation retrieve top k = 3 documents, with at most 5 search calls per trajectory.
  • B.2 Case Study (Q5): Case studies compare SAAS with correctness-only GRPO on unnecessary search and redundant search after search has already been triggered.The first case concerns questions answerable from parametric knowledge, while the second examines continued searching after evidence is sufficient.

B.3 Efficiency Analysis (Q6)

SAAS improves search efficiency relative to outcome-based GRPO by substantially reducing average search counts while largely preserving answer accuracy. GRPO’s outcome-only reward encourages repeated searching because it lacks an explicit signal for distinguishing necessary information acquisition from redundant tool use.

  • Accuracy-efficiency trade-off: SAAS reduces average search count while preserving competitive accuracy across model backbones.On Qwen2.5-3B-Instruct, average RC falls from 2.26 to 1.13, while average ACC changes from 46.5% to 45.8%.
  • Accuracy-efficiency trade-off: 50.0% reduction in search count occurs on Qwen2.5-3B-Instruct, with average ACC decreasing only slightly from 46.5% to 45.8%.This result corresponds to average RC decreasing from 2.26 to 1.13.
  • Accuracy-efficiency trade-off: 67.0% reduction in search count occurs on Qwen2.5-7B-Instruct, while average ACC remains 48.7% versus 50.2% for GRPO.SAAS lowers average RC from 2.94 to 0.97 on this backbone.
  • Outcome-based GRPO burden: GRPO incurs high average search counts of 2.26, 2.94, and 2.21 on Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-4B-Instruct, respectively.The baseline is trained solely with an outcome-based F1 reward.
  • Outcome-based GRPO burden: Outcome-only reward treats search as uniformly beneficial, so GRPO repeatedly searches after sufficient information has been obtained.GRPO lacks an explicit signal distinguishing necessary information acquisition from redundant tool use, whereas SAAS introduces search-necessity aware supervision.

B.4 Parameter Sensitivity Analysis

The sensitivity analysis finds that δ = 2 provides the most effective search-necessity estimates, achieving the highest accuracy while keeping search counts low. Both coarser and finer grouping reduce estimation quality without meaningful efficiency gains.

  • Grouping hyperparameter δ: δ = 2 achieves the highest average ACC of 45.8% while maintaining a low average RC of 1.13.The default setting is reported as most effective for downstream policy learning.
  • Grouping hyperparameter δ: δ = 1 keeps average RC comparable at 1.05 but lowers average ACC to 43.1%, weakening the reward signal for distinguishing necessary from unnecessary searches.Coarse grouping can suppress searches, but its necessity estimates are less accurate.
  • Grouping hyperparameter δ: δ = 3 and δ = 4 reduce average ACC to 43.7% and 43.0%, respectively, while average RC remains close to the default.Finer grouping may produce noisier and less stable evidence-demand estimates, leading to less consistent search decisions during training.

B.5 Over-search Behavior Compared with Outcome-Based GRPO · C Implementation Details

SAAS reduces over-search relative to outcome-based GRPO at both the question level, by avoiding unnecessary search initiation, and the step level, by limiting redundant searches after retrieval begins. These gains hold across Qwen2.5-3B, Qwen2.5-7B, and Qwen3-4B backbones while preserving the reported correct-answer behavior.

  • B.5 Over-search Behavior Compared with Outcome-Based GRPO: SOR measures redundant searches after retrieval begins, whereas QOR measures unnecessary searches for questions answerable from parametric knowledge.The two metrics capture complementary levels of search overhead.
  • B.5 Over-search Behavior Compared with Outcome-Based GRPO: On Qwen2.5-3B-Instruct, SAAS lowers average QOR from 100.0% with GRPO to 60.6%, a 39.4% reduction in question-level over-search.GRPO triggers search for all questions answerable from the model’s parametric knowledge.
  • B.5 Over-search Behavior Compared with Outcome-Based GRPO: On Qwen2.5-7B-Instruct, SAAS reduces average QOR from 100.0% with GRPO to 45.9%.The passage describes this as a more pronounced improvement than on Qwen2.5-3B-Instruct.
  • B.5 Over-search Behavior Compared with Outcome-Based GRPO: On Qwen3-4B-Instruct, GRPO has QOR 100.0 across all datasets, while SAAS lowers average QOR to 47.5%.This result further shows reduced question-level over-search across model backbones.
  • B.5 Over-search Behavior Compared with Outcome-Based GRPO: A case study shows both methods answering correctly, but SAAS uses one search call versus four for GRPO.The example asks for an industrial city in Germany on the Rhine-Herne canal; the gold answer is Duisburg.
  • B.5 Over-search Behavior Compared with Outcome-Based GRPO: SAAS consistently lowers average SOR across all three backbones, including reductions from 15.6% to 10.5%, 15.4% to 6.3%, and 13.3% to 6.1%.The Qwen2.5-7B-Instruct improvements are particularly large on TriviaQA, PopQA, HotpotQA, and MuSiQue.
  • B.5 Over-search Behavior Compared with Outcome-Based GRPO: Together, QOR and SOR reductions show that SAAS avoids unnecessary search initiation and redundant searches after search begins.The behavior reflects a policy that encourages search when it is expected to provide meaningful additional information.

C.1 Benchmark Dataset … F The Use of Large Language Models

The paper evaluates SAAS across diverse open-domain QA benchmarks, baselines, and metrics, situates it within agentic-search research, and documents its evaluation prompts and limited use of LLMs. The supplementary sections emphasize search reliance, baseline behavior, over-search measurement, unresolved knowledge-boundary control, reproducibility, and author-reviewed language assistance.

  • C.1 Benchmark Dataset: Seven open-domain QA benchmarks span single-hop and multi-hop questions with differing reliance on search, using normalized answers and external evidence documents for checking.The single-hop setting includes TriviaQA, PopQA, and Natural Questions; multi-hop datasets include HotpotQA and 2WikiMultiHopQA.
  • D Related Work: Together, the supplementary material defines the benchmark, comparison framework, metrics, related-work context, prompt protocol, and disclosure of author-reviewed LLM assistance.These components support interpretation and reproducibility of the evaluation.
  • C.2 Baseline Details: The comparison includes Direct Inference, rejection-sampling fine-tuning, Search-R1, StepSearch, and HiPRAG, covering parametric answering, successful trajectories, outcome-based RL, and search planning.Rejection Sampling Fine Tuning retains correct search-enabled and search-disabled trajectories, while Search-R1 primarily rewards final-answer correctness and StepSearch optimizes search scheduling.
  • C.3 Evaluation Metrics: Evaluation measures answer quality with Accuracy (ACC) and search behavior with Search Count (SC), Question-level Over-search Ratio (QOR), and Step-level Oversearch Ratio (SOR).SC counts average search calls, QOR captures unnecessary question-level searches, and SOR captures redundant search steps.
  • D.1 Agentic Search: Agentic search dynamically interleaves reasoning with external search, extending beyond static retrieve-then-generate RAG through systems such as ReAct, Chain-of-Retrieval, Search-o1, DeepResearcher, and WebThinker.The literature also includes reinforcement-learning approaches for search-augmented reasoning and tool use.
  • D.2 Efficient Agentic Search: Efficient agentic-search methods address planning, uncertainty, adaptive exploration, and process supervision, but often lack a unified mechanism to trigger and halt search according to knowledge boundaries.This limitation motivates SAAS’s focus on regulating both unnecessary and redundant search behavior.
  • E Prompt Set: Table 9 provides prompt sets for answer evaluation, SOR semantic-consistency evaluation, and search-enabled and search-disabled question answering.These prompts standardize evaluation and orchestrate the reasoning–search process used in the experiments.
  • F The Use of Large Language Models: The paper reports limited LLM use for writing polish, grammar, clarity, readability, formatting, and table presentation without altering scientific content.Authors reviewed and revised model-assisted suggestions and retain responsibility for the final work.
Loading 2605.29796v3…