PaperlayerView exact revision on arXiv

Source-linked AI summary

GRASP: GRanularity-Aware Search Policy for Agentic RAG

Varun Gandhi, Jaewook Lee, Shantanu Todmal, Franck Dernoncourt, Ryan Rossi, Zichao Wang, Andrew Lan

arXiv:2607.10463v1cs.AIcs.IR

TL;DR

Agentic RAG must adapt retrieval strategy and context granularity during multi-step reasoning, because noisy or coarse context can dilute evidence. GRASP trains an RL policy to coordinate complementary search and paragraph reading, improving retrieval recall and question-answering performance across multi-hop benchmarks.

  • Problem

    Agentic RAG must coordinate lexical and semantic retrieval while selecting context granularity, because noisy or coarse evidence can dilute information needed for multi-hop reasoning.

  • Method

    GRASP trains an RL agent to choose semantic search, keyword search, and paragraph reading during multi-step reasoning using rewards for accuracy, grounded reading, complementarity, and efficiency.

  • Results

    GRASP improves retrieval recall and question-answering performance over single-step retrieval, prompting-based agentic RAG, and RL-based agentic RAG on three multi-hop benchmarks.

  • Takeaways & Limitations

    The learned policy develops skimming and scanning behavior while controlling tool use and context expansion during agentic reasoning.

  • Takeaways & Limitations

    GRASP’s reward design depends on gold supporting-fact annotations, limiting application to datasets that provide such annotations.

Abstract

from arXiv · show

Agentic retrieval-augmented generation (RAG) extends static RAG by allowing language models to iteratively reason, generate search queries, retrieve evidence, and predict answers. However, it remains challenging for models to decide when to retrieve, whether to use lexical matching or semantic similarity, and how to control context granularity to prevent irrelevant tokens from interfering with agent reasoning. In this paper, we introduce GRASP, a reinforcement learning (RL) framework for training agents to adaptively coordinate complementary retrieval tools during multi-step reasoning. GRASP provides the agent with semantic search, keyword search, and paragraph-reading actions, enabling it to retrieve sentence-level evidence and expand further context only when needed. We train the policy with a reward that jointly accounts for answer accuracy, grounded reading, complementary search, and turn efficiency. Experiments on multi-hop reasoning benchmarks show that GRASP improves both retrieval recall and downstream question answering performance compared with single-step retrieval, prompting-based agentic RAG, and RL-based retrieval baselines. Qualitative and ablation analyses show that the learned policy develops interpretable skimming and scanning behavior: it uses semantic search for broad exploration, paragraph reading for local verification, and keyword search for entity-specific evidence. These results suggest that learning to coordinate retrieval signals and context granularity is critical for agent's correct reasoning.

1 Introduction

Agentic RAG extends static retrieval into iterative reasoning but must adapt retrieval strategy and context granularity to avoid noisy evidence and cascading errors. GRASP addresses this challenge by training agents with reinforcement learning to coordinate lexical search, semantic search, and context expansion using a reward that balances accuracy, grounded reading, complementary search, and efficiency.

  • RAG supplements LLMs’ limited and potentially outdated parametric knowledge with external knowledge at inference time.
  • Agentic RAG requires models to iteratively reason, generate queries, retrieve evidence, and decide how retrieval should occur at each step.
  • Coarse-grained, noisy, or distractor-heavy retrieved context can cause hallucinations, while incorrect retrieval decisions or intermediate conclusions can derail later reasoning.
  • GRASP formulates adaptive retrieval tool selection as reinforcement learning, allowing agents to learn when to invoke lexical search, semantic search, and context expansion.
  • Its reward jointly accounts for answer accuracy, grounded reading, complementary search, and turn efficiency while controlling tool usage and context granularity.

2 Problem Formulation

The framework models agentic RAG as dynamic retrieval, management, and composition of evidence across multiple retrieval signals and context granularities. It decomposes a corpus of gold and distractor paragraphs into sentences, then seeks answers from sufficient gold evidence while avoiding distractors.

  • Corpus and evidence: The retrieval corpus D = D_g ∪ D_d contains gold paragraphs with supporting evidence and distractor paragraphs.Gold paragraphs contain ground-truth evidence needed to answer query q, whereas distractors do not constitute the intended evidence source.
  • Corpus and evidence: Each paragraph d ∈ D is decomposed into sentences, forming a sentence-level corpus S.A mapping f: S → D records the paragraph associated with each sentence.
  • Retrieval representation: Each sentence exposes complementary lexical and semantic representations for retrieval.These representations support retrieval through multiple signals while preserving each sentence’s originating paragraph via f.
  • Answer objective: Given q, the framework generates an answer ŷ by selecting and composing sufficient evidence from D_g while avoiding reliance on D_d.The objective combines evidence selection and composition with avoidance of distractor evidence.

3 Methodology

GRASP models agentic RAG as a finite-horizon policy that alternates reasoning with retrieval actions, using adaptive evidence acquisition, multi-component rewards, and GRPO training. Its design combines semantic and keyword sentence search with paragraph expansion while tracking interaction history and turn budget.

  • Agent formulation: The agent state includes the query, interaction history, retrieved evidence, invoked tools, and remaining turn budget before each decision.The policy generates a reasoning segment, selects a tool action, receives an observation, and updates the interaction history for subsequent generation.
  • Action design: GRASP provides semantic search, keyword search, and paragraph reading actions for sentence-level retrieval, lexical or semantic matching, and broader-context expansion.Semantic search uses dense representations, keyword search favors exact or near-exact overlap, and paragraph reading returns the parent paragraph of a retrieved sentence.
  • Reward design: The trajectory reward combines answer accuracy, grounded reading, complementary search, and turn efficiency to optimize both final answers and intermediate evidence seeking.Answer accuracy is token-level F1, grounded reading rewards expansion of gold-document evidence, complementary search requires both search tools to retrieve gold evidence, and efficiency depends on turns used.
  • Reward design: The total reward has maximum value 2.0, with auxiliary weights α = 0.7, β = 0.15, and γ = 0.15.Most auxiliary weight is assigned to grounded reading, while smaller weights balance search quality and efficiency without encouraging unnecessary retrieval or premature guessing.
  • Learning algorithm: GRPO trains the retrieval policy by comparing groups of multi-turn trajectories for the same query rather than relying on an absolute reward scale.Each trajectory records states, retrieval decisions, observations, predicted answers, retrieved or read documents, and turns used.
  • Implementation: Training uses a sentence-indexed corpus with parent-paragraph metadata and masks retrieved-document tokens from the policy loss.This setup retains paragraph and document-title metadata for evidence tracking while training only on model-generated reasoning, tool calls, and final answers.

4 Experimental Setup

The experiments evaluate GRASP on three multi-hop question-answering benchmarks using controlled validation-based test sets and retrieval corpora. Comparisons span lexical, semantic, hybrid, prompting-based, RL-based, and base-model settings, with retrieval and QA quality assessed separately.

  • Benchmarks: Experiments use HotpotQA, 2WikiMultiHopQA, and MuSiQue, treating each validation split as a held-out test set because public gold-labeled test splits are unavailable.The main model is trained on HotpotQA and evaluated on all three datasets.
  • Data Construction: Each dataset contributes 500 validation questions sampled uniformly with seed 42, while retrieval uses a corpus of unique supporting and distractor paragraphs.The corpus is built from paragraphs appearing across the validation questions.
  • Retrieval Setup: BM25 provides lexical retrieval, Qwen3-0.6B provides semantic retrieval, and every method retrieves at top-k = 5.The hybrid single-step baseline retrieves 50 candidates from each retriever, deduplicates and reranks them with Qwen3-Reranker-0.6B before keeping five.
  • Baselines: Baselines include lexical, semantic, and hybrid single-step retrieval; IRCoT and Search-R1 agentic RAG; and a Base setting using GRASP’s prompt and tools without RL training.IRCoT uses lexical paragraph retrieval, Search-R1 uses semantic paragraph retrieval, and the Base setting uses Qwen 2.5 3B Instruct.
  • Evaluation Metrics: Retrieval quality is evaluated before generation, while QA quality uses Exact Match (EM), token-level F1 (F1), and LLM-as-a-judge (JD).EM and F1 measure lexical overlap, whereas JD with gpt5-mini captures semantically flexible correctness.

5 Results

GRASP improves multi-hop retrieval and QA by learning to coordinate complementary search tools and context granularity. Its learned policy exhibits broad semantic exploration, local paragraph verification, targeted keyword refinement, and adaptive recovery when evidence is insufficient.

  • Quantitative results: Complementary retrieval signals improve recall, while lexical retrieval alone performs substantially worse than semantic or hybrid retrieval across multi-hop benchmarks.Multi-step retrieval further benefits multi-hop reasoning by allowing the agent to gather evidence across steps.
  • Quantitative results: GRASP performs best overall across multi-step QA datasets and metrics, whereas single-step hybrid retrieval remains insufficient for combining evidence across supporting passages.IRCoT often achieves stronger JD scores, but Search-R1 generally scores higher on EM and F1; the comparison highlights the value of policy learning.
  • Qualitative analysis: The learned policy typically starts with semantic search, reads a paragraph to verify context and extract bridge entities, then uses keyword search for targeted lexical retrieval.This sequence corresponds to broad exploration, local verification, and entity-specific refinement.
  • Qualitative analysis: When keyword search is insufficient, the agent broadens semantic search; when it finds a promising candidate, paragraph reading verifies the fact in context.This adaptive branching helps avoid relying solely on snippets and supports continued exploration when queries are too narrow.
  • Ablation studies: Removing either semantic or keyword search reduces EM and F1, with semantic-search removal causing the larger degradation; full-paragraph retrieval also produces less targeted follow-up queries.Paragraph-level results expose many entities and relations simultaneously, weakening the signal for selecting the next missing hop.

6 Conclusion and Future Work

GRASP trains agentic RAG agents to adaptively coordinate semantic search, keyword search, and paragraph reading during multi-step reasoning. Future work should expand tool spaces, develop weaker grounding and complementarity rewards, and enable dynamic retrieval granularity.

  • Conclusion: GRASP is an RL framework for coordinating semantic search, keyword search, and paragraph reading during multi-step reasoning.Its rewards jointly address answer accuracy, grounded reading, complementary search, and turn efficiency.
  • Future Work: Richer tool spaces could support both evidence acquisition and reasoning across different reasoning stages.Future studies should examine which tools are useful at each stage.
  • Future Work: Reward designs should reduce reliance on gold supporting-paragraph annotations through weaker or model-based signals for evidence grounding and tool complementarity.This direction targets supervision for grounding and tool coordination.
  • Future Work: Future systems should investigate dynamic retrieval granularity so agents can decide how much context to retrieve.The passage identifies retrieval granularity as a direction for future investigation.

Limitations

GRASP’s reward design depends on gold supporting-fact annotations, limiting its applicability to datasets that provide them. Future work should investigate weakly supervised or self-supervised surrogate evidence signals.

  • Limitations: GRASP requires gold supporting-fact annotations because both grounded reading and complementary search rewards depend on them.This limits the framework to datasets providing supporting-fact annotations.
  • Limitations: Future work should explore weakly supervised or self-supervised alternatives, including answer-conditioned attribution or self-consistency as surrogate evidence signals.

Ethical Considerations

Agentic RAG systems introduce ethical risks involving privacy, fairness, factuality, accountability, security, and human oversight because they iteratively retrieve information and may invoke external tools. Mitigation requires controls for access, data, provenance, citations, auditing, and evaluation beyond task performance.

  • Risks: Agentic RAG may expose sensitive retrieved information, amplify corpus biases, generate misleading outputs from unreliable results, or exceed user intent.These risks extend beyond those of static systems because agents reason about their actions, iteratively retrieve information, and potentially invoke external tools.
  • Mitigations: Mitigations should include strict access controls, data minimization, provenance tracking, citation mechanisms, and audit logs for retrieval and tool-use decisions.
  • Evaluation and Oversight: Evaluation should cover factual grounding, robustness to malicious or irrelevant retrieved content, fairness across user groups and topics, and privacy leakage, not only task performance.The passage also states that consequential actions or outputs in high-impact domains should remain subject to human oversight, although the supplied text ends before completing that statement.

AI Policy · Appendix · A Related Work

The paper reports AI-assisted research workflows with human review, while positioning GRASP as the first work to jointly address retrieval-signal selection, context granularity, and RL-based retrieval-policy learning in agentic RAG.

  • AI Policy: AI tools supported research design, hypothesis formulation, implementation, literature synthesis, and writing-related tasks.The listed tools included Claude Code, Antigravity, Gemini, ChatGPT, GitHub Copilot, NotebookLM, and SciSummary.
  • AI Policy: All AI-generated outputs were reviewed and edited before inclusion.
  • A Related Work: The related-work discussion organizes prior research around retrieval-signal selection, context granularity, and learning retrieval policies with RL.
  • A Related Work: The paper identifies its contribution as jointly studying retrieval-signal selection, context granularity, and RL-based retrieval-policy learning in agentic RAG.The authors state that, to their knowledge, this is the first work to study these three aspects jointly in training-based agentic RAG.
  • A Related Work: Recent agentic RAG systems interleave reasoning with retrieval but often restrict the interface to either sparse lexical retrieval or dense semantic retrieval.
  • A Related Work: Retrieval granularity affects evidence relevance and contextual noise, yet its role in agentic RAG remains less fully explored than in single-step RAG.Existing approaches often retrieve fixed-size or semantically segmented chunks, implicitly assuming a single retrieval-granularity level.
  • A Related Work: RL-based agentic retrieval research includes single-tool formulations optimized mainly through outcome-level answer rewards and newer multi-tool extensions.Jin et al. (2025) provides one retrieval tool, while Luo et al. (2025) extends the setting to multiple tools and incorporates semantic search.

B Reward Implementation Details

The reward implementation instantiates gold, read, and surfaced evidence sets from HotpotQA supporting facts and tracks retrieval, reading, complementary search, answer accuracy, and turn efficiency. It also standardizes string matching and adds safeguards for duplicate evidence, multiple answers, excessive turns, and answer-tag spam.

  • Gold evidence set: HotpotQA supporting facts are converted into gold evidence items identified by normalized Wikipedia article titles, with duplicate titles removed during preprocessing.Multiple supporting facts from one article therefore contribute a single evidence item.
  • String normalization: A common normalizer lowercases strings, removes articles and punctuation, and collapses whitespace before retrieval-reward and answer-F1 comparisons.This prevents case, article, and punctuation variation from producing spurious mismatches.
  • Read evidence set: Read evidence is collected from the parent paragraphs of previously retrieved sentences, while precision penalizes distractor reads and recall penalizes missed gold titles.Read titles are indexed by the paragraphs encountered during the trajectory.
  • Surfaced evidence sets: Semantic and keyword searches track surfaced sentence-level evidence separately, crediting a title whenever it appears in any result list regardless of rank or later reading.Reading behavior is rewarded separately from surfacing evidence.
  • Complementary search indicator: The complementary-search reward is positive exactly when both semantic and keyword search surface at least one gold title, allowing the same title to receive credit from both tools.The reward does not use winner-takes-all attribution.
  • Answer extraction and turn efficiency: The implementation uses the last extracted answer span for final commitment, supports multiple acceptable references, scales turn efficiency by used steps, and caps answer accuracy at 0.25 for excessive answer-tag spam.With T = 10, a correct one-step trajectory receives RE = 0.9, whereas a correct trajectory using all T turns receives RE = 0.

C Training Procedure Details

The main GRPO model was trained for 270 optimization steps on HotpotQA distractor data, reaching a validation-reward plateau before a full epoch. Training updates were restricted to policy-generated tokens, while the entropy coefficient was selected to avoid premature entropy collapse.

  • Training data and run length: 270 GRPO optimization steps, or approximately 17,280 prompts, produced the reported main-model checkpoint after validation reward plateaued.The HotpotQA distractor training split contains 90,447 examples; 1,413 steps at batch size 64 would constitute a full pass, so the run covered approximately 19% of an epoch.
  • Loss masking: Only policy-generated tokens contributed to the GRPO surrogate loss through a binary per-token mask.The masked spans included model reasoning and semantic_search, keyword_search, read_chunk, and answer actions; retrieval-observation tokens were excluded to prevent spurious gradients.
  • Entropy schedule: The entropy-bonus coefficient was 0.0021, while coefficients ≤0.001 caused entropy collapse during the first approximately 60 optimization steps.The lower coefficients collapsed before retrieval reward had stabilized, motivating the selected coefficient.
  • Backbone and infrastructure: The policy was initialized from Qwen/Qwen2.5-3B-Instruct and trained with full-parameter FSDP updates without LoRA adapters.The reference policy was frozen at the same checkpoint, and training used 2× NVIDIA A100 80 GB GPUs with Verl-Tool and vLLM.

D Dataset Sources · E Agent Training Prompt

The paper builds retrieval corpora from Hugging Face multi-hop QA validation splits and indexes their evidence at sentence level when possible. GRASP training, inference, and ablations use a shared prompt template whose tool descriptions and rollout stop tokens are selectively removed for single-tool variants.

  • D Dataset Sources: The three multi-hop QA datasets are obtained from the Hugging Face Hub through the datasets library.
  • D Dataset Sources: Evaluation uses validation splits as held-out test sets, with 500-question subsets sampled using random.seed(42).The retrieval corpus contains unique supporting and distractor paragraphs extracted across each dataset’s full validation split.
  • D Dataset Sources: Sentence-level indexing retains parent-paragraph metadata so retrieved sentences can be expanded into paragraph context.HotpotQA and 2WikiMultiHopQA provide sentence boundaries directly, whereas MuSiQue stores paragraphs as paragraph_text.
  • E Agent Training Prompt: GRASP uses the exact prompt in Table 6 for training, ablations, and inference, with the question appended at {question} at inference time.The system prompt and surrounding instructions remain unchanged in the single-tool ablations.
  • E Agent Training Prompt: The “without τk” ablation removes the keyword_search description and its corresponding action stop token.
  • E Agent Training Prompt: The “without τs” ablation removes the semantic_search description and its corresponding action stop token.
  • E Agent Training Prompt: The “without τr” ablation removes read_chunk and uses paragraph-granular retrieval returning full paragraphs rather than matched-sentence snippets.
Loading 2607.10463v1…