PaperlayerView exact revision on arXiv

Source-linked AI summary

A New Role for Relevance: Guiding Corpus Interaction in Agentic Search

Jiangnan Li, Yuqing Li, Mo Yu, Jinchao Zhang, Jie Zhou

arXiv:2607.24223v2cs.CL

TL;DR

Existing agentic search interfaces use relevance to select content but do not directly guide fine-grained corpus interaction. RARG turns relevance into an execution prior for document traversal, entry-point initialization, and match reranking, improving the accuracy–efficiency frontier across challenging QA and reasoning-intensive retrieval.

  • Problem

    Document-level relevance cannot by itself localize, connect, compare, or verify evidence needed for complex multi-step questions.

  • Method

    RARG uses relevance to order document traversal, initialize searches with query-relevant paragraphs, and rerank grep matches for coarse-to-fine corpus interaction.

  • Results

    Across challenging QA, corpus scaling, and reasoning-intensive retrieval, RARG improves the accuracy–efficiency frontier; RARG++ reaches 84% accuracy with 23.9 average tool calls on 100K-document BrowseComp-Plus.

  • Takeaways & Limitations

    Relevance-aware interaction supports faster and more reliable search convergence while preserving fine-grained, compositional corpus exploration.

  • Takeaways & Limitations

    Evaluation is limited to BrowseComp-Plus and four BRIGHT subsets with GPT-5.x models, leaving broader validation across backbones, open-web settings, and domains for future work.

Abstract

from arXiv · show

Relevance is a query-dependent estimate of whether a document or excerpt contains useful evidence. Existing retrieval agents use relevance to select top-$k$ content, but document relevance alone cannot localize, compose, or verify the evidence required by complex questions. Direct Corpus Interaction (DCI) enables such fine-grained operations through grep-style exploration, but its relevance-agnostic search can expose useful clues late and delay convergence. Recent advances use relevance to narrow the corpus into a working space for interaction. Once interaction begins, however, relevance still does not directly guide which documents grep searches first or distinguish informative excerpts from a broad set of matches to let LLMs see them first. We introduce the Relevance-Aware RipGrep Search Agent (RARG), which turns relevance into an execution prior for corpus interaction. RARG provides coarse-to-fine relevance guidance: it orders documents for sequential 'ripgrep' traversal to expose globally relevant clues earlier, initializes promising entry points with query-relevant paragraphs, and reranks grep matches to surface informative excerpts that document-level ranking may otherwise obscure. Across challenging browse question answering and reasoning-intensive retrieval, RARG improves the accuracy--efficiency frontier over retrieval-based and direct-interaction agents. These results demonstrate that relevance-aware interaction enables faster and more reliable search convergence.

1 Introduction

The introduction argues that relevance should guide corpus interaction itself, not merely select documents, because document-level ranking cannot localize, compose, or verify evidence. RARG operationalizes this through coarse-to-fine guidance over traversal order, entry points, and matched excerpts, improving search convergence and reported accuracy.

  • Motivation: Document relevance alone cannot expose decisive spans omitted by snippets or evidence whose usefulness emerges only after intermediate entities are discovered.Retrieval agents conventionally rank documents or snippets for top-k exposure, conflating document relevance with evidence utility.
  • Motivation: Direct Corpus Interaction enables exact-context inspection, lexical constraint composition, and entity following, but its relevance-agnostic scans treat locations as equally promising.DCI uses terminal tools such as pattern matching and local file reads, while lacking corpus-level relevance guidance.
  • Method: RARG turns relevance into an execution prior by traversing documents sequentially in embedding-retriever order, exposing matches from more relevant documents earlier.The agent generates a query, ranks documents with an embedding retriever, and preserves that ranking during rg traversal.
  • Method: RARG+ initializes interaction with query-relevant paragraphs, while match-level reranking provides local relevance guidance when raw grep output exceeds the agent’s observation budget.Together, these mechanisms provide coarse-to-fine guidance over entry points and the excerpts shown to the model.
  • Evaluation: 84% accuracy was achieved by RARG++ on 100K-document BrowseComp-Plus with GPT-5.4-mini, versus 78% for both RISE and DCI.The evaluation also covers expansion from 100K to 1M documents and reasoning-intensive retrieval across four BRIGHT domains.

2 Related Work

Prior work uses relevance to rank or construct searchable context, while agentic retrieval and Direct Corpus Interaction enable iterative, fine-grained corpus access. RARG addresses the remaining gap by carrying relevance into interaction as an execution prior that prioritizes traversal, entry points, and observed matches.

  • Relevance organizes modern retrieval through sparse lexical ranking such as BM25 and dense semantic matching in a shared representation space.
  • Retrieval-augmented generation assembles context before reasoning, so longer ranked contexts do not guarantee reliable use of decisive evidence buried among less relevant content.
  • Agentic retrieval interleaves reasoning with retrieval actions, but conventional top-k interfaces can conflate document-level relevance with evidence utility.
  • Direct Corpus Interaction exposes raw files for pattern matching and local reads, but unrestricted operations may treat all files as equally promising and increase cost with distractors.
  • RISE and DR-DCI bound direct interaction by constructing or expanding searchable workspaces, yet do not explicitly propagate relevance to order local traversal or prioritize matches.
  • RARG carries corpus-level relevance into interaction as an execution prior, guiding search starts, document traversal order, and which local matches reach the model under a limited observation budget.RARG's overview describes document-level relevance as determining where ripgrep searches first and match-level relevance as determining which matches reach the model.

3 Method

RARG injects relevance into direct corpus interaction by ordering grep traversal with document-level guidance, initializing searches with relevant paragraphs, and reranking matches with a constructed query. Built on DCI-Agent-Lite, it progressively narrows and refines evidence while preserving sequential search behavior; its generative reranking variant degrades performance.

  • Relevance-guided traversal: RARG uses retrieval as relevance guidance for grep, treating it as a support mechanism rather than the evidence channel.The approach addresses embedding retrieval’s weakness when top-ranked documents are fed directly to the LLM.
  • Relevance-guided traversal: The embed_recall tool writes up to 10,000 ranked document paths to a scope file, which rg searches in that order with injected -j1 sequential scanning.The LLM begins each episode with the original question, may create new scopes, and retains scope mappings during compaction.
  • Query-relevant initialization: RARG+ appends the top-10 query-relevant paragraphs from the top-X scope documents as an initial entry point for the LLM.Paragraphs are 400–1000 characters, selected by embedding similarity, and cleared during compaction.
  • Match-level reranking: RARG++ reranks up to M grep matches and exposes the top m, using a constructed query that combines the scope query with local grep keywords.Reranking is skipped when fewer than m matches exist, helping surface informative excerpts hidden by document-level relevance.
  • Match-level reranking: The generative reranking-query variant degrades performance, so RARG++ uses the constructed query by default.The generative query is supplied through RerankAwareBash only for intended rg calls.

4 Experiments

Experiments evaluate RARG on browse question answering and reasoning-intensive retrieval, comparing relevance-guided interaction with retrieval-based and direct-interaction agents. RARG achieves the strongest reported accuracy-efficiency tradeoffs, while scaling, task search shape, and progressively finer guidance reveal important conditions on its behavior.

  • Main results: RARG++ reaches 84%/79% on GPT-5.4-mini/nano and 91% on GPT-5.4, exceeding the strongest baseline by 6/8/9 points while using fewer tool calls.It uses 23.9/25.43 tool calls against 99.1 for DCI and 28.7/34.3 for RISE, except in the GPT-5.4-nano setting.
  • Main results: Retrieval-centric Embedding and NeMo agents remain weakest on BrowseComp-Plus because they expose relevance as top-ranked content rather than as an exploration guide.NeMo provides stronger document coverage through recall and ranking, but both agents retain the same retrieval-interface bottleneck.
  • Variant analysis: Within RARG, mini accuracy rises 80→81→84 while tools fall 29.8→29.6→23.9 from RARG to RARG+ to RARG++, with nano showing the same ordering at 73/74/79.The generative variant converges fastest at 17.8 tools but drops to 75%, attributed to a train–evaluation gap.
  • Scaling: Scaling BrowseComp-Plus from 100K to 1M documents lowers all methods, with RISE-BM25 falling from 77% to 69% and RARG/RARG+/RARG++ from 80/81/84 to 78/78/79.The added long FineWeb-Edu articles increase incidental lexical matches and distracting context in ripgrep scans.
  • Cross-task behavior: RARG+ attains the best average nDCG@10 on BRIGHT despite fast convergence being less aligned with its breadth-first ranking objective than with depth-first question answering.Tool counts reflect different interfaces and are therefore not directly comparable across agents.

5 Conclusion

The conclusion argues that relevance should guide corpus interaction, not merely select inputs, and introduces RARG as a coarse-to-fine execution prior for grep exploration. RARG preserves DCI’s fine-grained, compositional interaction while restoring relevance guidance across challenging retrieval settings.

  • 5 Conclusion: RARG turns retrieval scores into an execution prior for grep exploration at document and match resolutions.Document-level relevance orders rg traversal, while match-level reranking controls which local excerpts remain visible under a limited observation budget.
  • 5 Conclusion: Entry-point initialization gives the agent a precise place to begin searching.
  • 5 Conclusion: RARG’s coarse-to-fine design preserves DCI’s fine-grained, compositional interaction while restoring the relevance guidance that unrestricted grep lacks.
  • 5 Conclusion: Across challenging QA, corpus scaling, and reasoning-intensive retrieval, RARG advances the accuracy–efficiency frontier.

Limitations

RARG’s effectiveness is limited by the quality of its relevance signal, the backbone’s instruction-following ability, and interference from noisy corpora. Its sequential scanning and match reranking also introduce a performance–time trade-off, while generative reranking-query behavior can reduce accuracy despite faster convergence.

  • Relevance guidance: RARG depends on relevance-model quality, with document ranking and short-match reranking imposing different demands that one embedding model may not satisfy equally.On BRIGHT, NV ranks documents well but handles short matches less reliably, prompting a fallback to Qwen3-Embedding.
  • Backbone dependence: RARG is sensitive to backbone instruction-following ability: GPT-5.4-nano follows the multi-stage protocol less reliably and converges more slowly than GPT-5.4-mini and GPT-5.4.RARG still improves over baselines on nano.
  • Corpus interference: Long, noisy documents introduce incidental lexical matches that degrade both rg output and embedding scores, which match-level reranking only partially mitigates.A generative reranking-query variant converges fastest but loses accuracy, suggesting a train–evaluation gap from perturbing learned Bash/rg behavior.

A Agent Implementation Details

The compared agents are configured for fair evaluation across BRIGHT and BC+, with benchmark-specific prompting, retrievers, truncation, and context-management procedures. Embedding and NeMo agents are adapted to match the methods’ retrieval settings while preserving their distinct interaction and ranking designs.

  • Fair comparison: The study configures all compared agents for a fair comparison on both benchmarks.Configurations are benchmark-specific and account for retrieval, prompting, and context handling.
  • BRIGHT prompting: On BRIGHT, grep-style agents are prompted to rank 10 document names, matching nDCG@10 and the shared output format.This modification addresses the substantially worse performance of the default DCI IR prompt.
  • Embedding Agent: On BC+, the Embedding Agent uses Q3E retrieval without grep interaction, recalls 5 documents per call, and truncates each to 512 Qwen3 tokens.Tool results from earlier turns are cleared after the context exceeds the compaction threshold, following DCI/RARG.
  • NeMo Agent: NeMo uses Q3E on BC+ and NV on BRIGHT, matching each method’s benchmark retriever while truncating recalled documents to the first 512 Qwen3 tokens.As a retrieval-specialized agent, NeMo recalls more documents to improve recall and reranks them to improve precision.
  • NeMo Agent adaptation: For BC+, NeMo is adapted to handle context overflow by RRF-ranking recalled documents and passing them to a fresh answer agent, dropping the lowest 5 documents if needed.This adapts NeMo’s recall-and-rank design for a QA-style task lacking a dedicated configuration.

B Prompts for RARG

RARG’s prompts tailor corpus interaction to BrowseComp-Plus and BRIGHT while enforcing scoped search over a large corpus. They combine semantic document recall with ordered ripgrep exploration, and expose tools for reading, shell execution, and optional match reranking.

  • B.1 System Prompt of BC+: On BrowseComp-Plus, RARG forbids commands that list all filenames because the corpus exceeds 100,000 files across 30,000 directories.The prompt warns that listing the corpus will timeout and waste turns.
  • B.1 System Prompt of BC+: The BrowseComp-Plus prompt uses embed_recall to create scope files of relevant document paths and return reference paragraphs that suggest ripgrep directions.This narrows the search space to a manageable subset before interaction.
  • B.1 System Prompt of BC+: Within BrowseComp-Plus scopes, RARG pipes each scope file to rg, returns matches by document embedding similarity, and prohibits combining multiple scope files.The prescribed command is cat /path/to/scope_N.txt | xargs -d ’\n’ rg [OPTIONS] "PATTERN".
  • B.2 System Prompt of BRIGHT: On BRIGHT, RARG is instructed to maximize recall without sacrificing precision and return a ranked list of exactly ten document paths.The task is framed as retrieval rather than question answering.
  • B.2 System Prompt of BRIGHT: The BRIGHT prompt requires an original-query embed_recall call, diverse rg exploration within its scope, and another recall call only when the scope clearly misses the topic.It prohibits repeated embed_recall calls without intervening rg exploration and forbids direct corpus-wide rg.
  • B.2 System Prompt of BRIGHT: BRIGHT defines relevance as usefulness for answering or supporting a query, including key entities, mechanisms, theorems, examples, facts, APIs, or other clues beyond surface wording.The prompt also directs searches from multiple angles while avoiding tangential documents.
  • B.3 Tool Specifications: RARG exposes embed_recall, read, and bash tools, while RARG++ additionally uses RerankAwareBash with an optional rerank_query for semantic reranking of rg matches.The rerank query consists of two short sentences describing the target fact and the current verification step.

C Case Study: Compressed Agent Trajectories

This case study shows that all three RARG trajectories correctly solve a multi-document BrowseComp-Plus query, while relevance guidance progressively reduces interaction. Initialization advances decisive evidence earlier, and match-level reranking preserves that benefit while further reducing tool use.

  • Case takeaway: All three RARG variants correctly identify Russell David Lyons in a 100K-document BrowseComp-Plus query requiring multiple biographical and bibliographic joins.The query combines a mathematics Ph.D. and AMS Fellowship with a three-author paper, an award, and a second paper title ending in “Line.”
  • Case takeaway: Tool use decreases strictly from 33 to 18 to 10 across the three RARG variants, while total turns fall from 17 to 11 to 10.These totals are computed from the uncompressed trajectories and include every original tool call.
  • How the search state changes: Initialization produces the largest evidence-timing improvement, moving the answer-bearing CV from turn T7 to T2.The first decisive observation changes the search from an open-ended person search into verification of a specific mathematician and coauthor chain.
  • Case takeaway: Match-level reranking retains early exposure to the answer-bearing evidence while further reducing tool use.The case illustrates how relevance guidance can reduce interaction needed to resolve a multi-document clue chain without implying a general ordering from one example.
Loading 2607.24223v2…