PaperlayerView exact revision on arXiv

Source-linked AI summary

Am I on the Right Track? What Can Predicted Query Performance Tell Us about the Search Behaviour of Agentic RAG

Fangzheng Tian, Jinyuan Fang, Debasis Ganguly, Zaiqiao Meng, Craig Macdonald

arXiv:2507.10411v1cs.IR

TL;DR

Agentic RAG’s generated queries and the retriever’s role in answer quality remain understudied. This study applies QPP to Search-R1 and R1-Searcher, finding that more effective retrieval tends to yield better answers with fewer iterations and that QPP estimates are positively associated with final-answer quality.

  • Problem

    The properties of Agentic RAG’s generated queries and retrieval results, and how retrieval contributes to answer quality across multiple iterations, remain underexplored.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates QPP estimates of generated queries in Search-R1 and R1-Searcher alongside retrieval effectiveness and answer quality.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    More effective retrieval models tend to yield higher-quality answers with fewer reasoning-retrieval iterations, and first-query QPP estimates show a weak positive correlation with final-answer quality.

    Source: §6 · p. 8

  • Takeaways & Limitations

    QPP estimates may help assess whether Agentic RAG retrieval results are useful enough to guide reasoning, an adaptive workflow left for future work.

    Source: §3.3 · p. 4, §6 · p. 8

    The experiments cover only Search-R1 and R1-Searcher on the NQ dataset, so other models and multi-hop QA remain outside the study’s scope.

    Source: §6 · p. 8

Abstract

from arXiv · show

Agentic Retrieval-Augmented Generation (RAG) is a new paradigm where the reasoning model decides when to invoke a retriever (as a "tool") when answering a question. This paradigm, exemplified by recent research works such as Search-R1, enables the model to decide when to search and obtain external information. However, the queries generated by such Agentic RAG models and the role of the retriever in obtaining high-quality answers remain understudied. To this end, this initial study examines the applicability of query performance prediction (QPP) within the recent Agentic RAG models Search-R1 and R1-Searcher. We find that applying effective retrievers can achieve higher answer quality within a shorter reasoning process. Moreover, the QPP estimates of the generated queries, used as an approximation of their retrieval quality, are positively correlated with the quality of the final answer. Ultimately, our work is a step towards adaptive retrieval within Agentic RAG, where QPP is used to inform the model if the retrieved results are likely to be useful.

1 INTRODUCTION

Agentic RAG lets a reasoning model choose when and what to retrieve, but the properties of its generated queries and retrieval results—and their contribution to answer quality—remain underexplored. This study examines retrieval models and query-performance estimates in Search-R1 and R1-Searcher.

1 INTRODUCTION

  • Standard RAG applies one retrieval strategy to every question, while Agentic RAG can issue iterative searches as reasoning proceeds.Answer quality may therefore reflect the cumulative influence of multiple retrievals.Source: §1 · pp. 1–2
  • Intermediate generated queries lack ground-truth relevance judgments, so the study uses QPP estimates to approximate their retrieval quality.It examines how estimates vary across iterations and correlate with answer quality.Source: §1 · p. 2
  • More effective retrieval models can reduce reasoning length and enhance answer quality, while generated-query QPP may signal final-answer quality.The study evaluates 3,610 Natural Questions test queries using Search-R1 and R1-Searcher.Source: §1 · p. 2

2 RELATED WORK

Related work covers retrieval in RAG, iterative retrieval integrated with reasoning, and query performance prediction. It motivates examining retrieval across Agentic RAG’s multiple reasoning-retrieval iterations, where query-level relevance judgments are unavailable.

2.1 The Role of Retriever in RAG

  • RAG passes retrieved documents to an LLM, and prior work examines document characteristics affecting downstream performance, but mostly studies single-retrieval pipelines.In Agentic RAG, answer quality may reflect the cumulative influence of multiple retrievals.Source: §2.1 · pp. 2–3

2.2 Iterative/Agentic RAG

  • Iterative RAG addresses questions that may need multiple searches; Agentic RAG integrates dynamic retrieval into LLM reasoning, with Search-R1 and R1-Searcher trained to invoke retrieval autonomously.The paper focuses on the retriever’s role in these models’ reasoning-retrieval iterations.Source: §2.2 · p. 3

2.3 Query Performance Prediction

  • QPP estimates retrieval-result quality without human relevance judgments; post-retrieval methods use result features, while query-only pre-retrieval methods are generally less accurate.Because generated Agentic RAG queries lack ground truth, QPP offers a way to investigate retrieval’s role.Source: §2.3 · pp. 3–4

3 AGENTIC RAG

Agentic RAG interleaves model reasoning with retrieval, while this study examines how retrieved-document quality may relate to reasoning and final answers. Because generated subqueries lack relevance ground truth, it investigates post-retrieval QPP as an estimate of their usefulness.

3.1 Overview

  • Each reasoning–retrieval cycle adds retrieved documents to the ongoing reasoning process, continuing until the model produces an answer; the cycle count is Iter.The generator issues a query when it needs external information, and the retriever returns documents for the next reasoning step.Source: §3.1 · p. 4

3.2 Retrieval

  • Retrieval pipelines can vary across sparse, reranked, and dense retrieval, but whether their retrieved documents improve answers in Agentic RAG remains an open question.The paper motivates testing whether more effective top-k results can reduce the iterations needed for an accurate answer.Source: §3.2 · p. 4

3.3 QPP of Generated Queries

  • Post-retrieval QPP estimates retrieved-document quality from document properties, serving as a possible proxy for usefulness because generated subqueries lack relevance judgments.The study focuses on post-retrieval predictors based on retrieved-document scores or embeddings.Source: §3.3 · p. 4
  • This initial study analyzes QPP estimates and their relation to answer quality, leaving adaptive use of those estimates in the reasoning process for future work.A QPP component could estimate retrieval quality during each iteration, but the adaptive workflow is not explored here.Source: §3.3 · p. 4
  • Dense QPP approaches are not a pragmatic fit for sparse retrievers such as BM25, and available embeddings may instead be used to improve the retriever.The practical applicability of QPP depends on the retrieval model type.Source: §3.3 · pp. 4–5

4 EXPERIMENTAL SETUP

The experiments examine retriever effects on answer quality and iteration count, query-performance estimates across iterations, and their association with answer quality. They use two Agentic RAG models, several retrieval configurations, and unsupervised QPP predictors.

4.1 Research Questions

  • The study asks how retriever configurations affect answer quality and iteration count, how query estimates vary across iterations, and how they correlate with answer quality.The query-quality estimates are used because dynamically generated queries lack human-assessed relevance judgments.Source: §4.1 · p. 5

4.2 Agentic RAG Pipeline Configurations

  • The evaluation uses 3,610 Natural Questions test questions with gold answers, measuring generated answers by F1 and Exact Match.The document corpus is a 2018 English Wikipedia snapshot.Source: §4.2 · p. 5
  • Search-R1 and R1-Searcher are tested with BM25, BM25 reranked by MonoT5, and dense E5 retrieval, incorporating the top three documents.R1-Searcher was not trained on NQ, so its evaluation here is out of domain.Source: §4.2 · p. 5

4.3 Query Performance Predictors

  • The predictors include NQC for all retrievers, Max(Score) for neural retrievers, and embedding-based A-Pair-Ratio and Dense-QPP for E5.NQC uses the top 100 retrieval scores; Dense-QPP uses the top three documents, and answer-quality correlations use only the first generated query.Source: §4.3 · pp. 5–6

5 RESULTS

More effective retrievers generally yield higher answer quality with fewer reasoning iterations in both Agentic RAG models. QPP estimates tend to decline over iterations, while first-query estimates are weakly but positively associated with final answer quality.

5.1 RQ-1: Effect of different retriever configurations on answer quality and number of iterations

  • Searcher-R1's F1 rose from 0.4185 with BM25 (2.52 iterations) to 0.5687 with E5 (2.0 iterations).Across Search-R1 and R1-Searcher, MonoT5 and E5 generally led to fewer reasoning-retrieval iterations than BM25.Source: §5.1 · p. 6
  • Iteration count had a moderate negative correlation with F1 for Search-R1 and a weaker negative correlation for R1-Searcher; the negative association was stronger with E5 and MonoT5 than BM25 for Search-R1.The authors suggest more relevant retrieved documents may support better follow-up queries and fewer iterations.Source: §5.1 · p. 6

5.2 RQ-2: Predicted Generated Query Quality versus Iterations

  • QPP estimates generally decreased over iterations for both models, except when NQC was applied to MonoT5, where its score distribution conflicts with statistical QPP assumptions.The authors caution that NQC may then misrepresent the relevance or usefulness of retrieved documents for LLM reasoning.Source: §5.2 · p. 6
  • For MonoT5, Max(Score) serves as an upper bound on query-document topical relevance; low scores may signal unmet information needs and potentially longer reasoning.Decreasing QPP trends also appeared for better-performing predictors such as Max(Score) for MonoT5.Source: §5.2 · pp. 6–7

5.3 RQ-3: Correlation between Predicted Query Performance and Answer Quality

  • First-query QPP estimates were weakly but consistently positively correlated with final answer quality, with correlations below the reported standard-retrieval levels of Spearman’s ρ > 0.5.The table covers significant Spearman correlations between first-iteration QPP estimates and F1 across both models and retrievers.Source: Table 3 · p. 7, §5.3 · pp. 7–8
  • For BM25, NQC correlations were around 0.12–0.13; with MonoT5, Max(Score) reached around 0.2, while Search-R1 with E5 and A-ratio reached 0.25.Correlations were higher for Search-R1 than R1-Searcher, and may be stronger for more effective RAG systems.Source: §5.3 · p. 8
  • The study correlates answer quality with QPP for the first generated query; improved aggregation of QPP estimates across iterations may yield higher correlations.The authors present QPP as a potentially useful signal for predicting answer quality in Agentic RAG.Source: §5.3 · p. 8

6 CONCLUSIONS

The study finds that retrieval quality and answer quality are linked in Search-R1 and R1-Searcher, and that the first retrieval may offer a useful signal for answer quality. It also identifies limits in the study’s scope and points to possible uses of QPP for adaptive retrieval and training.

6 CONCLUSIONS

  • More effective retrievers tend to produce higher-quality answers with fewer reasoning-retrieval iterations, while first-query QPP estimates show a weak positive correlation with final answer quality.Retrieval quality tends to decrease over the reasoning process; a case study suggests ambiguity in the original query may contribute, though retrieval quality does not necessarily decline monotonically.Source: §6 · p. 8
  • The study covers only Search-R1 and R1-Searcher on the NQ dataset, so its findings may not reflect other Agentic RAG models or multi-hop QA settings.The authors identify Search-O1 and HotpotQA as examples of models and tasks for future study.Source: §6 · p. 8
  • QPP could inform whether retrieved results are useful enough to show the reasoning model, or provide a training reward for better queries.The proposed training signal would go beyond rewards for valid search requests and correctly formatted responses.Source: §6 · p. 8

A APPENDIX

The appendix examines one negative and one positive Search-R1 example with E5 to explore why estimated retrieval quality declines in later iterations.

A APPENDIX

  • The appendix analyzes one negative and one positive Search-R1 example with E5 to investigate later-iteration declines in average estimated retrieval quality.It also offers a brief insight from the example analysis.Source: A APPENDIX · p. 10

A.1 Negative Example: Ambiguous Question Causes Drifted Sub-Queries and Low-Quality Answer

The negative example shows how an ambiguous question can lead Search-R1’s queries away from the information need and produce an incorrect answer. Its QPP estimates are lower than those in the positive example, though they do not decline monotonically.

A.1 Negative Example: Ambiguous Question Causes Drifted Sub-Queries and Low-Quality Answer

  • Search-R1’s first query repeats the input, then its second query drops “Chris Cornell,” and the third search retrieves documents about a weakly related song.The model initially treats the lyric excerpt as a song title, after which the omission of a key entity leads to topic drift.Source: §A.1 · p. 10
  • Across three query iterations, Max(Score) QPP estimates were 0.8370, 0.7870, and 0.8511, below the positive examples’ Max(Score)> 0.9; ambiguity in the original question led the model to misinterpret the user’s information need and answer incorrectly.Source: §A.1 · p. 10

A.2 Positive Example: Specific Question with High Quality Retrievals

In the positive example, Search-R1 answers a clear question correctly after two reasoning-retrieval iterations, with high Max(Score) QPP estimates. The example also notes repeated intermediate queries as an issue in agentic models.

A.2 Positive Example: Specific Question with High Quality Retrievals

  • The two iterations have Max(Score) QPP estimates of 0.9263 and 0.9268, and Search-R1 correctly answers 2018 after two iterations.The question asks when Villanova last won the NCAA basketball championship; the model reformulates it for both searches.Source: §A.2 · p. 10
  • With E5 and a three-iteration cutoff, 262 of 3610 generated queries contain identical sub-queries, indicating insufficient use of retrieved documents.The passage suggests QPP or other signals could help achieve more efficient use of retrieved information.Source: §A.2 · p. 10

A.3 Insights from the Case Study

The case study links shorter reasoning trajectories with higher predicted retrieval quality, while longer trajectories tend to include lower-quality intermediate retrievals. Example trajectories illustrate both drift toward a wrong answer and successful reformulation, alongside repeated queries in a successful example.

A.3 Insights from the Case Study

  • Longer reasoning trajectories tend to contain lower-quality intermediate retrievals, while correct-answer trajectories generally use fewer iterations and higher-quality retrievals.Early iterations have higher mean predicted retrieval quality than later iterations.Source: §A.3 · p. 10
  • Even in the positive example, current Agentic RAG models repeat identical intermediate queries, a redundancy that can reduce efficiency.Source: §A.3 · p. 10
  • Search-R1’s negative example drifts from the original information need and reaches a wrong answer, whereas its positive example reformulates the query and reaches the correct answer.Source: Figure 6 · p. 12, Figure 7 · p. 13

Related papers

Loading 2507.10411v1…