PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation

Fangzheng Tian, Debasis Ganguly, Craig Macdonald

arXiv:2609.16453v1cs.IR

TL;DR

Agentic RAG evaluations provide limited insight into how answer quality changes across retrieval-reasoning iterations. This paper probes intermediate answers and predicts their quality and utility from trajectory signals; its analysis finds sparse improvements and motivates prediction-guided early stopping.

  • Problem

    End-to-end evaluations provide limited insight into intermediate answer states, and the utility of individual retrieval-reasoning iterations is rarely observed or explicitly supervised.

    Source: §1 · p. 1

  • Method

    The framework probes an answer after each iteration, defines its partial quality and change in quality, and predicts these measures from intra-iteration, inter-iteration, and query-iteration signals.

    Source: §1 · pp. 1–2

  • Results

    Most trajectories contain only one iteration that substantially changes partial answer quality, while many later iterations provide no measurable improvement.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Online predictions can guide stopping when predicted quality remains consistently high or further iterations are unlikely to improve the answer, reducing redundant iterations and computational cost while preserving final answer quality.

    Source: §1 · p. 2

    The study focuses on prediction rather than optimized controller design, uses QA-style metrics that may miss finer-grained answer changes, and calls for evaluation on advanced pipelines and complex tasks.

    Source: §7 · p. 10

Abstract

from arXiv · show

Agentic Retrieval-Augmented Generation (RAG) has become a promising paradigm for multi-hop question answering, where a reasoning model iteratively issues queries to a retriever and incorporates newly retrieved context into subsequent reasoning steps. While this iterative process can improve final answer quality, current evaluations of agentic RAG largely focus on end-to-end outcomes and provide limited visibility into how a model's answer state changes during generation. In this work, we introduce an in-trajectory probing framework to study intermediate answer states in agentic RAG. Specifically, after each retrieval-reasoning iteration, we force an agentic model to stop reasoning and generate an intermediate answer based on its current state. This allows us to define two iteration-level measures: partial answer quality at each iteration, and partial utility as the change in partial answer quality across iterations. Our analysis across multi-hop QA benchmarks reveals that partial answer quality often plateaus before natural termination, with many later iterations contributing only small measurable improvements. Accordingly, we formulate two prediction tasks, partial answer quality prediction and partial utility prediction, and study trajectory-derived signals from intra-iteration, inter-iteration, and query-iteration perspectives. Experiments show that partial answer quality is more predictable than partial utility, with supervised models achieving Pearson's r above 0.43 for quality prediction. Finally, using predicted answer quality and utility for early stopping reduces average iteration count by about 11% while preserving about 98% of the final answer quality achieved by natural stopping.

1 Introduction

The introduction argues that end-to-end evaluation obscures how individual retrieval-reasoning iterations affect answer quality. It probes answers after each iteration, studies trajectory-based prediction, and finds that quality often plateaus before the process naturally ends.

1 Introduction

  • Agentic RAG iteratively retrieves evidence and reasons before producing a final answer, extending conventional single-step RAG [26] [49].The introduction situates this approach among frameworks reported to improve benchmark QA effectiveness [1] [16] [23] [52] [66].Source: §1 · p. 1
  • Existing agentic RAG evaluations offer limited visibility into intermediate answer states and rarely observe or supervise the utility of each iteration.The introduction notes that repeated iterations may prolong the process and degrade answer quality [49].Source: §1 · p. 1
  • Probing the model after each iteration yields partial answer quality and partial utility, the change in quality between adjacent iterations.Partial answer quality estimates how well the model would answer if the process stopped at that iteration.Source: §1 · p. 1, Figure 1 · p. 2
  • Most trajectories contain only one iteration that substantially changes partial answer quality; many later iterations provide no measurable improvement.The paper describes this pattern across two basic agentic RAG pipelines.Source: §1 · pp. 1–2
  • The paper formulates quality and utility prediction tasks, using intra-iteration, inter-iteration, and query-iteration signals to guide online early stopping.Stopping can be based on consistently high predicted quality or predicted utility indicating further iterations are unlikely to improve the answer.Source: §1 · p. 2

2 Related Work

Related work spans iterative retrieval systems, methods for probing intermediate states, and prediction of retrieval utility. The paper positions its contribution as trajectory-level answer-quality and utility prediction rather than static context assessment or direct stopping-policy learning.

2 Related Work

  • Agentic RAG systems interleave reasoning and retrieval, with approaches ranging from prompted retrieval such as iRCoT [53] and FLARE [20] to reinforcement-learned systems such as Search-R1 [21] and R1-Searcher [46].Other systems add reflection, refinement, or query-path mechanisms, including Self-RAG [40], Search-O1 [28], and CoRAG [59].Source: §2 · pp. 2–3
  • The paper studies Search-R1 [21] and R1-Searcher [46] in a single retrieval-reasoning trajectory, without added self-reflection, document refinement, or path selection.It distinguishes its standalone prediction tasks and interpretable trajectory signals from SIM-RAG's critic-based evidence-sufficiency judgment [65].Source: §2 · p. 3
  • Existing probing approaches use uncertainty, internal representations, or self-evaluation, but these signals do not directly measure iteration-level utility and can inherit calibration limitations [2] [41] [43].The paper contrasts its approach with [58]: it evaluates the model's most likely probed answer against ground truth, rather than measuring ground-truth answer probability, and notes adaptability to long-form QA [48] [54].Source: §2 · p. 3
  • Prior RAG prediction methods estimate context utility or answer quality mainly in single-step settings, while agentic RAG makes prediction trajectory-level because iterations affect later reasoning and retrieval [6] [51] [53].Related work also describes adaptive controllers, including methods estimating evidence sufficiency [36] [65] and Stop-RAG's finite-horizon stopping formulation [35].Source: §2 · p. 3

3 Predictors for Agentic RAG

The section defines how intermediate answer states are observed and measured, then develops trajectory-derived signals for predicting answer quality and iteration utility. Its prediction scheme combines current-iteration evidence, changes across iterations, and alignment with the original question.

3.1 Trajectories and Intermediate Answer States

  • Probing after each retrieval-reasoning iteration produces an answer from the current trajectory state, making intermediate answer quality and iteration-level utility measurable.Partial answer quality compares the probed answer with the ground truth; partial utility is the change in quality from the preceding iteration.Source: §3.1 · p. 4

3.2 Prediction Tasks

  • The prediction tasks estimate partial answer quality or partial utility from the trajectory prefix available at the current iteration, without using the ground-truth evaluation of the probed answer.Quality prediction estimates P_i; utility prediction estimates U_i, the change relative to the preceding iteration.Source: §3.2 · p. 4, §3.1 · p. 4

3.3 Prediction Scheme

  • Predictive signals capture current-iteration properties, changes between iterations, and alignment of the current query and reasoning with the original question.Current-iteration signals examine query, retrieved context, reasoning, and probed answer; inter-iteration changes are especially relevant to utility, while alignment can indicate focus or topic drift.Source: §3.3 · p. 4
  • Unsupervised signals act as interpretable indirect indicators, while supervised cross-encoders directly estimate partial answer quality or partial utility from structured trajectory inputs.The signal overview distinguishes these predictor families across the three signal sources.Source: §3.3 · p. 5, Table 1 · p. 5
  • A prediction head can combine estimator outputs from a recent-iteration window, allowing it to use complementary signals and temporal patterns when estimating either target.The windowed feature vector concatenates outputs across the selected iterations; unavailable positions before the trajectory begins are padded with missing-value indicators.Source: §3.3 · p. 5

4 Prediction-Guided Early Stopping

The early-stopping heuristic uses predicted answer quality and utility together to decide whether to stop, continue, or return an earlier probed answer. Combining both predictions avoids treating high utility or low utility alone as a sufficient stopping signal.

4 Prediction-Guided Early Stopping

  • Predicted quality estimates whether the current answer is good enough, while predicted utility estimates whether the iteration improves on the previous answer; the heuristic uses them jointly.Either signal alone can be ambiguous: high utility may accompany low answer quality, while low utility may reflect either a good-answer plateau or poor-answer stagnation.Source: §4 · p. 5
  • Figure 2 partitions predicted partial answer quality and predicted partial utility into four states using separate thresholds.For State 2 and cases not covered by the stopping rules, generation continues until natural termination.Source: Figure 2 · p. 6, §4 · p. 6
  • The heuristic stops at State 0 and returns the current probed answer, or rolls back to the earliest prior State 0 or State 1 answer when State 3 follows.State 0 means predicted high quality and high utility; State 1 means high quality and low utility; State 3 means low quality and low utility. Rollback costs count through the later State 3 trigger.Source: §4 · p. 6

5 Experimental Setup

The experimental setup asks how intermediate answer states evolve, whether trajectory-derived signals can predict them, and whether those predictions can guide more efficient stopping. It evaluates predictor families on two RL-trained agentic RAG pipelines and three multi-hop QA datasets.

5.1 Research Questions

  • The experiments ask whether answer quality plateaus before natural termination, whether quality and utility are predictable, and whether state predictions can improve process efficiency.The early-stopping evaluation considers whether the system can stop with limited answer-quality loss.Source: §5.1 · p. 6

5.2 Implemented Predictors

  • Predictor implementations include unsupervised trajectory measurements, supervised cross-encoder regressors, and answer-confidence signals, with combinations and context-window sizes evaluated as ablations.Confidence signals require answer probing at inference time and add computational overhead; fixed-cap stopping is the reference baseline for the early-stopping experiment.Source: §5.2 · pp. 6–7

5.3 Agentic RAG Pipelines and Datasets

  • The study evaluates Search-R1 and R1-Searcher on HotpotQA, 2Wiki, and MuSiQue, using F1 for partial answer quality and utility evaluation.The systems use Qwen2.5-7B and the study focuses on the basic iterative paradigm; supervised predictors are trained on combined training splits and evaluated on development splits.Source: §5.3 · p. 7

6 Results

Across three research questions, the results show that answer-quality changes are concentrated in a few often-early iterations, intermediate answer quality is more predictable than utility, and prediction-guided stopping can reduce iteration costs while preserving most answer quality.

6.1 RQ-1: Decisive Iterations are Sparsely Distributed in Agentic RAG

  • Partial utility distributions for both pipelines are sharply concentrated around zero, so most iterations do not substantially change probed answer quality.The less frequent non-zero iterations can improve or degrade quality and help determine a trajectory’s net utility.Source: §6.1 · pp. 7–8, Figure 3 · p. 8
  • Positive-utility trajectories often improve early and plateau, negative-utility trajectories often drop early without recovery, and zero-utility trajectories show no net improvement.Figure 4 groups trajectories by net utility and shows their partial answer quality over iterations for both pipelines.Source: Figure 4 · p. 8, §6.1 · p. 8

6.2 RQ-2: Intermediate Answer State Can Be Predicted from Trajectory-Derived Signals

  • For Search-R1, the best partial answer quality prediction reaches Pearson’s r=0.438, versus Pearson’s r=0.321 for partial utility prediction.Quality prediction is more reliable than utility prediction across almost all settings; the reported best Kendall’s τ values are 0.345 for quality and 0.208 for utility.Source: §6.2 · p. 8, Table 3 · p. 9
  • Supervised predictors are the strongest individual signal group, while combining them with unsupervised diagnostics and probing confidence provides complementary evidence.Probing-based confidence contributes less, especially for R1-Searcher; combining unsupervised and supervised signals yields only slight prediction gains but can improve interpretability.Source: §6.2 · p. 9
  • Intra-iteration signals are generally the strongest individual unsupervised sources, and combining all three signal types improves over each alone.A window of w=3 helps unsupervised prediction, while extending it to w=5 adds little; larger windows help most when unsupervised signals are included.Source: §6.2 · p. 9, Table 4 · p. 9

6.3 RQ-3: Prediction-Guided Stopping Improves Efficiency

  • With θP=0.3 and θU=0.2, stopping averages 2.86 iterations—10.89% fewer than natural stopping’s 3.21—while preserving 97.60% of its answer quality.Increasing θU to 0.3 preserves 98.49% of answer quality with only 0.06 additional average iterations versus θU=0.2.Source: Figure 5 · p. 10, §6.3 · p. 10
  • Prediction-guided stopping makes trace-specific decisions, unlike fixed iteration caps, which truncate every trajectory at the same length.Lowering the quality and utility thresholds makes stopping more aggressive, reducing iterations with greater risk of answer-quality loss.Source: §6.3 · p. 10
  • Non-probing predictors use signals already produced in the trajectory and compact SBERT-based regressors, while probing requires an additional answer generation.Adding probing brings only limited answer-quality improvement, so stopping can often avoid that additional generation cost.Source: §6.3 · p. 10

7 Future Work

The paper’s in-trajectory probing measures intermediate answer quality and utility, and its prediction tasks support early stopping. The study is limited to prediction rather than optimized controller design, QA-style quality metrics, and basic pipelines and tasks.

7 Future Work

  • In-trajectory probing measures partial answer quality after each retrieval-reasoning iteration and partial utility as its change across the trajectory.The analysis finds decisive iterations are sparse and often early, while later iterations may preserve a plateau or fail to recover degradation.Source: §7 · p. 10
  • Trajectory-derived signals predict both intermediate targets, with partial answer quality more reliable; a prediction-guided controller reduces iterations while retaining most natural-stop answer quality.The study also finds supervised predictors strong, with unsupervised diagnostics and probing-based confidence providing complementary information.Source: §7 · p. 10
  • The study focuses on prediction rather than optimized controller design, uses QA-style metrics such as F1 that may miss finer answer changes, and leaves advanced pipelines and long-form tasks for future work.Adaptive policies may further improve the efficiency-effectiveness trade-off.Source: §7 · p. 10

Related papers

Loading 2609.16453v1…