PaperlayerView exact revision on arXiv

Source-linked AI summary

Stop-RAG: Value-Based Retrieval Control for Iterative RAG

Jaewan Park, Solbee Cho, Jay-Yoon Lee

arXiv:2510.14337v1cs.LGcs.AI

TL;DR

Iterative RAG must decide when further retrieval is worthwhile, but existing approaches use fixed iteration counts or stopping signals that may not capture future retrieval value. Stop-RAG frames retrieval as a finite-horizon MDP and uses value-based control; across multiple multi-hop QA benchmarks it improves performance, while a HotpotQA case shows a boundary to that result.

  • Problem

    Iterative RAG needs an adaptive stopping strategy because additional retrieval has costs, and existing approaches may not reflect whether further retrieval will help.

    Source: §1 · p. 1

  • Method

    Stop-RAG frames iterative RAG as a finite-horizon MDP and trains a Q-network with Q(λ) to estimate the future benefits of continuing retrieval.

    Source: §1 · p. 2

  • Results

    Stop-RAG consistently improves performance across multiple multi-hop QA benchmarks and outperforms fixed-iteration or prompting-based baselines.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Value-based stopping can balance retrieval recall and precision while remaining modular and compatible with black-box LLMs.

    Source: §7 · p. 9

    On HotpotQA, Stop-RAG slightly underperforms both baselines; the broader, more challenging corpus makes retrieval recall critical, and full-iteration retrieval may outweigh distractor costs.

    Source: §6.1 · pp. 7–8

Abstract

from arXiv · show

Iterative retrieval-augmented generation (RAG) enables large language models to answer complex multi-hop questions, but each additional loop increases latency, costs, and the risk of introducing distracting evidence, motivating the need for an efficient stopping strategy. Existing methods either use a predetermined number of iterations or rely on confidence proxies that poorly reflect whether more retrieval will actually help. We cast iterative RAG as a finite-horizon Markov decision process and introduce Stop-RAG, a value-based controller that adaptively decides when to stop retrieving. Trained with full-width forward-view Q($\lambda$) targets from complete trajectories, Stop-RAG learns effective stopping policies while remaining compatible with black-box APIs and existing pipelines. On multi-hop question-answering benchmarks, Stop-RAG consistently outperforms both fixed-iteration baselines and prompting-based stopping with LLMs. These results highlight adaptive stopping as a key missing component in current agentic systems, and demonstrate that value-based control can improve the accuracy of RAG systems.

1 Introduction

Iterative RAG supports complex questions through repeated retrieval, but deciding when to stop remains underexplored despite the query-dependent costs and benefits of additional loops. Stop-RAG uses value-based control to estimate whether continuing retrieval will help, and improves performance on multi-hop QA benchmarks while remaining compatible with black-box LLMs and existing pipelines.

1 Introduction

  • Additional retrieval iterations incur latency and token costs and may introduce distractors, while their benefits vary across queries.The passage identifies optimal stopping as important because these costs and benefits differ by query.Source: §1 · p. 2
  • Existing stopping methods use self-assessment, telemetry, or current-step answer-quality supervision, which can misjudge whether future retrieval will help.Irrelevant documents can lower a current-step score despite sufficient evidence, while early confidence can prompt stopping before completeness improves.Source: §1 · p. 2
  • Stop-RAG formulates iterative RAG as a finite-horizon MDP and trains a Q-network with Q(λ) for forward-looking stopping estimates.The approach compares immediate and future gains without requiring internal telemetry.Source: §1 · p. 2
  • The paper identifies adaptive stopping in agentic AI, especially iterative RAG, as an underexplored decision-making capability.Prior work emphasizes multi-turn tool use, while comparatively little attention has gone to judging when further calls are unnecessary (Sapkota et al., 2025; Kimi Team, 2025; GLM-4.5 Team, 2025).Source: §1 · pp. 1–2
  • Stop-RAG improves performance on multi-hop QA benchmarks while remaining compatible with black-box LLMs and existing iterative RAG pipelines.The controller frames stopping as a finite-horizon MDP and uses Q(λ) to estimate stopping quality.Source: §1 · p. 2

2 Related work

Iterative RAG methods use repeated retrieve-generate loops for complex queries, but fixed iteration budgets can waste computation or stop before enough evidence is gathered. Adaptive RAG methods use prompting, confidence signals, or trained stopping mechanisms; Stop-RAG instead estimates forward-looking value in a finite-horizon MDP.

2 Related work

  • Fixed iteration budgets can waste computation on simple questions or halt difficult ones before enough evidence is gathered.CoRAG learns step-wise retrieval and Iter-RetGen uses prior outputs to guide later retrieval, but both rely on fixed iteration counts (Wang et al., 2025; Shao et al., 2023).Source: §2 · p. 2
  • Adaptive RAG addresses fixed iteration budgets by dynamically deciding when to stop retrieval using query-specific characteristics and intermediate results; approaches use prompting, confidence signals, or trained stopping mechanisms.Examples include IRCoT and IterDRAG for prompting, FLARE and DRAGIN for confidence, and Self-RAG for trained stopping mechanisms (Trivedi et al., 2023; Yue et al., 2025; Jiang et al., 2023; Asai et al., 2024).Source: §2 · p. 2
  • Unlike methods using internal or present-focused signals, Stop-RAG frames stopping as a finite-horizon MDP to provide forward-looking value estimates.Adaptive-RAG operates only before execution and cannot be applied within the iterative loop (Jeong et al., 2024).Source: §2 · p. 3

3 Preliminaries

The section introduces value-based Q-learning for sequential decisions and the multi-step targets used to estimate action values over a finite horizon.

3 Preliminaries

  • Q-learning approximates the optimal action-value function and derives a greedy policy by choosing the action with the highest Q-value.The cited convergence result holds under mild assumptions.Source: §3 · p. 3
  • Unrolling a one-step bootstrap into n-step backups reduces bias at the expense of higher variance.Source: §3 · p. 3
  • Full-width backups evaluate all possible actions while sampling transitions, removing variance introduced by action-level sampling.The paper uses this construction when the action space is sufficiently small.Source: §3 · p. 3
  • Forward-view Q(λ) continuously interpolates between discrete n-step targets and serves as an alternative training target.The resulting method is called full-width forward-view Q(λ).Source: §3 · p. 3

4 Stop-RAG

Stop-RAG models iterative RAG as a finite-horizon MDP and uses full-width forward-view Q(λ) with offline trajectories to learn adaptive stopping decisions.

4 Stop-RAG

  • The method formulates iterative RAG as an MDP and applies full-width forward-view Q(λ) to derive a stopping strategy from offline data, without online interaction.Source: §4 · p. 4

4.1 Iterative RAG as an MDP

  • Each MDP state contains the question, answer, and interaction history; STOP terminates, while CONT runs another RAG iteration, with rewards only at terminal states.Reaching the maximum iteration limit also makes a state terminal.Source: §4.1 · p. 4

4.2 Stop-RAG training

  • Full-width action backups are tractable because the action space has two choices and STOP deterministically ends the process; the method assumes no discounting.Source: §4.2 · p. 4
  • Offline training data includes every prefix of full trajectories generated by continuing retrieval to the horizon, with rewards estimated from multiple answer trials scored against ground truth.Source: §4.2 · p. 5
  • The offline dataset removes terminal states and states with no STOP reward and no CONT target, since they provide no useful training signal.This filtering is described as stabilizing training.Source: §4.2 · p. 5
  • Training starts with λ = 1 and gradually reduces it, moving from long-horizon, higher-variance targets toward shorter-horizon, lower-variance backups.The schedule balances the bias-variance tradeoff to stabilize learning.Source: §4.2 · p. 5

5 Experiments

The experiments evaluate Stop-RAG across three multi-hop QA benchmarks, two iterative RAG pipelines, and comparisons with fixed-iteration and prompting-based stopping. Implementation uses a state-based Q-network and a validation-tuned margin rule to choose when to stop.

5.1 Evaluation setup

  • The evaluation uses MuSiQue, HotpotQA, and 2WikiMultihopQA, with exact match, F1, and accuracy as answer metrics.The authors sample 1,000 validation and 1,000 test questions from each development partition.Source: §5.1 · p. 6
  • For MuSiQue and 2WikiMultihopQA, the retrieval corpus aggregates contexts from the train, development, and test partitions.HotpotQA uses the processed Wikipedia corpus released by its authors.Source: §5.1 · p. 6

5.2 Baselines

  • The comparison tests raw fixed-iteration pipelines, LLM-Stop prompting, and Stop-RAG on both a reproduced CoRAG pipeline and a separately constructed pipeline.CoRAG represents a pipeline with generator fine-tuning; the constructed pipeline represents systems without generator training.Source: §5.2 · p. 6, Table 1 · p. 7

5.3 Implementation details

  • The constructed pipeline uses Llama-3.1-8B-Instruct for query and answer generation, Contriever-MSMARCO for retrieval, and bge-reranker-v2-m3 for reranking.It retrieves 10 documents per iteration, selects the top-ranked one, and permits up to 10 iterations; CoRAG uses its official checkpoint and prompts with the same retriever and corpus.Source: §5.3 · p. 7
  • Stop-RAG's DeBERTa-v3-large Q-network reads the question and retrieved documents, omitting intermediate queries and answers from its state.It has separate STOP and CONT heads; at inference, retrieval stops when the head-output margin exceeds a validation-tuned threshold.Source: §5.3 · p. 7

6 Results

Stop-RAG improves over fixed-iteration and LLM-Stop baselines in the authors' pipeline and also improves CoRAG on two benchmarks, with a HotpotQA exception. Analysis reports higher recall than LLM-Stop alongside slightly longer retrieval, and the Q(λ) objective performs best among the tested targets.

6.1 Main results

  • Stop-RAG outperforms both baselines across the authors' pipeline benchmarks and improves CoRAG on MuSiQue and 2WikiMultihopQA, but slightly underperforms on HotpotQA.The HotpotQA result is attributed in part to CoRAG's generator being fine-tuned for a fixed iteration count.Source: §6.1 · p. 7
  • HotpotQA's broader, more challenging retrieval corpus makes recall especially important, and full-iteration retrieval appears to outweigh the cost of added distractors.The authors suggest this may explain the fixed-iteration baseline's stronger performance on that dataset.Source: §6.1 · p. 8

6.2 Analysis

  • Compared with LLM-Stop, Stop-RAG takes slightly more retrieval steps on average, achieves higher recall, and yields better performance on MuSiQue and 2WikiMultihopQA.The analysis describes this as a better balance between retrieval depth and precision.Source: §6.2 · p. 8, Table 2 · p. 8
  • On MuSiQue, Q(λ) achieves the highest F1 and accuracy; MC has high variance, Q(0) is myopic, and the binary variant lags.The ablation compares annealed Q(λ) targets with Monte Carlo returns, one-step TD targets, and binary labels.Source: §6.2 · p. 8, Table 3 · p. 8

7 Conclusion

Stop-RAG uses value-based control to adaptively stop retrieval in iterative RAG. Across multiple multi-hop QA benchmarks, it consistently improves performance while balancing retrieval recall and precision and remaining compatible with black-box LLMs.

7 Conclusion

  • Stop-RAG frames iterative RAG as a finite-horizon MDP and trains with Q(λ) targets to estimate the benefits of continued retrieval.These estimates support more optimal stopping decisions than fixed-iteration or prompting-based baselines.Source: §7 · p. 8
  • Stop-RAG consistently improves performance across multiple multi-hop QA benchmarks while balancing retrieval recall and precision.It remains modular and compatible with black-box LLMs.Source: §7 · pp. 8–9
  • The results highlight value-based stopping control as effective for iterative RAG and adaptive decision-making as a potential direction for broader LLM-driven systems.Source: §7 · p. 9

A Performance summary of existing iterative RAG pipelines

Table 4 reports prior iterative RAG performance on MuSiQue, HotpotQA, and 2WikiMultihopQA, but methods vary in their prompts, retrievers, and generators, and reported results are inconsistent across datasets.

A Performance summary of existing iterative RAG pipelines

B Hyperparameters

The Stop-RAG Q-network is trained with standard settings, whose hyperparameters are listed in Table 5.

B Hyperparameters

  • Stop-RAG’s Q-network uses standard training settings.The passage directs readers to Table 5 for the full list.Source: B Hyperparameters · p. 13
  • Table 4 covers reported performance of existing iterative RAG methods on MuSiQue, HotpotQA, and 2WikiMultihopQA.Missing results are marked with a dash; gray highlighting marks lower-spec results that outperform larger variants.Source: Table 4 · p. 13
  • Table 5 lists the hyperparameters used to train the Stop-RAG Q-network.Source: Table 5 · p. 13

C Training objectives of ablation variants

The ablation compares Monte Carlo, one-step Q(0), and binary-classification training objectives for the stopping process.

C Training objectives of ablation variants

D Prompts

This section presents prompts for query generation, intermediate answer generation, and LLM-Stop, with bracketed placeholders to fill in.

D Prompts

  • Prompt 3 implements the LLM-Stop baseline, asking the generator to stop when current information supports a complete answer or continue when required information is missing.It evaluates gathered follow-up documents and intermediate answers against the original question.Source: D Prompts · pp. 14, 16
  • Final answer generation uses IRCoT-style prompting, following the original work’s prompt template and in-context examples..Source: D Prompts · p. 14
  • Query generation targets a specific missing fact needed to answer the original question and uses existing follow-up traces to generate the next question.The prompt asks for only the new question, without explanations or repeated information.Source: D Prompts · p. 15
  • Intermediate answers are generated concisely using only the Wikipedia snippet paired with each question.The prompt directs the model to output only the answer, without additional text.Source: D Prompts · p. 15

Related papers

Loading 2510.14337v1…