PaperlayerView exact revision on arXiv

Source-linked AI summary

S2G-RAG: Structured Sufficiency and Gap Judging for Iterative Retrieval-Augmented QA

Minghan Li, Junjie Zou, Xinxuan Lv, Chao Zhang, Guodong Zhou

arXiv:2604.23783v1cs.IRcs.AI

TL;DR

Multi-hop QA requires iterative systems to judge whether accumulated evidence is sufficient and what to retrieve next, while noisy retrieval contexts can destabilize control. S2G-RAG uses a structured judge to predict sufficiency and information gaps, alongside compact sentence-level evidence selection. It improves EM/F1 across three QA benchmarks under BM25 and E5 retrieval, with the authors noting that sentence selection may omit useful context or multi-sentence evidence.

  • Problem

    Iterative RAG must decide whether accumulated evidence supports answering and what to retrieve next, while noisy contexts and under-specified information needs challenge reliable turn-level control.

    Source: §1 · pp. 1–2

  • Method

    S2G-RAG uses S2G-Judge to predict evidence sufficiency and structured gaps for the next query, and maintains a compact sentence-level Evidence Context.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    S2G-RAG improves EM/F1 on TriviaQA, HotpotQA, and 2WikiMultiHopQA under both BM25 and E5 retrieval.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The judge and sentence-level extraction can be integrated into existing RAG pipelines as lightweight components without modifying the search engine or retraining the generator.

    Source: §5 · p. 9

    Sentence-level extraction may omit useful surrounding context or evidence spanning multiple sentences, and sufficiency predictions can be conservative.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) grounds language models in external evidence, but multi-hop question answering remains difficult because iterative pipelines must control what to retrieve next and when the available evidence is adequate. In practice, systems may answer from incomplete evidence chains, or they may accumulate redundant or distractor-heavy text that interferes with later retrieval and reasoning. We propose S2G-RAG (Structured Sufficiency and Gap-judging RAG), an iterative framework with an explicit controller, S2G-Judge. At each turn, S2G-Judge predicts whether the current evidence memory supports answering and, if not, outputs structured gap items that describe the missing information. These gap items are then mapped into the next retrieval query, producing stable multi-turn retrieval trajectories. To reduce noise accumulation, S2G-RAG maintains a sentence-level Evidence Context by extracting a compact set of relevant sentences from retrieved documents. Experiments on TriviaQA, HotpotQA, and 2WikiMultiHopQA show that S2G-RAG improves multi-hop QA performance and robustness under multi-turn retrieval. Furthermore, S2G-RAG can be integrated into existing RAG pipelines as a lightweight component, without modifying the search engine or retraining the generator.

1 Introduction

Multi-hop QA requires later retrieval to build on intermediate evidence, but iterative systems must decide both whether evidence is sufficient and what to retrieve next. S2G-RAG makes those decisions explicit with a structured judge and compact, gap-aware evidence context; the paper reports improved EM/F1 across three QA benchmarks.

1 Introduction

  • Iterative RAG must assess evidence sufficiency and identify the next information need, or risk incomplete answers and unfocused retrieval that accumulates distractors.Multi-hop retrieval depends on intermediate findings; generic follow-up queries can repeatedly retrieve near-duplicate pages.Source: §1 · p. 1
  • S2G-Judge makes per-turn control explicit by predicting whether evidence supports answering and, if not, producing structured gaps for the next query.The controller is decoupled from answer generation.Source: §1 · p. 2
  • S2G-RAG distills turn-level supervision from multi-turn execution traces, using intermediate evidence contexts to train the judge.A teacher labels snapshots with sufficiency decisions and gap items under a context-only evidence constraint.Source: §1 · p. 2
  • A gap-aware sentence-level Evidence Context retains compact, auditable evidence and prioritizes sentences relevant to predicted gaps to reduce distractor interference.The introduction presents this as a complement to structured retrieval control.Source: §1 · p. 2
  • S2G-RAG improves EM/F1 over strong baselines across TriviaQA, HotpotQA, and 2WikiMultiHopQA under sparse and dense retrieval settings.The framework pairs explicit sufficiency and gap predictions with a compact sentence-level evidence context.Source: §1 · p. 2

2 Related Work

Related work moves from single-round RAG toward iterative retrieval and reasoning for questions whose evidence spans documents. It covers approaches to query generation, stopping, turn-level supervision, and context management.

2.2 Multi-Round RAG

  • Multi-round RAG alternates retrieval and reasoning because one pass often misses distributed evidence and later-hop retrieval depends on earlier findings.Standard RAG remains effective when one retrieval pass covers the required evidence.Source: §2.2 · p. 2, §2.1 · p. 2
  • Prior multi-round work addresses query generation, stopping, turn-level supervision, and context management, but these lines leave gaps in specifying and auditing next-step information needs.Free-form queries can drift under noisy contexts; stopping signals may lack standardized states, supervision faces distribution mismatch, and compression alone does not specify what to retrieve next.Source: §2.2 · pp. 2–3

3 Methodology

S2G-RAG makes iterative multi-hop retrieval control explicit through turn-level sufficiency judgments and structured gaps, while maintaining a compact evidence context. Its method combines a judge-first loop, trajectory-distilled judge training, gap-guided query construction, and sentence-level extraction.

3.1 S2G-RAG Inference Framework

  • S2G-Judge checks whether the current context supports answering; otherwise, gap items guide retrieval, extracted sentences update the context, and a reasoner answers from it.The loop can stop when evidence is sufficient or the retrieval budget is reached.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2.1 Output Schema and Constraints

  • Each gap item records a category, target, slot, and clarification, while sufficiency judgments must rely strictly on evidence in the current context.When evidence is sufficient, the judge outputs an empty gap set.Source: §3.2.1 · p. 4

3.2.2 Training via Trajectory Distillation from Execution Traces

  • S2G-Judge is fine-tuned on structured labels from multi-turn execution traces, using teacher judgments, conflict filtering, and LoRA-based supervised training.The traces expose the judge to intermediate contexts that include accumulated redundancy and distractors.Source: §3.2.2 · p. 4

3.3 Gap-Guided Query Construction

  • When evidence is insufficient, predicted gaps become query phrases, prioritizing target-plus-slot when available and otherwise using the description before appending phrases to the original question.The system uses up to the first K valid phrases in the judge's order, falling back to the original question if none can be formed.Source: §3.3 · p. 4

3.4 Sentence-level Evidence Extractor

  • The sentence-level extractor selects salient sentences from retrieved documents using the question and predicted gaps, then appends a compact evidence block to the accumulated context.Its index-based selection maps back to original sentences and is designed to retain answer-bearing, bridging, and disambiguating evidence while limiting redundant or distractor-heavy text.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate S2G-RAG on three QA benchmarks with sparse and dense retrieval, finding gains over strong baselines and benefits from explicit control and compact evidence memory. Ablations and analyses examine the judge, extractor, sufficiency predictions, compression, and retrieval-budget behavior.

4.1 Task and Datasets

  • Evaluation uses TriviaQA for single-hop QA and HotpotQA and 2WikiMultiHopQA for multi-hop QA, reporting EM/F1 with BM25 or E5 retrieval on official development splits.Unless otherwise noted, runs use at most four turns, six documents per turn, and title-based de-duplication; baselines include single-round RAG and multi-round retrieval-control methods.Source: §4.1 · p. 5, §4.2 · p. 5, §4.3 · p. 5

4.4 Main Results

  • Under matched BM25 settings, S2G-RAG improves over SIM-RAG by +1.3 EM/+2.3 F1 on TriviaQA, +10.6 EM/+13.2 F1 on HotpotQA, and +7.6 EM/+8.4 F1 on 2WikiMultiHopQA; it also leads the dense-retrieval panel on all three benchmarks.The comparison covers end-to-end EM/F1 with the same answer reasoner; the E5 results include comparisons with Standard RAG and learned-control baselines such as RAG-Critic.Source: §4.4 · p. 6, Table 1 · p. 6

4.5.1 Ablation Study

  • Removing S2G-Judge causes the largest HotpotQA drop (−15.8 EM, −19.0 F1), while removing the extractor drops scores by −3.8 EM and −4.0 F1.Replacing the trained judge with its unfine-tuned backbone drops performance by −4.1 EM and −5.7 F1; that variant still exceeds prior BM25 baselines.Source: §4.5.1 · p. 6, Table 2 · p. 7

4.5.2 Analysis on Sufficiency Predictions

  • On HotpotQA, S2G-Judge’s false-positive rate for sufficiency is 6.44%, while 31.60% of instances satisfying the retrieval-only truth label are predicted insufficient.The truth label requires accumulated retrieved titles to cover all gold supporting document titles; the pattern indicates more under-claiming than premature sufficiency declarations.Source: §4.5.2 · p. 7, Figure 2 · p. 7

4.5.3 Efficiency of Evidence Memory Compression

  • Evidence Context is roughly 4.5×–6.4× more compact than raw concatenated context, and sentence-level extraction reduces per-sample latency from 1.9552 to 1.6085 seconds while improving EM/F1 by +3.8/+4.0.The latency change corresponds to a 17.7% reduction; compression benefits become more pronounced for examples ending after more retrieval turns.Source: §4.5.3 · p. 7, Table 3 · p. 7, Figure 3 · p. 8

4.5.4 Comparison with Evidence Compression Baselines

  • Against LLM summarization and ReComp, the extractor achieves the best EM/F1 while retaining a compression ratio of 0.3461; abstractive compression shortens contexts but lowers EM/F1.The comparison uses the same iterative QA pipeline, varying only the evidence-compression module.Source: §4.5.4 · pp. 7–8, Table 4 · p. 8

4.5.5 Teacher–Student Gap under Varying Retrieval Budgets

  • As the maximum retrieval budget increases, F1 gains are large at small budgets and then saturate; the trained judge closely tracks the teacher across the sweep.The unfine-tuned controller performs worse and scales less reliably, while NaiveGen remains flat across budgets; once key evidence is gathered, additional turns are less likely to add answer-relevant information.Source: §4.5.5 · p. 8, Figure 4 · p. 8

5 Conclusion

S2G-RAG makes iterative retrieval control explicit through structured sufficiency judgments and gap items, and uses sentence-level evidence extraction to preserve a compact context. The paper reports improvements across three QA datasets under two retrieval methods.

5 Conclusion

  • S2G-Judge makes two turn-level decisions explicit: whether evidence is adequate and, if not, what information remains missing.Its structured gap items provide a concrete basis for constructing the next retrieval query.Source: §5 · pp. 8–9
  • Sentence-level extraction retains a compact, provenance-preserving Evidence Context for later judging and final answering.This complements explicit retrieval control under distractor-heavy retrieval.Source: §5 · p. 9
  • S2G-RAG improves EM/F1 on TriviaQA, HotpotQA, and 2WikiMultiHopQA under both BM25 and E5 retrieval.The paper's analysis suggests the judge reduces answers made under insufficient evidence, while sentence-level extraction limits redundant or distracting text.Source: §5 · p. 9

Limitations

The limitations concern the gap schema's expressiveness, evidence extraction's compactness–recall trade-off, and conservative sufficiency predictions. Supplementary results also report compatibility with a stronger downstream reasoner and extractor setting.

Limitations

  • The gap-item schema favors stability and ease of use over expressiveness, so complex queries may require structures beyond its current fields.Examples include multi-entity joins, temporal constraints, and relations spanning several intermediate variables.Source: Limitations · p. 9
  • Sentence-level extraction may omit useful surrounding context or evidence spanning multiple sentences, while sufficiency predictions can be conservative.The authors identify improved calibration and adaptive evidence selection as future directions.Source: Limitations · p. 9

A Additional Results

  • The appendix reports supplementary results using the same retrieval corpus and evaluation protocol as the main experiments unless otherwise noted.Each subsection changes only the stated component.Source: A Additional Results · p. 11

A.1 Strong Reasoner Setting

  • Under a stronger reasoner setting, reported gains indicate compatibility with stronger downstream components without changing the retriever or retraining the judge.Table 5 replaces the answer reasoner and sentence-level evidence extractor with GPT-4o-mini while keeping retrieval the same.Source: §A.1 · p. 11, Table 5 · p. 11

A.2 Query Construction Variants

The query-construction analysis favors structured gap-based queries over free-text additions and finds modest sensitivity to the number of gap items. A separate backbone comparison reports similar end-to-end accuracy across two lightweight judges.

A.2 Query Construction Variants

  • Structured queries improve EM by 1.8 points, F1 by 2.0 points, and final-turn Correct Retrieval by 2.2 points over free-text queries.The comparison is on HotpotQA dev under BM25; structured queries form phrases from missing-fact fields such as target and slot.Source: §A.2 · p. 11, Table 6 · p. 11
  • Across K ∈ {1, 2, 3}, EM/F1 varies within 1 point and Correct Retrieval stays within 52.4–52.7%.K=1 yields the best EM/F1, with modest differences; the authors default to K=1 for concise queries and strongest average accuracy.Source: §A.2 · p. 11, Table 6 · p. 11

A.3 Effect of Judge Backbone

  • The Llama-3.2-3B-Instruct and Qwen-3-4B-Instruct judge backbones yield similar end-to-end accuracy across datasets.Qwen-3-4B gives small TriviaQA and HotpotQA EM gains with comparable F1, while 2WikiMultiHopQA results are close; other components are held fixed.Source: §A.3 · p. 11, Table 7 · p. 11

A.4 Robustness to Retrieval Breadth

S2G-RAG's EM/F1 remains stable across a wide range of per-turn BM25 retrieval breadth. The authors attribute this behavior largely to sentence-level evidence management that limits irrelevant text in the accumulated context.

A.4 Robustness to Retrieval Breadth

  • EM/F1 remains stable with only modest fluctuations as BM25 top-k increases across the tested per-turn retrieval breadth.Figure 5 varies k ∈ {2, 4, 6, 8, 10, 20} while holding the judge, query construction, extractor, maximum turns, and decoding settings fixed.Source: §A.4 · p. 12, Figure 5 · p. 12
  • Sentence-level extraction forms a compact evidence block, limiting irrelevant text as wider retrieval adds increasingly redundant or distractor-heavy documents.The authors say this helps keep gap identification and answer generation stable as k grows.Source: §A.4 · p. 12

A.5 Effect of Evidence Extraction Cap Ke

The extractor cap has little effect on HotpotQA performance across the tested range, and the method performs well on both Bridge and Comparison questions.

A.5 Effect of Evidence Extraction Cap Ke

  • On HotpotQA (dev) under BM25, EM varies within 43.2–43.4 and F1 within 55.8–56.5 across caps, with no clear monotonic trend.The extractor may return fewer than the cap; other components are held fixed.Source: §A.5 · p. 12, Table 8 · p. 12
  • Ke is a ceiling rather than a target, and modest caps can preserve key two-hop evidence while limiting distractor text.HotpotQA instances are typically supported by a small number of key evidence sentences.Source: §A.5 · p. 12

A.6 Breakdown by HotpotQA Question Type

  • S2G-RAG performs well on both Bridge and Comparison questions, so its effectiveness is not restricted to bridge-style reasoning.Table 9 reports the HotpotQA breakdown by question type.Source: §A.6 · p. 12, Table 9 · p. 12

B Training Details

S2G-Judge training uses turn-level pipeline traces and teacher labels to supervise sufficiency decisions and structured gaps. Filtering checks output format and conflicts without supplying gold signals at inference.

B Training Details

  • Unspecified training settings use library defaults.The appendix details turn-level supervision construction and LoRA-based supervised fine-tuning.Source: B Training Details · p. 12

B.1 Turn-Level Supervision Construction

  • Training traces roll out the iterative pipeline to a fixed maximum budget and record each turn’s question and accumulated evidence context.An unfine-tuned judge backbone under the final prompt and schema produces intermediate contexts containing natural redundancy and distractors.Source: §B.1 · pp. 12–13
  • A snapshot is initially tagged sufficient only when retrieved titles cover all gold supporting-document titles; otherwise it is tagged insufficient.A stronger teacher labels each snapshot using only its accumulated evidence context, producing a binary decision and structured gap items.Source: §B.1 · p. 13

B.2 Supervision Filtering

  • Filtering removes malformed teacher outputs and low-confidence pairs with conflicts between the teacher label and a retrieval-derived sufficiency tag.The gold-derived tag is only a weak sanity check and is never given to the judge or used at inference.Source: §B.2 · p. 13

B.3 Supervision Dataset

The section describes the cleaned turn-level supervision set, its split, and the LoRA and supervised fine-tuning setup used for S2G-Judge.

B.3 Supervision Dataset

  • The cleaned supervision set contains 2,804 snapshots, with a relatively even distribution across turns 1–4 and approximately balanced sufficiency labels.Table 10 summarizes the cleaned set’s statistics.Source: §B.3 · p. 13, Table 10 · p. 13
  • A random split with seed 42 yields 2,523 training snapshots and 281 validation snapshots from the cleaned set.The split assigns 90% to training and 10% to validation.Source: §B.3 · p. 13

B.4 LoRA Configuration

  • LoRA adapters are inserted into attention and feed-forward projection layers in each decoder block for parameter-efficient adaptation.The listed target modules are q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.Source: §B.4 · p. 13

B.5 Supervised Fine-Tuning Setup

  • Supervised fine-tuning uses chat-formatted prompts, gradient accumulation 8, and a maximum sequence length of 2048.Evaluation and checkpointing occur every 200 steps, the latest three checkpoints are kept, and adapter weights and tokenizer are saved for inference.Source: §B.5 · pp. 13–14

C Pseudocode for the Evaluation Pipeline

The evaluation loop repeatedly judges accumulated sentence-level evidence, retrieves against predicted gaps when needed, and stops at sufficiency or the turn limit. Case studies illustrate both successful gap refinement and failure from retrieval coverage and ambiguity.

C Pseudocode for the Evaluation Pipeline

  • The evaluation loop stops when the judge predicts sufficient evidence or the maximum turn budget is reached, then reasons over the accumulated context.Otherwise, it builds a query from structured gaps, retrieves documents, extracts sentences, appends them to context, and records the trace.Source: C Pseudocode for the Evaluation Pipeline · p. 14

D Case Studies

  • In the successful two-hop case, a relation gap surfaces Michael Swango, then an evidence-span gap verifies the remaining constraint before the system answers correctly.Table 11 describes the sequence as a bridge-entity gap followed by verification of the remaining constraint.Source: D Case Studies · p. 14, Table 11 · p. 15
  • In the failure case, retrieval never finds Valhalla Highlands Historic District and instead accumulates evidence for Cold Spring Historic District before the budget runs out.The judge remains insufficient throughout, but the final prediction is incorrect.Source: D Case Studies · p. 14, Table 12 · p. 15

E.3 Sentence-level Evidence Extractor Prompts

  • The S2G-Judge prompt specifies a binary sufficiency decision and structured gap items when evidence is insufficient; teacher prompts label training snapshots.Tables 13–15 provide the judge and teacher prompts.Source: Table 13 · p. 16, Table 14 · p. 16, Table 15 · p. 17
  • The Evidence Extractor prompt returns sentence indices as pointers rather than rewriting evidence text.Table 16 describes the sentence-level extractor prompt.Source: Table 16 · p. 17

Related papers

Loading 2604.23783v1…