PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG

Jiaqi Wei, Wenxuan Huang, Chengdong Xu, Xiang Zhang, Wanli Ouyang

arXiv:2610.21184v1cs.CL

TL;DR

Multi-hop RAG can miss required evidence because a passage is unreachable or because search drifts away from an unanswered sub-question. ExecutionRAG uses a requirement register to guide search while keeping retrieval fixed; it reduces missing-evidence failures, but shifts the remaining bottleneck toward verifying evidence sufficiency.

  • Problem

    Standard retrieval evaluations conflate missing required passages that the retriever cannot reach with passages the agent never requests, obscuring distinct causes of apparent retrieval failure.

    Source: §2.1 · p. 2

  • Method

    ExecutionRAG maintains a register of unresolved sub-questions and supporting passages to guide search without changing the retriever, corpus, or generation model.

    Source: §1 · p. 2

  • Results

    382 to 234 and 383 to 274: on MuSiQue, wrong answers with unretrieved required evidence decrease under two fixed retrievers, while premature resolutions and abstentions increase.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Maintaining unresolved requirements helps retrieve reachable evidence, but correct answers still require more reliable verification after retrieval.

    Source: §6 · p. 9

    The register does not verify evidence reliably or make genuinely difficult passages retrievable, and missing-evidence failures remain highest on four-hop questions.

    Source: §4.4 · p. 5

Abstract

from arXiv · show

Multi-hop retrieval-augmented generation (RAG) requires an agent to retrieve evidence for a sequence of dependent sub-questions. When a wrong answer lacks a required passage, the passage may be inaccessible to the retriever, or the agent may have failed to issue a query that would retrieve it. Standard retrieval evaluations conflate these two causes. We study how much apparent retrieval failure is instead caused by search drift: the search no longer tracks an unanswered sub-question. Holding the retriever, corpus, and generation model fixed, we introduce ExecutionRAG, a training-free requirement register that records open sub-questions and their supporting evidence. The register changes search guidance but not evidence access, so each gold passage it recovers certifies that the passage was reachable by the original retriever. On 1,000 MuSiQue questions, the number of wrong answers with unretrieved required evidence falls from 382 to 234 with all-MiniLM-L6-v2 and from 383 to 274 with Qwen3-Embedding-0.6B. These counts correspond to 69% to 44% and 66% to 48% of each system's wrong answers, respectively. Crucially, evidence recovery does not produce a proportional reduction in total error: premature resolutions and abstentions increase as missing-evidence failures decrease. The central finding is therefore a shift in the remaining bottleneck, from guiding search toward required evidence to verifying that the retrieved evidence is sufficient.

1 Introduction

The paper argues that missing evidence in multi-hop RAG can reflect search drift, not just retriever inaccessibility. ExecutionRAG tracks unresolved requirements to guide search, reducing missing-evidence failures while exposing evidence verification as a remaining bottleneck.

1 Introduction

  • 66–69% of wrong answers by a strong agentic baseline omit required evidence, rising from 62% at two hops to 81% at four.This missing-passage symptom can arise even when the passage remains in the corpus, because search loses the intermediate target.Source: §1 · p. 1
  • Evidence recovery shifts the remaining bottleneck from search guidance toward checking whether retrieved evidence is sufficient.Explicit resolutions can be compared with gold supporting evidence to identify premature answers.Source: §1 · p. 2
  • 382 to 234 and 383 to 274: on MuSiQue, ExecutionRAG reduces wrong answers with unretrieved required evidence under two fixed retrievers.Premature resolutions and abstentions increase, so total error falls by a much smaller amount.Source: §1 · p. 2
  • ExecutionRAG records unresolved sub-questions and their supporting passages, guiding later searches without changing the retriever, corpus, or generator parameters.The register is initialized from the question and seed evidence; an answer is produced only when no item remains open.Source: §1 · p. 2

2 Problem Setup: What “Retrieval Failure” Hides

The section separates missing required evidence into retriever inaccessibility and search drift, then explains how a fixed retriever makes recovered evidence diagnostic. It proposes an explicit requirement register to keep open sub-questions in view and make their supporting passages inspectable.

2 Problem Setup: What “Retrieval Failure” Hides

  • Conventional RAG retrieves context for a query and generates an answer from that context, while multi-hop search must also track which sub-questions remain open.The register directs the next search toward an open item rather than repeatedly searching from the original question alone.Source: §2 · p. 2, §2.3 · p. 2

2.1 Two Causes Inside One Symptom

  • A missing gold passage may be unreachable by any query or unasked because the agent never issued a query that would retrieve it; only the former is a retrieval problem.Both causes look identical in a search trace because each ends with a missing passage.Source: §2.1 · p. 2

2.2 What a Fixed Retriever Identifies

  • Recovering a needed passage while retriever and corpus remain fixed certifies that the passage was reachable; failure to recover it proves nothing.Aggregate reductions are evidence of a recoverable component, not a numerical lower bound on search-drift prevalence, because systems may fail on different questions.Source: §2.2 · p. 2

2.3 The Probe: An Explicit Requirement Register

  • The probe is an explicit requirement register: for a question, it records the sub-questions the answer must satisfy and their status and supporting passages.Its purpose is to change search guidance while leaving retrieval unchanged.Source: §2.3 · p. 2

3 Method

ExecutionRAG changes what the agent searches for, not how retrieval works: it maintains a requirement register initialized from seed evidence and updates it as new passages arrive. It keeps searching until requirements are resolved or the budget runs out, while deterministic bookkeeping preserves the record.

3 Method

  • ExecutionRAG leaves the retriever, corpus, tool interface, and generator untouched, so recovered evidence was reachable by the same retriever.Its intervention is limited to search guidance.Source: §3 · p. 2

3.1 What the Register Records

  • Each register entry tracks a requirement’s status, evidence note, and supporting passages; an item is resolved only when enough evidence supports it.The four statuses are pending, active, resolved, and retired; retirement requires an explicit reason, and recording resolution makes the decision checkable against gold evidence.Source: §3.1 · pp. 2–3

3.2 Building the Register from Seed Evidence

  • The register is built from the question and an initial seed retrieval, grounding its requirements in evidence the corpus can actually return.One requirement becomes the first search target while the others remain pending.Source: §3.2 · p. 3

3.3 Updating the Register and Steering the Search

  • At each step, the agent selects an open item, queries for what remains missing, and updates items touched by the new evidence.Supporting passages are added; untouched items remain unchanged, and resolution or retirement requires sufficient support or an explicit reason.Source: §3.3 · pp. 3–4

3.4 When to Answer, and Two Bookkeeping Rules

  • The agent answers when no requirement remains open, or returns a bounded partial answer when its search budget runs out.It shows only progress deltas, and deterministic rules prevent open items from disappearing and previously recorded supporting passages from being erased; these rules do not ensure correct resolution.Source: §3.4 · p. 4

4 Experiments

The experiments test whether tracking unresolved sub-questions reduces missing-evidence failures under fixed retrieval, and whether the change persists across embeddings and benchmarks. Evidence recovery shifts remaining errors toward verification and answer commitment.

4.1 Experimental Questions

  • The study asks whether search guidance reduces missing-evidence failures, whether effects persist across embeddings, and whether context volume or register components explain them.It also tests answer accuracy, particularly when sub-questions must be combined across documents.Source: §4.1 · p. 4

4.2 Experimental Setup

  • The evaluation uses 1,000 questions each from HotpotQA, MuSiQue, and 2Wiki, with GPT-4o-mini generation and MiniLM top-5 retrieval in the main comparison.Qwen3-Embedding-0.6B is also tested; baselines span direct inference, vanilla RAG, and structured or agentic RAG.Source: §4.2 · p. 4

4.3 RQ1: Reduction in Failures Caused by Missing Evidence

  • 382 to 234 missing-evidence failures with MiniLM and 383 to 274 with Qwen show reductions under fixed-retriever comparisons.These counts are 69.3% to 43.9% and 66.4% to 47.9% of each method's failures, respectively.Source: §4.3 · p. 5

4.4 RQ2: Evidence Recovery Shifts the Remaining Failures

  • Reasoning-error shares rise from 9% to 26% with MiniLM and 8% to 21% with Qwen, while abstentions also increase.The register makes premature resolutions observable, but this comparison does not establish that the generation model reasons worse; four-hop missing-evidence failures remain most common.Source: §4.3 · p. 5, §4.4 · p. 5, Figure 2 · p. 6

4.5 RQ3: The Reduction Persists Across Embedding Models

  • Under Qwen3-Embedding-0.6B, ExecutionRAG exceeds A-RAG on all three benchmarks, with LLM-Evaluation Accuracy of 80.0, 68.3, and 42.8.A-RAG scores 76.4, 55.2, and 42.3, respectively; the largest gain is on 2Wiki.Source: §4.5 · p. 5, Figure 3 · p. 6

4.6 RQ4: Answer Accuracy

  • ExecutionRAG has the highest LLM-Evaluation Accuracy on all three benchmarks: 79.8 on HotpotQA, 74.3 on 2Wiki, and 50.2 on MuSiQue.Gains concentrate on 2Wiki and MuSiQue, which require maintaining dependent sub-questions across documents; long evidence chains remain difficult.Source: §4.6 · p. 5

4.7 RQ5: Retrieved Context and Accuracy

  • On 2Wiki, ExecutionRAG gets its largest accuracy improvement with comparable or fewer retrieved tokens, consistent with better targeting rather than more context.Search is limited to fifteen retrieval loops and 128,000 cumulative context tokens.Source: §4.7 · pp. 5–6

4.8 RQ6: Effects of Seed Grounding and Iterative Updates

  • Building requirements from seed evidence yields 50.2 LLM-Acc, versus 48.7 from the question alone; removing iterative updates lowers it to 48.3.Both seed grounding and iterative updates contribute to the observed gain.Source: §4.8 · p. 6

5 Related Work

Related work addresses evidence access, retrieval control, and evidence sufficiency; ExecutionRAG is positioned as a fixed-retriever empirical study of how an explicit register changes multi-hop failure patterns.

5 Related Work

  • Corpus-structuring approaches organize knowledge to improve retrieval, while ExecutionRAG instead tracks unresolved requirements during search [6] [8] [31] [12] [17] [54] [27] [26].The cited approaches include graphs, hierarchical retrieval trees, and graph-based retrieval; ExecutionRAG does not restructure the corpus.Source: §5 · pp. 6–7
  • Agentic methods optimize what to retrieve next, but ExecutionRAG adds no new retrieval policy and instead preserves open sub-questions until the current question is answered [35] [15] [1] [5] [4].Unlike persistent episodic-memory work, its register tracks only the current question's open requirements and is discarded after answering.Source: §5 · p. 7
  • Evidence-gap and sufficiency methods include corrective retrieval, sequential reading notes, and sufficiency checks [46] [49] [21] [29]; ExecutionRAG measures a register's effect with the retriever and generator fixed.The register records sub-questions, their states, and evidence provenance, and the study finds fewer missing-evidence failures but more post-retrieval reasoning errors.Source: §5 · p. 8
  • The implications distinguish search-guidance failures from retrieval-coverage failures and identify evidence verification as a separate target.The paper predicts that a verifier can improve answer accuracy without changing retrieved evidence, while improved coverage alone will not fix failures from unasked queries.Source: §5 · p. 8

6 Conclusion

The conclusion finds that maintaining unresolved requirements recovers reachable evidence but does not proportionally reduce answer errors. The appendices list further material on the algorithm, implementation, breakdowns, efficiency, failures, case studies, prompts, and broader impacts.

6 Conclusion

  • 382 to 234 and 383 to 274: on MuSiQue, the register reduces wrong answers with unretrieved required evidence under two retrievers, but premature resolutions and abstentions increase as missing-evidence failures decrease.The remaining bottleneck is verifying that retrieved evidence is sufficient to support a correct answer.Source: §6 · pp. 8–9

Appendix Overview

  • Appendices A, B, and G cover the inference algorithm, extended implementation details, and prompt templates.The implementation material includes model configuration, retrieval tools, budgets, and engineering choices.Source: Appendix Overview · p. 11
  • Appendices C through F cover per-category results, retrieval efficiency, failure analysis, and a case study of register-guided search; Appendix H discusses broader impacts.Source: Appendix Overview · p. 11

B Extended Implementation Details

The implementation uses GPT-4o-mini with semantic, keyword, and direct-chunk retrieval, alongside two fixed embedding backbones. It also specifies retrieval and context budgets, deduplication, progress updates, and budget-aware prompting.

B Extended Implementation Details

  • GPT-4o-mini is the default language model; retrieval uses semantic search, keyword search, and direct chunk lookup, with top-k=5 defaults for both search tools.Identical queries are deduplicated using normalized search keys and return previously retrieved chunk IDs.Source: B Extended Implementation Details · p. 11
  • The primary all-MiniLM-L6-v2 and alternative Qwen3-Embedding-0.6B indices are precomputed, with no re-indexing between runs.Evaluation uses the same retrieval corpora and 1,000-question splits from the three benchmarks for all baselines.Source: B Extended Implementation Details · p. 11
  • Each question allows at most 15 retrieval loops and a cumulative 128,000-token context budget; exceeding it disables further tool calls and forces an answer.Budget-exhausted answers are categorized as “Gave Up” in the error analysis.Source: B Extended Implementation Details · p. 11
  • Progress-delta injection sends only materially changed register items and reduces register-injection overhead by approximately 40% on 4-hop questions versus replaying the full list.Source: B Extended Implementation Details · p. 11
  • At 80% of the loop budget, a soft nudge encourages synthesis when evidence is sufficient or more targeted retrieval otherwise; Algorithm 1 also includes initial register, progress-delta, and budget-nudge injections.Source: B Extended Implementation Details · p. 11

C Per-Category Breakdown

The appendix tables report Contain-Match and LLM-Evaluation Accuracy by category across three benchmarks and two embedding backbones; the supplied captions identify the breakdowns but provide no category-level values.

C Per-Category Breakdown

D Retrieval Efficiency Analysis

The analysis compares ExecutionRAG’s retrieval efficiency with A-RAG and ablation variants, using accuracy alongside retrieved-token and cost measures.

D Retrieval Efficiency Analysis

  • The efficiency analysis compares ExecutionRAG with A-RAG and ablation variants using GPT-4o-mini, all-MiniLM-L6-v2, and full 1,000-question evaluation sets.The comparison covers three benchmarks.Source: D Retrieval Efficiency Analysis · p. 12, Table 9 · p. 12
  • The supplied category tables report accuracy by question type on HotpotQA and 2WikiMultiHopQA using Qwen3-Embedding-0.6B.Both tables use GPT-4o-mini and 1,000-question evaluations.Source: Table 7 · p. 12, Table 8 · p. 12
  • Retrieval efficiency is measured by average retrieved tokens per question and average API cost per question in USD, alongside accuracy.The ablation table reports tokens and cost alongside the accuracy numbers from Table 2.Source: Table 9 · p. 12, Table 10 · p. 12

E Failure Case Analysis

The failure analysis classifies wrong answers using MuSiQue’s gold supporting paragraphs and examines how failure categories change across embedding backbones and hop counts.

E Failure Case Analysis

  • Gold supporting paragraphs are mapped to corpus chunks by 40-character fingerprint matching, with paragraph-level recall of 87.9%.Hop-level evidence recall is the fraction of supporting paragraphs for which at least one matching chunk was retrieved.Source: E Failure Case Analysis · pp. 12–13
  • Each wrong answer receives one prioritized category: Judge Error, Gave Up, Reasoning Error, or Missing Evidence.Missing Evidence includes both unreachable and unasked evidence, while ExecutionRAG’s Reasoning Error also includes false resolution.Source: E Failure Case Analysis · p. 13
  • Failure percentages use each method’s own total failures; the MiniLM ExecutionRAG total slightly exceeds the count implied by main-table accuracy.The passage characterizes the reported reduction for that pairing as conservative.Source: E Failure Case Analysis · p. 13
  • Missing Evidence falls from 382 to 234 cases with all-MiniLM-L6-v2 and from 383 to 274 with Qwen3-Embedding-0.6B.These correspond to 69.3%→43.9% and 66.4%→47.9% of each method’s failures, respectively.Source: E Failure Case Analysis · p. 13, Table 12 · p. 14, Table 14 · p. 14
  • Reasoning Error becomes more prominent: 9.4%→25.9% with MiniLM and 7.8%→21.2% with Qwen.The category includes false resolution despite evidence gaps and incorrect synthesis when supporting evidence was retrieved.Source: E Failure Case Analysis · pp. 13–14
  • Gave Up rises from 6.5% to 13.1% under MiniLM and from 12.0% to 17.3% under Qwen.The analysis describes this as greater willingness to stop when tracked evidence remains insufficient.Source: E Failure Case Analysis · p. 14
  • Missing Evidence remains substantial at four hops, reaching 55.3% for ExecutionRAG with MiniLM; the hop-distribution tables report reductions at every hop level.The reported overall reductions are −25.4pp for MiniLM and −18.5pp for Qwen.Source: E Failure Case Analysis · p. 14, Table 13 · p. 14, Table 15 · p. 14

F Case Study: How the Register Guides the Search

Three MuSiQue cases illustrate how a requirement register anchors entities, sharpens follow-up retrieval, and allows correction when evidence contradicts an intermediate item; prompt templates specify the workflow.

Case 1: Preventing Entity Confusion (2-hop, Success)

Case 2: Completing a Long Dependency Chain (4-hop, Success)

  • Progress deltas surface newly answered items, so recording “Santa Monica” focuses the next search on the remaining gap in the four-hop chain.A-RAG instead repeatedly searches a flatter space and accumulates irrelevant context before stabilizing the decisive entity.Source: Case 2: Completing a Long Dependency… · p. 14, Table 17 · p. 15

Case 3: Self-Correction via Requirement Retirement (4-hop, Success)

A.1 Default QA Agent System Prompt

  • The default QA agent iteratively searches, reads, evaluates, and updates progress, using the register to track established and unresolved requirements.The prompts use GPT-4o-mini unless otherwise stated; the agent can use keyword search, semantic search, and chunk reading.Source: G Prompt Templates · p. 16, §A.1 · p. 16

A.2 Seed-Grounded Requirement Initialization Prompt

  • Initialization uses the question and seed evidence to create concrete, searchable requirements grounded in known evidence rather than speculation.The register includes requirement text, status, evidence notes, and one active target.Source: §A.2 · pp. 16–17

A.3 Requirement-State Injection

  • Requirement-state injection presents the current register as guidance without overriding agent autonomy, directing search toward unresolved requirements and away from unproductive repeats.Progress messages may report requirement-state changes after new evidence is read.Source: §A.3 · p. 17

A.4 Requirement Register Update Prompt

  • After each retrieval loop, the update prompt lets the LLM decide whether new evidence advances, resolves, or invalidates tracked requirements.The prompt requests a register update based on the newly retrieved information.Source: §A.4 · p. 17

A.5 Force Final Answer Prompt

  • At the loop or token budget limit, the force-final prompt disables further tool calls and requires an answer from the current evidence state.The prompt is appended when the agent reaches either limit.Source: §A.5 · p. 17

H Broader Impacts

The broader-impact discussion connects explicit intermediate requirements to multi-source agent workflows and notes limits on deployment where accountability and verification are unavailable.

H Broader Impacts

  • ExecutionRAG maintains an explicit register of open sub-questions during multi-step retrieval to improve multi-hop QA.The paper links explicit evidence support and provenance to transparency and auditability in workflows integrating information from multiple sources.Source: H Broader Impacts · p. 17
  • The method depends on its underlying retriever and LLM, so source-document biases or errors may propagate.The paper recommends deployment where accountability and verification are feasible, warning about automated decisions without human oversight.Source: H Broader Impacts · p. 17
Loading 2610.21184v1…