PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing the Fact-Grounding Gap in Multi-Hop Question Answering

Kevin Mo, Nathan Mo, Richard Zhu

arXiv:2609.17043v1cs.CLcs.AIcs.IR

TL;DR

Multi-hop QA research has largely examined whether the right documents are retrieved, not whether they contain the facts needed at each reasoning step. The paper measures hop-level fact presence across three benchmarks and finds extraction failures alongside retrieval failures. These extraction failures remain unresolved by retrieval interventions, revealing a limit to retrieval-only improvements.

  • Problem

    Document-level retrieval analyses leave unclear whether retrieved passages contain the specific facts required at each reasoning step.

    Source: §1 · p. 1

  • Method

    The paper measures per-hop fact presence across three benchmarks and uses a human-validated LLM judge and a lightweight predictor to identify deficient hops.

    Source: §1 · p. 2, §5 · p. 5

  • Results

    Extraction failures reach 47% of all per-hop deficiencies on MuSiQue, recur across the evaluated benchmarks, and are not recovered by tested retrieval interventions.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Retrieval-only approaches cannot recover a fact missing from a retrieved passage, so retrieval failures and extraction failures require different solutions.

    Source: §1 · p. 1, §7 · p. 9

    The study uses a single reasoning model, so specific failure rates and intervention effects may differ with other reasoning models.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multi-hop question answering requires combining information from multiple documents to answer complex questions. These systems have grown increasingly capable, yet when they fail, the error is typically attributed to not finding the right documents. Whether this holds at the level of individual reasoning steps remains largely unexamined. We investigate this across three standard multi-hop QA benchmarks and find that failures decompose into two distinct modes: retrieval failures, where the needed passage was not retrieved, and extraction failures, where the passage was retrieved but the needed fact could not be extracted - a phenomenon we term the fact-grounding gap. Extraction failures account for nearly half of all per-hop deficiencies and are invisible to standard retrieval metrics. They remain unresolved by every retrieval intervention we test, establishing a ceiling for retrieval-only improvements. The gap's severity varies across benchmarks and question types, but extraction failures appear on every dataset we measure. Our findings reveal that retrieval failures and extraction failures are fundamentally different bottlenecks requiring different solutions - a distinction absent from current evaluation practice.

1 Introduction

The introduction argues that multi-hop QA research commonly treats retrieval quality as the bottleneck, while document-level evaluation leaves open whether retrieved passages contain the facts needed at each hop. The paper measures this gap across three benchmarks and finds extraction failures persist despite retrieval interventions.

1 Introduction

  • Whether retrieved passages contain the specific facts needed at each reasoning step has not been systematically studied.Prior analyses ask whether the right document was retrieved, rather than whether it contains the relational fact required by the current hop.Source: §1 · p. 1
  • The study measures hop-level fact presence across three standard multi-hop QA benchmarks and distinguishes retrieval failures from extraction failures.Figure 1 illustrates the distinction: a retrieved gold passage can still omit the fact needed for a hop.Source: §1 · p. 1, Figure 1 · p. 2
  • Extraction failures account for 47% of per-hop deficiencies on MuSiQue and persist across every tested retrieval intervention.These failures occur when the needed passage is retrieved but does not contain the needed fact.Source: §1 · p. 2
  • Retrieval augmentation and reranking do not resolve extraction failures, establishing a ceiling for retrieval-only approaches.A passage can be retrieved while still lacking the fact needed for the reasoning step.Source: §1 · pp. 1–2
  • Targeted re-retrieval on predictor-flagged hops matches every-hop intervention at half the cost.The fact-presence evaluation uses an LLM judge validated against human annotation (κ = 0.840).Source: §1 · p. 2

2 Related Work

Related work improves multi-hop retrieval and evaluates whether the right documents were found, but does not generally test whether those documents contain the facts needed at each reasoning step. The paper locates its contribution at that finer granularity and uses a human-validated LLM judge in its evaluation approach.

2 Related Work

  • Multi-hop systems commonly decompose, retrieve, and reason iteratively, while assuming that surfacing the right documents is sufficient; this paper examines a shared failure mode rather than proposing a new pipeline.The cited systems include IRCoT, Self-Ask, and ReAct.Source: §2 · p. 2
  • The Figure 2 pipeline decomposes each question, retrieves passages, checks fact presence with a DeBERTa predictor, and retrieves supplementary passages when a hop is flagged insufficient.Source: Figure 2 · p. 3
  • Retrieval methods are commonly evaluated by whether the right documents appear, not whether they contain the fact required for the current reasoning step.The paper distinguishes this per-hop fact-presence question from document-level retrieval metrics and system-level retriever-versus-reader decompositions.Source: §2 · pp. 2–3
  • LLM-based fact-presence judging is validated against human annotation (κ = 0.840), and its labels train a lightweight classifier with a documented conservative bias.The paper follows recommendations to validate LLM judges against task-specific human annotations before use.Source: §2 · p. 3

3 Experimental Setup

The experimental setup evaluates three standard multi-hop QA benchmarks with a Self-Ask-style reasoning pipeline and judges answer correctness using an LLM validated against human annotations.

3 Experimental Setup

  • The study evaluates MuSiQue-Ans, HotpotQA, and 2WikiMultihopQA, developing and validating its methodology on MuSiQue before applying it to the other benchmarks.MuSiQue provides gold question decompositions, intermediate answers, and supporting paragraph indices.Source: §3 · p. 3
  • A Self-Ask-style pipeline uses GPT-4.1-mini and retrieves k = 3 passages per hop with BM25, with additional Contriever runs on MuSiQue and HotpotQA.Retrieved passages accumulate across hops and are deduplicated by content.Source: §3 · p. 3
  • Answer correctness is judged by GPT-4.1-mini, which is validated against human annotation on 200 stratified examples (Cohen’s κ = 0.840).The LLM judge accommodates synonyms, verbose answers, and partial overlaps beyond exact-match scoring.Source: §3 · p. 3

4 The Fact-Grounding Gap

This section measures whether retrieved passages contain each hop’s needed fact and distinguishes missing gold passages from retrieved passages that lack the required fact.

4.1 Measuring Per-Hop Fact Presence

  • For each sub-question, an LLM judge evaluates deduplicated passages accumulated through that hop without seeing the expected intermediate answer.On 200 examples, judge–human agreement was 92.0% (κ = 0.840); disagreements skewed toward marking answerable hops as NOT-ANSWERABLE.Source: §4.1 · p. 4
  • String matching overstates fact presence: on MuSiQue, it reports 59.8% answerability versus 49.1% for the LLM judge, a 10.7-point gap.An answer string may appear without the relational fact required by the hop.Source: §4.1 · p. 4

4.2 Results

  • 49.1% of 6,404 MuSiQue hop steps are answerable from retrieved passages, falling from 68.5% at hop 1 to 18.5% at hop 4.Later-hop evidence cannot be retrieved ahead of time when it concerns entities revealed during earlier reasoning.Source: Table 1 · p. 4, §4.2 · p. 4
  • 30.7% of hops are retrieval failures and 27.3% are extraction failures, separating missing gold passages from retrieved passages that lack the needed fact.The decomposition covers 6,401 hops; document-recall metrics can count extraction failures as successful retrievals.Source: §4.2 · p. 4, Table 2 · p. 5
  • Extraction failures account for 47% of per-hop deficiencies and stay around 9% at each hop depth, while retrieval failures drive the sharp depth-related decline in fact presence.Even a perfect retriever would leave nearly half of per-hop deficiencies unresolved.Source: §4.2 · p. 5
  • Retrieval recall and fact presence are nearly uncorrelated among incorrectly answered hops (ρ = 0.042), and the best nonoracle predictor reaches only 68.1% accuracy.That leaves a 15-point gap to oracle baselines.Source: Table 3 · p. 5, §4.2 · p. 5

4.3 Robustness to Retriever Choice

  • Under dense retrieval, extraction failures persist: MuSiQue counts rise from 598 with BM25 to 725 with Contriever, and 77% of BM25 failures also occur under Contriever.Overall MuSiQue accuracy is nearly unchanged (51.2% vs. 51.9%); HotpotQA shows the same persistence pattern (3.1% vs. 4.5% extraction failures).Source: §4.3 · p. 5

5 Learning to Predict Fact-Presence Deficiency

This section trains a lightweight classifier to identify hops whose retrieved passages lack the needed fact. Its results show that fact-presence prediction is learnable and that flagged-hop counts track QA accuracy.

5 Learning to Predict Fact-Presence Deficiency

  • The classifier predicts whether accumulated retrieved passages contain a hop’s needed fact, using the sub-question and passages as input.It learns from LLM-judge labels as a smaller, more practical alternative to judging each hop individually.Source: §5 · p. 5

5.1 Training Setup

  • The DeBERTa-v3-large classifier is trained on 46,610 MuSiQue hop examples labeled by the LLM judge and evaluated on 6,404 held-out dev examples.At inference, hops with predicted probability of at least 0.5 are classified as ANSWERABLE.Source: §5.1 · p. 5

5.2 Classification Performance

  • DeBERTa-v3-large achieves the highest F1 (0.785), and all three predictors outperform the 68.1% retrieval-recall accuracy baseline.The comparison covers BERT-base, RoBERTa-large, and DeBERTa-v3-large on MuSiQue dev.Source: §5.2 · pp. 5–6, Table 4 · p. 6
  • The DeBERTa predictor flags 3,201 of 6,404 hops (50.0%) as NOT-ANSWERABLE, close to the ground-truth rate of 50.9%.The authors suggest the predictor could replace the LLM judge at lower computational cost.Source: §5.2 · p. 6
  • LLM-judge labels improve accuracy by 8.9 points over string-match labels, while adding retrieved passages yields a further 15.8-point improvement.The ablation indicates that answer-string presence alone is unreliable and the classifier must read passage content.Source: Table 5 · p. 6, §5.2 · p. 6
  • QA accuracy drops from 68.7% with zero flagged hops to 8.5% with four flagged hops on MuSiQue dev.Accuracy declines steadily as the number of flagged hops increases.Source: Table 6 · p. 6, §5.2 · p. 6

6 Intervention Experiments

The experiments test whether predictor-guided re-retrieval helps specifically at fact-deficient hops, using augmentation and reranking across three datasets. Targeting generally matches or improves on every-hop intervention at lower cost, but gains vary by dataset, question depth, and failure type.

6.1 Experimental Setup

  • The experiments use the Self-Ask pipeline and compare baseline, every-hop, predictor-flagged-hop, and oracle re-retrieval under a shared corpus and reasoning model.Augmentation adds three passages retrieved from a rephrased sub-question; reranking selects the top three from 20 candidates.Source: §6.1 · pp. 6–7

6.2 Main Results

  • On MuSiQue with augmentation, intervening only at predictor-flagged fact-deficient hops improves accuracy +3.7% over baseline, versus +2.7% for every-hop intervention, despite using half the re-retrieval calls.Source: §6.2 · p. 7
  • On HotpotQA, the MuSiQue-trained predictor improves accuracy by +5.1% over baseline while flagging 30.7% of hops; every-hop intervention gains +9.0%.With Contriever, re-retrieval improves accuracy by +3.5% on MuSiQue and +10.0% on HotpotQA.Source: §6.2 · p. 7, Table 7 · p. 7

6.3 Flag-Count Degradation

  • As flagged-hop count rises, MuSiQue baseline accuracy falls from 68.7% at zero flags to 8.5% at four, while intervention gains rise from +0.4% to +17.0%.HotpotQA shows the same direction without retraining: gains rise from +0.1% at zero flags to +11.8% at two.Source: §6.3 · p. 8, Figure 3 · p. 8

6.4 Selectivity: When Targeting Matters

  • At three-hop questions, every-hop intervention drops accuracy 2.1% below baseline, while targeted intervention raises it by 2.1%, a 4.2-point difference.Every-hop intervention performs better at two hops; both strategies produce similar large gains at four hops.Source: §6.4 · p. 8

6.5 Failure Mode Decomposition

  • Re-retrieval improves accuracy on retrieval failures by +8.6% with augmentation and +25.3% with reranking, but extraction-failure accuracy changes by only −0.5%.The gold passage was already retrieved in extraction-failure cases.Source: §6.5 · p. 8
  • For extraction failures, the needed relational fact is absent from corpus passages containing the answer string in 97.7% of MuSiQue, 100% of HotpotQA, and 97.6% of 2WikiMultihopQA cases.Extraction failures account for 27.3% of all hops, establishing a ceiling for retrieval-only improvements.Source: §6.5 · p. 9

6.6 Entity Familiarity and Limits of Transfer

  • On 2WikiMultihopQA, intervention yields no meaningful gains (78.8% baseline, 78.9% DeBERTa); extraction failures are 2.7% for comparison questions and 16.6% for multi-step questions.Retraining does not improve the result, and the passage attributes the small aggregate gap partly to well-known entities and extensive Wikipedia coverage.Source: §6.2 · p. 7, §6.6 · p. 9

7 Conclusion

The conclusion identifies the fact-grounding gap as a recurring multi-hop QA failure: retrieved passages can be topically relevant yet lack the facts needed for reasoning. Across the evaluated settings, retrieval interventions do not recover missing facts, leaving a ceiling for retrieval-only approaches.

7 Conclusion

  • The fact-grounding gap describes retrieved documents that lack the specific facts needed for individual reasoning steps.Per-hop analysis across MuSiQue, HotpotQA, and 2WikiMultihopQA distinguishes retrieval failures from extraction failures.Source: §7 · p. 9
  • On MuSiQue, extraction failures account for 27.3% of all hops and are invisible to standard retrieval metrics.Source: §7 · p. 9
  • Neither augmentation nor reranking recovers facts missing from retrieved passages, establishing a ceiling for retrieval-only approaches.On HotpotQA, accuracy degradation with more flagged hops and larger intervention gains on flagged questions appear without retraining the predictor.Source: §7 · p. 9

Limitations

The paper bounds its conclusions by the reasoning model, judge, interventions, and evaluated datasets. Its appendices describe the fact-presence prompt, MuSiQue sub-question normalization, and checks against gold paragraph annotations.

Limitations

  • The study uses one reasoning model, GPT-4.1-mini, so failure rates and intervention effects may differ with other models.Source: Limitations · p. 9

A LLM Judge Prompt

  • The LLM judge achieved κ = 0.840 human agreement, but may have systematic biases on question types underrepresented in its few-shot examples.The prompt asks whether retrieved passages contain the specific fact needed for a sub-question, including when it is implied or paraphrased.Source: Limitations · p. 9, A LLM Judge Prompt · p. 12

B Sub-Question Normalization Examples

C Gold Paragraph Consistency

  • The judge labels gold paragraphs answerable in 81.7% of cases, while 15.0% of hops without the gold paragraph are answerable from another passage.The remaining 18.3% with a retrieved gold paragraph are described as genuinely ambiguous because that paragraph does not self-sufficiently answer the sub-question.Source: C Gold Paragraph Consistency · p. 12

D Retrieval Recall Correlation Analysis

On MuSiQue, retrieval recall tracks fact presence across all hops but offers almost no signal on incorrectly answered hops. Content-based predictors outperform retrieval recall, though the strongest require gold answers unavailable at inference.

D Retrieval Recall Correlation Analysis

  • Among 5,283 incorrectly answered MuSiQue dev hops, retrieval recall and fact presence are nearly uncorrelated (Spearman ρ = 0.042), despite ρ = 0.449 across all 6,404 hops.The broader correlation largely reflects easy cases where retrieval and fact presence succeed or fail together.Source: D Retrieval Recall Correlation Analysis · p. 12
  • Accuracy is 68.1% for retrieval recall, 80.8% for string-matching the intermediate answer against retrieved text, and 83.4% for gold-paragraph presence.The 80.8% and 83.4% methods require gold answers and cannot be applied at inference time.Source: D Retrieval Recall Correlation Analysis · p. 12
  • The 15-point gap between retrieval recall (68.1%) and gold-paragraph presence (83.4%) indicates that passage content predicts fact presence beyond what standard retrieval metrics capture.Gold-answer-dependent predictors cannot be applied at inference time.Source: D Retrieval Recall Correlation Analysis · p. 12

E Predictor Complementarity Analysis

DeBERTa and BM25-threshold flags identify substantially different deficient hops. Their intersection gives the strongest failure signal, while the types of questions each flags show different accuracy profiles.

E Predictor Complementarity Analysis

  • The comparison uses DeBERTa, a BM25-threshold flag for maximum retrieval scores below the corpus median, and a random predictor flagging each hop with probability 0.5.The analysis relates the number of flagged hops per question to QA failure.Source: E Predictor Complementarity Analysis · pp. 12–13
  • The intersection of DeBERTa and BM25-threshold flags gives the strongest failure signal (ρ = −0.282, p < 10^-45), versus individual correlations of −0.216 and −0.254.The individual correlations are −0.216 for DeBERTa and −0.254 for BM25-threshold.Source: Table 9 · p. 13, E Predictor Complementarity Analysis · p. 13
  • 42.9% of hop-level flags overlap between DeBERTa and BM25-threshold, indicating that they identify substantially different deficient hops.The overlap is reported for 6,401 hops.Source: Table 10 · p. 13, E Predictor Complementarity Analysis · p. 13
  • Baseline accuracy is 71.6% for questions flagged only by DeBERTa and 63.7% for those flagged only by BM25, suggestive of extraction-type and retrieval-risk cases, respectively.Questions flagged by both signals produce the strongest failure correlation.Source: E Predictor Complementarity Analysis · p. 13, Table 11 · p. 13

F Supervised Fine-Tuning Experiment

Fine-tuning GPT-4.1-mini on successful Self-Ask traces improves accuracy by 3.5 points, comparable to the DeBERTa intervention gain. The accompanying appendices specify retrieval corpora, statistical tests, predictor training, compute, and licenses.

F Supervised Fine-Tuning Experiment

  • Fine-tuning GPT-4.1-mini on 654 successful Self-Ask traces raises LLM-judged accuracy from 48.1% to 51.6% (+3.5 points; n = 2,417), comparable to DeBERTa’s +3.7-point intervention gain.The training data contains no intervention traces; the paper characterizes fine-tuning and DeBERTa-targeted intervention as complementary training-time and inference-time approaches.Source: F Supervised Fine-Tuning Experiment · p. 13

G Retrieval Corpus Details

  • Elasticsearch uses default BM25 scoring for passage contexts; dense-retrieval experiments encode MuSiQue and HotpotQA passages with Contriever and index them with FAISS.Table 12 reports retrieval corpus statistics per dataset.Source: G Retrieval Corpus Details · p. 13, Table 12 · p. 13

H Statistical Testing Details

  • Paired accuracy comparisons use McNemar’s test, key effects use 10,000-resample bootstrap 95% intervals, and all Table 13 targeted-intervention gains are significant at p < 0.001.Spearman rank correlations are used for monotonic associations, and reported p-values are two-sided unless noted otherwise.Source: H Statistical Testing Details · pp. 13–14, Table 13 · p. 14

I DeBERTa Training Details

  • On 2WikiMultihopQA, retraining the same architecture achieves 97.4% accuracy (F1 = 0.983), while zero-shot MuSiQue transfer yields 0% flag rate and requires domain-specific retraining.Source: I DeBERTa Training Details · p. 14

J Compute Infrastructure

  • DeBERTa training cost approximately $80 on NVIDIA A100 GPUs; the QA pipeline and experiments ran on a 2-vCPU VM, with GPT-4.1-mini API calls costing approximately $100.Source: J Compute Infrastructure · p. 14

K Dataset and Model Licenses

  • Dataset licenses are CC BY 4.0 (MuSiQue-Ans), CC BY-SA 4.0 (HotpotQA), and Apache 2.0 (2WikiMultihopQA); model and retrieval-tool licenses also vary.Source: K Dataset and Model Licenses · p. 14

Related papers

Loading 2609.17043v1…