Revealing the Importance of Semantic Retrieval for Machine Reading at Scale

1 Introduction

• MRS requires both precise retrieval of sparsely available information and understanding of the selected knowledge and query.The task was introduced as requiring semantic retrieval and comprehension at different granularities (Chen et al., 2017).Source: §1 · p. 1
• Prior evaluations focused mainly on final-task performance, while retrieval was often optimized for information coverage rather than exactness.The authors identify this as misaligned with MRS’s joint retrieval-and-comprehension setting.Source: §1 · p. 1
• The proposed system combines term-based retrieval with neural paragraph- and sentence-level retrieval before downstream task modeling.The system uses Wikipedia as its external knowledge source.Source: §1 · p. 1
• 45.32% answer EM and 25.14% joint EM on HOTPOTQA, and 67.26% FEVER score, were reported as state-of-the-art results.The authors report absolute improvements of 8% in HOTPOTQA answer EM and 3% in FEVER score over previous best results.Source: §1 · p. 2
• Ablations and analyses indicate that both retrieval levels matter and that stronger semantic retrieval shapes downstream data and provides more relevant, higher-quality examples.The authors connect these effects to high retrieval recall, a high downstream upper bound, and better downstream modeling on QA and fact verification.Source: §1 · p. 2

Machine Reading at Scale

• Earlier MRS systems used paragraph retrieval mainly to improve recall or sentence retrieval as an auxiliary task, while this work optimizes both granularities for downstream comprehension.The authors state that, to their knowledge, this is the first application and optimization of neural semantic retrieval at both levels for MRS.Source: §2 · p. 2

Information Retrieval

• MRS retrieval selects facts needed to answer a question or verify a statement, unlike typical IR’s ranking of collection elements by query relevance.The paper calls this fact-oriented process semantic retrieval because it emphasizes semantic understanding.Source: §2 · p. 2

Automatic Fact Checking

• Fact checking is treated as MRS with verification or natural language inference as the downstream task rather than question answering.The passage describes the two task setups as otherwise almost the same.Source: §2 · p. 2

3 Method

• The pipeline maps a query and Wikipedia knowledge base to a prediction and selected supporting sentences, using a unified setup for interpretability.For QA, the query is a question and the prediction an answer; for verification, the query is a claim and the prediction its truthfulness.Source: §3 · p. 2
• Term-based retrieval combines TF-IDF and rule-based keyword matching to narrow Wikipedia to candidate paragraphs.The step aims to cover as much relevant information as possible while keeping the candidate set manageable for downstream processing.Source: §3 · p. 2
• Neural paragraph retrieval ranks candidate paragraphs by query relatedness and retains top-scoring paragraphs subject to a threshold.The number selected and threshold are chosen to balance paragraph retrieval recall and precision.Source: §3 · p. 3
• Sentence retrieval scores sentences from retained paragraphs against the query, then selects top-scoring sentences above a threshold for downstream modeling.The selected set is evaluated against annotated evidence sentences.Source: §3 · p. 3

3.1 Modeling and Training

• The downstream modules consume the selected sentences, and their training contexts include upstream-sampled sentences to reflect inference-time data distributions.Verification uses three-way NLI classification; the authors also note that training modules in isolation can cause upstream data-distribution misalignment.Source: §3 · p. 3, §3.1 · p. 4
• Paragraph- and sentence-level retrieval are binary classifiers trained with binary cross-entropy; ground-truth evidence supplies positives and upstream-selected material supplies negatives.At paragraph level, paragraphs containing any ground-truth sentence are positive; at sentence level, ground-truth sentences are positive.Source: §3.1 · pp. 3–4

4 Experimental Setup

• The unified experimental setup evaluates semantic retrieval’s effects on multi-hop QA and fact verification.Source: §4 · p. 4

4.1 Tasks and Datasets

• HOTPOTQA’s fullwiki setting requires finding answers across all of Wikipedia and provides human-annotated supporting facts for evaluating retrieval and QA.Source: §4.1 · p. 4
• FEVER asks systems to retrieve Wikipedia evidence and classify claims as SUPPORT, REFUTE, or NOT ENOUGH INFO; annotated facts support retrieval evaluation.Source: §4.1 · p. 5
• Wikipedia is the shared knowledge base for both tasks, standardizing analysis across QA and verification.Source: §4.1 · p. 5

4.2 Metrics

• QA evaluation uses answer EM and F1, plus joint scores combining answer-span and supporting-fact performance.Source: §4.2 · p. 5
• FEVER evaluation uses Label Accuracy and FEVER Score; Oracle Score estimates the final-score upper bound at an intermediate retrieval layer assuming perfect downstream modules.Source: §4.2 · p. 5

5 Results on Benchmarks

• HOTPOTQA results set a new state of the art across metrics, with answer EM up ∼8 absolute points, answer F1 up ∼9, and joint EM doubling over prior best results.Source: §5 · pp. 5–6
• FEVER evidence F1 reaches 74.62, 22 points above the second system, while Label Accuracy and FEVER Score improve by ∼4 and ∼3 absolute points over prior published results.Source: §5 · p. 6
• The authors contrast their fully hierarchical pipeline with prior systems that treated sentence retrieval as auxiliary or used only one neural retrieval module, motivating ablation and analysis experiments.Source: §5 · p. 6

6 Analysis and Ablations

• Both neural retrieval stages shape the immediate retrieval distribution and provide downstream modules with better training and inference data.The retrieval sub-modules also help speed downstream processing.Source: §6 · p. 6

6.1 Ablation Studies

• Removing paragraph-level retrieval lowers sentence-level retrieval precision and downstream scores on both HOTPOTQA and FEVER, despite higher recall and upper bounds.The authors caution that optimizing only recall or the final score’s upper bound can jeopardize overall performance.Source: §6.1 · p. 6
• Without sentence-level retrieval, QA EM and F1 drop by ∼2 points, while FEVER Score drops by ∼15 points; paragraph removal drops answer EM by 11 points.The authors report that sentence-level retrieval removal hurts verification more than QA, while QA is more reliant on paragraph-level retrieval; FEVER NOT ENOUGH INFO F1 also drops significantly without retrieval.Source: §6.1 · pp. 6–7

6.2.1 Effects of Paragraph-level Retrieval

• On HOTPOTQA, most scores rise sharply when k_p increases from 1 to 2, then decline as more paragraphs are selected; supporting-fact recall peaks at k_p=4.The authors attribute the decline to noise in retrieved facts and note that answer prediction is sensitive to paragraph-level input.Source: §6.2.1 · p. 7

6.2.2 Effects of Sentence-level Retrieval

• On HOTPOTQA, supporting-fact retrieval and joint performance peak at h_s=0.5, but answer prediction peaks at h_s=0.2, where recall exceeds precision.On FEVER, Label Accuracy and FEVER Score peak at h_s=0.2, while retrieval F1 peaks at h_s=0.5.Source: §6.2.2 · p. 8

6.3 Answer Breakdown

• Among 200 sampled HOTPOTQA examples, Person was the most frequent answer type (24%) and Event the least frequent (2%); Yes/No accuracy was 70.6%.Source: §6.3 · p. 8

6.4 Examples

• The full pipeline filters a distracting fact in the example, whereas the system without paragraph-level retrieval gives a wrong answer; the authors say such distractors are difficult to filter downstream.They distill three guidelines: paragraph-level retrieval is imperative, downstream modules tolerate some sentence-level noise, and paragraph-level changes may have cascade effects.Source: §6.4 · p. 9

• The proposed hierarchical pipeline achieves state-of-the-art results on two MRS tasks.Source: §7 · p. 9
• Ablation studies demonstrate the importance of semantic retrieval at both paragraph and sentence levels.Source: §7 · p. 9
• The work offers general MRS modeling guidelines and aims to inspire research on retrieval-comprehension relationships in joint settings.Source: §7 · p. 9

• Table 6 presents hyper-parameter selection for the full pipeline, including paragraph- and sentence-level retrieval settings.Source: Table 6 · p. 10
• Hyper-parameters were chosen based on dev-set performance.Source: A Training Details · p. 10
• The learning rate was 10^-5 in all experiments.Source: A Training Details · p. 10

• For FEVER, keyword matching and TF-IDF top-5 documents were combined into the final term-based retrieval set.The merged set contained a mean of 8.06 paragraphs per query, with standard deviation 4.88.Source: B Term-Based Retrieval Details · p. 10
• For HOTPOTQA, the procedure added the top five TF-IDF-ranked hyperlinked documents from the initial candidate set’s retrieved documents.The final set averaged 39.43 paragraphs per query, with standard deviation 16.05.Source: B Term-Based Retrieval Details · pp. 10–11
• Table 7 reports detailed sentence-level retrieval and question-answering results for different h_s values on HOTPOTQA.Source: Table 7 · p. 11

• Table 7 reports sentence-level retrieval and downstream QA results on HOTPOTQA for different hs values.Source: C Detailed Results · p. 11
• Table 8 reports sentence-level retrieval and downstream verification results on FEVER for different hs values.Source: C Detailed Results · p. 11
• Table 9 reports downstream sentence-level retrieval and QA results on HOTPOTQA for different kp values.Source: C Detailed Results · p. 11

• The case studies show high semantic diversity, while errors often arise from extracting wrong, surplus, or insufficient knowledge-base information.Source: D Examples and Case Study · p. 11
• Detailed results are tabulated for different hs values on FEVER and different kp values on HOTPOTQA.Source: Table 8 · p. 12, Table 9 · p. 12
• Correct examples are identified for HOTPOTQA and FEVER when sufficient evidence is present.Source: Table 10 · p. 12, Table 13 · p. 13
• Incorrect examples on both datasets are classified as involving insufficient or wrong evidence, or extra incorrect evidence.Source: Table 11 · p. 13, Table 12 · p. 13, Table 14 · p. 13