Source-linked AI summary
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, Barlas Oğuz
TL;DR
Complex multi-hop open-domain questions require retrieving and combining evidence across documents, while existing approaches may depend on corpus-specific structure and face efficiency challenges. The paper introduces a recursive dense retrieval framework that conditions each retrieval step on prior passages and evaluates it across HotpotQA and multi-evidence FEVER. It achieves state-of-the-art downstream results on both datasets and matches the best published HotpotQA result while being 10x faster at inference.
Problem
Multi-hop questions require sequential evidence retrieval across documents, but search spaces grow rapidly and common pruning methods rely on corpus-specific entity links or hyperlinks.
Method
The method recursively encodes the question and previously retrieved passages into a dense query vector, retrieves next passages with MIPS, and uses beam search over sequence candidates.
Results
The approach achieves state-of-the-art downstream results on HotpotQA and multi-evidence FEVER, matching the best published HotpotQA result while being 10x faster at inference.
Takeaways & Limitations
Dense multi-hop retrieval can discover useful passage sequences from unstructured text without corpus-specific hyperlinks and can be applied across two multi-hop tasks.
Takeaways & Limitations
Bridge-question errors include failures to capture exact n-gram matches, motivating a possible hybrid term-and-dense retriever.
Abstract
from arXiv · showhide
We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datasets, HotpotQA and multi-evidence FEVER. Contrary to previous work, our method does not require access to any corpus-specific information, such as inter-document hyperlinks or human-annotated entity markers, and can be applied to any unstructured text corpus. Our system also yields a much better efficiency-accuracy trade-off, matching the best published accuracy on HotpotQA while being 10 times faster at inference time.
1 INTRODUCTION
Complex open-domain questions require aggregating evidence across multiple documents, but existing multi-hop retrieval can suffer from expanding search spaces, corpus-specific assumptions, and efficiency challenges. The paper proposes recursive multi-hop dense retrieval that discovers relevant passage sequences from unstructured text and achieves strong accuracy with improved efficiency.
- Background: Dense retrieval represents questions and documents in a vector space and uses maximum inner-product search to improve retrieval beyond lexical matching.Prior dense retrieval methods had already demonstrated gains over traditional term-based information retrieval.
- Problem: Complex questions require sequentially aggregating information from multiple documents rather than retrieving evidence from a single text.Single-shot retrieval is insufficient because each retrieval step may depend on information obtained earlier.
- Problem: The multi-hop search space grows exponentially with each retrieval hop, motivating pruning strategies based on entity links or hyperlink graphs.These strategies bound search using corpus structure but may not generalize beyond domains such as Wikipedia.
- Contribution: The proposed recursive framework conditions each new query on the question and previously retrieved documents, avoiding reliance on corpus-specific hyperlinks.It is designed to discover sufficient passage sequences directly from unstructured text.
- Results: On HotpotQA and multi-evidence FEVER, the approach improves over traditional linking-based retrieval and achieves state-of-the-art downstream results on both datasets.On HotpotQA, it matches the best published result while being 10x faster when limiting contexts passed to downstream models.
2 METHOD
The method retrieves passage sequences iteratively by reformulating a dense query from the question and prior passages, then searching the corpus with maximum inner-product search. Training uses positive evidence and multiple negative passages, while inference uses beam search over a dense passage index.
- Problem Definition: The retrieval task is to find a short sequence of passages from a large corpus that provides sufficient information to answer a multi-hop question.The retriever returns k sequence candidates, with k much smaller than the corpus size, while balancing recall and downstream processing cost.
- Model: The system models passage-sequence probability as a product of stepwise probabilities and solves retrieval iteratively.Each step selects a passage conditioned on the question and passages retrieved previously.
- Model: At each hop, a query encoder reformulates the question using previously retrieved passages, and MIPS selects passages from dense corpus representations.The formulation uses qt = g(q, p1, ..., pt−1) and passage vectors produced by h(·).
- Model: A shared RoBERTa-base encoder produces dense query and passage representations, with the question and retrieved passages concatenated for query reformulation.Layer normalization over RoBERTa’s start-token representations yields the final vectors.
- Training and Inference: Training pairs each stepwise query with gold evidence and multiple in-batch, hard, and memory-bank negatives to approximate the corpus-wide softmax.Hyperlinked pages are used as additional hard negatives in experiments but are neither necessary nor critical to the approach.
- Training and Inference: At inference, beam search generates top-k passage-sequence candidates using MIPS and scores beams by summed inner products.The procedure requires only a dense passage index and query representations, without explicit graph structure.
3 EXPERIMENTS
Experiments evaluate MDR on HotpotQA and multi-evidence FEVER, comparing retrieval strategies, reranking, ablations, question decomposition, readers, and existing systems. MDR substantially outperforms retrieval baselines and achieves strong results without corpus-specific hyperlink information, while analysis identifies bridge-question and exact-match limitations.
- 3.1.1 DIRECT: MDR substantially outperforms all retrieval baselines on HotpotQA and multi-evidence FEVER.HotpotQA uses recall at top-k paragraphs, while FEVER reports precision, recall, and F1.
- 3.1.2 RERANKING: After reranking the top-100 dense-retrieved sequences, MDR achieves better passage recall than Graph Recurrent Retriever while processing fewer passages.Graph Recurrent Retriever uses BERT to process more than 500 passages; MDR reranks the top 100 sequences.
- 3.1.3 ANALYSIS: Comparison questions are easier than bridge questions because both entities needed for retrieval appear in the question.Among 50 sampled erroneous bridge cases, half produced an alternative passage sequence that was also valid, implying about 90% top-1 passage-sequence accuracy.
- 3.1.2 RERANKING: Shared encoders and passage ordering are crucial for MDR, whereas memory-bank training yields only modest gains and linked-paragraph negatives have minor impact.Order-agnostic training hardly works and underperforms even the single-hop baseline.
- 3.1.3 ANALYSIS: Explicit oracle question decomposition produces no strong improvement, while replacing the first decomposed hop with the original question causes no performance degradation.The result suggests pretrained encoders can select necessary information from the multi-hop question at each retrieval step.
- 3.2.1 RESULTS: MDR outperforms prior HotpotQA systems by large margins, especially on supporting-fact prediction, despite not using hyperlink structure.On FEVER, replacing existing retrieval with MDR significantly improves verification, particularly on the strict multi-evidence subset, without downstream-module finetuning.
4 RELATED WORK
The paper situates its approach among dense retrieval, query expansion, and multi-hop QA research, emphasizing the need for open-domain retrieval across multiple documents.
- Dense retrieval captures semantic matching beyond the simple n-gram overlaps of sparse term-index methods.
- Query expansion methods update retrieval representations using previously retrieved information, paralleling the paper’s iterative query construction.
- Many other multi-hop QA datasets are built from knowledge bases and are less diverse than HotpotQA, limiting their design for open-domain multi-hop retrieval.
5 CONCLUSION
The work extends dense retrieval to multi-hop questions and applies the framework across tasks and downstream modules. Its corpus-independent design supports broader application across domains and settings.
- The authors extend successful dense retrieval methods to the multi-hop setting for complex queries.
- The approach is demonstrated on two different tasks with a variety of downstream modules.
- Its simple framework does not depend on corpus-specific graph structures, enabling easier application across domains and settings.
A.1 FALSE BRIDGE QUESTION ERROR CASES
The appendix examines bridge-question retrieval errors and finds that many predicted passage sequences are alternative valid evidence chains rather than genuine errors.
- Half of the sampled bridge-question errors correspond to alternative passage sequences that could also answer the questions.The analysis manually inspects 50 randomly sampled erroneous examples after reranking.
- Treating valid alternative sequences as correct changes the interpretation of the reported bridge-question errors.
- Table 8 presents cases where the model’s predicted passage sequence is also correct.Important clues in these cases are marked in blue.
A.2 EXAMPLES FROM THE QUESTION DECOMPOSITION ANALYSIS
The decomposition analysis illustrates retrieval errors caused by removing information from the original question or previous-hop results. These clues can remain available when retrieval uses the full question context.
- Table 9 shows retrieval errors made only by the decomposed system.
- Some decomposition errors could potentially be avoided by retaining the full original question or previous-hop results.
- Decomposition can remove marked information from one retrieval hop even when that information is important for retrieving the correct documents.
A.3 EXTRACTIVE & GENERATIVE READER MODEL
The section describes extractive and generative readers for multi-hop retrieval, including extractive supervision and multi-hop RAG’s retrieval-generation formulations.
- Extractive & Generative Reader Models: Extractive readers predict answers from the top 250 retrieved passage sequences, while generative readers use fewer passages because they are more computationally expensive.The passage sequences pair passages from hop one and hop two.
- Extractive Reader: The extractive reader jointly predicts passage-sequence relevance, answer spans, and supporting sentences using four loss functions.Its heads operate on the [CLS] token, token representations, and sentence-level special-token representations.
- FiD: FiD uses T5-large with 770M parameters and feeds the top 50 retrieved passage sequences, concatenating passages within each sequence.Checkpointing is used to fit training within GPU memory.
- Multi-Hop RAG: Multi-hop RAG models answer generation from a question and documents retrieved at hop one and hop two.The hop-two retriever conditions on both the question and the hop-one document, while the generator conditions on both retrieved documents and prior tokens.
- Multi-Hop RAG Sequence Model: The RAG sequence model marginalizes over the two retrieval-hop document variables to generate an answer sequence.The retrieved documents are the top-k documents from their respective retrieval modules.
- Multi-Hop RAG Token Model: The RAG token model can make predictions using different passages extracted at each token.Its formulation specifies a predicted probability for each token.
C RETRIEVAL-FREE APPROACHES
Retrieval-based methods are especially important for multi-hop QA because retrieval-free methods show a larger performance gap there than on simple single-hop questions.
- Retrieval-Free Approaches: The performance gap between retrieval-based and retrieval-free methods is much larger for multi-hop QA than for simple single-hop questions.The comparison uses experiments on HotpotQA with BART as a retrieval-free baseline.
D A UNIFIED QA RETRIEVAL SYSTEM
The unified QA retrieval system adaptively supports single-hop and multi-hop questions with one dense retriever, maintaining competitive performance across NaturalQuestions-Open and HotpotQA.
- Motivation: A unified system is needed because incoming questions may require either single or multiple pieces of text evidence.The proposed framework can be extended to adaptively retrieve for multiple hops.
- Adaptive Retrieval: Adaptive retrieval adds a binary stop-prediction head that uses the question and first retrieved passage to decide whether to stop after one hop.The head uses the final hidden state of the question encoder’s first token.
- Experimental Setup: The unified model is evaluated on a merged corpus containing NaturalQuestions-Open as single-hop QA and HotpotQA as multi-hop QA.The NQ Wikipedia corpus is taken from DPR.
- Results: The unified dense retrieval model maintains competitive performance on both datasets despite their different question and evidence characteristics.For NQ, performance is very similar to the single-dataset DPR model; HotpotQA performance decreases more.
- Evaluation: Answer recall at top-K retrieved passage sequences is the retrieval metric used for comparing the unified and task-specific models.All three comparison models use BERT-base encoders.