Source-linked AI summary
Adaptive Document Retrieval for Deep Question Answering
Bernhard Kratzwald, Stefan Feuerriegel
TL;DR
The interplay between document retrieval and deep machine comprehension, especially the choice of candidate count, remains poorly understood. This paper introduces adaptive retrieval to choose that count and reports stronger benchmark performance than state-of-the-art methods, including when corpus size varies.
Problem
How retrieval candidate count interacts with machine comprehension remains insufficiently understood, although count affects relevant-information recall and extraction noise.
Source: §1 · p. 1Method
The paper proposes adaptive document retrieval that selects an optimal document count for each retrieval, with a trainable ordinal-regression approach among its methods.
Source: §1 · p. 1, §4.2 · p. 4Results
The adaptive scheme is best on three of four datasets and consistently outperforms the naïve DrQA baseline across all datasets.
Source: §5.1 · p. 4Takeaways & Limitations
The optimal document count depends considerably on corpus size, and adaptive retrieval achieves the best performance in terms of regret compared with a fixed count.
Source: §1 · p. 1, §6 · p. 5Reported numbers vary slightly from the original paper because the public repository was optimized for runtime performance.
Source: §5.3 · p. 5
Abstract
from arXiv · showhide
State-of-the-art systems in deep question answering proceed as follows: (1) an initial document retrieval selects relevant documents, which (2) are then processed by a neural network in order to extract the final answer. Yet the exact interplay between both components is poorly understood, especially concerning the number of candidate documents that should be retrieved. We show that choosing a static number of documents -- as used in prior research -- suffers from a noise-information trade-off and yields suboptimal results. As a remedy, we propose an adaptive document retrieval model. This learns the optimal candidate number for document retrieval, conditional on the size of the corpus and the query. We report extensive experimental results showing that our adaptive approach outperforms state-of-the-art methods on multiple benchmark datasets, as well as in the context of corpora with variable sizes.
1 Introduction
Deep QA retrieves candidate documents before a comprehension module extracts an answer. Because candidate count affects both relevant-information recall and extraction noise, a fixed choice can be suboptimal; the paper proposes adaptive selection.
1 Introduction
- More candidate documents improve retrieval recall and the chance of including relevant information, but can add noise that reduces answer-extraction accuracy.A top-1 system can potentially outperform systems selecting more than one document (Kratzwald and Feuerriegel, 2018).Source: §1 · p. 1
- Adaptive retrieval selects the optimal document count for each retrieval, addressing the risk that a static choice yields suboptimal QA performance.The paper analyzes the interaction between retrieval and machine comprehension in neural QA systems.Source: §1 · p. 1
- The paper reports that its adaptive system outperforms state-of-the-art neural QA methods on multiple benchmark datasets.It also identifies corpus size as an important factor in the optimal document count.Source: §1 · p. 1
2 Related Work
Related work distinguishes knowledge-based QA from retrieval-based systems that pass candidate documents to an answer-extraction module. Prior deep QA work used fixed candidate counts, while the interaction between retrieval and deep machine comprehension remained underexplored.
2 Related Work
- Knowledge-based QA uses rules, ontologies, and knowledge graphs, whereas retrieval-based QA selects candidate documents for a subsequent answer-extraction module.The passage gives examples of prior work in each paradigm.Source: §2 · p. 1
- Figure 1 compares end-to-end exact-match performance, top-n answer recall, and the average number of answer-containing documents as corpus size changes.Its caption notes that declining recall need not compromise top-n systems because the correct answer may occur more than once.Source: Figure 1 · p. 2
- The approach extracted answers from the top n = 5 documents using dot-product retrieval with tf-idf-weighted hashed bigram counts; Wang et al. (2018) added neural reranking while keeping n fixed.The interplay between document retrieval and machine comprehension, particularly how many candidate documents to select, had not yet been studied.Source: §2 · p. 2
- The interaction findings from traditional and machine-learning QA do not directly transfer to deep comprehension, whose attention mechanisms and multi-paragraph inputs pose distinct challenges.Deep networks can struggle to select the correct answer across multiple input paragraphs (Clark and Gardner, 2017).Source: §2 · p. 2
3 Noise-Information Trade-Off in Document Retrieval
Experiments compare retrieval recall with end-to-end QA performance while varying top-n and corpus size. They show a noise-information trade-off: top-1 can suit small corpora, while larger corpora can benefit from multiple answer-bearing documents, motivating adaptive retrieval.
3 Noise-Information Trade-Off in Document Retrieval
- The experiments vary both retrieved-document count and corpus size, comparing retrieval recall with the complete QA system’s exact-match performance.They use the TREC QA dataset and the DrQA system as a baseline.Source: §3 · p. 2
- In the TREC QA experiments, top-1 performs better for a small corpus, but this pattern changes as corpus size increases.The study ranges from one answer-containing Wikipedia article plus 50% noise per question to a full dump of more than five million documents.Source: §3 · p. 2
- As corpus size grows, top-1 recall drops quickly, while top-n recall remains fairly stable because the answer can appear in any selected document.The answer is often returned more than once, increasing the chance of answer extraction.Source: §3 · pp. 2–3
- With corpus size fixed, increasing top-n makes recall converge while the average number of relevant documents keeps growing, increasing relevant-information density.This pattern is shown for the full Wikipedia dump.Source: §3 · p. 3, Figure 2 · p. 3
4 Adaptive Document Retrieval
Adaptive retrieval chooses a query-specific document count rather than using a fixed n. The paper presents a cumulative-confidence threshold heuristic and a trainable ordinal regression model for predicting the retrieval cutoff.
4 Adaptive Document Retrieval
- Adaptive retrieval selects ni between 1 and a maximum τ using document relevance scores, while treating the scoring function as a black box.The scoring function can range from tf-idf similarity to more complex probabilistic models.Source: §4 · p. 3
4.1 Threshold-Based Retrieval
- The threshold heuristic retrieves documents until their cumulative confidence score surpasses a fixed threshold θ.When retrieval is confident the answer lies within a particular document, the heuristic selects fewer documents.Source: §4.1 · p. 3
4.2 Ordinal Regression
- Ordinal ridge regression predicts the position of the first relevant document containing the answer, using a ceiling-rounded rank and absolute-error loss.A linear offset helps ensure the predicted cutoff includes that first relevant document; nonlinear predictors showed no significant improvement justifying their added complexity.Source: §4.2 · pp. 3–4
5 Experiments
The experiments test adaptive retrieval against established QA systems and under changes in corpus size, including with a second paragraph-retrieval system. Across these settings, adaptive retrieval generally improves end-to-end performance, while fixed top-n choices vary with corpus size.
5 Experiments
- Experiments replace DrQA's document retrieval module with adaptive retrieval while leaving document scoring and answer extraction unchanged, across several QA benchmarks.The benchmark set includes SQuAD, Curated TREC, WikiMovies, and WebQuestions.Source: §5 · p. 4, §5.1 · p. 4
5.1 Overall Performance
- Adaptive retrieval yields the best-performing system on three of four datasets and consistently outperforms naïve DrQA across all four.Table 1 reports exact-match performance on the full Wikipedia dump.Source: §5.1 · p. 4, Table 1 · p. 5
5.2 Sensitivity: Adaptive QA to Corpus Size
- Adaptive retrieval is overall superior across corpus sizes; top-1 is slightly better on small corpora but ranks last on the complete corpus, while top-10 leads there.The top-1 result partly reflects the setup's inclusion of one correct document per question with little added noise.Source: §5.2 · p. 4, Figure 3 · p. 5
- The best top-n system yields regrets of 0.83 and 1.12, while adaptive control reaches 0.70.Regret measures the performance gap from the best system at an evaluation corpus size, averaged across observations and weighted by their span.Source: §5.2 · p. 4
5.3 Robustness Check
- In the second, paragraph-level QA system, both tested configurations outperform a system without adaptive retrieval, and top-n passage count gains even more importance.The configurations use top-50 and top-80 paragraphs; the comparison also includes adaptive cutoffs selected within the first 250 paragraphs.Source: §5.3 · p. 5, Table 2 · p. 5
6 Conclusion
Deep QA faces a noise-information trade-off, so the number of retrieved documents should be tailored to the task. Adaptive retrieval adjusts that count across corpus sizes and achieves the best performance in terms of regret compared with a fixed count.
6 Conclusion
- Deep QA is subject to a noise-information trade-off, so the number of selected documents should be tailored to the QA task rather than treated as fixed.The proposed adaptive schemes determine the document count.Source: §6 · p. 5
- For SQuAD train-split configurations, the best settings were selected from document counts of 30 through 100.Source: §6 · p. 5
- Across different corpus sizes, the adaptive strategy achieves the best performance in terms of regret compared with a fixed document count.Source: §6 · p. 5