Source-linked AI summary
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, Tat-Seng Chua
TL;DR
OpenQA research has advanced through neural MRC-integrated systems, but existing QA surveys have rarely covered these developments comprehensively. This paper reviews OpenQA’s origins, Retriever-Reader architecture, component techniques, challenges, and benchmarks. It identifies continuing interest in improved retrieval, end-to-end systems, knowledge enhancement, and dialogue-like interaction, while noting unresolved knowledge-representation questions.
Problem
Recent neural MRC-integrated OpenQA research has advanced substantially but has been rarely covered in existing QA surveys.
Method
The paper conducts a comprehensive literature review covering OpenQA development, Retriever-Reader systems and techniques, challenges, and QA benchmarks.
Results
The survey presents a Retriever-Reader architecture and reviews diverse OpenQA systems, approaches, challenges, and QA benchmarks.
Takeaways & Limitations
The review identifies improved retrieval, end-to-end OpenQA, knowledge-enhanced systems, and dialogue-like interfaces as promising directions for further research.
Takeaways & Limitations
How to represent and fully use knowledge for OpenQA still needs more research.
Abstract
from arXiv · showhide
Open-domain Question Answering (OpenQA) is an important task in Natural Language Processing (NLP), which aims to answer a question in the form of natural language based on large-scale unstructured documents. Recently, there has been a surge in the amount of research literature on OpenQA, particularly on techniques that integrate with neural Machine Reading Comprehension (MRC). While these research works have advanced performance to new heights on benchmark datasets, they have been rarely covered in existing surveys on QA systems. In this work, we review the latest research trends in OpenQA, with particular attention to systems that incorporate neural MRC techniques. Specifically, we begin with revisiting the origin and development of OpenQA systems. We then introduce modern OpenQA architecture named "Retriever-Reader" and analyze the various systems that follow this architecture as well as the specific techniques adopted in each of the components. We then discuss key challenges to developing OpenQA systems and offer an analysis of benchmarks that are commonly used. We hope our work would enable researchers to be informed of the recent advancement and also the open challenges in OpenQA research, so as to stimulate further progress in this field.
1 INTRODUCTION
OpenQA seeks to answer natural-language questions from large collections of unstructured documents, unlike MRC, which supplies context passages. The survey reviews OpenQA’s development, modern neural techniques, challenges, and benchmarks, emphasizing neural MRC-integrated systems.
- OpenQA answers natural-language questions by inferring answers from collections of unstructured text documents.
- Textual QA includes MRC, which uses specified context passages, and OpenQA, which must first search for relevant documents.
- Traditional OpenQA systems commonly use Question Analysis, Document Retrieval, and Answer Extraction in a sequential pipeline.
- Deep learning has been applied across OpenQA stages, including neural question classification, neural retrieval, and advanced neural MRC techniques.
- The survey reviews OpenQA development, Retriever-Reader systems, key challenges, and applicable QA benchmarks to identify research gaps.
2 DEVELOPMENT OF OPENQA
OpenQA developed from restricted-domain QA toward systems that retrieve answers from large unstructured collections, with deep learning increasingly integrated throughout the pipeline. Traditional systems analyze questions, retrieve and process documents, and extract answers, while neural methods enable end-to-end training.
- 2.1 Origin of OpenQA: Early QA systems focused on restricted domains, including Baseball, LUNAR, and MURAX, using dictionaries, databases, or encyclopedias.
- 2.1 Origin of OpenQA: 1999 marked an early OpenQA definition in TREC as extracting the top five probable answer-containing snippets from news articles.
- 2.2 Traditional OpenQA: Traditional OpenQA uses Question Analysis, Document Retrieval, and Answer Extraction to transform questions, find relevant documents, and return answers.
- 2.2.1 Question Analysis: Query formulation addresses term mismatch through linguistic processing, query expansion, and paraphrasing, while question classification predicts expected answer types.
- 2.2.2 Document Retrieval: Document retrieval has used Boolean, vector-space, probabilistic, and language models, followed by filtering, re-ranking, or selection when retrieved sets contain noise.
- 2.3 Application of Deep Neural Networks in OpenQA: Deep learning supports dense semantic retrieval and end-to-end OpenQA training, including systems that treat document retrieval as a latent variable.
3 MODERN OPENQA: RETRIEVING AND READING
Modern OpenQA is organized around a Retriever-Reader architecture: the Retriever finds question-relevant documents, and the Reader infers the final answer from them. The Reader is usually a neural MRC model, with auxiliary modules also possible.
- The Retriever-Reader architecture contains two major components: a Retriever for relevant documents and a Reader for final answer inference.
- The Retriever can be regarded as an information-retrieval system, while the Reader is usually a neural Machine Reading Comprehension model.
- Modern OpenQA systems may also include auxiliary modules in addition to the Retriever and Reader.
3.1 Retriever
Retrievers locate and rank documents relevant to a question within the Retriever-Reader architecture. The survey distinguishes sparse, dense, and iterative approaches, each balancing retrieval effectiveness, computation, interpretability, and adaptive search.
- Retrievers search for documents or passages likely to contain the answer and rank them by relevance.They are treated as information-retrieval systems within OpenQA.
- Sparse Retriever: Sparse retrievers use classical information-retrieval methods such as TF-IDF and BM25.DrQA combines bi-gram hashing and TF-IDF matching with neural MRC over Wikipedia.
- Dense Retriever: Dense retrievers encode questions and documents in neural representations, with representation-based, interaction-based, and representation-interaction variants.These variants differ in how they encode questions and documents and score their similarity.
- Dense Retriever: Representation-interaction retrievers combine independent encoding with token-level interaction to trade off retrieval effectiveness and efficiency.ColBERT computes token-level question-document scores and sums them into a final relevance score; the approach remains underexplored.
- Dense Retriever: Dense retrieval can impose heavy computation, while offline document encoding improves speed by sacrificing some question-dependent effectiveness.Offline representations are computed in advance and remain unchanged after encoding.
- Iterative Retriever: Iterative retrievers search in multiple steps, reformulating explicit natural-language or implicit dense queries from previous retrieval results.They are especially used for complex questions requiring multi-hop reasoning; explicit queries are interpretable, whereas implicit queries avoid vocabulary limits but lack interpretability.
3.2 Document Post-processing
Document post-processing reduces and re-ranks retrieved documents before they reach the Reader. Neural passage rankers and paragraph selectors address retrieval noise, although trainable dense retrievers can make this module unnecessary.
- Document post-processing filters retrieved documents so that only the most relevant candidates are passed to the Reader.It is needed because retrieval can return irrelevant or excessively many documents.
- Neural passage rankers and paragraph selectors remove noisy candidates using learned relevance or answer-containing probabilities.R3 jointly trains a neural Passage Ranker with the Reader through reinforcement learning, while DS-QA selects paragraphs by answer probability.
- Recent systems may omit post-processing because trainable dense retrievers can learn ranking and retrieval simultaneously.This development reduces the need for a separate document re-ranking module.
3.3 Reader
The Reader is the neural MRC component that infers an answer from ranked documents, making OpenQA harder than reading a single specified passage. Existing Readers are extractive or generative.
- The Reader infers an answer from a set of ordered documents and is usually implemented as a neural MRC model.Its input setting is more challenging than MRC over only one passage.
- Extractive Reader: Extractive Readers assume the answer exists in context and predict the start and end positions of an answer span.They may process retrieved documents independently or jointly.
- Extractive Reader: Some extractive systems rank documents by answer probability and extract from the most probable document or documents.DS-QA uses a dedicated Paragraph Selector to choose a paragraph before span extraction.
- Extractive Reader: Other extractive systems process all retrieved documents jointly when predicting answer spans.DrQA decomposes documents into paragraphs, extracts linguistic features, and uses a multi-layer Bi-LSTM Reader.
- Generative Reader: Generative Readers synthesize natural-language answers with sequence-to-sequence models rather than only extracting spans.S-Net first extracts evidence boundaries and then feeds the evidence to a Seq2Seq answer synthesis model.
3.4 Answer Post-processing
Answer post-processing selects the final answer from candidates produced by the Reader. This module uses learned re-ranking to address the difficulty of identifying answers supported by distributed evidence.
- Neural MRC models can fail when the correct answer depends on multiple evidences across narrative or multiple documents.Answer post-processing is introduced to help detect the final answer from extracted candidates.
- A feed-forward network or recurrent neural network can re-rank answer candidates and select the final answer.
3.5 End-to-end Methods
Modern OpenQA systems extend beyond the standard Retriever-Reader pipeline through end-to-end training, Retriever-only designs, and Retriever-free pretrained language models. These approaches trade retrieval and reading stages for joint optimization, efficient phrase retrieval, or parametric knowledge.
- System Types: OpenQA systems include Retriever-Reader, Retriever-only, and Retriever-free architectures.The survey distinguishes systems combining retrieval and reading, systems using only retrieval, and systems answering without retrieval.
- Retriever-Reader: End-to-end methods jointly train Retriever and Reader components to improve OpenQA modeling.Examples jointly optimize passage-question similarity and answer-span prediction, or treat retrieval as a latent variable.
- Retriever-only: Retriever-only systems omit the time-consuming Reader and use offline phrase-level indexes for efficient answer retrieval.DenSPI represents candidate phrases with concatenated sparse and dense vectors and retrieves them during inference.
- Retriever-free: Pretrained Seq2Seq models can answer questions without external knowledge by storing information in their parameters.GPT-2 can answer questions without fine-tuning, while GPT-3 achieves competitive performance with few-shot learning.
- Retriever-free: Retriever-free methods provide a fundamentally different approach to OpenQA system construction.Their answers are generated from knowledge learned during pretraining rather than retrieved documents.
4 CHALLENGES AND BENCHMARKS
OpenQA faces challenges in data quality, retrieval, knowledge incorporation, conversational interaction, and benchmark suitability. The survey organizes these issues around effectiveness, efficiency, knowledge representation, dialogue capabilities, and dataset access to supporting document collections.
- Challenges: OpenQA training data is difficult to collect at sufficient quality, so distant supervision introduces noisy and sometimes incorrect labels.Automatic labeling from corpora such as Wikipedia suffers from the wrong-label problem and complicates modeling and training.
- Retrieval Effectiveness and Efficiency: Retrieval must distinguish relevant documents from both term-mismatched evidence and irrelevant documents containing overlapping terms or answer spans.These errors make accurate context understanding during answer inference more difficult.
- Retrieval Effectiveness and Efficiency: Retrieval effectiveness and efficiency must be improved together, with an appropriate trade-off for practical and real-time OpenQA deployment.Neural retrieval methods improve effectiveness but often suffer from low efficiency.
- Knowledge Incorporation: Knowledge-enhanced OpenQA must represent and use world, commonsense, or domain-specific knowledge through explicit or implicit approaches.Explicit approaches use knowledge bases and graphs, whereas implicit approaches store knowledge in pretrained language-model parameters.
- Knowledge Incorporation: Knowledge incorporation supports both answer generation and interpretation, but its representation and use still require further research.The survey identifies this as an unresolved research need.
- Conversational OpenQA: Conversational OpenQA uses dialogue to decompose complex questions, clarify ambiguity, and gather missing background information.Conversational systems can ask follow-up questions before producing a final answer.
- Conversational OpenQA: Conversational OpenQA must detect unanswerable questions, generate follow-up questions, and model conversation history in both Reader and Retriever.Current systems rarely determine unanswerability, while retrieval remains necessary in open-retrieval dialogue settings.
- Conversational OpenQA: F1: 29.4 on OR-QuAC is far below F1: 74.41 on QuAC, highlighting the greater difficulty of open-retrieval conversational QA.OR-QuAC requires passage retrieval before answer inference, unlike the context-provided QuAC setting.
5 CONCLUSION
The survey synthesizes recent OpenQA progress, presents the Retriever-Reader architecture, reviews systems and approaches, and analyzes challenges and benchmarks. It identifies neural retrieval, end-to-end systems, knowledge use, and dialogue interfaces as important directions for further research.
- The survey reviews recent OpenQA systems through the Retriever-Reader architecture and analyzes their different approaches.
- The survey discusses salient OpenQA challenges and summarizes QA benchmarks applicable to the task.
- Neural retrievers, including single-step and multi-step designs, are expected to attract increasing attention because more accurate retrieval of related documents is needed.
- Knowledge-enhanced OpenQA is promising for answer generation and answer interpretation, but representing and using knowledge fully still requires further research.
- Dialogue-like OpenQA interfaces are expected to receive increasing attention because they support information exchange between users and systems in real-world application scenarios.