Source-linked AI summary
Reading Wikipedia to Answer Open-Domain Questions
Danqi Chen, Adam Fisch, Jason Weston, Antoine Bordes
TL;DR
Open-domain QA over Wikipedia requires finding relevant articles and extracting answer spans from them without relying on multiple knowledge sources. The paper introduces DrQA, combining retrieval with recurrent reading, and reports competitive components plus improved multi-dataset performance from multitask learning and distant supervision. The authors identify paragraph-level training and independent retriever-reader training as directions for improvement.
Problem
Open-domain factoid QA must use Wikipedia as a unique knowledge source while combining large-scale document retrieval with machine comprehension.
Method
DrQA combines a bigram-hashing and TF-IDF Document Retriever with a recurrent-neural-network Document Reader for answer-span extraction.
Results
Across multiple benchmarks, the components are competitive and multitask learning with distant supervision improves performance across all datasets over single-task training.
Takeaways & Limitations
The results support machine reading at scale as a challenging task requiring integrated search, distant supervision, and multitask learning rather than machine comprehension alone.
Takeaways & Limitations
SQuAD questions were written for specific paragraphs, and future work should train document aggregation and the retriever-reader pipeline end to end.
Abstract
from arXiv · showhide
This paper proposes to tackle open- domain question answering using Wikipedia as the unique knowledge source: the answer to any factoid question is a text span in a Wikipedia article. This task of machine reading at scale combines the challenges of document retrieval (finding the relevant articles) with that of machine comprehension of text (identifying the answer spans from those articles). Our approach combines a search component based on bigram hashing and TF-IDF matching with a multi-layer recurrent neural network model trained to detect answers in Wikipedia paragraphs. Our experiments on multiple existing QA datasets indicate that (1) both modules are highly competitive with respect to existing counterparts and (2) multitask learning using distant supervision on their combination is an effective complete system on this challenging task.
1 Introduction
The paper frames open-domain factoid QA over Wikipedia as machine reading at scale, requiring both large-scale retrieval and detailed comprehension. DrQA combines these components and improves performance across multiple datasets through multitask learning and distant supervision.
- Problem: Machine reading at scale requires retrieving relevant articles from more than 5 million Wikipedia items before identifying answer spans.The task uses Wikipedia as a collection of articles rather than relying on its internal graph structure.
- Motivation: A single knowledge source forces precise search because evidence may appear only once, while preserving the challenge of deep text comprehension.This contrasts with systems that exploit redundancy across Wikipedia, knowledge bases, dictionaries, news, and books.
- Approach: DrQA pairs a Document Retriever using bigram hashing and TF-IDF matching with a recurrent-neural-network Document Reader that detects answer spans.The retriever returns relevant articles, which the reader scans for answers.
- Evaluation: The evaluation requires one open-domain system to perform well across multiple existing QA datasets at once.This operationalizes machine reading at scale as a cross-dataset evaluation setting.
- Results: Multitask learning with distant supervision improves performance across all evaluated datasets compared with single-task training.The paper evaluates both individual components and the complete system.
2 Related Work
Related work spans open-domain QA with unstructured documents, knowledge bases, machine comprehension, and multi-resource pipelines. DrQA instead emphasizes comprehension while restricting the system to a single Wikipedia resource.
- Open-domain QA: Open-domain QA originated as finding answers in collections of unstructured documents, whereas later work often used knowledge bases such as Freebase.Knowledge bases offer structured representations but have limitations including incompleteness and fixed schemas.
- Machine comprehension: Machine-comprehension research advanced through attention and memory-augmented architectures and datasets built from news, books, and Wikipedia.The paper tests these methods in an open-domain QA framework.
- Wikipedia QA: Prior Wikipedia QA systems commonly combined article text with semi-structured knowledge, other documents, or additional answer-matching resources.These approaches differ from using Wikipedia alone as the knowledge source.
- Full pipelines: Full-pipeline systems such as AskMSR, DeepQA, and YodaQA use search or multiple heterogeneous resources, often exploiting redundancy or structured evidence.DrQA makes comprehension more challenging by using only a single resource.
- Multitask learning: Multitask learning in QA has been used both to transfer knowledge across datasets and to build one system for differing question distributions.The related work describes these as complementary motivations for combining QA training datasets.
3 Our System: DrQA
DrQA narrows Wikipedia search with a classical retriever, then uses a recurrent reader to encode paragraphs, align them with questions, and predict answer spans. Its design combines lexical, contextual, and manually specified token features.
- System overview: DrQA consists of a Document Retriever for relevant articles and a Document Reader for extracting answers from one or several documents.The retriever narrows the search space before the comprehension model processes candidate documents.
- Document Retriever: The retriever compares questions and articles with TF-IDF bag-of-word vectors, adds n-gram features for local word order, and returns five Wikipedia articles.It uses an inverted index lookup followed by term-vector scoring.
- Document Reader: The reader applies a recurrent model to each paragraph and aggregates predicted answers across the retrieved documents.Paragraph tokens are encoded as feature vectors before recurrent processing.
- Paragraph encoding: A multi-layer bidirectional LSTM concatenates hidden units from its layers to encode contextual paragraph representations.The reader is inspired by neural machine-comprehension models.
- Token representation: Token features combine word embeddings, exact-match indicators, POS, NER, and normalized term frequency.Exact-match features indicate matching in original, lowercase, or lemma form; manual features capture token properties in context.
- Question alignment: Aligned question embeddings softly match each paragraph token with question words, including similar non-identical words such as car and vehicle.Attention scores are computed from nonlinear mappings of word embeddings.
- Prediction: Two independent classifiers predict answer-span start and end positions, and the system selects the highest-scoring span up to 15 tokens long.Final span scores combine start and end probabilities across paragraphs.
4 Data
The paper uses Wikipedia and several QA datasets to train and evaluate machine reading at scale, including distant supervision for datasets without associated paragraphs.
- Data sources: Wikipedia is the knowledge source for finding answers, while SQuAD trains and evaluates the Document Reader on paragraph-based machine comprehension.
- Data sources: CuratedTREC, WebQuestions, and WikiMovies provide additional open-domain QA evaluation data with different construction processes.
- Dataset accounting: Table 2 records question counts and distantly supervised training data across datasets, marking datasets without paragraph associations and the SQuAD development set.
- Training data: CuratedTREC, WebQuestions, and WikiMovies lack associated documents or paragraphs, so they cannot directly train the Document Reader.
- Training data: Distant supervision retrieves the top 5 Wikipedia articles, filters paragraphs using answer matches, length, named entities, and question–answer overlap, then retains high-scoring candidates.
- Training data: Additional distant-supervision examples for SQuAD can come from other pages or the same page, with around half originating outside SQuAD’s articles.
5 Experiments
Experiments evaluate DrQA’s retriever, reader, and full Wikipedia QA system across multiple benchmarks. The results show strong component performance, while multitask learning with distant supervision improves the unified system across datasets.
- Retriever Evaluation: The Document Retriever’s bigram-hashing and TF-IDF approach outperforms Wikipedia Search across all QA datasets, especially with bigram hashing.Okapi BM25 and cosine distance in word-embedding space perform worse.
- Reader Evaluation: The Document Reader achieves 70.0% exact match and 79.0% F1 on SQuAD’s test set, surpassing published results at the time.The reader uses a multi-layer recurrent neural network, implemented here with 3-layer bidirectional LSTMs.
- Full Wikipedia QA: The full DrQA system is evaluated on four datasets using SQuAD-only, dataset-specific distant-supervision fine-tuning, and jointly trained multitask distant supervision.The Document Retriever remains unchanged across these comparisons.
- Full Wikipedia QA: DrQA provides reasonable performance across all four datasets despite searching full Wikipedia rather than receiving the correct paragraph or using redundant external resources.On CuratedTREC, DrQA reaches 25.4 compared with YodaQA’s 31.3.
- Full Wikipedia QA: Multitask training with distant supervision outperforms SQuAD-only training on all four datasets, while dataset-specific distant-supervision fine-tuning also improves performance.The authors attribute much of the improvement to additional same-domain data rather than task transfer alone.
- Full Wikipedia QA: Full-Wikipedia SQuAD performance drops from 69.5 to 27.1 because questions written for specific paragraphs can become ambiguous without that context.With the correct document but not the paragraph, performance reaches 49.4; the retriever finds the answer 77.8% of the time.
6 Conclusion
The paper presents machine reading at scale as a challenging task requiring Wikipedia-based search and comprehension, and finds that integrating both supports an effective complete system across benchmarks. It also identifies two directions for improving DrQA’s training setup.
- Machine reading at scale uses Wikipedia as the unique knowledge source for open-domain question answering.
- Machine comprehension alone cannot solve the overall task, so DrQA integrates search, distant supervision, and multitask learning.
- Evaluations of individual components and the full system across multiple benchmarks showed the efficacy of the approach.
- Future improvements should train the reader over multiple paragraphs and documents and train the retriever-reader pipeline end to end.