Source-linked AI summary
Simple and Effective Multi-Paragraph Reading Comprehension
Christopher Clark, Matt Gardner
TL;DR
The paper addresses how to adapt paragraph-level neural question answering models to documents, where multiple paragraphs and noisy supervision complicate confidence estimation. It samples paragraphs during training and applies shared normalization while combining the method with a document-QA pipeline. The resulting system achieves 71.3 F1 on TriviaQA web, substantially above prior work, while showing strong performance across several document-QA settings.
Problem
Adapting paragraph-level neural question answering to entire documents requires reliable confidence scores across paragraphs and handling noisy document-level supervision.
Method
The method samples paragraphs, including those without answers, uses shared normalization across paragraphs, and combines this training with TF-IDF selection and summed supervision.
Results
71.3 F1 on TriviaQA web is a 15 point absolute gain over prior work, alongside strong results on TriviaQA unfiltered and modified SQuAD.
Takeaways & Limitations
The combined approach advances document-level question answering and can be applied to building end-to-end open question answering systems.
Takeaways & Limitations
Confidence scores from naively adapted paragraph models may be poorly calibrated because their training does not make scores comparable across paragraphs.
Abstract
from arXiv · showhide
We consider the problem of adapting neural paragraph-level question answering models to the case where entire documents are given as input. Our proposed solution trains models to produce well calibrated confidence scores for their results on individual paragraphs. We sample multiple paragraphs from the documents during training, and use a shared-normalization training objective that encourages the model to produce globally correct output. We combine this method with a state-of-the-art pipeline for training models on document QA data. Experiments demonstrate strong performance on several document QA datasets. Overall, we are able to achieve a score of 71.3 F1 on the web portion of TriviaQA, a large improvement from the 56.7 F1 of the previous best system.
1 Introduction
The paper adapts paragraph-level neural question answering to entire documents by combining improved paragraph selection and noisy-supervision handling with calibrated multi-paragraph confidence training. Experiments report strong document-level performance, including a 71.3 F1 score on TriviaQA web.
- Document-level question answering remains difficult because extracting answers from retrieved texts is still an open challenge, and processing entire documents is computationally expensive.
- The paper improves pipelined question answering and introduces training that produces comparable confidence scores across paragraphs.
- The method samples answer-containing and non-answer paragraphs, then uses shared normalization to marginalize answer probability across paragraphs from the same document.
- 71.3 F1 on TriviaQA web represents a 15 point absolute gain over prior work on the test set.
2 Pipelined Method
The pipelined method selects relevant paragraphs with TF-IDF, handles distantly supervised answer-span noise by summing probability over matching spans, and uses a neural reading-comprehension model. Its paragraph-level architecture combines recurrent encoding, attention, residual self-attention, dropout, and span prediction.
- 2 Pipelined Method: The pipeline selects a single paragraph heuristically and passes it to a paragraph-level question-answering model.
- 2.1 Paragraph Selection: The TF-IDF selector chooses the paragraph closest to the question by cosine distance, improving answer-containing paragraph selection from 83.1% to 85.1% on TriviaQA web.
- 2.2 Handling Noisy Labels: Distant supervision labels every span matching the answer text, so the summed objective marginalizes probability over all matching answer spans.
- 2.3 Model: The model uses shared bidirectional GRU encodings and BiDAF-style bidirectional attention to build query-aware context representations.
- 2.3 Model: Residual self-attention, variational dropout, and concatenated word- and character-level embeddings provide additional model components.
- 2.3 Model: A final bidirectional GRU and linear layers predict answer start and end scores, whose softmax probabilities are trained against correct boundaries.
3 Confidence Method
The confidence method trains paragraph-level QA models to produce scores comparable across paragraphs, including those without answers, enabling selection of the highest-confidence span across a document.
- Confidence Method: Unnormalized span scores are used as paragraph-level confidence, and the highest-confidence answer span is selected across paragraphs at test time.For boundary-based models, a span score is the sum of its start and end token scores.
- Confidence Method: Naive paragraph training does not require confidence scores to be comparable between paragraphs, making cross-paragraph selection unreliable.The paper reports that such models can be poor at providing good confidence scores.
- 3.1 Shared-Normalization: The method samples paragraphs without answers and uses shared normalization across paragraphs from the same context.The shared objective marginalizes an answer candidate’s probability over all sampled paragraphs from the document.
- 3.1 Shared-Normalization: Shared normalization forces scores to be comparable across independently processed paragraphs without giving the model access to the other paragraphs.The normalization factor is shared across paragraphs from the same context.
- Alternative Training: An alternative concatenates sampled paragraphs during training, adding a learned separator token to test whether additional text improves irrelevant-text handling.The paragraphs are concatenated only for the alternative training approach.
- 3.3 No-Answer Option: The no-answer variant adds a score for selecting a special no-answer option, with δ indicating whether an answer exists.The model computes an additional score z representing the weight assigned to the no-answer possibility.
4 Experimental Setup
The experiments evaluate the methods on TriviaQA unfiltered, TriviaQA web, and SQuAD, using paragraph selection, document preprocessing, and training configurations tailored to each dataset.
- Datasets: The evaluation covers TriviaQA unfiltered, TriviaQA web, and SQuAD.TriviaQA uses trivia questions paired with web-search documents, while SQuAD contains Wikipedia articles and crowdsourced questions.
- Dataset Processing: TriviaQA web training uses all 530k question-document pairs rather than the subsampling used by prior work.The paper also adjusts answer-span labeling to match spans accepted as exact matches after official text normalization.
- Paragraph Processing: TriviaQA documents are merged into paragraphs of up to 400 tokens, with learned separator embeddings preserving formatting information.The maximum paragraph size is 400 unless otherwise stated.
- Paragraph Selection: For SQuAD and TriviaQA web, the system selects the top four TF-IDF-ranked paragraphs and samples two different paragraphs from that set each epoch.Higher-ranked paragraphs are more likely to contain the context needed to answer the question.
- Paragraph Selection: TriviaQA unfiltered uses a learned five-feature linear paragraph ranker combining TF-IDF distance, document position, preceding-token count, and question-word matches.The ranker is trained using the distantly supervised paragraph-selection objective.
- Results: The pipelined TriviaQA web system significantly improves over its baseline by combining preprocessing, TF-IDF selection, the sum objective, and the model design.The comparison is reported in Table 2.
5 Results
The proposed confidence-training methods improve paragraph-level QA when models process multiple paragraphs, with shared normalization generally strongest. The resulting pipeline achieves strong TriviaQA performance and adapts effectively to document-level SQuAD.
- TriviaQA Web: Shared-norm training was significantly ahead on verified TriviaQA web and tied with merge on the general set.No-answer training also improved performance as more paragraphs were used.
- TriviaQA Web: 71.32 F1 was achieved on the TriviaQA web test set, ahead of prior work.The corresponding EM score was 66.37.
- TriviaQA Unfiltered: The base model loses performance as more paragraphs are added on TriviaQA unfiltered, whereas shared normalization is strongest.This setting requires confidence scores that remain reliable across paragraphs.
- SQuAD: 67.4% of sampled SQuAD questions were context-independent, 22.6% document-dependent, and 10% paragraph-dependent.Documents were constructed by concatenating paragraphs from each article.
- SQuAD: 72.37 F1 and 64.08 EM were reached on document-level SQuAD using shared normalization with 15 paragraphs.The base model began dropping after more than two paragraphs, while shared normalization peaked at 15.
- SQuAD: 59.14 EM and 67.34 F1 were obtained on the re-evaluated document-level SQuAD dataset, exceeding the reported 49.7 EM.The source articles were downloaded at different dates and some questions may no longer have been answerable.
- Discussion: Training only on answer-containing paragraphs can perform very poorly across multiple paragraphs, especially on SQuAD.The authors identify shared normalization as the most effective remedy; no-answer and merge are moderately effective.
6 Related Work
Prior reading-comprehension work established large datasets and attention-based neural architectures, while open-QA systems commonly used retrieval and answer-selection pipelines. This paper focuses specifically on applying neural QA models across multiple paragraphs.
- Reading Comprehension Datasets: Large reading-comprehension datasets helped drive advances in neural models, beginning with Cloze-style benchmarks and later including SQuAD and WikiReading.
- Neural Reading Comprehension: Neural reading-comprehension systems typically use attention, with later work adding architectures such as self-attention and bidirectional attention.The paper follows this general neural-attention approach while incorporating recent architectural advances.
- Open QA: Open-QA systems use knowledge bases or natural-language resources to retrieve text, construct answer candidates, and select final answers.These systems are often organized as pipelines.
- Neural Open QA: Neural open-QA work combined SQuAD-trained models with retrieval engines, while other pipelines used rankers to select paragraphs for reading comprehension.The paper differs by explicitly addressing multi-paragraph processing.
7 Conclusion
The combined training, paragraph-selection, and model-design methods substantially advance TriviaQA performance and support direct application to open question answering systems.
- The combined method advances the state of the art on TriviaQA by a large stride and can be directly applied to open question answering systems.It combines non-answer paragraph sampling, shared normalization, paragraph selection, summed training, and the proposed model design.