Source-linked AI summary
Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, Bowen Zhou
TL;DR
Multi-hop multi-document QA must reason across multiple sources while explaining answers with supporting evidence. SAE selects answer-related documents, then jointly predicts answers and supporting sentences through multi-task learning and attention-based interaction, achieving competitive HotpotQA distractor-setting performance.
Problem
Multi-hop QA requires reasoning across multiple documents and explaining answers with supporting evidence, while current approaches have limited explainability and often process unrelated documents.
Method
SAE filters answer-unrelated documents with pairwise learning-to-rank, then jointly predicts answer spans and supporting sentences using multi-task learning, GNN reasoning, and mixed attention.
Results
SAE achieves top competitive results against existing systems on the HotpotQA blind test set and improves over the baseline by more than 28% joint EM and 25% joint F1 on development data.
Takeaways & Limitations
Selecting relevant documents and jointly modeling answers with supporting sentences provides an effective interpretable approach for multi-hop reading comprehension.
Takeaways & Limitations
The approach assumes each example provides labeled support sentences and an answer type, with gold document labels derived from these annotations.
Abstract
from arXiv · showhide
Interpretable multi-hop reading comprehension (RC) over multiple documents is a challenging problem because it demands reasoning over multiple information sources and explaining the answer prediction by providing supporting evidences. In this paper, we propose an effective and interpretable Select, Answer and Explain (SAE) system to solve the multi-document RC problem. Our system first filters out answer-unrelated documents and thus reduce the amount of distraction information. This is achieved by a document classifier trained with a novel pairwise learning-to-rank loss. The selected answer-related documents are then input to a model to jointly predict the answer and supporting sentences. The model is optimized with a multi-task learning objective on both token level for answer prediction and sentence level for supporting sentences prediction, together with an attention-based interaction between these two tasks. Evaluated on HotpotQA, a challenging multi-hop RC data set, the proposed SAE system achieves top competitive performance in distractor setting compared to other existing systems on the leaderboard.
Introduction
Multi-hop multi-document QA requires models to reason across scattered documents while identifying supporting evidence, yet existing approaches often process distracting documents and provide limited explanations. SAE addresses these gaps by selecting answer-related documents before jointly predicting answers and supporting sentences.
- Introduction: SAE targets multi-hop QA settings where answers require reasoning across multiple documents and supporting evidence may be scattered across contexts.
- Introduction: The system first filters answer-unrelated documents using a classifier with a pairwise learning-to-rank loss to reduce distraction and identify gold documents.
- Introduction: SAE jointly predicts answer spans and supporting sentences through token-level and sentence-level multi-task learning on selected documents.
- Introduction: The system is evaluated on HotpotQA distractor setting and achieves top competitive results on the blind test leaderboard.
- Introduction: Mixed attentive pooling uses answer-span logits and token self-attention to connect answer prediction with supporting-sentence prediction.
Related work
Prior multi-hop QA work adapts single-document methods or uses graph neural networks, while explainable approaches differ in how they locate evidence. SAE distinguishes itself by filtering documents first and directly predicting supporting sentences.
- Related work: Prior multi-hop QA studies adapt single-document techniques or use graph neural networks for reasoning across documents.
- Related work: Existing explainable QA approaches include interpretable single-document models and systems that locate supporting entities before deriving support sentences.
- Related work: SAE differs by filtering answer-unrelated documents before answer prediction and directly finding supporting sentences through joint training.
Methodology
SAE first selects answer-related documents, then jointly predicts answer spans and supporting sentences using interacting multi-task components and sentence-level graph reasoning.
- Select gold documents: SAE ranks documents pairwise to select top-ranked answer-related documents and remove distracting information before downstream prediction.Document representations interact through multi-head self-attention, and relevance is computed from pairwise comparisons thresholded at 0.5.
- Select gold documents: The selector assigns higher scores to gold documents containing the answer span than to other gold or non-gold documents.Gold documents receive score 1, while a gold document containing the answer span receives score 2.
- Answer and Explain: On selected documents, BERT encodes the question and concatenated context for joint answer-span and supporting-sentence prediction.Answer prediction uses start and end position logits, while sentence prediction classifies contextual sentence representations as supporting evidence.
- Answer prediction: Answer prediction derives a span from start and end positions and trains with cross-entropy losses over possible token indices.The two output dimensions represent start and end positions, with labels in the range [0, L-1].
- Supporting sentence prediction: Mixed attentive pooling combines self-attention with answer start and end logits to summarize each sentence for support prediction.The resulting sentence embeddings initialize a multi-relational GCN whose message passing supports reasoning across sentence nodes and documents.
Experiments
SAE is evaluated on HotpotQA using answer and supporting-sentence prediction, document-selection ablations, reasoning-type analysis, and attention visualization. The system achieves strong benchmark performance, while ablations show benefits from document filtering, attention-based sentence summarization, and GNN message passing.
- Results: SAE improves over the baseline by more than 28% joint EM and 25% joint F1 on HotpotQA development data, and exceeds DFGN and QFE by over 5% on both metrics.These comparisons are reported for the development set in Table 1.
- Results: SAE-large ranks No. 2 on the HotpotQA blind-test leaderboard and outperforms the best single model in support sentence prediction EM and joint EM.The ranking and comparisons apply at the time of submission.
- Ablation studies: Document-selection ablations show that multi-head self-attention provides the largest gain, pairwise ranking further improves selection, and better filtering improves answer and support prediction.The study measures selection accuracy and recall, including selection of the document containing the answer span.
- Ablation studies: Replacing attention-based sentence summarization with averaging lowers joint EM by almost 2%, whereas using only one attention source causes a marginal decline.The results support attention-based sentence summarization, with mixed attention providing a smaller additional benefit.
- Ablation studies: Removing GNN message passing reduces EM by ∼1.4% and F1 by ∼0.9%, while removing cross-document edge types further reduces performance.Connections among sentences within the same document contribute most, but cross-document information remains useful.
- Result analysis: SAE performs better for both bridge and comparison reasoning, with a larger improvement over DFGN for bridge reasoning; its attention heatmap focuses on supporting tokens such as “american”.The heatmap example concerns whether Scott Derrickson and Ed Wood shared a nationality.
Conclusion
The paper proposes an effective, interpretable system for multi-hop reading comprehension over multiple documents. It filters unrelated documents before jointly predicting answers and supporting evidence, achieving competitive HotpotQA blind-test results.
- The proposed system addresses multi-hop reading comprehension over multiple documents with an effective and interpretable approach.
- It first filters out unrelated documents, then jointly predicts the answer and supporting evidence.
- The system attains competitive results on the HotpotQA blind test set compared with existing systems.