Source-linked AI summary
Multi-hop Question Answering via Reasoning Chains
Jifan Chen, Shih-ting Lin, Greg Durrett
TL;DR
Multi-hop QA asks systems to gather clues across text, yet strong end-to-end performance may not reflect the intended reasoning. This paper extracts heuristic-supervised sentence chains and uses them for answer prediction, reporting state-of-the-art performance on WikiHop and strong performance on HotpotQA.
Problem
Multi-hop QA requires combining information across documents, but high-performing models may not perform the expected reasoning and intermediate behavior is difficult to evaluate.
Source: §1 · p. 1Method
The model learns to extract discrete sentence chains from heuristic training labels and passes the chains to a BERT-based answer predictor.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Extracted chains provide an intermediate evidence representation that can help gauge model behavior beyond final answer accuracy.
Source: §1 · pp. 1–2On HotpotQA, BERT with annotated supporting facts achieves 75.4% F1, only 5% higher than BERT-Para with top 5 extracted chains.
Source: §5.3 · p. 7
Abstract
from arXiv · showhide
Multi-hop question answering requires models to gather information from different parts of a text to answer a question. Most current approaches learn to address this task in an end-to-end way with neural networks, without maintaining an explicit representation of the reasoning process. We propose a method to extract a discrete reasoning chain over the text, which consists of a series of sentences leading to the answer. We then feed the extracted chains to a BERT-based QA model to do final answer prediction. Critically, we do not rely on gold annotated chains or "supporting facts:" at training time, we derive pseudogold reasoning chains using heuristics based on named entity recognition and coreference resolution. Nor do we rely on these annotations at test time, as our model learns to extract chains from raw text alone. We test our approach on two recently proposed large multi-hop question answering datasets: WikiHop and HotpotQA, and achieve state-of-art performance on WikiHop and strong performance on HotpotQA. Our analysis shows the properties of chains that are crucial for high performance: in particular, modeling extraction sequentially is important, as is dealing with each candidate sentence in a context-aware way. Furthermore, human evaluation shows that our extracted chains allow humans to give answers with high confidence, indicating that these are a strong intermediate abstraction for this task.
1 Introduction
Multi-hop QA requires integrating clues across documents, yet high-performing models may not perform the intended reasoning and their intermediate behavior can be difficult to evaluate. The paper extracts discrete reasoning chains before answer prediction, training on heuristic chains and achieving strong results on WikiHop and HotpotQA.
1 Introduction
- Multi-hop datasets test aggregation of clues across documents, but models with high scores may not perform the expected reasoning.Evaluating intermediate components such as attention is difficult, motivating inductive bias for more appropriate reasoning.Source: §1 · p. 1
- The proposed two-stage model extracts discrete sentence chains that connect a question to answer-relevant facts, then predicts the answer separately.The extractor scores sentence sequences and produces an n-best list of chains by beam search.Source: §1 · p. 1
- Heuristic chains use coreference and named-entity recognition to find paths between sentences, without requiring these annotations at test time.The method runs from raw text at test time and does not use answers or answer candidates to construct training graphs.Source: §1 · p. 2
- WikiHop performance is state of the art by a substantial margin, while HotpotQA results are strong and outperform several recent systems.On HotpotQA, derived chains are essentially as effective as the dataset’s ground-truth supporting facts.Source: §1 · p. 2
- The extracted chains are presented as generalizing across datasets, comparable to human-annotated chains, and a useful evidence representation for question answering.They also provide an intermediate output for gauging model behavior beyond final task accuracy.Source: §1 · pp. 1–2
2 Question Answering via Chain Extraction
A reasoning chain is a sentence sequence connecting a question to an answer-relevant fact through relations that let a reader link adjacent sentences. Such chains can use shared entities or coreference, but a connected sequence may still be irrelevant to the question.
2 Question Answering via Chain Extraction
- A reasoning chain links the question to an answer-relevant fact through adjacent sentences related by a shared entity, event, temporal structure, or other textual relation.The relation should let a human reader connect the information in the sentences.Source: §2 · p. 2
- In the HotpotQA example, a chain can follow Shirley Temple to information about her government position, while another route makes a less justified inference from a sequel.Other connected sentence sequences may not imply the answer when their shared entity is unrelated to the question.Source: §2 · p. 2
- The extractor learns from training-time coreference and temporal cues but does not require a coreference system at test time.It extracts chains directly from learned patterns in the text.Source: §2 · p. 2
- The approach addresses three linked tasks: selecting pseudo-groundtruth chains, modeling chain extraction, and converting extracted chains into answers.Source: §2 · p. 2
3 Learning to Extract Chains
The method constructs heuristic oracle chains, learns to extract sentence sequences, and uses the top predicted chains for answer prediction. Its extractor combines question-aware sentence representations with sequential pointer-based selection.
3.1 Heuristic oracle chain construction
- The heuristic graph links sentences that share a named entity and links all sentence pairs within a paragraph.Shared-entity links can be noisy, while within-paragraph links are treated as connected based on coreference or bridging relations.Source: §3.1 · p. 3
- Oracle chains are selected either by shortest path or by the highest question-overlap Rouge-F1 score, which favors more complete chains over shortcuts.The search begins at the question node and enumerates possible chains leading to the answer.Source: §3.1 · p. 3
3.2 Chain extraction model
- BERT-Para encodes each paragraph jointly with the question and uses max-pooling over sentence spans to form sentence representations.This paragraph-factored scheme is described as more efficient and scalable than encoding a full context that may run to thousands of words.Source: §3.2 · p. 3
- An LSTM pointer network selects sentences sequentially from the encoded representations, and reinforcement-learning fine-tuning did not improve on oracle pretraining for the two datasets considered.At each step, sentence selection is conditioned on earlier chain choices.Source: §3.2 · p. 4
3.3 Answer prediction
- At evaluation, the answer module receives sentences from the top k predicted chains to account for multiple plausible chains.The answer prediction module is a BERT-based QA system run on the pruned context.Source: §3.3 · p. 4
4 Experimental Setup
The experiments use WikiHop and HotpotQA, two multi-document QA datasets with different answer formats. Implementation details describe chain-graph construction and task-specific answer prediction, and Table 1 lists chain and QA measures on HotpotQA.
4.1 Datasets
- WikiHop contains around 40k questions with passages and multiple-choice answers, averaging around 10 choices per question.Source: §4.1 · p. 4
- HotpotQA contains 113k Wikipedia-based question-answer pairs, uses variable-length answer spans, and annotates relevant sentences as supporting facts.Source: §4.1 · p. 4
4.2 Implementation Details
- The chain graph uses an off-the-shelf AllenNLP NER system and ignores entities appearing explicitly more than five times across sentences.The stated rationale is that very common entities can introduce spurious edges rather than useful ones.Source: §4.2 · p. 4
- Table 1 reports HotpotQA dev-set chain length, answer-containing fraction, supporting-fact F1, and final QA F1 for different models and chain oracles.It uses only the first-beam chain.Source: Table 1 · p. 5
- For WikiHop, candidate answers are encoded with BERT and scored by the dot product of each candidate representation with the question-plus-chains representation.Source: §4.2 · p. 5
- For HotpotQA, the answer module predicts span start and end positions and a question type, using bert-large-uncased.Source: §4.2 · p. 5
5 Results
The final system performs strongly on both blind-test benchmarks, with extracted chains effectively substituting for supporting-fact supervision on HotpotQA. The analyses find benefits from context-aware and sequential extraction, while showing that chains can also contain distractors.
5.1 Comparison of Chain Extraction Methods
- Around 5% better answer prediction performance comes from encoding more context in BERT-Para than BERT-Sent, with other parts fixed.The authors suggest context may help BERT capture cross-sentence relations and find supporting evidence.Source: §5.1 · p. 5
- 76% of the time, BERT-Para’s one-best chain of just 2 extracted sentences contains the answer.The average human-labeled supporting facts comprise 2.4 sentences.Source: §5.1 · p. 6
- 64 vs. 56 supporting-fact F1 favors Q-Overlap over the Shortest oracle, which supplies only the answer-containing sentence.Q-Overlap includes additional relevant sentences.Source: §5.1 · p. 6
- Unioning the top 5 chains and truncating them to 5 sentences greatly increases answer recall and downstream F1.The chains contain permutations of roughly the same sentences, so unioning them does not greatly increase average length; retaining multiple chains also preserves uncertainty and handles comparison questions.Source: §5.1 · p. 6
5.2 Results compared to other systems
- On the blind test sets, the model significantly outperforms prior models on WikiHop and performs strongly on HotpotQA, including against some systems using human-labeled supporting facts.On HotpotQA, comparisons with systems using annotated supporting facts as extra supervision are not direct.Source: Table 2 · p. 6, §5.2 · p. 6
5.3 Evaluation of chains
- On all datasets, sequential extraction yields higher QA performance than unordered extraction, especially on HotpotQA-Hard; its WikiHop improvement is more marginal.The authors note that question-answer option correlations on WikiHop may make evidence quality less influential on downstream performance.Source: §5.3 · pp. 6–7, Table 3 · p. 7
- Human answer F1 and confidence are similar across evidence sets; BERT reaches 75.4% F1 with annotated facts, only 5% above BERT-Para with top 5 extracted chains.The authors note that other QA challenges, including question decomposition, remain complementary to chain extraction.Source: Table 4 · p. 7, §5.3 · p. 7
- The extractor can learn coreference jumps and propose separate chains for comparison entities, but repeated entities can also distract it into selecting unrelated sentences.For comparison questions, unioning the entity-specific chains lets the answer predictor use both.Source: §5.3 · pp. 7–8
6 Related Work
Prior multi-hop systems iteratively gather information or search entity graphs, while related work also selects sentences or documents before answering. This model is positioned as a finer-grained account of how an answer is derived.
6 Related Work
- Entity-graph methods construct graphs at training and test time, whereas this model needs entities only during training.Source: §6 · p. 8
- Memory Network models gather information iteratively from different passage regions using a memory cell.Source: §6 · p. 8
- Coarse-to-fine systems select relevant sentences before using a more expensive model to produce answers, while other extractors use distant supervision or answer scores.Choi et al. (2017) combine a fast sentence selector with an RNN; Wang et al. (2019) use distant-supervision labels, and Wang et al. (2018), Indurthi et al. (2018), and Min et al. (2018) use answer-prediction scores as supervision.Source: §6 · p. 8
- Open-domain multi-hop systems iteratively retrieve relevant documents, whereas this model operates at sentence level to show how the answer is derived.Source: §6 · p. 8
7 Discussion and Conclusion
The authors report effective chains and strong results on two large datasets, while noting that both datasets may impose less multi-hop reasoning than intended. They frame explicit reasoning as a first step toward systems for more challenging QA settings.
7 Discussion and Conclusion
- The extracted chains are as effective as human annotations and achieve strong performance on two large datasets.Source: §7 · p. 8
- HotpotQA and WikiHop have aspects that make them require multi-hop reasoning less strongly than they otherwise might.Source: §7 · p. 8
- The authors view explicit reasoning and chain-like representations as critical for QA on more challenging datasets, and this work as a first step toward that goal.Source: §7 · p. 8