Source-linked AI summary
Multi-hop Reading Comprehension through Question Decomposition and Rescoring
Sewon Min, Victor Zhong, Luke Zettlemoyer, Hannaneh Hajishirzi
TL;DR
Multi-hop reading comprehension requires aggregating evidence across several paragraphs, while decomposition annotations are difficult to obtain. DECOMPRC generates single-hop sub-questions through span prediction and rescoring, achieving state-of-the-art HOTPOTQA results with explainable evidence. Its scope excludes some implicit, commonsense, and counting or calculation-based reasoning cases.
Problem
Multi-hop reading comprehension requires aggregating evidence across several paragraphs, and question decomposition lacks readily available annotations.
Method
DECOMPRC decomposes compositional questions into single-hop sub-questions using span prediction, answers them with a single-hop RC model, and rescored decompositions to select the final answer.
Results
DECOMPRC achieves state-of-the-art results on HOTPOTQA while providing explainable evidence through sub-questions.
Takeaways & Limitations
With 400 labeled decomposition examples, DECOMPRC’s generated sub-questions are as effective as human-written ones, and answer-aware rescoring improves performance.
Takeaways & Limitations
DECOMPRC cannot handle some implicit multi-hop questions, sub-questions requiring commonsense inference, or reasoning beyond its supported types such as counting or calculation.
Abstract
from arXiv · showhide
Multi-hop Reading Comprehension (RC) requires reasoning and aggregation across several paragraphs. We propose a system for multi-hop RC that decomposes a compositional question into simpler sub-questions that can be answered by off-the-shelf single-hop RC models. Since annotations for such decomposition are expensive, we recast sub-question generation as a span prediction problem and show that our method, trained using only 400 labeled examples, generates sub-questions that are as effective as human-authored sub-questions. We also introduce a new global rescoring approach that considers each decomposition (i.e. the sub-questions and their answers) to select the best final answer, greatly improving overall performance. Our experiments on HotpotQA show that this approach achieves the state-of-the-art results, while providing explainable evidence for its decision making in the form of sub-questions.
1 Introduction
DECOMPRC addresses multi-hop reading comprehension by decomposing compositional questions into single-hop sub-questions and rescoring alternative decompositions. On HOTPOTQA, it outperforms published methods while providing explainable evidence.
- Multi-hop reading comprehension requires aggregating evidence across several paragraphs to answer a question.The paper illustrates this with a question requiring identification of an MVP from one paragraph and that player’s team from another.
- DECOMPRC decomposes compositional questions into simpler single-hop sub-questions using spans from the original question.The resulting sub-questions can be answered sequentially, with the first answer replacing ANS in the next sub-question.
- Sub-question generation is recast as span prediction, allowing competitive decomposition modeling with only 400 decomposition examples.This avoids reliance on distant supervision built from underlying relational logical forms.
- DECOMPRC rescored alternative decompositions using their answers before selecting the final answer.The approach considers multiple possible decompositions rather than deciding on one solely from the question at the beginning.
- DECOMPRC outperforms other published methods on HOTPOTQA and provides explainable evidence through sub-questions.The introduction also reports robustness relative to an end-to-end BERT baseline and effectiveness comparable to human-written sub-questions.
2 Related Work
Prior multi-hop reading-comprehension work often targets relational queries and uses attention over coreference chains. DECOMPRC instead decomposes natural-language questions over unstructured text, requiring only 400 decomposition examples and using context-aware rescoring.
- Most reading-comprehension questions can be answered from a single sentence, unlike multi-hop questions whose evidence spans multiple paragraphs.
- HOTPOTQA contains natural, hand-written multi-hop questions that are not based on relational queries.
- Prior multi-hop methods emphasize relational queries and attention models that reason over coreference chains.
- DECOMPRC answers natural-language questions through decomposition into single-hop sub-questions over unstructured text.The decomposed questions also make the model’s decisions explainable.
- Unlike approaches using relational-query supervision, DECOMPRC requires only 400 decomposition examples and selects among decompositions using all given context.The paper identifies context-aware decomposition selection as crucial to improving performance.
3 Model
DECOMPRC answers multi-hop questions by generating single-hop sub-questions, answering them with an off-the-shelf RC model, and rescoring alternative decompositions. Its span-based generation reduces supervision requirements, while context-aware selection chooses the final answer.
- Question decomposition: DECOMPRC decomposes an original multi-hop question into several single-hop sub-questions using span predictions and predefined reasoning types.The reasoning types are bridging, intersection, and comparison; a sample of 200 HOTPOTQA questions found that 92% of multi-hop cases belonged to these types.
- Decomposition rescoring: DECOMPRC generates answers for multiple reasoning types and uses a decomposition scorer to select the most suitable decomposition and final answer.The approach uses all available context when rescoring decompositions instead of choosing an operation from the question alone.
- Question decomposition: Sub-questions are formed by copying and lightly editing key spans from the original question, avoiding word-by-word generation.The Pointerc model predicts c indices in the question, which are used to compose sub-questions for each reasoning type.
- Supervision and evaluation: 400 decomposition examples suffice to train a competitive decomposition model, whose sub-questions are as effective as human-written ones in ablation studies.The span-prediction formulation was introduced because decomposition annotations are difficult to obtain and word-by-word generation requires substantial training data.
- Single-hop reading comprehension: For each decomposition, a single-hop RC model predicts an answer and supporting paragraph independently for each sub-question.The model can return a paragraph span, yes, no, or no answer, but independently processes paragraphs rather than jointly reasoning across them.
4 Experiments
Experiments on HOTPOTQA evaluate DECOMPRC across standard settings, baselines, robustness tests, and ablations. DECOMPRC outperforms the compared methods, while limited supervision produces sub-questions as effective as human-authored ones and rescoring improves performance.
- Results: DECOMPRC outperforms all baselines on the HOTPOTQA development set in both distractor and full wiki settings.The comparison includes BERT, BiDAF, and variants trained only on single-hop data.
- Results: DECOMPRC outperforms BERT by a large margin on development examples that are not solvable by single-hop models.The development set is divided into single-hop solvable and single-hop non-solvable examples using a BERT single-hop ensemble.
- Results: DECOMPRC achieves the best test-set result among models reporting both distractor and full wiki settings.The comparison uses official leaderboard results, including concurrent unpublished models.
- Robustness: Under modified distractors, DECOMPRC is more robust than BERT, while DECOMPRC–1hop train drops only 3.41 F1.Both methods degrade when the distractor paragraphs are changed, but the single-hop-trained DECOMPRC variant degrades less.
- Ablations: On 50 bridging questions, span-based sub-questions reach 69.44 F1 with 400 annotations, close to human span-based and free-form results.The reported scores are 69.44 F1 for Span trained on 400 annotations, 70.41 for Span human, and 70.76 for Free-form human.
- Ablations: The decomposition scorer outperforms confidence-based and pipeline selection, while remaining 6.2 F1 below the oracle.Using answer and evidence context from decompositions helps avoid cascading pipeline errors, according to the analysis.
5 Conclusion
DECOMPRC decomposes multi-hop questions into simpler single-hop sub-questions and achieves state-of-the-art performance on both HOTPOTQA settings. It also provides explainable evidence and greater robustness to adversarial settings than strong baselines.
- DECOMPRC decomposes multi-hop questions into simpler, single-hop sub-questions.
- 400 labeled decomposition examples suffice to generate high-quality sub-questions.
- Decomposition scoring provides further performance gains.
- DECOMPRC achieves state-of-the-art results on HOTPOTQA distractor and full wiki settings.
- Sub-questions provide explainable evidence, and the system is more robust to adversarial settings than strong baselines.
A Span Annotation
The span annotation procedure collects question spans for bridging and intersection questions, then uses them to generate heuristic sub-questions. Annotators first classify the reasoning type and exclude comparison questions from annotation.
- The procedure collects three points for bridging questions and two points for intersection questions.
- Annotators classify each question as bridging, intersection, one-hop, or neither before annotating spans.
- For bridging questions, annotators click the span start, span end, and head-word.
- Comparison questions are excluded from annotation because HOTPOTQA already labels them.
- For intersection questions, annotators mark the start and end of the second segment among three segments.
- The interface displays heuristically generated sub-questions after the required clicks are completed.
B Decompotision for Comparison
Comparison questions are decomposed through predefined discrete operations. The procedure extracts the compared entities, identifies an operation, and generates two sub-questions for each entity.
- The comparison procedure uses 10 predefined discrete operations to cover comparison operations.
- It first extracts the two entities being compared using four pointer positions.
- The first two pointer positions mark the first entity, while the latter two mark the second entity.
- The suitable discrete operation is then identified according to Algorithm 2.
- Two sub-questions are generated for each extracted entity.
- Given a question and two comparison entities, queries and the discrete operation can be obtained heuristically.
C Implementation Details
The implementation fine-tunes a lowercased pretrained BERT-BASE model using PyTorch and Hugging Face's BERT implementation. Training uses Adam with a learning rate of 5 × 10^-5.
- The system is implemented in PyTorch on top of Hugging Face's BERT implementation.
- The model is initialized from Google's pretrained lowercased BERT-BASE, with 12 Transformer layers and hidden dimension 768.
- Optimization uses Adam with learning rate 5 × 10^-5.
- Models receiving both question and paragraph inputs use maximum sequence length |S| = 300.
D Creating Inverted Binary Comparison Questions
The method automatically inverts seven of ten discrete comparison operations, producing 665 inverted questions.
- 665 inverted questions are produced through this process.
E A Set of Samples used for Ablations
Table 11 presents question IDs from samples used for ablations in Section 4.6.
- Table 11 lists question IDs from samples used for ablations in Section 4.6.