Source-linked AI summary
Commonsense for Generative Multi-Hop Question Answering Tasks
Lisa Bauer, Yicheng Wang, Mohit Bansal
TL;DR
The paper addresses generative multi-hop QA, where models must synthesize non-span answers and may need unstated commonsense relations. It combines multi-hop attention, pointer generation, and grounded ConceptNet knowledge with gated incorporation, achieving state-of-the-art NarrativeQA performance.
Problem
Generative multi-hop QA requires synthesizing non-span answers from long contexts, often using relations not directly stated in the text.
Method
The model combines multi-hop attention and a pointer-generator decoder with PMI- and term-frequency-based ConceptNet path selection and selectively gated commonsense incorporation.
Results
The combined system achieves new state-of-the-art performance on NarrativeQA, while commonsense incorporation significantly improves performance across all metrics.
Takeaways & Limitations
Grounded commonsense paths can fill inference gaps between document-context reasoning hops in multi-hop generative QA.
Takeaways & Limitations
The reported comparisons use the standard non-oracle, unmasked, non-validated dataset setting, so results from different settings are not directly comparable.
Abstract
from arXiv · showhide
Reading comprehension QA tasks have seen a recent surge in popularity, yet most works have focused on fact-finding extractive QA. We instead focus on a more challenging multi-hop generative task (NarrativeQA), which requires the model to reason, gather, and synthesize disjoint pieces of information within the context to generate an answer. This type of multi-step reasoning also often requires understanding implicit relations, which humans resolve via external, background commonsense knowledge. We first present a strong generative baseline that uses a multi-attention mechanism to perform multiple hops of reasoning and a pointer-generator decoder to synthesize the answer. This model performs substantially better than previous generative models, and is competitive with current state-of-the-art span prediction models. We next introduce a novel system for selecting grounded multi-hop relational commonsense information from ConceptNet via a pointwise mutual information and term-frequency based scoring function. Finally, we effectively use this extracted commonsense information to fill in gaps of reasoning between context hops, using a selectively-gated attention mechanism. This boosts the model's performance significantly (also verified via human evaluation), establishing a new state-of-the-art for the task. We also show promising initial results of the generalizability of our background knowledge enhancements by demonstrating some improvement on QAngaroo-WikiHop, another multi-hop reasoning dataset.
1 Introduction
The paper targets NarrativeQA, a generative multi-hop task requiring synthesis beyond extractive fact finding, and introduces a strong pointer-generator baseline plus grounded commonsense integration.
- Earlier datasets either emphasize fact-based reading comprehension or extractive multi-hop fact linking rather than generating new information.
- NarrativeQA requires multi-hop reasoning over long, complex stories and synthesis of non-span answers.
- The Multi-Hop Pointer-Generator Model combines multiple attention hops with a pointer-generator decoder to read long passages and synthesize coherent responses.
- 41.49 Rouge-L and 17.33 METEOR were achieved on NarrativeQA’s summary subtask, substantially exceeding previous generative models.
- Grounded multi-hop ConceptNet paths selected with PMI and term-frequency scoring, then inserted through selectively gated attention, further improved NarrativeQA performance.
- The commonsense enhancements also produced promising initial improvement on QAngaroo-WikiHop, an extractive multi-hop dataset from another domain.
2 Related Work
Related work spans multi-hop reading comprehension and external commonsense incorporation. The paper differs by using multi-hop ConceptNet paths alongside multi-hop reasoning and answer synthesis.
- bAbI established reasoning over disjoint evidence, while newer QAngaroo and NarrativeQA datasets require synthesis across longer contexts.
- Prior MRC datasets and models often focused on attention-based understanding, extractive answering, or reasoning settings with simpler text.
- ConceptNet represents intricate commonsense relations as a graph, but goal-oriented graph traversal had not been extensively used for commonsense incorporation.
- Earlier external-knowledge approaches incorporated lexical information, single-hop ConceptNet relations, or commonsense triples into task models.
- This work uses multi-hop commonsense paths and incorporates different path aspects at successive reasoning hops to bridge inference gaps.
3 Methods
The methods combine a multi-hop pointer-generator baseline with grounded ConceptNet commonsense selection and a selectively gated mechanism for inserting external reasoning paths.
- 3.1 Multi-Hop Pointer-Generator Baseline: MHPGM embeds context and questions, performs iterative query-guided reasoning, resolves long-term dependencies with self-attention, and generates answers with a pointer-generator decoder.The architecture combines learned and ELMo embeddings, k reasoning cells, residual self-attention, and attention-based copying from context.
- 3.1 Multi-Hop Pointer-Generator Baseline: Each reasoning cell updates the context representation using bidirectional attention to focus on query-relevant information and emulate a reasoning hop.The initial representation is the embedded context, and the final reasoning output is produced by the last cell.
- 3.1 Multi-Hop Pointer-Generator Baseline: The decoder combines vocabulary generation with direct copying from the context through a learned selection distribution.This supports coherent generation while preserving rare context-specific tokens such as entities.
- 3.3 Commonsense Model Incorporation: The Necessary and Optional Information Cell incorporates selected commonsense paths between document-reasoning hops to fill gaps in inference.The cell extends the base reasoning cell and is designed to use optional helpful information.
- 3.2 Commonsense Selection and Representation: Commonsense selection constructs candidate ConceptNet paths grounded in the context and question, then ranks and filters them for relevance, quality, and variety.The process uses high-recall tree construction followed by initial node scoring, cumulative node scoring, and path selection.
- 3.2 Commonsense Selection and Representation: Term frequency scores context concepts, while PMI scores unconstrained ConceptNet nodes for logical consistency with the preceding path.These heuristics reduce noise and prioritize grounded, coherent multi-hop relations.
4 Experimental Setup
The experiments evaluate the model on generative NarrativeQA summaries and extractive WikiHop, using task-specific metrics and reported benchmark settings.
- Datasets: The evaluation uses the NarrativeQA summary subtask and the extractive QAngaroo WikiHop dataset.For WikiHop, candidate responses are ranked by generation probability.
- Evaluation Metrics: NarrativeQA is evaluated with BLEU-1, BLEU-4, METEOR, Rouge-L, and CIDEr metrics.
- NarrativeQA Evaluation: The NarrativeQA results table reports different metrics on the summaries test set and marks span-prediction models trained on the Rouge-L retrieval oracle.
- WikiHop Evaluation: The WikiHop results table reports model performance as percentage accuracy.
5 Results
The model outperforms prior generative systems on NarrativeQA and reaches competitive performance with span-prediction models, while commonsense integration further improves results and shows promising transfer to WikiHop.
- 5.1 Main Experiment: The model outperforms all generative models on NarrativeQA and is competitive with the top span-prediction models.Adding NOIC commonsense integration further improves performance on all metrics with p < 0.001.
- 5.1 Main Experiment: The model performs reasonably well on WikiHop and obtains promising initial improvements when commonsense information is added.The authors suggest smaller gains may reflect that approximately 11% of WikiHop examples require commonsense and that its commonsense is more fact-based.
- 5.1 Main Experiment: WikiHop results use the standard non-oracle, unmasked, and not-validated dataset, making them incomparable to results from different data settings.
- 5.2 Model Ablations: A single attention hop decreases performance by over 12 Rouge-L points, demonstrating the importance of multi-hop attention.ELMo embeddings and self-attention also contribute significantly to performance.
- 5.3 Commonsense Ablations: The commonsense selection and incorporation mechanism improves performance significantly across all metrics, whereas naïvely initializing embeddings with ConceptNet information does not yield statistically significant improvements.
6 Human Evaluation Analysis
Human evaluation examined both the relevance of selected commonsense relations and the comparative quality of answers produced with commonsense integration. The results supported the automatic evaluation findings and showed that selected commonsense was often useful.
- Commonsense Selection: 50% of sampled NarrativeQA cases required external commonsense knowledge, and relevant commonsense was selected in 34% of those cases.The analysis used 50 test examples and evaluated whether commonsense was necessary and whether the selected relations were relevant.
- Commonsense Selection: The commonsense selection algorithm provided useful commonsense in 48% of the evaluated cases.
- Ablation Analysis: All three ablation approaches showed no statistically significant Rouge-L or METEOR improvement, with bootstrap-test p ≥ 0.15.
- Model Performance: Human evaluation compared MHPGM+NOIC with MHPGM for answer correctness using blinded judgments from three external evaluators per example.The study randomly selected 100 NarrativeQA test examples and allowed evaluators to mark one response better or both responses similar in quality.
- Model Performance: The human evaluation results agreed with the automatic evaluation metrics and verified that the commonsense-incorporated model was better than MHPGM.
7 Conclusion
The paper combines a multi-hop generative QA architecture with grounded commonsense selection and NOIC integration. Together, these components achieve a new state-of-the-art result on NarrativeQA.
- Conclusion: The architecture combines multiple hops of bidirectional attention with a pointer-generator decoder to reason across context and synthesize answers.
- Conclusion: Grounded commonsense paths are selected to fill inference gaps, then incorporated during multi-hop reasoning through Necessary and Optional Information Cells.
- Conclusion: The complete approach achieves a new state-of-the-art on NarrativeQA.
A.1 Experimental Setup
Experiments evaluate the model with and without commonsense on generative NarrativeQA and multiple-choice WikiHop. The setup uses standard task metrics and a three-hop attention model with specified training settings.
- Datasets: NarrativeQA uses story or story-summary passages with questions about complex narrative aspects, and its human-written answers usually cannot be found directly in the passage.The paper focuses on the summary subtask, whose summaries can contain up to 1000 words.
- Datasets: WikiHop is a fact-based multiple-choice multi-hop dataset requiring models to read several documents before ranking candidate answers.Candidate responses are ranked by generation probability, while candidate documents are first ranked using TF-IDF cosine distance.
- Evaluation Metrics: NarrativeQA is evaluated with Bleu-1, Bleu-4, METEOR, Rouge-L, and CIDEr, whereas WikiHop is evaluated with accuracy.
- Model Analysis: Increasing attention hops produces more focused activation on passage keywords relevant to the question.
- Training Details: Training minimizes negative log-likelihood with Adam, using a 0.001 learning rate, 0.2 dropout, batch size 24, 256-dimensional embeddings, hidden size 128, and k = 3 hops.Inference uses greedy decoding.
A.2 Commonsense Extraction Examples
The paper illustrates commonsense extraction on questions whose answers require bridging implicit connections between context or question concepts. Highlighted words identify the relevant bridge and supporting context.
- Commonsense Extraction Examples: The examples show extracted commonsense relations for questions that require commonsense to reach an answer.
- Commonsense Extraction Examples: Bolded question and commonsense words mark cases where the extracted knowledge bridges implicitly connected concepts.
- Commonsense Extraction Examples: Bolded context words identify the information that is key to answering the question through commonsense.
A.3 Commonsense Integration Visualization
The section visualizes how selectively gated commonsense attention is integrated across three reasoning hops and three examples. It uses gate-derived activation values and selected commonsense paths to inspect this integration.
- Commonsense Integration Visualization: The visualization uses z_i gate values to show how much commonsense-attention representation contributes to the output context representation.For each hop i ∈ {1, 2, 3}, darker blue corresponds to a larger average of (1 − z_i), normalized within that hop.
- Example 1: Example 1 is visualized across first, second, and third attention hops using activation values (1 − z_1), (1 − z_2), and (1 − z_3).The corresponding selected commonsense paths are listed for Example 1.
- Example 2: Example 2 is visualized across first, second, and third attention hops using activation values (1 − z_1), (1 − z_2), and (1 − z_3).The corresponding selected commonsense paths are listed for Example 2.
- Example 3: Example 3 is visualized across first, second, and third attention hops using activation values (1 − z_1), (1 − z_2), and (1 − z_3).The corresponding selected commonsense paths are listed for Example 3.