Source-linked AI summary
Cognitive Graph for Multi-Hop Reading Comprehension at Scale
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, Jie Tang
TL;DR
Multi-hop machine reading requires stronger reasoning, explainability, and scalability than existing retrieval-extraction systems provide. CogQA coordinates iterative entity extraction and graph reasoning through System 1 and System 2, with BERT and GNN implementation achieving state-of-the-art HotpotQA results.
Problem
Multi-hop question answering remains challenging because existing systems provide limited reasoning, unordered sentence-level explanations, and restricted source scalability.
Method
CogQA iteratively builds a cognitive graph by coordinating System 1 entity extraction with System 2 graph-based reasoning to discover answers and reasoning paths.
Results
CogQA surpasses previous works and competitors substantially across all metrics, with joint correct answers reaching 33.4% versus 7.9% for Yang et al. (2018) and 30.3% for QFE.
Takeaways & Limitations
The cognitive graph supports ordered, entity-level explainability and relational reasoning while enabling multi-hop reading comprehension at scale.
Takeaways & Limitations
The paper identifies improving reasoning reliability and System 1 capacity as future research directions.
Abstract
from arXiv · showhide
We propose a new CogQA framework for multi-hop question answering in web-scale documents. Inspired by the dual process theory in cognitive science, the framework gradually builds a \textit{cognitive graph} in an iterative process by coordinating an implicit extraction module (System 1) and an explicit reasoning module (System 2). While giving accurate answers, our framework further provides explainable reasoning paths. Specifically, our implementation based on BERT and graph neural network efficiently handles millions of documents for multi-hop reasoning questions in the HotpotQA fullwiki dataset, achieving a winning joint $F_1$ score of 34.9 on the leaderboard, compared to 23.6 of the best competitor.
1 Introduction
CogQA addresses multi-hop QA's reasoning, explainability, and scalability challenges by coordinating implicit retrieval with explicit graph reasoning. Its BERT- and GNN-based implementation organizes entity-level reasoning and reports strong performance on HotpotQA.
- Multi-hop QA is presented as the next frontier because single-paragraph systems often select question-matched sentences without complex reasoning.
- Explicit reasoning paths are needed for verification, with ordered entity-level explanations offering more structure than unordered sentence-level supporting evidence.
- Scalability remains essential because retrieval-extraction systems pre-retrieve only a few paragraphs, limiting access to information distributed across massive document collections.
- CogQA coordinates System 1 extraction with System 2 reasoning, iteratively building a cognitive graph of entities and answer candidates until answers are found.
- The framework's contributions include ordered entity-level explainability, relational reasoning, and BERT-GNN performance that surpasses previous works across metrics.
2 Cognitive Graph QA Framework
CogQA iteratively expands a directed cognitive graph while System 1 extracts spans and System 2 performs graph-based reasoning. Frontier nodes and predecessor clues guide continued exploration, supporting scalable and explicit multi-hop deduction.
- The cognitive graph represents entities or possible answers as nodes connected by directed edges for step-by-step deduction.
- CogQA initializes the graph with question entities, visits frontier nodes, collects predecessor clues, fetches paragraphs, and generates semantic representations.
- System 2 updates node representations using graph structure and prepares clues for successor nodes, enabling relational reasoning over the cognitive graph.
- The process stops when no frontier nodes remain or the graph is large enough, then returns the answer node with the highest predicted score.
- System 1 extracts answer candidates and next-hop entities from each paragraph, expanding the graph with new hop and answer nodes.
- New nodes and nodes receiving incoming edges become frontier nodes and may be revisited when additional predecessor clues emerge.
- Because all-paragraph access is limited to specific title-index lookups, the framework is designed to scale without time consumption growing significantly with paragraph count.
3 Implementation
CogQA implements System 1 with BERT and System 2 with a graph neural network, iteratively extracting spans, expanding the cognitive graph, and propagating node representations. Its predictor handles HotpotQA question categories with span or classification outputs, while training uses supervised span annotations and negative sampling.
- System 1 and System 2: BERT serves as System 1 and a GNN serves as System 2 in the CogQA implementation.BERT extracts entities, answer candidates, and semantic vectors; the GNN updates graph-node representations through propagation.
- System 1: System 1 visits nodes using question, predecessor-sentence clues, and the node paragraph as BERT input.Answer nodes may lack paragraphs, in which case semantic vectors are computed from the clues portion without span extraction.
- Span extraction: Top K start positions determine candidate span ends within the maximum span length, while probabilities below a negative threshold are discarded.The implementation uses the [CLS] start probability as the negative threshold and expands remaining answer spans and next-hop spans into graph nodes.
- System 2: GNN propagation updates hidden representations from neighbor-aggregated vectors, and jointly updating all finalized graph nodes is adopted for efficiency.The paper reports no apparent performance difference between asynchronous frontier-node updates and multiple-step updates after graph finalization.
- Prediction: HotpotQA special questions use a two-layer FCN to predict answer spans, whereas alternative and general questions use binary classification over paired node representations.Alternative questions return an entity name; general questions return “yes or no.”
- Training: Training uses pre-extracted next-hop and answer spans, fuzzy entity matching, negative nodes, and cross-entropy objectives for span extraction and answer-node prediction.Multiple next-hop spans receive distributed ground-truth weights, while each paragraph has at most one answer span.
4 Experiment
Experiments on HotpotQA fullwiki evaluate CogQA’s scalability, reasoning, robustness across hop counts, and explainability. CogQA substantially outperforms prior systems, while ablations indicate that graph-based reasoning—not BERT alone—is central to the gains.
- Dataset: 112,779 questions in HotpotQA fullwiki include 84% requiring multi-hop reasoning, with hard multi-hop development and test sets.The training, development, and test sets contain 90,564, 7,405, and 7,405 questions, respectively.
- Metrics: Evaluation jointly measures answer and supporting-fact Exact Match and F1, requiring both answer and evidence correctness for Joint EM.Joint precision and recall multiply answer and supporting-fact precision and recall before calculating joint F1.
- Overall performance: CogQA outperforms all baselines on all metrics by a significant margin in the HotpotQA fullwiki setting.The reported results are summarized in Table 1, whose test set is not public and provides EM and F1 through the benchmark maintainer.
- Logical rigor: 33.4% of CogQA’s answers are jointly correct according to all necessary supporting facts, compared with 7.9% for Yang et al. and 30.3% for QFE.AnsEM is defined as the proportion of joint-correct answers among correct answers and is used to assess logical rigor.
- Multi-hop reasoning: As hop number increases, Yang et al. and Yang et al.-IR decline dramatically, whereas CogQA remains surprisingly robust across eight question types.CogQA shows no improvement on alternative and general questions because judgment evidence cannot be inferred from supporting facts, leaving a supervision gap.
- Ablation studies: CogQA-onlyR remains significantly better than the baseline despite using the same initial paragraphs, while CogQA-onlyQ also outperforms all baselines using question entities alone.These ablations attribute much of the improvement to the framework rather than elaborate retrieval initialization.
- Ablation studies: Vanilla BERT performs similarly to or slightly worse than Yang et al. on multi-hop QA, indicating that BERT is necessary but not the key improvement factor.The authors suggest Yang et al.’s architecture better leverages supporting-fact supervision.
- Ablation studies: Removing System 2 reduces CogQA’s improvement over the best competitor on answer metrics by about 50%, highlighting the GNN’s reasoning role.CogQA-sys1 retains the iterative framework but outputs answer spans with maximum predicted probability instead of cascading graph reasoning.
5 Related work
Related work progresses from single-paragraph machine reading comprehension toward multi-hop and open-domain QA. Existing retrieval-extraction approaches improve scalable answer extraction, but complex multi-hop reasoning remains insufficiently addressed.
- Machine Reading Comprehension: Neural models shifted machine reading comprehension from cloze-style tasks toward more complex question-answering benchmarks.BiDAF and R-net improved answer extraction, while BERT-based models approached or exceeded human performance on single-paragraph MRC-QA.
- Multi-Hop QA: Multi-hop QA datasets historically used limited knowledge-base schemas or multiple-choice settings, while HotpotQA provided a higher-quality benchmark.Noise in earlier datasets restricted multi-hop QA development.
- Open-Domain QA: Open-domain QA addresses settings where the supporting-evidence search space is extremely large.Information retrieval methods for paragraph-level answers have been studied since the 1990s.
- Open-Domain QA: DrQA advanced open-domain QA through a retrieval-extraction framework that extracts answers from retrieved paragraphs.Later work enhanced retrieval using heuristic sampling or reinforcement learning.
- Open-Domain QA: Prior open-domain approaches largely neglected the framework revisions needed for complex reasoning and necessary revisits.This limitation is presented in the context of retrieval improvements based on heuristic sampling and reinforcement learning.
6 Discussion and Conclusion
CogQA organizes multi-hop reading at scale as a cognitive graph and reports state-of-the-art results on HotpotQA. The authors identify reliability, system interaction, and cross-task generalization as future research directions.
- CogQA organizes multi-hop machine reading at scale through a cognitive graph.
- CogQA obtains state-of-the-art results on the HotpotQA dataset, supporting the framework’s efficacy.
- Future directions include neural logic for reliability, improved System 1 and System 2 interaction, and generalization to conversational AI and sequential recommendation.