Source-linked AI summary
Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering
Mingxuan Ju, Wenhao Yu, Tong Zhao, Chuxu Zhang, Yanfang Ye
TL;DR
Open-domain QA readers can miss relationships among entities in questions and retrieved passages, producing answers that contradict facts. GRAPE incorporates KG relations through localized graphs and a graph neural network in the reader; on three benchmarks, it improves exact-match performance over state-of-the-art readers with the retriever and passages held fixed.
Problem
State-of-the-art open-domain QA readers can miss complex relationships between entities in questions and retrieved passages, leading to answers that contradict facts.
Source: §1 · p. 1Method
GRAPE builds a localized bipartite graph for each question-passage pair, initializes entity representations from reader hidden states, and uses a GNN to incorporate KG relations into the reader.
Source: §3.2.1 · pp. 3–4Results
Takeaways & Limitations
GRAPE demonstrates reader-side use of KG facts can improve performance without changing the retriever or retrieved passages.
Source: §1 · p. 2, §2 · p. 3GRAPE only solves errors from fact-related examples and uses Wikidata relations, potentially omitting facts from sources such as Freebase.
Source: §6 · p. 9
Abstract
from arXiv · showhide
A common thread of open-domain question answering (QA) models employs a retriever-reader pipeline that first retrieves a handful of relevant passages from Wikipedia and then peruses the passages to produce an answer. However, even state-of-the-art readers fail to capture the complex relationships between entities appearing in questions and retrieved passages, leading to answers that contradict the facts. In light of this, we propose a novel knowledge Graph enhanced passage reader, namely Grape, to improve the reader performance for open-domain QA. Specifically, for each pair of question and retrieved passage, we first construct a localized bipartite graph, attributed to entity embeddings extracted from the intermediate layer of the reader model. Then, a graph neural network learns relational knowledge while fusing graph and contextual representations into the hidden states of the reader model. Experiments on three open-domain QA benchmarks show Grape can improve the state-of-the-art performance by up to 2.2 exact match score with a negligible overhead increase, with the same retriever and retrieved passages. Our code is publicly available at https://github.com/jumxglhf/GRAPE.
1 Introduction
State-of-the-art readers can miss factual relationships between entities in questions and retrieved passages, motivating reader-side use of knowledge-graph facts. GRAPE builds a localized graph for each question-passage pair and improves exact-match performance over state-of-the-art readers with the same retriever and passages.
1 Introduction
- State-of-the-art readers can produce answers that contradict facts because they fail to understand factual evidence in retrieved passages.The introduction proposes improving readers with structured KG triples rather than increasing retriever answer coverage.Source: §1 · p. 1
- In WebQ, 58.1% of examples can be matched to related KG fact triplets, yet FiD frequently answers these examples incorrectly.The related triples connect question entities to answer entities in retrieved passages.Source: §1 · p. 2
- GRAPE constructs a localized bipartite graph per question-passage pair, initializes entity nodes from reader hidden states, and uses a GNN to return relationally informed representations to the reader.The design uses KG relations while retaining contextual representations from the reader.Source: §1 · p. 2
- Up to 2.2 exact-match score improvement over state-of-the-art readers is reported for GRAPE with the same retriever and retrieved passages.The improvement gain nearly doubles on the subset that can be enhanced by KG fact triplets.Source: §1 · p. 2
2 Related Work
Prior open-domain QA work has advanced both retrieval and reading, while KG-enhanced approaches have primarily used graph relations to improve retrieval. GRAPE instead enhances the reader, complementing those retrieval-focused approaches.
2 Related Work
- Open-domain QA retrievers have progressed from sparse passage matching to dense contextual vectors, with further work on training, reranking, and passage generation.Readers include extractive span-based systems and generative FiD and RAG models.Source: §2 · p. 2
- Text-only readers such as FiD and RAG use retrieved text but can miss complex entity relationships, leading to answers that contradict facts.The paper positions reader-side KG use as a response to this limitation.Source: §2 · p. 2
- Earlier KG-enhanced QA methods improve retrieved evidence through indexing, passage retrieval, or reranking, whereas GRAPE enhances the reader and is orthogonal to those frameworks.With the retriever and retrieved passages held fixed, GRAPE outperforms FiD.Source: §2 · pp. 2–3
3 Proposed Method: GRAPE
GRAPE retains the retriever-reader pipeline but adds a localized question–passage graph and a relation-aware GNN to bring structured facts into the reader’s representations.
3 Proposed Method: GRAPE
- GRAPE builds a localized question–passage graph, learns structured facts with a relation-aware GNN, and fuses them into token-level entity representations.The workflow then uses the reader’s encoder and decoder to generate an answer.Source: §3 · p. 3
3.1 Passage Retrieval
- Separate BERT encoders represent the question and each passage, whose [CLS] representations provide similarity scores for ranking.Source: §3.1 · p. 3
3.2.1 Graph Construction
- GRAPE keeps only KG relations between question entities and passage entities in each bipartite graph, pruning peripheral relations and removing isolated nodes.Question and passage entities form the two node sets; edges connect entities linked by KG facts.Source: §3.2.1 · pp. 3–4
3.2.2 Factual Relation Fusion
- GRAPE splits the reader encoder, applies a relation-aware GNN to intermediate entity representations, then fuses the enriched nodes back before further encoder processing.The GNN combines contextualized node states with representations of factual relations.Source: §3.2.2 · pp. 4–6
4 Experiments
Experiments evaluate GRAPE on three open-domain QA benchmarks and compare it with established retrieval and reader systems. The setup uses Wikipedia passages, Wikidata, and exact-match evaluation.
4 Experiments
- Experiments use NQ, TriviaQA, and WebQ with the same train/dev/test splits and preprocessing as prior work.Source: §4 · p. 6
4.1 Experimental Setup
- The retrieval corpus contains 21 million disjoint 100-word Wikipedia passages, and English Wikidata supplies entities, relations, and triplets for graph construction.Source: §4.1 · p. 6
- ELQ identifies mentions in questions and retrieved passages, while the default setup uses 100 passages, two GNN layers, three encoder layers below fusion, and eight GNN heads.Source: §4.1 · p. 6
- Encoder-layer choices had little performance impact, lower values converged faster, and results use average exact match across three random-seed runs.Source: §4.1 · p. 6
4.2 Baseline Models
- Baselines span closed-book, extractive, KG-retrieval, and generative-reader approaches; GRAPE is positioned as orthogonal to KG methods for retrieval or reranking.Source: §4.2 · pp. 6–7
4.3 Experimental Results
GRAPE outperforms FiD across all three datasets in base and large configurations. Ablations, KG-favorable subsets, passage-count comparisons, and case studies examine where its gains arise.
4.3.1 Comparison with Baselines
- GRAPE improves FiD by 0.5, 1.2, and 1.6 EM on base, and 2.1, 2.2, and 1.2 EM on large, across NQ, TriviaQA, and WebQ, respectively.GRAPE also significantly outperforms the best-performing baselines across all datasets and both configurations; gains are larger on TriviaQA and WebQ than on NQ.Source: §4.3.1 · p. 7
4.3.2 Ablation Study
- Removing either relation modeling or attention lowers performance, and relation modeling contributes more than the attention mechanism.The ablations remove relations alone (w/o Rel) or relations together with neighbor-difference attention (w/o Att).Source: §4.3.2 · pp. 7–8, Table 3 · p. 8
4.3.3 Improvement Analysis by KG Relation
- On KG-enhanceable subsets, base-model gains are 2.6, 3.8, and 3.7 EM on NQ, TriviaQA, and WebQ, nearly doubling the overall margins.The subset includes questions whose constructed graph has an edge linking an answer entity to a question entity.Source: §4.3.3 · p. 8, Table 4 · p. 8
4.3.4 Scaling with Number of Passages
- At the same passage count, GRAPE consistently outperforms FiD, with larger gains as passage count increases; from 25 passages, GRAPE matches FiD with half as many.Source: §4.3.4 · p. 8, Figure 4 · p. 8
4.3.5 Case Studies on KG Relations
- Case studies show KG relations correcting FiD errors, including links identifying Arges as a Cyclops and the UK Parliament’s jurisdictional relation.The third example uses the triplet (UK, applies to jurisdiction, Parliament of UK) to correct FiD’s partially correct answer.Source: §4.3.5 · p. 8, Figure 5 · p. 9
5 Conclusion
GRAPE combines a localized graph of factual relations with a relation-aware GNN to improve open-domain QA; the authors also plan to enrich its graphs with external structured information.
5 Conclusion
- Up to 2.2 exact match score improvement over state-of-the-art readers is reported on three open-domain QA benchmarks.The comparison uses the same retriever and retrieved passages.Source: §5 · p. 9
- The authors plan to enrich the structured information contained in GRAPE’s graphs using other external resources.Source: §5 · p. 9
6 Limitations
GRAPE’s stated correction scope is limited to fact-related errors, and its use of Wikidata may omit relations from other sources.
6 Limitations
- GRAPE only solves errors from fact-related examples, and its Wikidata-based relations may omit facts from sources such as Freebase.Source: §6 · p. 9
A.1 Software and Hardware Environment
The reported implementation used specified software versions and an eight-GPU server; training time varied by model configuration.
A.1 Software and Hardware Environment
- GRAPE used Transformers 4.18.0, DGL 0.8.2, and PyTorch 1.11.0.Source: §A.1 · p. 12
- Experiments ran on servers with 8 Tesla V100 32GB GPUs and RAM usage below 50GB.Source: §A.1 · p. 12
- Updating parameters for 10K training steps took 72 GPU hours at base configuration and 170 GPU hours at large configuration.These times include intermediate validations.Source: §A.1 · p. 12
A.2 Hyper-parameter Sensitivity
The authors found little performance sensitivity to encoder layer index L, but lower L converged faster; they selected L = 3 for faster convergence.
A.2 Hyper-parameter Sensitivity
- The default GNN setup uses N = 2 layers and M = 8 heads, choices described as stable and able to capture 2-hop neighbor information.The GNN hidden dimension matches the language model dimension.Source: §A.2 · p. 12
- Different L selections from {3, 4, 6, 8, 9} had little performance impact, while lower L values converged faster; L = 3 was selected for faster convergence.Source: §A.2 · p. 12
A.3 Dataset and Graph Statistics
The section analyzes constructed-graph statistics across datasets using the same training, development, and test split as FiD and RAG. It relates graph availability to KG-favorable questions and performance gains, and compares dataset graph profiles.
A.3 Dataset and Graph Statistics
- The experiments use the same training, development, and test split as FiD and RAG.Source: §A.3 · p. 12
- The number of constructed graphs per question is proportional to both the share of questions favorable to KG triples and the performance gain introduced by GRAPE.Source: §A.3 · p. 12
- TriviaQA and WebQ are more entity-focused than NQ, while the authors report no other graph differences among the three datasets.Source: §A.3 · p. 12
- Table 5 gives the best training hyperparameters for the results reported in Table 2.Source: Table 5 · p. 13
- Table 6 reports constructed-graph statistics for all datasets, with means and standard deviations, using 100 retrieved passages per question.Source: Table 6 · p. 13