PaperlayerView exact revision on arXiv

Source-linked AI summary

QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering

Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec

arXiv:2104.06378v5cs.CLcs.LG

TL;DR

QA systems must identify relevant knowledge in large knowledge graphs and jointly reason over that knowledge and the question context. QA-GNN uses language-conditioned node relevance scoring and a joint graph, improving question answering and structured reasoning, including negation handling.

  • Problem

    LM+KG question answering must identify informative knowledge in large knowledge graphs and jointly reason over QA contexts and graph structure.

  • Method

    QA-GNN scores KG-node relevance conditioned on the QA context and jointly updates QA and KG representations through graph neural network message passing.

  • Results

    4.6% improvement over fine-tuned LMs on questions with negation, compared with 0.6% for existing LM+KG models.

  • Takeaways & Limitations

    QA-GNN supports interpretable structured reasoning by extracting general KG subgraphs as reasoning processes, including correct handling of negation.

Abstract

from arXiv · show

The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG. In this work, we propose a new model, QA-GNN, which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to estimate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph neural networks. We evaluate our model on QA benchmarks in the commonsense (CommonsenseQA, OpenBookQA) and biomedical (MedQA-USMLE) domains. QA-GNN outperforms existing LM and LM+KG models, and exhibits capabilities to perform interpretable and structured reasoning, e.g., correctly handling negation in questions.

1 Introduction

QA-GNN addresses the challenges of selecting informative knowledge from large knowledge graphs and jointly reasoning over QA contexts and graph structure. It does so through relevance scoring and a unified working graph, improving question-answering performance and structured reasoning while supporting explanations.

  • Motivation: Question answering requires systems to access relevant knowledge and reason over both language-model knowledge and structured knowledge graphs.Language models encode knowledge implicitly in pre-trained text, while knowledge graphs represent entities and relations explicitly.
  • Challenges: Combining language models and knowledge graphs requires identifying informative graph knowledge and jointly modeling QA-context nuance with graph structure.Prior methods separately apply language models to QA contexts and graph neural networks to knowledge graphs, which may limit structured reasoning such as negation handling.
  • QA-GNN: QA-GNN uses language-model relevance scoring and joint reasoning to address both challenges in an end-to-end question-answering model.It encodes the QA context, retrieves a knowledge-graph subgraph, and weights graph information using node relevance conditioned on the QA context.
  • QA-GNN: QA-GNN forms a working graph by representing the QA context as a node connected to topic entities, augmenting node features with relevance scores, and simultaneously updating both modalities with an attention-based graph neural network.The working graph unifies the QA context and retrieved knowledge graph into one graph for reasoning.
  • Evaluation: 4.7% improvement over strong fine-tuned language-model baselines and 2.3% over the existing best language-model-plus-knowledge-graph model demonstrate QA-GNN’s performance across three knowledge-intensive QA datasets.The datasets are CommonsenseQA, OpenBookQA, and MedQA-USMLE, spanning commonsense and biomedical domains.
  • Results: 4.6% improvement over fine-tuned language models on questions with negation, versus +0.6% for existing language-model-plus-knowledge-graph models, reflects improved structured reasoning.QA-GNN also supports extracting general knowledge-graph subgraphs as reasoning processes for explaining predictions.

2 Problem statement

QA-GNN answers natural-language questions by combining knowledge from a pretrained language model with a structured knowledge graph. For each question–answer pair, it links mentioned entities to the KG and extracts a relevant subgraph.

  • The task is to answer natural-language questions using knowledge from both a pretrained language model and a structured knowledge graph.
  • A language model is represented as f_head(f_enc(x)), where the encoder produces contextualized representations and the head performs the desired task.
  • The KG contains nodes V connected by typed edges E ⊆ V×R×V, with R denoting relation types.
  • For each question q and answer choice a, entities mentioned in the QA pair are linked to the KG, forming V_q,a := V_q∪V_a.
  • The model extracts G_q,a, a KG subgraph comprising all nodes on k-hop paths between entities mentioned in the question and answer choice.

3 Approach: QA-GNN

QA-GNN combines a language-model representation of the question–answer context with a retrieved KG subgraph in a joint working graph. It scores node relevance and uses relation-, type-, and score-aware graph attention to jointly reason over both sources before predicting the answer.

  • Joint graph construction: QA-GNN concatenates each question and answer choice into a QA context, retrieves a KG subgraph, and connects a context node z to topic entities in a joint working graph.The new edges distinguish whether an entity comes from the question or answer portion of the context.
  • Joint graph construction: The working graph assigns nodes four types—context, question entities, answer entities, and other nodes—and initializes z with the LM’s QA-context representation.KG nodes are initialized with their entity embeddings before graph reasoning begins.
  • Node relevance scoring: QA-GNN uses an LM to score each retrieved KG node’s relevance to the QA context, using the score to guide reasoning or prune the working graph.This addresses irrelevant or generic retrieved nodes that can complicate reasoning or cause overfitting.
  • Joint reasoning: A GAT-based GNN performs iterative message passing on the working graph, jointly updating QA-context and KG representations with node-type-, relation-, and relevance-aware messages and attention.The final representations combine the LM context embedding, the GNN representation of z, and pooled KG-node representations for answer scoring.
  • Answer prediction: The model predicts answer-choice probabilities from the LM and GNN representations and trains both components end-to-end with cross-entropy loss.For a question, the training set provides multiple answer choices with one correct choice.
  • Computational complexity: QA-GNN has time complexity constant in the number of relation types and linear in the number of nodes, while matching MHGRN’s space complexity.It uses different edge embeddings rather than independent graph networks for each relation type.

4 Experiments · 4.1 Datasets

QA-GNN is evaluated on three multiple-choice QA datasets spanning commonsense, elementary science, and biomedical or clinical knowledge. The datasets contain 5,957–12,723 questions and use either official, original, or in-house splits, with CommonsenseQA’s test evaluation constrained by leaderboard access.

  • 4.1 Datasets: QA-GNN is evaluated on CommonsenseQA, OpenBookQA, and MedQA-USMLE.These benchmarks cover commonsense, elementary science, and biomedical or clinical knowledge.
  • 4.1 Datasets: Main CommonsenseQA experiments use the in-house data splits from Lin et al. (2019), alongside the final system’s official test-set score.The official test set is accessed through the leaderboard.
  • 4.1 Datasets: OpenBookQA is a 4-way multiple-choice task requiring elementary science reasoning and containing 5,957 questions.The experiments use the official data splits from Mihaylov and Frank (2018).
  • 4.1 Datasets: MedQA-USMLE is a 4-way multiple-choice task requiring biomedical and clinical knowledge, with 12,723 questions from United States Medical License Exams practice tests.The experiments use the original data splits from Jin et al. (2021).

4.2 Knowledge graphs

QA-GNN uses domain-specific knowledge graphs for its benchmark datasets, initializing node representations from pretrained language-model embeddings. For each QA context, it retrieves a two-hop subgraph and prunes it to the 200 most relevant nodes.

  • Commonsense knowledge graph: ConceptNet supplies structured knowledge for CommonsenseQA and OpenBookQA, containing 799,273 nodes and 2,487,810 edges.Node embeddings use entity representations prepared by applying pretrained language models to all ConceptNet triples.
  • Biomedical knowledge graph: MedQA-USMLE uses a self-constructed graph integrating UMLS Disease Database and DrugBank, with 9,958 nodes and 44,561 edges.Node embeddings are initialized from pooled entity-name representations produced by SapBERT.
  • Subgraph retrieval and pruning: For each QA context, QA-GNN retrieves a subgraph with hop size k=2 and retains the top 200 nodes by the relevance score computed in §3.2.The paper subsequently uses “KG” to refer to this retrieved and pruned subgraph.

4.3 Implementation & training details

QA-GNN uses a five-layer GNN with dropout and RAdam optimization, trained on two GPUs for approximately 20 hours. Batch size and separate LM and GNN learning rates are selected from predefined grids using the development set.

  • The GNN module has L = 5 layers, with dropout rate 0.2 applied to every layer.
  • Training uses the RAdam optimizer on two GeForce RTX 2080 Ti GPUs and takes approximately 20 hours.
  • Batch size is tuned over {32, 64, 128, 256}.
  • The LM learning rate is tuned over {5e-6, 1e-5, 2e-5, 3e-5, 5e-5}, while the GNN learning rate uses {2e-4, 5e-4, 1e-3, 2e-3}.
  • The listed hyperparameters are tuned on the development set.

4.4 Baselines

The evaluation compares QA-GNN against fine-tuned language models without knowledge graphs and existing LM+KG methods using alternative KG reasoning modules. Baselines span commonsense, biomedical, and path-based KG reasoning settings.

  • Fine-tuned LM: Fine-tuned LM baselines omit the knowledge graph: RoBERTa-large is used for CommonsenseQA, RoBERTa-large and AristoRoBERTa2 for OpenBookQA, and SapBERT for MedQA-USMLE.These models provide vanilla LM comparisons across the three evaluation benchmarks.
  • Existing LM+KG models: Existing LM+KG baselines replace QA-GNN’s KG reasoning module with Relation Network, RGCN, GconAttn, KagNet, or MHGRN.Relation Network, RGCN, and GconAttn are relation-aware KG GNNs, whereas KagNet and MHGRN additionally model KG paths.

4.5 Main results

QA-GNN consistently improves over fine-tuned language models and existing LM+KG systems on CommonsenseQA and OpenBookQA, including gains of +4.7% over RoBERTa and +2.3% over MHGRN. It also outperforms fine-tuned LMs on MedQA-USMLE and achieves top performance among comparably sized, non-ensemble systems across the two commonsense datasets.

  • CommonsenseQA and OpenBookQA: On CommonsenseQA, QA-GNN improves +4.7% over RoBERTa and +2.3% over the prior best LM+KG system, MHGRN.The authors attribute the boost over MHGRN to better use of knowledge graphs for joint reasoning.
  • CommonsenseQA and OpenBookQA: QA-GNN shows consistent improvements over fine-tuned LMs and existing LM+KG models on both CommonsenseQA and OpenBookQA.The paper also reports competitive results on the official leaderboards.
  • CommonsenseQA and OpenBookQA: Excluding the two larger top systems and ensemble systems, QA-GNN achieves top performance on both commonsense datasets with comparable size and data.QA-GNN has ∼360M parameters, while T5 and UnifiedQA use 8x to 30x more parameters and more training data.
  • MedQA-USMLE: On MedQA-USMLE, QA-GNN outperforms state-of-the-art fine-tuned LMs such as SapBERT.The result supports QA-GNN as an augmentation of LMs and KGs across commonsense and biomedical domains.

4.6 Analysis

QA-GNN’s ablations show that joint graph connections, KG node relevance scoring, and GNN features improve performance, with five layers performing best. Attention analyses further indicate that QA-GNN identifies general KG reasoning structures, adapts to negation and entity changes, and benefits from relevance scoring on larger retrieved graphs.

  • Ablation analysis: Removing connections between the QA context node and QA entity nodes lowers performance from 76.5% to 74.8%, while connecting the context to all KG nodes slightly reduces performance by 0.16%.The joint graph enables mutual representation updates between the QA context and KG.
  • Ablation analysis: KG node relevance scoring improves performance from 75.56% to 76.54%; contextual embeddings alone reach 76.31% and adding them to scores performs on par with scores alone.The comparison evaluates variants of the relevance-scoring component.
  • Ablation analysis: Ablations indicate that node type, relation, and relevance-score features improve GNN performance, while five GNN layers perform best on the development set.Five layers support message-passing patterns between the QA context and KG, including paths that return to the context after three KG hops.
  • Interpreting reasoning: Attention tracing reveals KG concepts that bridge question and answer entities, and QA-GNN finds broader reasoning structures such as subgraphs with multiple anchor nodes rather than only individual paths.Examples include “sea” and “ocean” bridging “crab” and “salt water,” and “building” linking concepts associated with “elevator” and “basement.”
  • Structured reasoning: On CommonsenseQA questions containing negation, QA-GNN improves over RoBERTa by +4.6%, compared with +0.6% for prior LM+KG models, and removing joint graph connections reduces performance toward MHGRN.Qualitatively, QA-GNN adapts predictions to negation and entity substitutions, though it fails on the shown double-negation example.
  • Structured reasoning: KG node relevance scoring is helpful for questions with more than 10 entities, where retrieved subgraphs average 160 nodes and are larger and noisier than those for questions with 10 or fewer entities.The corresponding smaller retrieved subgraphs average 90 nodes.

5 Related work and discussion

QA-GNN extends LM+KG question answering by jointly modeling QA contexts and knowledge graphs, while scoring KG-node relevance conditioned on language. Its approach differs from related work in how it constructs joint representations and applies graph neural networks.

  • Knowledge-aware methods for NLP: Earlier knowledge-aware NLP work explored pretrained LMs as latent knowledge bases, while other methods integrated structured knowledge such as KGs into LMs.The cited works span both latent and explicit knowledge integration approaches.
  • Question answering with LM+KG: QA-GNN’s key novelties are a joint QA-context/KG graph with mutually updated LM and KG representations, and language-conditioned KG-node relevance scoring.These mechanisms address joint reasoning and knowledge selection within the same model.
  • Other QA tasks: Unlike passage-based QA and KBQA, QA-GNN answers questions using knowledge available in language models and knowledge graphs.Passage-based QA uses given or retrieved documents, whereas KBQA parses questions and executes queries on knowledge bases.
  • Knowledge representations: QA-GNN constructs a graph connecting each question and a KG, complementing work on joint representations of textual and structured knowledge.Prior work studied combinations such as Wikipedia articles and KGs rather than the question–KG graph used here.
  • Graph neural networks (GNNs): Unlike prior GNN applications to text or KGs alone, QA-GNN jointly models language and KGs and builds on attention-based message passing from GATs.GATs induce graph representations through attention-based message passing.

6 Conclusion

QA-GNN is an end-to-end question-answering model that leverages language models and knowledge graphs through relevance scoring and joint reasoning. Quantitative and qualitative analyses show improvements over existing LM and LM+KG models, alongside interpretable, structured reasoning such as correctly handling negation.

  • QA-GNN is an end-to-end question-answering model that leverages language models and knowledge graphs.
  • Relevance scoring computes knowledge-graph node relevance conditioned on the given question-and-answer context.
  • Joint reasoning connects the question-and-answer context and knowledge graphs through a working graph, jointly updating their representations with graph neural network message passing.
  • QA-GNN improves over existing language-model and language-model-plus-knowledge-graph models on question-answering tasks.The conclusion reports this finding from both quantitative and qualitative analyses.
  • QA-GNN performs interpretable and structured reasoning, including correctly handling negation in questions.
Loading 2104.06378v5…