Source-linked AI summary
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, Jure Leskovec
TL;DR
Complex question answering requires combining textual constraints with unstated world knowledge, but language models and knowledge graphs remain difficult to integrate effectively. GREASELM enables multilayer, bidirectional exchange between language-model and graph representations, outperforming prior LM-only and KG+LM systems across commonsense and medical benchmarks, including models with substantially more parameters.
Problem
Question answering requires reasoning over explicit textual constraints and relevant unstated world knowledge, while effectively integrating language context with knowledge-graph representations remains an open challenge.
Method
GREASELM combines language-model layers with graph neural networks over retrieved knowledge-graph subgraphs, exchanging information bidirectionally across multiple cross-modal layers.
Results
GREASELM outperforms prior LM-only and KG+LM baselines across CommonsenseQA, OpenbookQA, and MedQA-USMLE, with gains of 5.5%, 6.6%, and 1.3% over fine-tuned LM baselines, respectively.
Takeaways & Limitations
Multilayer fusion improves handling of questions involving textual nuances such as negation, hedging, and multiple constraints across commonsense and medical domains.
Abstract
from arXiv · showhide
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GreaseLM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.
1 INTRODUCTION
Question answering requires combining explicit textual constraints with unstated world knowledge, but existing language-model and knowledge-graph approaches do not fully support deep cross-modal interaction. GREASELM addresses this gap through multilayer information exchange and improves performance across commonsense and medical QA benchmarks.
- QA requires reasoning over both explicit constraints in context and relevant unstated world knowledge.
- Existing methods often fuse language and graph modalities shallowly, limiting their ability to exchange useful information.
- GREASELM uses an LM and a GNN with multilayer bidirectional interaction through a special interaction token and node.
- 0.9%, 1.8%, and 0.5% gains over state-of-the-art KG+LM models are reported on the same three benchmarks, respectively.
2 RELATED WORK
Prior work combines knowledge graphs and language models through separate towers, one-way grounding, or deeper pretraining integrations, but these approaches retain important interaction limitations. GREASELM is positioned against methods that do not sustain bidirectional, multilayer exchange between modalities.
- Two-tower QA models fuse graph and language representations without interaction between them.
- Some approaches use language models to generate local knowledge graphs, but discard the static graph structure after training.
- Knowledge-graph integration during pretraining typically lets knowledge feed language rather than designing interactions across multiple layers.
3 PROPOSED APPROACH: GREASELM
GREASELM augments pretrained language models with graph reasoning over a retrieved local knowledge graph. Its cross-modal layers repeatedly exchange information through interaction bottlenecks so language and graph representations contextualize one another.
- 3 PROPOSED APPROACH: GREASELM: GREASELM stacks N unimodal LM layers with M cross-modal layers that jointly represent the language sequence and linked knowledge graph.
- 3.1 INPUT REPRESENTATION: The model concatenates context, question, and candidate answer, then retrieves a relevant KG subgraph for each QA example.
- 3.3 GREASELM: Special interaction token and node representations act as bottlenecks for transferring information between the language and graph modalities.
- 3.3 GREASELM: The LM continues encoding tokens while a GNN propagates relation-aware messages between neighboring graph nodes.
- 3.3 GREASELM: Across multiple layers, KG knowledge grounds language representations while contextual constraints inform knowledge representations.
- 3.3 GREASELM: The model predicts the most plausible multiple-choice answer from an attention-pooled graph representation queried by the final interaction-token representation.
4 EXPERIMENTAL SETUP
The experiments evaluate GreaseLM across commonsense, scientific, and clinical multiple-choice QA, using domain-appropriate language models, knowledge graphs, and controlled baselines.
- GreaseLM is evaluated on CommonsenseQA, OpenbookQA, and MedQA-USMLE, covering commonsense reasoning and medical question answering across two domains.
- CommonsenseQA contains 12,102 five-way questions requiring background commonsense knowledge beyond surface language understanding.Experiments use the in-house split of Lin et al. (2019).
- OpenbookQA contains 5,957 four-way questions testing elementary scientific knowledge with an accompanying open book of scientific facts.Experiments use the official data splits from Mihaylov & Frank (2018).
- MedQA-USMLE contains 12,723 four-way questions requiring biomedical and clinical knowledge from United States Medical License Exam practice tests.Experiments use the original data splits from Jin et al. (2021).
- The model uses RoBERTa-Large, AristoRoBERTa, and SapBERT as seed language models for CommonsenseQA, OpenbookQA, and MedQA-USMLE, respectively.This setup demonstrates generality across language-model initializations.
- ConceptNet supplies external knowledge for CommonsenseQA and OpenbookQA, while a self-constructed UMLS–DrugBank graph supports MedQA-USMLE.ConceptNet has 799,273 nodes and 2,487,810 edges; the medical graph has 9,958 nodes and 44,561 edges.
5 EXPERIMENTAL RESULTS
GREASELM consistently improves over language-model and LM+KG baselines on commonsense QA, with evidence that multi-layer cross-modal fusion helps on semantically nuanced and complex questions. Qualitative analysis further shows attention patterns that better reflect contextual cues than QA-GNN.
- Benchmark performance: 5.5% and 0.9%: GREASELM improves over fine-tuned LMs and existing LM+KG models on CommonsenseQA.On OpenbookQA, the reported improvements are 6.4% over raw LMs and 2.0% over QA-GNN.
- Benchmark performance: GREASELM posts the third-highest OpenbookQA leaderboard score and the highest score among models with comparable parameter counts.T5 and UnifiedQA have 8× and 30× more parameters, respectively.
- Complex reasoning: GREASELM generally outperforms RoBERTa-Large and QA-GNN on questions containing negation, hedging, and prepositional phrases.Prepositional phrases serve as a proxy for reasoning complexity, while negation and hedging capture nuanced constraints.
- Complex reasoning: GREASELM’s multi-layer fusion is associated with stronger performance as question complexity increases, whereas QA-GNN and GREASELM perform comparably without prepositional phrases.QA-GNN’s end-fusion approach compresses the language context into a single vector before KG interaction.
- Qualitative analysis: In a CommonsenseQA example, GREASELM predicts “airplane” correctly while QA-GNN predicts “motor vehicle,” with attention traced through the retrieved KG subgraph.Both models use Best First Search to trace high-attention paths from the interaction node.
- Qualitative analysis: GREASELM’s attention on “bug” rises in intermediate layers and falls by the final layer, better matching the hedge “unlikely” than QA-GNN’s pattern.Attention on “windshield” consistently increases across GREASELM’s layers.
6 CONCLUSION
GREASELM uses interactive fusion to exchange information between language models and knowledge graphs, achieving superior performance across commonsense and medical QA benchmarks. The paper also identifies ethical risks from inherited biases and cautions against real-world clinical prediction use.
- GREASELM enables joint information exchange between language-model and knowledge-graph representations for reasoning.
- GREASELM outperforms prior KG+LM and LM-only baselines across standard commonsense and medical datasets.
- Its analysis reports improved capability on questions involving textual nuances such as negation and hedging.
- Ethics statement: GREASELM may reproduce biases and toxic behaviors present in its pretrained language models and knowledge graphs, including stereotypes encoded in ConceptNet.
- Ethics statement: The authors do not encourage using these models for real-world clinical prediction, particularly at the reported MedQA-USMLE performance levels.
B EXPERIMENTAL SETUP DETAILS
The experimental setup retrieves a question-specific subgraph from the full knowledge graph through entity linking, bridge-entity expansion, and relevance-based pruning.
- Entity linking first retrieves an initial set of graph nodes connected to the QA context.
- The retrieval procedure adds bridge entities lying on 2-hop paths between linked entities.
- Retrieved nodes are then pruned using a computed relevance score for each node.
B.2 GRAPH INITIALIZATION
GREASELM initializes graph nodes with language-model-derived entity representations, using different encoders for ConceptNet and MedQA-USMLE. When MedQA retrieval fails, it uses a zero-initialized dummy node.
- ConceptNet triples are converted into relation-specific sentences before BERT-large computes their token representations.
- For each ConceptNet entity, token representations from its mention spans are mean-pooled to initialize the entity embedding.
- MedQA-USMLE initializes node embeddings from pooled entity-name token outputs produced by SapBERT.
- 5% of MedQA-USMLE examples yield no retrieved entity and therefore use a dummy graph node initialized with 0.
- With the dummy-node fallback, graph propagation contributes no information and the model relies only on language-model representations.
C.1 ABLATION STUDIES
Ablation studies show that modality interaction, graph connectivity, and language-model-based node initialization materially affect GREASELM performance, while performance is relatively stable across nearby layer counts.
- Modality interaction: 78.5% falls to 76.5% when modality interaction is removed, approximately matching QA-GNN performance.
- Modality interaction: Sharing parameters across modality-interaction layers outperforms not sharing them, possibly because the datasets are small.
- Number of GREASELM layers: M = 5 GREASELM layers achieves the highest performance, while M = 4 and M = 6 remain relatively close.
- Graph connectivity: Connecting the interaction node to all subgraph nodes hurts performance by -0.9% compared with connecting it to explicitly linked entities.
- KG node embedding initialization: Random KG-node initialization reduces performance from 78.5% to 60.8%, while TransE recovers performance to 77.7%.
C.2 EFFECT OF LM INITIALIZATION ON GREASELM
The study tests whether GreaseLM's gains depend on its language-model initialization. Across CommonsenseQA and MedQA-USMLE, replacing the original language models with alternative initializations yields consistent improvement.
- C.2 EFFECT OF LM INITIALIZATION ON GREASELM: GreaseLM is evaluated with multiple language-model initializations across CommonsenseQA and MedQA-USMLE.The experiments are reported in Table 9 for CommonsenseQA and Table 10 for MedQA-USMLE.
- C.2 EFFECT OF LM INITIALIZATION ON GREASELM: RoBERTa-large is replaced by RoBERTa-base for CommonsenseQA, while SapBERT is replaced by BioBERT and PubmedBERT for MedQA-USMLE.These substitutions test whether the method is agnostic to the language model used to seed GreaseLM layers.
- C.2 EFFECT OF LM INITIALIZATION ON GREASELM: GreaseLM provides consistent improvement across the tested language-model initializations in both domains.The authors frame GreaseLM as a modality junction between knowledge graphs and language.