Source-linked AI summary
Fusing Context Into Knowledge Graph for Commonsense Question Answering
Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu, Michael Zeng, Xuedong Huang
TL;DR
Commonsense QA methods combine language models with knowledge graphs, but graphs lack rich context for understanding concepts. DEKCOR adds Wiktionary descriptions to graph information as input to ALBERT. It reports state-of-the-art results on CommonsenseQA and OpenBookQA, while two large-scale T5 models remain exceptions among OpenBookQA baselines.
Problem
Knowledge graphs encode concept relations but lack rich contextual information, while pre-trained language models lack modules to explicitly handle knowledge and commonsense.
Source: §1 · p. 1Method
DEKCOR selects a ConceptNet triple for the question and choice concepts, retrieves their Wiktionary definitions, and feeds the triple and descriptions with the question and choice into ALBERT.
Source: §1 · pp. 1–2Results
DEKCOR exceeds the previous state-of-the-art result by 1.2% (single model) on CommonsenseQA and outperforms OpenBookQA baselines except two large-scale T5 models.
Source: §1 · p. 2Takeaways & Limitations
Ablations demonstrate the effectiveness of using knowledge descriptions and knowledge triples in commonsense question answering.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Commonsense question answering (QA) requires a model to grasp commonsense and factual knowledge to answer questions about world events. Many prior methods couple language modeling with knowledge graphs (KG). However, although a KG contains rich structural information, it lacks the context to provide a more precise understanding of the concepts. This creates a gap when fusing knowledge graphs into language modeling, especially when there is insufficient labeled data. Thus, we propose to employ external entity descriptions to provide contextual information for knowledge understanding. We retrieve descriptions of related concepts from Wiktionary and feed them as additional input to pre-trained language models. The resulting model achieves state-of-the-art result in the CommonsenseQA dataset and the best result among non-generative models in OpenBookQA.
1 Introduction
DEKCOR combines knowledge-graph triples with Wiktionary descriptions to address the context missing from graph-based commonsense QA. It reports gains on CommonsenseQA and OpenBookQA.
1 Introduction
- Pre-trained language models are effective in language understanding but lack modules for explicitly handling knowledge and commonsense.The introduction also notes that structured data can represent commonsense more efficiently than unstructured text.Source: §1 · p. 1
- Knowledge graphs encode relations among concepts but lack the rich context that can clarify what an entity means, as the Mona Lisa example illustrates.The passage contrasts graph neighborhood information with a Wiktionary definition identifying the Mona Lisa as a painting.Source: §1 · p. 1
- The proposed representation combines neighboring concepts and their relations with a definitive description of each concept.The paper frames this panoramic view as a way to represent structured data for integration into language models.Source: §1 · p. 1
- The model selects a ConceptNet triple for question and choice concepts, retrieves their Wiktionary definitions, and feeds these with the question and choice into ALBERT.When no direct edge exists, it selects the highest-scoring triple containing the choice concept.Source: §1 · pp. 1–2
- DEKCOR outperforms the previous state-of-the-art result by 1.2% (single model) and 3.8% (ensemble model) on the CommonsenseQA test set, and all OpenBookQA baselines except two large-scale T5 models.The authors also report ablations demonstrating the effectiveness of fusing context into the knowledge graph.Source: §1 · p. 2
2 Related work
Prior work uses knowledge graphs, paths, and external text to support question answering and other knowledge tasks. DEKCOR is compared with work that also adds descriptions to commonsense QA.
2 Related work
- Commonsense QA methods use passage graphs, inferential knowledge graphs, schema graphs, path-based LSTMs, graph relation networks, and retrieved evidence from structured and text sources.Other work uses GPT-2 to generate paths between knowledge-graph concepts.Source: §2 · p. 2
- Knowledge descriptions have supported knowledge-text co-pretraining, knowledge-graph completion, entity classification, and cross-lingual entity alignment.The cited approaches use Wikipedia text or entity descriptions alongside knowledge-graph representations.Source: §2 · p. 2
- Chen et al. (2020) also add knowledge descriptions to commonsense QA, while DEKCOR is described as less complex and as outperforming that work on CommonsenseQA.The passage characterizes Chen et al.'s method as involving additional rankers trained on retrieved text.Source: §2 · p. 2
3 Method
DEKCOR retrieves a knowledge triple and entity descriptions for each question–choice pair, then scores the pair using ALBERT representations. The method combines graph relations with textual context.
3.1 Knowledge Retrieval
- For a question and choice entity pair, DEKCOR uses a direct ConceptNet edge if available; otherwise, it scores triples containing the choice entity and selects the highest-weight one.The score combines ConceptNet triple weight with relation-type weight, which favors rarer relation types.Source: §3.1 · p. 2
3.2 Contextual information
- Question, choice, entity names, descriptions, and the selected triple form ALBERT's input, with graph entities and relations represented by their surface forms.CommonsenseQA provides question and choice entities; for OpenBookQA, the method chooses entities most frequent in retrieved facts.Source: §3.2 · pp. 2–3, Figure 1 · p. 3
- The method retrieves Wiktionary definitions from a dump covering 999,614 concepts, matching each concept by original form, lemma, then base word.For each concept, it uses the first definition entry.Source: §3.2 · p. 3
3.3 Reasoning
- An attention-based weighted sum of ALBERT's output representations feeds a softmax layer that produces a relevance score across the answer choices.The score is used with a cross-entropy loss over the choices.Source: §3.3 · p. 3
4 Experiments
DEKCOR improves on prior state-of-the-art results on CommonsenseQA and is the strongest non-T5 baseline on OpenBookQA. The experiments also assess dataset setup, knowledge-component contributions, and examples of how descriptions affect answers.
4.1 Datasets and baselines
- CommonsenseQA questions are built from ConceptNet entities and relations, while OpenBookQA tests elementary science knowledge using a book of 1,326 facts.For OpenBookQA, the study appends five retrieved Aristo facts and pre-trains on CommonsenseQA training data.Source: §4.1 · pp. 3–4
- The CommonsenseQA test comparison separates single models from an ensemble of seven differently seeded models whose choices are combined by majority vote.All benchmark comparisons use pre-trained-model baselines, some with additional knowledge-processing modules.Source: §4.2 · p. 4, §4.1 · p. 4
4.2 Results
- DEKCOR exceeds prior CommonsenseQA state of the art by 1.2% as a single model and 3.8% as an ensemble, and leads non-T5 baselines on OpenBookQA.The OpenBookQA comparisons are single-model results; two T5-based systems perform better.Source: §4.2 · p. 4, Table 2 · p. 4, Table 3 · p. 4
- On CommonsenseQA dev, descriptions from Wiktionary and ConceptNet triples improve DEKCOR accuracy by 2.7% and 4.4%, respectively; similar results appear on OpenBookQA.The two T5-based OpenBookQA systems have 11B and 3B parameters and are described as computationally very expensive.Source: §4.2 · p. 4
- The reported language-model accuracy order is BERT < RoBERTa < XLNet < ALBERT < T5; knowledge modules also improve accuracy in the cited comparisons.RoBERTa+KEDGN exceeds vanilla RoBERTa by 1.9%, and DEKCOR exceeds vanilla ALBERT by 6.8%.Source: §4.2 · p. 4
- The two examples show descriptions changing the answer-relevant cues: bat diet information and alligator teeth and jaw details support different commonsense inferences.The alligator example also uses the description's link between reptiles and being cold-blooded.Source: §4.2 · p. 5, Table 5 · p. 5
5 Conclusions
The conclusion presents DEKCOR as a way to add contextual descriptions to knowledge graphs for commonsense QA. It reports strong benchmark results and ablation evidence for descriptions and triples.
5 Conclusions
- DEKCOR adds Wiktionary entity descriptions as additional input to ALBERT for commonsense question answering.The paper frames descriptions as definitive text for entities that may lack descriptions in a knowledge graph.Source: §5 · p. 5
- The authors report state-of-the-art results on CommonsenseQA and OpenBookQA.The conclusion states this benchmark outcome for the resulting DEKCOR model.Source: §5 · p. 5
- Ablation studies show the effectiveness of using both knowledge descriptions and knowledge triples in commonsense question answering.Source: §5 · p. 5
A Implementation Details
The implementation selects ConceptNet triples and Wiktionary descriptions using entity occurrence counts, then trains ALBERT with specified input and optimization settings.
A Implementation Details
- For OpenBookQA retrieval, the implementation counts ConceptNet entity occurrences in retrieved text and selects the highest-weight edge or a triple for the most frequent answer entity.The triple weight is the sum of the question- and choice-entity occurrence counts.Source: A Implementation Details · p. 7
- Wiktionary descriptions are selected for the question and choice entities with the most occurrences, and triples are restricted to ten listed ConceptNet relation types.The listed relations include CausesDesire, HasProperty, CapableOf, PartOf, AtLocation, Desires, HasPrerequisite, HasSubevent, Antonym, and Causes.Source: A Implementation Details · p. 7
- Training uses AdamW at a 2e-5 learning rate, batch size 8, a 192-token input limit, and 10 epochs.The ALBERT model uses the Hugging Face implementation.Source: A Implementation Details · p. 7
B Baseline Methods
The baseline set includes methods that retrieve, represent, or generate knowledge for question answering, alongside general-purpose QA models.
B Baseline Methods
- GraphReason retrieves from structured knowledge bases and plain text, while PG-FULL fine-tunes GPT-2 to generate knowledgeable ConceptNet paths.Source: B Baseline Methods · p. 7
- MHGRN uses multi-hop graph relation reasoning, HyKAS consumes ConceptNet triples through option comparison, and ALBERT+KRD weights triple representations with self-attention.Source: B Baseline Methods · p. 7
- UnifiedQA pre-trains T5 on diverse QA datasets, while BERT + Selection uses abductive retrieval, information-gain reranking, passage selection, and weighted scoring on OpenBookQA.Source: B Baseline Methods · p. 7