Source-linked AI summary
Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
Robert L. Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, Sameer Singh
TL;DR
Traditional language models struggle to generate factually correct text and rare entities because they rely on memorized training information. This paper introduces KGLM, which selects and copies relevant facts from a knowledge graph, and Linked WikiText-2, an aligned benchmark. KGLM outperforms the compared models on language modeling and factual completion, while its reliance on inference-time marginalization and annotated tokens defines important limitations.
Problem
Traditional language models have limited ability to generate factually correct text and to represent rare entities, numbers, and dates from memorized training data.
Method
KGLM uses a dynamically growing local knowledge graph to select and copy entities and facts, while Linked WikiText-2 aligns WikiText-2-like text with Wikidata.
Results
KGLM achieves lower overall and unknown-penalized perplexity than the compared models and is significantly more accurate at factual completion.
Takeaways & Limitations
Knowledge graphs support language modeling that generates rare tokens and accurate facts, including facts about rare entities and facts controlled through graph modifications.
Takeaways & Limitations
KGLM requires marginalization during inference and relies on annotated tokens.
Abstract
from arXiv · showhide
Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge. However, traditional language models are only capable of remembering facts seen at training time, and often have difficulty recalling them. To address this, we introduce the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. We also introduce the Linked WikiText-2 dataset, a corpus of annotated text aligned to the Wikidata knowledge graph whose contents (roughly) match the popular WikiText-2 benchmark. In experiments, we demonstrate that the KGLM achieves significantly better performance than a strong baseline language model. We additionally compare different language model's ability to complete sentences requiring factual knowledge, showing that the KGLM outperforms even very large language models in generating facts.
1 Introduction
Existing language models generate fluent text but struggle with factual accuracy, rare entities, and tokens such as dates and numbers. The paper introduces KGLM and Linked WikiText-2 to address these limitations through knowledge-graph-based generation and aligned evaluation data.
- Language models can generate grammatical sentences but have limited ability to produce factually correct text about the world.
- Memorization-based models may prefer an incorrect common token over the correct rare entity, even when the factual sentence appeared in training.For example, AWD-LSTM assigns higher probability to “PlayStation” than “Game Boy” in a sentence containing “Game Boy.”
- KGLM selects and copies information from an external knowledge graph while maintaining a local graph of mentioned and related entities.It can render new entities or local-graph facts and combine graph tokens with the standard vocabulary.
- Linked WikiText-2 aligns WikiText-2-like text with Wikidata entities and relations to support knowledge-graph language-model research.Its annotations combine human-provided links with automatic linking and coreference models.
- KGLM is evaluated against AWD-LSTM on Linked WikiText-2 and achieves lower overall and unknown-penalized perplexity.The paper also evaluates factual completion, rare-entity generation, and control through knowledge-graph modifications.
2 Knowledge Graph Language Model
KGLM conditions language generation on a dynamically maintained local knowledge graph, selecting entities and relations before rendering tokens from vocabulary and copied aliases.
- Generative KG Language Model: KGLM maintains a local knowledge graph containing facts involving entities already mentioned in the context.The graph grows when the model refers to previously unseen entities.
- Generative KG Language Model: The model computes p(xt, Et|x<t, E<t) and marginalizes the local knowledge graph to obtain token probabilities.This connects the entity-aware generative process to the language-modeling objective over observed tokens.
- Generative KG Language Model: In the Figure 2 example, the model selects Super Mario Land, Publisher, and Nintendo before generating “Nintendo” and adding related facts to the local graph.The illustration shows how entity selection expands the available word distribution.
- Generative KG Language Model: The generative process classifies each token as related, new, or non-entity, then selects entities and facts accordingly.For related entities, it chooses a parent, relation, and tail entity before generating the token.
- Parameterizing the Distributions: The recurrent state is split into components for predicting words, parents, and relations, while token type uses a softmax over the word component.Entity and relation embeddings support selecting new entities and related facts.
- Parameterizing the Distributions: When rendering an entity, KGLM combines the original vocabulary with tokens from its aliases and uses a copy mechanism for rare names and values.Without an entity to render, it uses the standard vocabulary distribution.
3 Linked WikiText-2
Linked WikiText-2 augments WikiText-2-like text with links to Wikidata, enabling annotations that connect entities and facts across documents. Its annotations have broad coverage but differ from WikiText-2 enough to prevent direct comparison.
- Dataset construction: Linked WikiText-2 links approximately the same articles as WikiText-2 to Wikidata, preserving a basis for comparing models trained on the two corpora.Wikidata also provides coverage of many facts expressed in the text.
- Dataset construction: The corpus combines human-provided Wikipedia links with entity linking and coreference models to identify entity mentions beyond first occurrences.Coreference resolution covers pronouns, nominals, and other tokens missed by entity linking.
- Annotation process: Entities are assigned plausible knowledge-graph relations by processing text token by token and allowing multiple facts when several relations explain a mention.Related entities seen later in a document can serve as parents for later entities.
- Annotation process: String matching expands annotations to aliases, dates, and quantities, including exhaustive date formats and quantity expressions.The procedure targets both missed entities and non-entity factual tokens.
- Corpus statistics: More than 10% of tokens are entity tokens, while each entity appears fewer than five times on average across more than a thousand relations.The corpus therefore contains sparse, long-tailed factual references.
- Differences from WikiText-2: Minor text variations and the use of article HTML rather than the WikiText-2 text API prevent direct comparison with WikiText-2.These differences arise from edits between download dates and discarded annotation information.
4 Training and Inference for KGLM
KGLM training maximizes the likelihood of annotated text, while inference estimates the marginal text probability without access to annotations. Pretrained Wikidata embeddings support predictions involving unseen entities and relations.
- Pretrained KG embeddings: Pretrained TransE embeddings represent entities and relations so evaluation can handle entities and relations unseen during training.The embeddings are fixed during KGLM evaluation.
- Pretrained KG embeddings: A max-margin loss learns the knowledge-graph embeddings by contrasting a true entity or parent with a randomly chosen replacement.The margin is γ, and either the parent or entity embedding may be replaced.
- Training: KGLM is trained by maximizing the likelihood of the training data under its generative process.The objective is the negative log-likelihood, with multiple viable parents marginalized during training.
- Inference: During evaluation, KGLM does not assume access to entity annotations and targets the marginal probability p(x), not the joint probability.Observed annotations simplify training but are unavailable at evaluation time.
- Inference: Importance sampling approximates the intractable marginal distribution using samples from a proposal distribution q(E|x).The proposal is computed with a discriminative version of KGLM that predicts annotations for the current token.
5 Experiments
The experiments evaluate KGLM against language-model baselines using perplexity, factual completion, and knowledge-graph interventions. KGLM improves modeling of rare factual tokens and produces more specific factual completions, while retaining limitations in relation selection and coverage.
- Evaluation setup: The evaluation compares KGLM with AWD-LSTM, ENTITYNLM, EntityCopyNet, and GPT-2 using held-out perplexity and factual completion tasks.Fact completion measures whether the first generated token correctly completes templates expressing relations such as birthplace.
- Perplexity: 44.1 vs. 76.1/85.4 perplexity shows KGLM substantially outperforming the other entity-based language models on Linked WikiText-2.The reported perplexities use importance sampling to marginalize over annotations.
- Perplexity: KGLM significantly outperforms all models in unknown penalized perplexity, demonstrating stronger generation of rare tokens.This metric addresses comparisons where rare entity aliases are out-of-vocabulary for traditional language models.
- Fact completion: Both KGLM variants significantly outperform AWD-LSTM in factual completion, producing accurate facts instead of generic common words.They are also competitive with models trained on orders of magnitude more data, including completions involving birthplaces, dates, and authors.
- Limitations: KGLM does not capture facts or relations that frequently appear in large corpora, such as cities within states.The paper attributes one reported error category to the model’s ability to pick the correct relation from the knowledge graph given the prompt.
- Qualitative comparison: KGLM often completes facts about rare entities with specific tokens, whereas GPT-2 more often handles popular entities using generic tokens.Examples contrast specific outputs such as exact dates with generic alternatives such as “January” or “New York.”
- Knowledge-graph control: Changing the knowledge graph’s birth-date fact changes KGLM’s top decoded tokens from “August”, “4”, “1961” to “March”, “21”, “2013”.This illustrates direct controllability through modifications to the external knowledge source.
6 Related Work
The paper connects KGLM to knowledge-based language modeling, data-to-text generation, and general language-model improvements. Its distinction is using broad structured knowledge to improve factual and rare-token handling rather than narrow-domain text generation.
- Knowledge-based language models: ENTITYNLM tracks entity mentions through named-entity recognition and coreference, while KGLM models entity linking and relation extraction to generate factual information.Both approaches track entities through documents, but KGLM incorporates external knowledge-graph information.
- Knowledge-based language models: NKLM established knowledge-graph use in neural language models, whereas KGLM operates on an entire graph and needs no entity supplied in advance.NKLM instead uses predefined edges from one entity and requires that entity as conditioning information.
- Data-to-text generation: Prior data-to-text methods target coherent text in narrow domains and optimize BLEU or METEOR, while KGLM targets rare tokens and facts across broad topics.KGLM emphasizes perplexity rather than those task-specific generation metrics.
- General language modeling: AWD-LSTM improvements on WikiText-2 are treated as compatible with KGLM because the techniques are considered orthogonal.The paper expects combining them with KGLM to yield further improvement.
7 Conclusions and Future Work
The paper concludes that external knowledge graphs and Linked WikiText-2 support more factually correct language generation involving rare entities, numbers, and dates. It also identifies inference and annotation requirements as directions for further work.
- Conclusions: KGLM uses an external knowledge graph to generate higher-quality factual text containing rare entities and specific tokens such as numbers and dates.The evaluation reports this capability as a consequence of utilizing the graph.
- Conclusions: Linked WikiText-2 aligns text with knowledge-graph facts, enabling efficient KGLM training and providing a freely available research resource.The dataset’s distantly supervised construction is presented as part of the paper’s contribution.
- Future work: KGLM’s limitations include the need for marginalization during inference and reliance on annotated tokens.The paper presents these constraints as research problems for advancing knowledge-aware neural NLP models.
- Future work: The distantly supervised dataset-construction approach can be applied to other knowledge graphs and text types for language modeling in new domains.This is identified as an opportunity rather than a demonstrated result in the paper.