A Neural Knowledge Language Model

Motivation and proposed model

• Rare and unknown words, especially named entities, are difficult for traditional language models to represent because vocabulary-wide modeling is computationally prohibitive and such words remain sparsely observed.These limitations matter for knowledge-related applications such as question answering and dialogue modeling.Source: §1 · p. 1
• NKLM combines symbolic knowledge from a knowledge graph with an RNN language model to address factual-knowledge limitations.The model predicts whether each next word has an underlying fact and carries both fact and word context through the RNN.Source: §1 · pp. 1–2

NKLM approach

• NKLM generates ordinary vocabulary words through a softmax or knowledge words by copying positions from a predicted fact description, enabling words outside the predefined vocabulary.Because the model predicts facts explicitly, it can also adapt to added or modified knowledge without retraining.Source: §1 · p. 2

Resources and evaluation

• The paper introduces WikiFact, a dataset aligned with facts at the word level for evaluating knowledge-related language models.The dataset addresses the limited named-entity coverage and alignment difficulties of existing language-modeling datasets.Source: §1 · p. 2

Reported contributions

• NKLM significantly improves performance and generates named entities that traditional models treat as unknown words.The paper also highlights explicit fact representations as relevant to knowledge use, while reasoning ability itself is not investigated.Source: §1 · pp. 1–2

Resources and evaluation

• The paper proposes evaluation metrics that address standard perplexity’s tendency to reward predicting unknown words as the generic UNK class.This issue is especially consequential when named entities carry important information.Source: §1 · p. 2

Neural language models

• RNN language models capture longer-term dependencies, and LSTM-based RNNLMs have surpassed carefully tuned traditional n-gram models.This work provides the neural language-modeling foundation related to NKLM.Source: §2 · p. 2

Large-vocabulary modeling

• Hierarchical softmax, importance sampling, and noise-contrastive estimation reduce the computational cost of large-vocabulary language models but still suffer from rare or unknown words.These methods mitigate vocabulary-scaling costs rather than eliminating the rare-word problem.Source: §2 · p. 2

Copy mechanisms

• Pointer and copy mechanisms have been applied to translation, summarization, context sentences, and Wikipedia infoboxes to handle rare or unknown words.They infer copied words from available contextual text.Source: §2 · p. 2
• Existing copy approaches cannot be directly applied when context is short and contains no known relevant words, such as person names.NKLM instead supplies knowledge through a structured knowledge representation.Source: §2 · p. 2

External memory and context models

• External-memory models frame knowledge-based next-word prediction as fill-in-the-blank question answering, while other work combines Wikipedia and knowledge graphs for word-embedding enrichment.Related topic-based models represent context using latent variables or LDA-derived topic features.Source: §2 · p. 2

3.1. Preliminary

• NKLM represents each topic with knowledge-graph facts and a textual description, where each fact is a subject–relationship–object triple with associated knowledge words.Knowledge words are restricted mainly to object-entity words, while a special Topic Itself fact exposes subject words.Source: §3.1 · pp. 2–3
• Supervised string matching augments each description word with a fact identifier and binary fact indicator, assigning unmatched function words to the Not-a-Fact type.The resulting tuples support labels for whether a word is generated from knowledge or the vocabulary.Source: §3.1 · p. 3

3.2. Inference

• At each time step, NKLM feeds previous word and fact context into an LSTM, predicts a relevant fact from the knowledge memory, selects a generation source, and emits the next word.The source is either the global vocabulary or a word copied from the selected fact’s knowledge words.Source: §3.2 · p. 3

3.2.1. INPUT REPRESENTATION AND LSTM CONTROLLER

• The LSTM input concatenates embeddings for the previous fact and whichever previous word source was active, zeroing the inactive word representation before state update.This lets both fact and word predictions flow through the recurrent controller.Source: §3.2.1 · p. 4

3.2.2. FACT EXTRACTION

• Fact extraction maps the LSTM state and pooled topic-knowledge context to a fact key, then performs key–value lookup in memory to retrieve the selected fact embedding.The key generator is an MLP, and the topic context is intended to adapt lookup to changes in available facts without retraining.Source: §3.2.2 · p. 4

3.2.3. SELECTING WORD GENERATION SOURCE

• NKLM uses a binary decision from the LSTM state and extracted fact to choose vocabulary softmax generation or copying from the selected fact’s knowledge words.The copy probability is produced by an MLP, after which the chosen source determines word generation.Source: §3.2.3 · p. 4

3.2.4. WORD GENERATION

• Vocabulary generation uses a softmax, whereas knowledge generation predicts a position in the selected fact’s knowledge-word sequence and copies that word using learned position embeddings.Position prediction avoids relying on embeddings for often-unknown, short knowledge words and matches their frequent increasing-order use.Source: §3.2.4 · p. 4

3.3. Learning

• Training maximizes the supervised augmented-observation log-likelihood with stochastic gradient optimization, while inference obtains fact and source decisions from model predictions.The inference procedure applies hard decisions for the predicted fact and source when computing the next-word probability.Source: §3.3 · p. 5, §3.2.4 · p. 5

4.1. The WikiFacts Dataset

• WikiFacts aligns Wikipedia descriptions with corresponding Freebase facts and supplies word-level associations for knowledge-language-model evaluation.The FilmActor version filters topics with both Freebase entities and Wikipedia descriptions, using summary text and alias-based matching.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2.1. SETUP

• The evaluation compares NKLM with RNNLM using two-layer LSTMs with dropout, tests 200, 500, and 1000 hidden units, and reports the 1000-unit model selected by validation results.NKLM uses 40-dimensional symbol embeddings and 400-dimensional word embeddings, with slightly fewer parameters than RNNLM.Source: §4.2.1 · pp. 6–7

4.2.2. THE UNKNOWN PENALIZED PERPLEXITY

• Unknown-Penalized Perplexity evaluates unknown words by assigning their likelihood across an out-of-vocabulary set, with UPP-fact restricting that set to topic knowledge words.The metric addresses cases where predicting UNK can yield deceptively good standard perplexity despite failing to identify useful named entities.Source: §4.2.2 · p. 7

4.2.3. OBSERVATIONS FROM EXPERIMENT RESULTS

• NKLM significantly outperforms RNNLM on all three perplexity measures at vocabulary sizes 5K, 10K, 20K, and 40K, with the PPL gap increasing as vocabulary grows.For UPP, the gap remains similar across vocabulary sizes, while UNK predictions contribute very little to total perplexity.Source: Table 3 · p. 6, §4.2.3 · p. 7
• The copy mechanism drives NKLM’s largest improvement over fact-aware no-copy variants, while TransE embeddings also contribute to performance.No-copy benefits from fact information but improves less, and random entity vectors reduce performance relative to TransE initialization.Source: Table 2 · p. 6, §4.2.3 · p. 8
• NKLM generates roughly half as many unknown words as RNNLM and produces named entities from provided facts instead of replacing them with UNK tokens.The sampled descriptions show named entities, movie titles, band names, and places generated through the knowledge mechanism.Source: Table 4 · p. 7, §4.2.3 · p. 8
• When no knowledge is available, NKLM performs as well as RNNLM, while changing a topic fact lets it update a generated entity without retraining.The reported update replaces Oklahoma with Chicago while preserving the other generated words.Source: Table 4 · p. 7, §4.2.3 · p. 8
• Standard perplexity is strongly influenced by unknown-word predictions, so it requires careful interpretation for knowledge-related language modeling.UPP reduces the contribution of UNK predictions and maintains a more stable comparison as vocabulary size changes.Source: Table 3 · p. 6, §4.2.3 · p. 8

• The NKLM significantly outperforms the RNNLM in perplexity, generates named entities unseen during training, and immediately adapts to changes in knowledge.Source: §5 · p. 8
• The investigated task assumes that the true topic of a description is known, so practical deployment would require searching for the proper topic on the fly.Source: §5 · p. 8