PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Relation Language Models

Hiroaki Hayashi, Zecong Hu, Chenyan Xiong, Graham Neubig

arXiv:1908.07690v1cs.CL

TL;DR

Neural language models can underperform on rare words, while matching text spans to knowledge-graph relations is uncertain. LRLMs model text with latent relation-linked spans and outperform previous approaches on open-vocabulary language modeling, with relation posteriors that align with human intuitions.

  • Problem

    Neural language models often underperform on rare words, and it is unknown which text spans match knowledge-graph relations, while simple matching can yield false positives.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    LRLMs jointly model text and latent generation choices, generating words from a fixed vocabulary or spans defined by relations to a topic entity.

    Source: §1 · p. 2

  • Results

    LRLMs outperform previous approaches on open-vocabulary language modeling, and their relation posteriors accord with human intuitions about how relations are expressed in text.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Posterior probabilities over latent relations can also be treated as entity links of interest to information extraction.

    Source: §1 · p. 2

    WikiFacts has no entity aliases, so its surface-form selection module acts as an oracle and assigns probability 1 to the correct surface form.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

In this paper, we propose Latent Relation Language Models (LRLMs), a class of language models that parameterizes the joint distribution over the words in a document and the entities that occur therein via knowledge graph relations. This model has a number of attractive properties: it not only improves language modeling performance, but is also able to annotate the posterior probability of entity spans for a given text through relations. Experiments demonstrate empirical improvements over both a word-based baseline language model and a previous approach that incorporates knowledge graph information. Qualitative analysis further demonstrates the proposed model's ability to learn to predict appropriate relations in context.

1 Introduction

Neural language models often underperform on rare words, motivating language models that use structured knowledge as context. LRLMs generate relation-defined entity spans as well as vocabulary words, and outperform previous approaches on open-vocabulary language modeling.

  • Neural language models often underperform on rare words because limited parameter budgets disadvantage them against non-parametric n-gram models.Source: §1 · p. 1
  • Earlier conditional language models use structured knowledge sources such as knowledge graphs, tables, or grammars as context.Source: §1 · p. 1
  • LRLMs condition on knowledge-graph relations and generate either vocabulary words or relation-defined entity spans, with latent generation choices jointly learned and marginalized.Source: §1 · p. 2
  • On open-vocabulary language modeling, LRLMs outperform previous approaches and assign relation posteriors that accord with human intuitions.Experiments use Wikipedia articles with relational information about each article's topic.Source: §1 · p. 2

2 Language Modeling Conditioned on Structured Knowledge

This section frames language modeling conditioned on structured data as predicting text from a topic-related knowledge graph. It defines the topic-conditioned task and explains that knowledge graphs offer rare-word coverage and potentially factually consistent entity associations.

2 Language Modeling Conditioned on Structured Knowledge

  • The section introduces open-vocabulary language modeling conditioned on structured data as its task.Source: §2 · p. 2

2.1 Task Definition

  • For a topic entity s, the task is to model P(X | G, s) using a subgraph of directly related knowledge-graph nodes and edges.The graph represents entities or attributes with surface forms, and objects may have multiple aliases.Source: §2.1 · p. 2

2.2 Why Condition on Knowledge Graphs?

  • Knowledge graphs have high coverage of rare words and may support factually consistent text through associations between entities.Source: §2.2 · p. 2

3 Latent Relation Language Models

LRLMs generate text either from a word vocabulary or by matching spans to knowledge-graph relations, treating those matches as latent choices. The model marginalizes over valid choices during training and parameterizes its generation decisions with neural networks.

3.1 Motivation

  • Unannotated relation-to-text matches and false positives from simple matching motivate treating spans and their generation sources as latent variables.A relation-generated span must match a valid surface form of an object related to the topic entity; the spans form a segmentation of the text.Source: §3.1 · p. 3
  • The conditional probability is obtained by marginalizing over valid latent-variable sequences, a marginalization the paper says is tractable.The model conditions on the KG subgraph and topic entity.Source: §3.1 · p. 3

3.2 Definition

  • Given a latent sequence, LRLM factors generation by time step; its process chooses either a word or a relation and, for a relation, a surface form.The preceding tokens provide the context for decisions.Source: §3.2 · p. 3, §3.3 · p. 4

3.3 Training

  • Training marginalizes over latent sequences with forward-backward because each time-step probability is independent of previous latent choices.The optimized marginal probability is αN; backward probabilities support gradient computation.Source: §3.3 · pp. 3–4

3.4 Parameterization

  • Neural networks parameterize the model’s probability distributions using a D-dimensional hidden state that may come from any neural sequence model.The paper experiments with multiple sequence models to demonstrate the approach’s generality.Source: §3.4 · p. 4

3.4.1 Source Selection

  • A linear model followed by softmax selects the generation source from the latest word-level hidden state.Source: §3.4.1 · p. 4

3.4.2 Word Generation

  • The word path uses a vocabulary distribution for ordinary words and a character language model to spell unknown words; its low-rank transform outperformed a full transform empirically.The character model is pretrained on unique training words and fixed during LRLM training.Source: §3.4.2 · p. 4

3.4.3 Relation Generation

  • Relation generation factors into relation selection and surface-form selection, using knowledge-graph embeddings for relations and objects and pretrained fastText embeddings for surface forms.The relation embedding concatenates the relation-type and object embeddings.Source: §3.4.3 · pp. 4–5

4 Datasets

The experiments use WikiFacts and two WikiText-derived datasets to evaluate language modeling with knowledge-graph information. WikiFacts is limited to film-actor articles and has a single alias per object, while WikiText-F and WikiText-S extend the data to open-domain articles.

4.1 WikiFacts

  • WikiFacts contains first-section Wikipedia articles about Freebase film-actor entities and uses a random 80/10/10% split.The dataset has no official evaluation splits.Source: §4.1 · p. 5
  • WikiFacts excludes linked-page gold entities from its main experiments because those entities would not be available when generating new articles.Results including those entities are reported in the appendix for consistency with prior work.Source: §4.1 · p. 5
  • WikiFacts has one alias per object, making surface-form selection an oracle that assigns probability 1 to the correct form.Source: §4.1 · p. 5

4.2 WikiText

  • To move beyond WikiFacts’ film-actor domain, the authors build open-domain WikiText data by linking WikiText-103 articles to Wikidata entities and their relations.They extract each article’s topic-entity neighbors as a star-shaped subgraph, match object surface forms in the article (allowing token overlaps), and rely on the model to discount false-positive matches.Source: §4.2 · p. 5
  • WikiText-F retains the articles, while WikiText-S truncates them after the first section to approximate WikiFacts article length and support comparisons across datasets.Source: §4.2 · p. 5

5 Experiments

The experiments define an open-vocabulary evaluation and compare LRLM with a vanilla language model and NKLM, using alternative sequence-model backbones and dataset-specific vocabularies.

5 Experiments

  • Open-vocabulary evaluation discounts unknown-word probability from spelling for every tested model, enabling evaluation beyond corpus-limited OOV adjustment.The paper reports token-level perplexity and contrasts spelling-based discounting with UPP.Source: §5 · pp. 5–6

5.1 Model Configuration

  • WikiFacts uses a fixed 40,000-word vocabulary, while WikiText-derived datasets include words occurring at least three times.Source: §5.1 · p. 6
  • Experiments use LSTMs and Transformer-XL as alternative sequence models for hidden-state computation.Source: §5.1 · p. 6

5.2 Baselines

  • LRLM is compared with a vanilla language model that removes its relation-generation module and is analogous to standard LSTM or Transformer-XL language models.Source: §5.2 · p. 6

Neural Knowledge Language Model (NKLM)

6 Results and Analysis

LRLM improves open-vocabulary perplexity over baselines across datasets, with a nonsignificant exception, and its generated spans and relation posteriors provide qualitative evidence of how it uses knowledge-graph context. Its gains are largest as the number of available relations increases.

6.1 Main Results

  • LRLM outperforms baselines on all datasets, though its LSTM+WikiText-S improvement is not statistically significant.On WikiText-derived datasets, it significantly improves over the vanilla LM; NKLM sometimes has worse perplexity than the vanilla LM.Source: §6.1 · p. 6, Table 2 · p. 7

6.2 Generated Samples

  • LRLM generates full entity mentions through span-level copying, while NKLM can produce partial mentions, false positives, and repetitions.Across 100 samples, Table 3 counts partially generated, fully generated, and valid mentions; LRLM sometimes predicts an article's theme incorrectly.Source: §6.2 · pp. 6–7, Table 3 · p. 7

6.3 Posterior Probability of Spans

  • Relation posteriors connect entities such as Madonna to the topic based on context, and the model tends to generate multiword spans through relations.For common phrases such as “the United States,” it often favors word-based generation even when a relation-based prediction is possible.Source: §6.3 · p. 7, Table 4 · p. 8

6.4 Effect of Subgraph Size

  • LRLM's gain over baselines grows as the average number of relations per WikiFacts article increases.All models have slightly higher word-average log-probabilities as relations increase; the authors attribute LRLM's larger gain to marginalization over latent variables helping disambiguate candidates.Source: §6.4 · p. 8, Figure 4 · p. 8

7 Related Work

Related work conditions language models on varied sources of entity or contextual information, while NKLM is the closest prior approach using relations. LRLM also relates to latent-source selection and differs from KG-to-text generation that does not explicitly model referent entities as latent variables.

Entity-aware language models

  • Entity-aware language models condition on sources including coreference or entity annotations, definitions, and keywords.The cited examples include expert coreference annotations, entity annotations, definitions, or keywords.Source: §7 · p. 8

Relational language models

  • NKLM is the closest prior work using relational information, while LRLM more successfully lowers perplexity and calculates relation posteriors.The passage characterizes LRLM relative to NKLM on both outcomes.Source: §7 · p. 8

Knowledge-graph generation

  • KG-to-text sequence-to-sequence work uses graph or table attention but does not explicitly generate referent entities as latent variables.The distinction is drawn against LRLM's latent-variable treatment.Source: §7 · p. 8

Latent-variable generation

  • LRLM adapts latent-source selection from Ling et al. (2016) to Wikipedia article language modeling with a knowledge-graph triple repository.Other latent-variable approaches address topics, n-gram chunking, source choice, continuous vectors, compressed tokens, or syntactic and semantic trees.Source: §7 · p. 8

8 Conclusion

Latent Relation Language Models condition language models on knowledge graphs and represent text as spans, including entity spans. The conclusion highlights improved conditional language modeling and posterior relation probabilities for spans; the article collection uses Wikipedia articles linked to Wikidata entities.

8 Conclusion

  • LRLMs model text as spans, some generated as entities included in a knowledge graph.The model is a class of conditional language models conditioned on knowledge graphs.Source: §8 · p. 8
  • Marginalizing latent variables lets LRLMs outperform previous conditional language modeling approaches and score spans with posterior relation probabilities.Source: §8 · p. 8

A Article Collection

  • The article collection starts from raw WikiText-103 and identifies each article’s Wikidata entity through the Wikipedia API.Some training articles were discarded when the API returned no Wikidata ID; missed development and test matches were manually recovered.Source: A Article Collection · p. 13

B Training Details and Hyperparameters

The experiments use Adam and truncated backpropagation through time, with a 150-token window. Transformer-XL and LSTM models use distinct training schedules, while LSTM hyperparameters vary by dataset.

B Training Details and Hyperparameters

C Utilization of Extra Entities

Adding extra entities significantly improves results for both models. The entities increase the average relations and mentions per article, and their rarity and predictable inclusion are offered as explanations for the gains.

C Utilization of Extra Entities

D Data Preprocessing for NKLM

NKLM preprocessing excludes some entity matches and handles overlap differently across datasets. The section also describes CVT relation handling and UPP’s treatment of unknown-word probabilities.

D Data Preprocessing for NKLM

  • Preprocessing removes relations whose subject is a Freebase CVT and expands object-CVT relations into multiple field-based relations.After removing CVT-based relations, each article has an average of 37.67 relations.Source: D Data Preprocessing for NKLM · p. 14
  • For WikiText, overlapping matches are pruned greedily, with longer spans prioritized.The pruning order is predefined, and selected spans cannot overlap.Source: D Data Preprocessing for NKLM · p. 14
  • For overlapping WikiFacts matches, preprocessing prefers earlier spans and canonical entity surface forms, then breaks ties by relation type and surface-form index.Source: D Data Preprocessing for NKLM · p. 14
  • NKLM preprocessing omits partial and arbitrary-order entity matches because they increase matching complexity and may add false positives.WikiFacts copying positions are assigned sequentially within matched spans.Source: D Data Preprocessing for NKLM · p. 14

E Comparison of Models using UPP

Loading 1908.07690v1…