PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models are Open Knowledge Graphs

Chenguang Wang, Xiao Liu, Dawn Song

arXiv:2010.11967v1cs.CLcs.AIcs.LG

TL;DR

Knowledge graph construction often requires costly human supervision, while pre-trained language models acquire factual knowledge from large-scale corpora. This paper introduces MAMA, an unsupervised single-pass approach for turning that knowledge into open KGs, and reports that its methods achieve over 60% precision and outperform two open IE systems.

  • Problem

    KG construction commonly requires costly human supervision, despite pre-trained language models acquiring factual knowledge from large-scale corpora.

    Source: §1 · p. 1

  • Method

    MAMA uses an unsupervised two-stage process to match corpus facts with language-model knowledge and map the matches to fixed or open schemas in one forward pass without fine-tuning.

    Source: §1 · pp. 1–2

  • Results

    MAMA's methods achieve greater than 60% precision and outperform two open IE systems; MAMA-GPT-2XL outperforms Stanford OpenIE by over 2.6% in F1.

    Source: §3.1.2 · p. 7

  • Takeaways & Limitations

    The results suggest that larger language models store richer knowledge than existing knowledge graphs.

    Source: §5 · p. 9

    The authors identify quantitative evaluation of unmapped facts as future work to better understand MAMA's strengths and shortcomings.

    Source: §3.2 · p. 9

Abstract

from arXiv · show

This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.

1 INTRODUCTION

MAMA uses factual knowledge stored in pre-trained language models to build open knowledge graphs from text in one forward pass, without fine-tuning or human supervision. It matches corpus facts to LM knowledge, then maps them to fixed or open schemas.

1 INTRODUCTION

  • Existing KG construction commonly depends on human supervision, either to write facts directly or to improve semi-automatic extraction, which can be expensive.The introduction contrasts supervised construction with semi-supervised extraction that still requires human interaction to improve fact quality.Source: §1 · p. 1
  • Pre-trained language models acquire factual knowledge from large-scale corpora, motivating the paper’s proposal to use that stored knowledge for KG construction.The authors frame this as an alternative to manually created knowledge and as a way to make learned knowledge explicit.Source: §1 · pp. 1–2
  • MAMA constructs knowledge graphs in one forward pass of a pre-trained language model, using an unsupervised Match stage followed by a Map stage.Match uses beam search over attention-weight matrices without fine-tuning; Map assigns matched facts to a fixed KG schema or preserves them in an open schema.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The resulting open KG combines facts mapped to existing schemas with unmapped facts retained in an open schema, including facts new to the reference schema.The paper names Wikidata and TAC KBP as the existing KG schemas used for mapped facts.Source: §1 · p. 2

2 MAMA

MAMA constructs open knowledge graphs from pre-trained language models in a single forward pass over the corpus, without fine-tuning. Match generates candidate facts, and Map places them in a fixed KG schema or an open schema.

2 MAMA

  • MAMA’s two stages match corpus text to language-model knowledge and map the resulting candidate facts into fixed-schema or open-schema portions of an open KG.The method uses a single forward pass of pre-trained language models over the corpus, without fine-tuning.Source: §2 · p. 2

2.1 MATCH

  • Matching treats candidate-fact generation as a search over pre-trained language-model attention weights, which are obtained in a forward pass without fine-tuning.The search returns candidates with the top matching degrees for an input sentence.Source: §2.1 · p. 3

2.1.1 BEAM SEARCH

  • Beam search maintains the k-best candidates for each head-tail pair, appending the next highest-attended token until the tail is reached, and can search in both directions.The breadth-first implementation has time complexity O(k · d), where d is the maximum search-tree depth.Source: §2.1.1 · pp. 3–4

2.1.2 FILTER

  • Candidate facts are retained only when their matching degree and relation’s distinct frequency exceed thresholds and the relation is contiguous in the sentence.These filters target poorly matched facts, over-specific relations, and relations without meaningful interpretations.Source: §2.1.2 · p. 4

2.2 MAP

  • Map creates an open KG containing facts mapped to an existing KG schema and unmapped facts represented in an open schema.The fixed schema can be an existing KG schema such as Wikidata.Source: §2.2 · p. 5

2.2.1 MAPPED FACTS IN KG SCHEMA

  • For fixed-schema mapping, an unsupervised mention-to-entity linker uses contextual word embeddings, while relation phrases are matched to KG relations using co-occurring linked head-tail pairs.Relation phrases are normalized, and one author manually checks the top 15 phrases for each KG relation; this takes approximately one day.Source: §2.2.1 · p. 5

2.2.2 UNMAPPED FACTS IN OPEN SCHEMA

  • Unmapped facts may be partially mapped or entirely unmapped; keeping them in an open schema lets the resulting KG extend fixed-schema coverage with flexible facts.Partial mappings retain any linked entities or relation, while completely unmapped facts retain all three components in the open schema.Source: §2.2.2 · p. 5

3 EXPERIMENTS

The experiments evaluate MAMA’s mapped facts against human-built knowledge graphs and analyze the quality of facts that remain in its open schema.

3.1.1 DATASETS

  • MAMA’s mapped facts were evaluated against TAC KBP 2013 oracle facts and Wikidata oracle facts, using English Wikipedia as the Wikidata corpus.The evaluation ran Match over the corresponding documents and then Map to each reference KG schema, with shared parameter settings.Source: §3.1.1 · p. 6

3.1.2 TAC KBP

  • Over 2.6% F1 gain over Stanford OpenIE for MAMA-GPT-2XL on TAC KBP; all MAMA methods outperform both open IE systems.All proposed methods achieve precision greater than 60%, despite MAMA’s unsupervised setup.Source: Table 2 · p. 7, §3.1.2 · p. 7
  • Larger/deeper LMs improve KG quality; at similar sizes, BERT beats GPT-2 in F1, while GPT-2 has higher precision and BERT higher recall.The paper attributes BERT’s recall advantage to its masked-language-model objective and GPT-2’s precision advantage to less pre-training noise from masking.Source: §3.1.2 · pp. 7–8

3.1.3 WIKIDATA

  • 5.6% F1 gain over Stanford OpenIE for MAMA-GPT-2XL on Wikidata; it also leads MAMA-BERTLARGE by over 1% F1.MAMA-GPT-2XL performs best in precision, recall, and F1; MAMA-BERTLARGE gains approximately 4.4% F1 over Stanford OpenIE.Source: Table 3 · p. 8, §3.1.3 · p. 8

3.2 ANALYSIS OF UNMAPPED FACTS

  • 35.3% of sampled unmapped Wikidata facts were true, and 83.2% of those true facts were partially unmapped.Among untrue facts, 45.5% were attributed to incorrect spaCy entity detection, 9.1% to incorrect or missing entity linking, and 4.5% to missing relation mapping; the paper identifies spaCy noun-chunk errors as the main root cause and proposes better entity detection, linking, and relation mapping.Source: §3.2 · pp. 8–9

4 RELATED WORK

Earlier knowledge graphs use human supervision or semi-supervised extraction, while MAMA uses pretrained language models to construct an open KG; related work also treats models as implicit knowledge stores or analyzes their internals.

  • Knowledge graphs traditionally use supervised structured sources or semi-supervised pattern- and schema-based extraction; MAMA instead uses pretrained-LM knowledge without human supervision to build a mixed fixed- and open-schema KG.Source: §4 · p. 9
  • MAMA explicitly generates whole fact triplets from sentences, unlike LAMA’s cloze-based fact recall (Petroni et al., 2019; 2020).Its benchmark datasets are larger than LAMA’s; Wikidata is three orders of magnitude larger than the largest LAMA dataset.Source: §4 · p. 9
  • Prior work uses pretrained language models as implicit knowledge sources for question answering and language modeling, while MAMA explicitly extracts their facts (Dhingra et al., 2020; Guu et al., 2020; Khandelwal et al., 2019).Source: §4 · p. 9
  • Whereas prior neural-network interpretation work visualizes or analyzes Transformer attention, MAMA generates structured knowledge graphs to recover what language models learn (Vig, 2019; Jain & Wallace, 2019; Clark et al., 2019).Source: §4 · p. 9

5 CONCLUSION

MAMA constructs open knowledge graphs from language models in a single forward pass, matching corpus facts to model knowledge and mapping the matched facts. The resulting graphs compare favorably in quality with Wikidata and TAC KBP, include new open-schema facts, and offer a way to inspect what models learn.

5 CONCLUSION

  • MAMA uses a two-stage unsupervised approach to match corpus facts with a language model’s internal knowledge, then map the matched facts into a knowledge graph in one forward pass.The pass runs over textual corpora and does not require fine-tuning.Source: §5 · p. 9
  • The resulting open knowledge graphs are evaluated against Wikidata and TAC KBP and feature new facts in an open schema.The conclusion describes the comparison as demonstrating graph quality, but reports no specific scores here.Source: §5 · p. 9
  • The results suggest that larger language models store richer knowledge than existing knowledge graphs, and that larger high-quality text corpora could further improve the graphs.Source: §5 · p. 9
  • The generated graphs can help researchers inspect what language models learn, establishing a bridge between deep learning and knowledge graph communities.The conclusion notes potential implications for knowledge graphs and their downstream applications.Source: §5 · p. 9

A ADDITIONAL DETAILS AND ANALYSIS OF MAMA · A.1 IMPLEMENTATION DETAILS

MAMA evaluates mapped facts by matching documents and mapping entities and relations. Its Wikidata oracle retains 27,368,562 facts after filtering.

A.1 IMPLEMENTATION DETAILS

  • MAMA generates candidate facts with Match, uses oracle span annotations on TAC KBP, and links Wikidata entities through Wikipedia anchors and a Wikidata-item dictionary.The Wikidata linking process also uses neuralcoref for pronoun heads or tails and GloVe embeddings for disambiguation.Source: §A.1 · p. 13
  • The Wikidata oracle contains 27,368,562 facts after excluding entity attributes and auxiliary relations and retaining only relations linkable to Wikipedia anchors.Relation mappings are constructed offline for TAC KBP and Wikidata.Source: §A.1 · p. 13
  • For Wikidata matching, English Wikipedia is split across 20 servers, each configured with four Tesla K80 12G GPUs; MAMA-BERTLARGE takes about 48 hours and MAMA-GPT-2XL about 96 hours.The setup uses a maximum sequence length of 256 and batch sizes of 32 for MAMA-BERTLARGE and 4 for MAMA-GPT-2XL.Source: §A.1 · p. 13
  • Wikidata Map stage takes about 18 hours, while TAC KBP Match stage completes within 48 hours across settings.A MongoDB database stores the Wikidata oracle and entity-linking results to support the Map stage.Source: §A.1 · p. 13
  • For MAMA-BERTBASE and MAMA-GPT-2, GPT-2MEDIUM, and GPT-2LARGE, batch sizes are 64, 32, 16, and 8, respectively.Pretrained language models are implemented with Transformers, and spaCy sentencizer segments documents into sentences.Source: §A.1 · p. 13
  • Across TAC KBP and Wikidata, MAMA sets Algorithm 1 beam size to 6, Constraint #1 matching degree threshold to 0.005, and Constraint #2 distinct head-tail pairs to 10.The parameter choices follow the study in Sec. A.3.Source: §A.1 · p. 13
  • To form a sentence attention-weight matrix, MAMA averages attention weights across every attention head in the pretrained model’s last layer.This setting is shared across both datasets.Source: §A.1 · p. 13

A.2 ERROR ANALYSIS

An error analysis of MAMA-GPT-2XL finds that precision errors arise from entity detection, missing relation mappings, and incorrect relation phrases, while some apparent errors are correct facts absent from reference KGs. The authors suggest improving extraction components and adding true facts to reference KGs.

A.2 ERROR ANALYSIS

  • In a 100-document Wikidata sample, 33.1% of precision errors involved incorrect entities, 18.3% missing relation mappings, and 23.8% were correct facts new to the reference KGs.Incorrect entities were linked to incorrect noun chunks detected by spaCy; the remaining errors were attributed to incorrect relation phrases, including uninformative ones.Source: §A.2 · p. 13
  • MAMA-GPT-2XL made similar errors on TAC KBP, and the authors identify better entity detection, entity linking, relation mapping, and relation generation as helpful.They also plan to use lifelong learning to add true facts to reference KGs and improve evaluation.Source: §A.2 · p. 13

A.3 PARAMETER STUDY

The parameter study uses MAMA-BERTBASE on TAC KBP, tuning on a random 20% hold-out of oracle query entities and measuring effects with F1. The best setting is then used for both TAC KBP and Wikidata experiments.

Effects of beam size

  • Larger beam sizes generally improve F1, but gains become subtle as computation costs rise; the study therefore sets beam size to 6.The larger beam preserves more potentially correct facts during the Match stage.Source: §A.3 · p. 14

Effects of search constraints

  • Constraint #1 performs best at a matching-degree threshold of 0.005, while Constraint #2 performs best with 10 distinct head-tail pairs, capturing some rare relations.The summed attention score is normalized by fact length to penalize cumbersome facts.Source: §A.3 · p. 14

Effects of attention weights

  • Last-layer attention weights outperform weights averaged across layers, and mean reduction across last-layer heads outperforms max reduction.The paper attributes the first result to lower layers encoding less relevant linguistic knowledge (Clark et al., 2019; Ramsauer et al., 2020), and the second to mean reduction being more sensitive to tokens' intensive attention to specific tokens.Source: §A.3 · p. 14

B SAMPLES FROM MAMA ON TAC KBP · B.1 MAPPED FACTS

This appendix samples sentences from TAC KBP documents to illustrate MAMA’s candidate-to-mapped facts and cases where its BERTLARGE and GPT-2XL methods produce different facts.

B.1 MAPPED FACTS

  • The examples begin with 100 randomly sampled TAC KBP documents, followed by randomly sampled sentences from those documents.The appendix shows uncurated candidate facts and mapped facts produced by MAMA-BERTLARGE and MAMA-GPT-2XL in Figures 5 and 6, respectively.Source: §B.1 · p. 15
  • Figure 5 and Figure 6 show candidate facts and their corresponding mapped facts for sampled sentences using MAMA-BERTLARGE and MAMA-GPT-2XL, respectively.In each table, facts to the left of the arrow are candidates, and facts to the right are mapped results.Source: §B.1 · p. 15
  • Figure 7 presents sampled sentences where MAMA-BERTLARGE and MAMA-GPT-2XL produce different resulting facts.The sentences are randomly sampled for comparison.Source: §B.1 · p. 15
  • Each table’s ID identifies the sampled sentence’s document in the TAC KBP corpus.The Sentence column gives the sampled sentence itself.Source: §B.1 · p. 15

B.2 UNMAPPED FACTS

The section examines manually verified unmapped facts from sampled TAC KBP documents and presents model outputs and comparisons.

B.2 UNMAPPED FACTS

  • For 100 randomly sampled TAC KBP documents, the authors manually check unmapped candidate facts for correctness under Sec. 3.2; Figures 8–10 present MAMA-BERTLARGE and MAMA-GPT-2XL outputs and comparisons.Figures 8 and 9 show candidate facts alongside their corresponding unmapped facts; Figure 10 provides a further comparison.Source: §B.2 · p. 15

C SAMPLES FROM MAMA ON WIKIDATA · C.1 MAPPED FACTS

The section samples Wikipedia sentences to illustrate how MAMA maps candidate facts, including cases where MAMA-BERTLARGE and MAMA-GPT-2XL produce different facts.

C SAMPLES FROM MAMA ON WIKIDATA

  • The Wikidata sample draws 100 documents at random from the English Wikipedia corpus, then randomly samples sentences from those documents.Source: §C.1 · p. 15

C.1 MAPPED FACTS

  • Figures 11 and 12 show uncurated candidate facts and corresponding mapped facts from sampled sentences using MAMA-BERTLARGE and MAMA-GPT-2XL, respectively.Source: §C.1 · p. 15
  • Figure 13 presents randomly sampled sentences where MAMA-BERTLARGE and MAMA-GPT-2XL differ in their resulting facts.Source: §C.1 · p. 15
  • Each table identifies the Wikipedia page title as “ID” and provides the sampled sentence in the “Sentence” column.Source: §C.1 · p. 15
  • The “Candidate facts to mapped facts” column places each candidate fact to the left of an arrow and its corresponding mapped fact to the right.Source: §C.1 · p. 15
  • The displayed Wikidata examples use the paper’s best methods, MAMA-BERTLARGE and MAMA-GPT-2XL.Source: §C.1 · p. 15

C.2 UNMAPPED FACTS

The section presents manually checked unmapped facts from sampled Wikidata-corpus documents and compares outputs from two language models.

C.2 UNMAPPED FACTS

  • 100 randomly sampled Wikidata-corpus documents supplied sentences whose manually checked correct unmapped facts were generated by MAMA-BERTLARGE and MAMA-GPT-2XL.Figures 14–15 pair candidate facts with their corresponding unmapped facts for sampled sentences; Figure 16 provides a further comparison.Source: §C.2 · p. 15

D ADDITIONAL OPEN KG SUBGRAPHS FROM MAMA ON WIKIDATA

This section presents additional MAMA-generated open knowledge graph subgraphs and example facts from sampled Wikidata-corpus documents and Wikipedia pages, including facts mapped to Wikidata entities and properties and facts left unmapped. Examples from MAMA-BERTLARGE and MAMA-GPT-2XL show both facts represented in Wikidata and open-schema facts, including correct facts new to Wikidata.

D ADDITIONAL OPEN KG SUBGRAPHS FROM MAMA ON WIKIDATA

Mapped facts

Unmapped facts

Loading 2010.11967v1…