PaperlayerView exact revision on arXiv

Source-linked AI summary

Crawling the Internal Knowledge-Base of Language Models

Roi Cohen, Mor Geva, Jonathan Berant, Amir Globerson

arXiv:2301.12810v1cs.CLcs.AI

TL;DR

The paper addresses how to represent the factual knowledge encoded in language models so it can be inspected. It crawls a knowledge graph outward from a seed entity using prompt-based subtasks, and reports graphs with 82–92% precision. The approach is limited to two-hop graphs because of cost and error propagation.

  • Problem

    The paper asks how to represent language models’ encoded factual knowledge explicitly so people can inspect and query it.

    Source: §1 · p. 1

  • Method

    The paper uses prompt-based subtasks to expand a knowledge graph around a seed entity by generating relations and objects.

    Source: §3 · p. 3

  • Results

    GPT-3 produced knowledge graphs with ∼82-92% precision across 140 seed entities, with a plausible number of facts per entity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work extracts a structured knowledge base from an LM as a step toward understanding the knowledge these models possess.

    Source: §7 · p. 9

    The study evaluates only two-hop graphs because of cost and the risk that generation errors propagate and multiply in later steps.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Language models are trained on large volumes of text, and as a result their parameters might contain a significant body of factual knowledge. Any downstream task performed by these models implicitly builds on these facts, and thus it is highly desirable to have means for representing this body of knowledge in an interpretable way. However, there is currently no mechanism for such a representation. Here, we propose to address this goal by extracting a knowledge-graph of facts from a given language model. We describe a procedure for ``crawling'' the internal knowledge-base of a language model. Specifically, given a seed entity, we expand a knowledge-graph around it. The crawling procedure is decomposed into sub-tasks, realized through specially designed prompts that control for both precision (i.e., that no wrong facts are generated) and recall (i.e., the number of facts generated). We evaluate our approach on graphs crawled starting from dozens of seed entities, and show it yields high precision graphs (82-92%), while emitting a reasonable number of facts per entity.

1 Introduction

The paper asks how to represent the factual knowledge encoded in language models so people can inspect and query it. It proposes prompt-based graph crawling from seed entities and reports high-precision graphs.

1 Introduction

  • Knowledge graphs offer readable, queryable representations, but manual curation and closed schemas limit their coverage.The paper asks whether an LM can be converted into a KG that combines these advantages with the broader coverage of knowledge in raw text.Source: §1 · p. 1
  • The crawling task starts from an entity without known relations, so the system must discover relations and their objects while maintaining high factual precision.The full graph is reduced to recursively expanding a graph around a seed entity.Source: §1 · pp. 1–2
  • The approach decomposes crawling into prompt-based subtasks, including relation generation and object generation using few-shot examples.It prompts for “Don’t know” when the model is not confident and uses entity and relation paraphrases to obtain additional triplets.Source: §1 · p. 2
  • GPT-3 produces graphs with ∼82-92% precision across 140 seed entities, with a plausible number of facts per entity.The procedure also generates facts outside Wikidata’s schema.Source: §1 · p. 2
  • The paper’s contributions are formulating LM-to-KG crawling, presenting a prompt-based decomposition, and evaluating it with GPT-3.The evaluation yields high-precision graphs.Source: §1 · p. 2

2 Problem Setup

The paper represents an LM’s knowledge base as a knowledge graph and studies expanding it around a given seed entity. Entities and relations are strings without a fixed vocabulary.

2 Problem Setup

  • A knowledge graph is defined as entities connected by subject-relation-object triplets.Formally, G = (N, R, E), with entities in N, relations in R, and triplets in E.Source: §2 · p. 3
  • The setup assumes a user-provided seed entity around which the graph is expanded.The paper describes this as a more realistic scenario than having the LM generate seed entities.Source: §2 · p. 3
  • Entities and relations are represented as strings and are not restricted to a predefined vocabulary.The paper compares this aspect to open information extraction (Vo and Bagheri, 2017).Source: §2 · p. 3

3 Crawling KGs via Prompting

LMCRAWL expands a seed entity into a graph by generating relations and objects, then using paraphrases and “Don’t know” outputs to balance recall with precision. Its components use few-shot prompts and can generate relations beyond Wikidata.

3 Crawling KGs via Prompting

  • Entity expansion generates relations for an entity, then finds objects for each relation; an empty object set adds no edge when the model does not know an answer.Both steps use fixed few-shot in-context examples.Source: §3 · p. 3

3.1 Relation Generation

  • Relation generation uses Wikidata entities and their relations as in-context examples, while allowing the model to produce relations beyond Wikidata.The generated relations form the set used in subsequent crawling.Source: §3.1 · p. 3

3.2 Relation Paraphrasing

  • Relation paraphrasing supplies alternative strings for each relation, and downstream crawling is run for each variant.The paraphrasing procedure uses an instruction prompt; in-context examples are not necessary.Source: §3.2 · p. 3

3.3 Object Generation

  • Object generation seeks objects for each entity-relation pair, and accepts objects only when at least two relation variants produce them.The method also includes sports-team relations such as MASCOT or LARGEST WIN, which are outside Wikidata.Source: §3.3 · pp. 3–4

3.4 Learning to Output “Don’t Know”

  • DK Object Generation uses erroneous model predictions as examples labeled “Don’t know,” alongside correct predictions, to teach when to abstain.The prompt contains equal numbers of failure and correct-prediction examples.Source: §3.4 · p. 4

3.5 Subject Paraphrasing

  • Subject paraphrasing generates up to three alternative names for an entity by sampling the model three times.The paper reports that DK handles most nonsensical paraphrases, supporting its use with paraphrasing to control recall and precision.Source: §3.5 · p. 5

3.6 LMCRAWL

  • The full LMCRAWL pipeline paraphrases the subject, generates relations, paraphrases those relations, and extracts known objects.The illustrated example starts from BARACK OBAMA and yields the triplet (BARACK OBAMA, SPOUSE, MICHELLE OBAMA).Source: §3.6 · p. 5, Figure 2 · p. 5

4 Experimental Setup

The experiments used 140 Wikidata-selected seed entities, divided between validation and test sets, to evaluate knowledge-graph construction. Because no ground-truth graph was available, evaluation relied on fact verification and reported verified-triplet counts rather than recall with a denominator.

4 Experimental Setup

  • The 140 Wikidata-selected seeds were split into 20 validation entities for design choices and 120 test entities for final evaluation; none appeared in prompt construction.The test set comprised 100 entities sampled across 25 categories and 20 popular entities; Table 2 lists validation and head-test seeds.Source: §4 · p. 5, Table 2 · p. 6
  • The 100-entity main test set sampled four entities from each of 25 categories, balancing two randomly selected entities with two sampled from the first 1,000.An additional 20 popular entities were randomly selected from a manually constructed pool of 1,000.Source: §4 · p. 5
  • Precision was checked manually for all head-test graphs and 1-hop main-test graphs, and for a random 20% of 2-hop triplets; remaining triplets were checked automatically.The automatic check searched Google results for the predicted object among the first 40 words after excluding HTML labels and URL links.Source: §4 · p. 6
  • Without a ground-truth graph, the authors report verified triplet counts as “# of facts,” not recall with a denominator.Wikidata omits correct facts and equivalent facts may be phrased differently, complicating direct comparison.Source: §4 · p. 6
  • Experiments used text-davinci-002 with greedy decoding or three sampled outputs at temperature 0.8, and one- or two-step expansion.Duplicates were removed when token-wise F1 exceeded 0.85.Source: §4 · p. 6
  • The base ablations combined Relation Generation with Pure Object Generation; variants added Don’t Know object generation or subject and relation paraphrasing.Pure-Greedy and Pure-Sampling distinguish the decoding choice in the base variants.Source: §4 · p. 6

5 Results

LMCRAWL produces high-precision graphs on main and head test sets, while ablations show how “Don’t Know” prompting and paraphrasing affect precision and coverage. Extracted fact counts correlate with WIKIDATA counts, and the authors caution that their automatic precision estimate may be imperfect.

5 Results

  • 83.3% and 82.0% precision for LMCRAWL’s 1-hop and 2-hop graphs, versus 54.6% and 43.4% for Pure-Greedy on the main test set.The paper describes the resulting graph sizes as reasonable despite a small hit in number of facts.Source: §5 · p. 7, Table 3 · p. 7
  • 91.5% and 90.9% precision for LMCRAWL’s 1-hop and 2-hop graphs, respectively, on the head test set.Both precision and number of facts are higher than on the main test set; the authors suggest popular entities may be easier to extract facts about or may have more facts encoded.Source: §5 · p. 7, Table 3 · p. 7

5.1 Ablations

  • “Don’t Know” prompting increases precision but reduces generated facts, while RP and SP separately increase coverage with minimal precision loss and together improve both measures.The authors report that subject and relation paraphrasing recover some facts lost through “Don’t Know” prompting.Source: §5.1 · p. 7

5.2 Coverage vs. Entity Frequency

  • Extracted fact counts for validation PERSON entities correlate with WIKIDATA fact counts at 0.61, despite identical expansion treatment for head and tail entities.LMCRAWL does not use entity frequency.Source: §5.2 · p. 8, Figure 3 · p. 8

5.3 Precision is Possibly Underestimated

  • Manual inspection found 4.1% of automatically correct-labeled triplets were wrong and 22% of automatically incorrect-labeled triplets were true.The authors call the automatic estimate conservative and note that exact precision estimation would require full manual annotation, which they avoided to minimize costs.Source: §5.3 · p. 8

6 Related Work

Prior knowledge-base construction relies substantially on manual effort or automated information extraction, while LM probing has mostly targeted specific facts. Related KG-extraction work also uses language models, but the cited approaches assume relations are supplied, unlike this paper’s seed-driven relation generation.

6 Related Work

  • WordNet, ConceptNet, and WIKIDATA rely heavily on manual effort, while information extraction methods have been developed to reduce that labor.LM knowledge research has mostly focused on probing for specific facts.Source: §6 · p. 8
  • Hao et al. (2022) and Alivanistos et al. (2022) assume relations are provided, whereas this approach starts with a seed entity and generates relevant relations.The paper describes its work as the first to construct a KG directly from an LM using only one seed entity, without other given relations or entities.Source: §6 · p. 8
  • The “Don’t know” mechanism is related to work on expressing certainty, but this approach uses it for KG crawling through in-context learning rather than fine-tuning.The comparison is specifically with Kadavath et al. (2022).Source: §6 · p. 8

7 Conclusion

The paper presents extracting a structured knowledge base from a language model as a step toward understanding the knowledge it possesses. It identifies larger graphs and more cost-effective approaches as future directions.

7 Conclusion

  • Extracting a structured knowledge base from an LM is presented as a step toward understanding the knowledge models possess.The authors connect this goal to using and revising that knowledge to avoid factual errors and biases.Source: §7 · pp. 8–9
  • Expanding the graph to more hops would require many more API calls, which are currently costly.The authors also suggest developing more cost-effective approaches.Source: §7 · p. 9
  • The authors suggest consistency constraints, such as checking whether generated FATHER OF and CHILD OF relations agree, as a possible precision improvement.They also identify querying a larger extracted knowledge graph for question answering as a future direction.Source: §7 · p. 9
  • The paper frames converting LMs to KGs for better interaction and control as a direction for future research.Source: §7 · p. 9

Limitations

The authors identify limits on graph expansion, precision evaluation, and interpreting inaccuracies; they also note that measuring knowledge coverage remains unresolved. Relation paraphrasing uses three manually constructed prompts per generated relation.

Limitations

  • Errors during graph generation may propagate at later steps, motivating the authors to limit experiments to two-hop graphs; expanding further also faces API cost.Source: Limitations · p. 9
  • Automatic precision evaluation is approximate, so reported 2-hop accuracy only approximates true precision.The authors believe true precision is higher.Source: Limitations · p. 9
  • The study does not determine whether inaccurate facts reflect extraction limitations or facts absent from the LM; generated graphs may also contain disinformation or misleading facts.Source: Limitations · p. 9
  • The authors leave unresolved whether their proposed recall metric better reflects coverage, since measuring knowledge coverage remains unclear.Source: Limitations · p. 9

A.1 Relation-Paraphrasing

  • For each generated relation, the relation-paraphrasing procedure makes three additional model calls using manually constructed prompts to produce paraphrases.Overlapping paraphrases may be eliminated.Source: §A.1 · p. 10

C Main Test Set

The main test set contains 100 seeds drawn from 25 predefined entity-group categories.

C Main Test Set

  • The main test set comprises 100 seeds, with four selected from each of 25 predefined entity-group categories.Source: C Main Test Set · p. 12

D Automatic Precision Evaluation

The automatic precision evaluation can fail when Google search or string matching does not recognize a fact. The examples include exact-string mismatch and equivalent paraphrases.

D Automatic Precision Evaluation

  • An exact-string mismatch can prevent verification: the Boston Celtics league fact is not verified with “NATIONAL BASKETBALL ASSOCIATION (NBA),” but succeeds when “(NBA)” is dropped.Source: D Automatic Precision Evaluation · p. 12
  • A paraphrase can also prevent verification: “UNITED KINGDOM” is not verified for Marble Arch’s country, while “ENGLAND” succeeds.Source: D Automatic Precision Evaluation · p. 12

E Additional Generated Graphs

Figures 4 and 5 add depth-2 LMCRAWL example graphs around Angela Merkel and the Boston Celtics, respectively, beyond Figure 1.

E Additional Generated Graphs

Related papers

Loading 2301.12810v1…