BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models

1 Introduction

• Crowdsourcing, text mining, and KG completion remain limited by predefined relations, corpus mentions, or preexisting schemas.ConceptNet, for example, contains 34 relations, while open-schema extraction misses knowledge not explicitly stated in the corpus.Source: §1 · pp. 1–2, Figure 1 · p. 1
• Prompted LMs contain substantial implicit knowledge but lack the access, browsing, editing, and quality-assurance properties of symbolic KGs.Existing symbolic distillation methods also depend on GPT-3 capabilities or quality discriminators tied to existing KGs.Source: §1 · p. 2
• BertNet uses a prompt, a few example entity pairs, and search-and-rescore mechanisms to extract arbitrary relations from pretrained LMs.Prompt paraphrasing and prompt weighting improve search efficiency and outcome accuracy.Source: §1 · p. 2
• The framework harvests over 400 new relations across BERT, RoBERTa, and DistilBERT, including complex and 3-ary relations.Examples include relations expressing capability without competence and actions performed at a specified location or time.Source: §1 · p. 2

Knowledge graph construction

• Traditional KG construction relies on substantial human labor, while text-mining pipelines require corpus selection, aggregation, filtering, and linguistic extraction components.These methods commonly target relations represented by linguistic patterns or existing schemas.Source: §2 · pp. 2–3
• LM-based construction methods include link prediction, entity-pair scoring, generative KG completion, and GPT-3 distillation, but many depend on existing relations or strong few-shot models.GPT-3-based distillation is not generally applicable to most other pretrained LMs.Source: §2 · p. 3

LMs as knowledge bases

• Factual probing methods optimize predictions for one or a few entities, whereas BertNet explicitly harvests extensive entity pairs into symbolic knowledge graphs.Prompt-learning approaches can improve probing accuracy but are not designed to construct full explicit KGs.Source: §2 · p. 3

Consistency of LMs

• Consistency research motivates BertNet’s use of diverse prompts and multiple scoring terms to enforce stable relation-compatible entity pairs.The framework regularizes extracted pairs so they satisfy varied prompt formulations consistently.Source: §2 · p. 3

3 Harvesting KGs from LMs

• The framework takes an initial relation prompt and a few example entity pairs, then automatically searches for many entity pairs forming the desired knowledge graph.The task is more challenging than standard probing because it seeks extensive valid pairs rather than one or a few tails.Source: §3 · p. 3

3.1 Creating Diverse Weighted Prompts

• It generates linguistically diverse prompts by paraphrasing example-filled sentences, removing entity names, filtering by edit distance, and collecting at least 10 prompts.The process begins with an example pair and repeatedly paraphrases newly created prompts until the minimum is reached.Source: §3.1 · p. 4
• Because generated prompts may be imprecise, the method weights them using compatibility with example pairs, combining joint and individual likelihoods before softmax normalization.The balancing factor is α = 2/3 in the reported experiments.Source: §3.1 · p. 4

3.2 Efficient Search for Consistent Knowledge

• Entity-pair consistency is the weighted average of compatibility across prompts, so pairs compatible with the diverse prompt set are treated as consistent knowledge.Prompt weights learned during prompt creation are reused in this consistency calculation.Source: §3.2 · pp. 4–5
• To avoid exhaustive O(V^2) enumeration, the search first proposes candidates with weighted minimum individual likelihoods and then reranks them using the full consistency score.Pruning with a heap removes entities outside the top-K during proposal generation, after which the final top-K pairs are selected.Source: §3.2 · p. 5
• The approach extends beyond binary predicates to customized relations such as “A is capable of, but not good at, B” and n-ary relations such as “A can do B at C.”The paper presents the approach as adaptable to arbitrary fill-in-the-blank language models and interprets consistently harvested tuples as symbolic views of LM knowledge.Source: §3.2 · pp. 5–6

4.1 Setup

• The evaluation covers ConceptNet, LAMA, 12 human-written relations, and 487 automatically derived relations, with human annotation judging extracted tuples as accepted, rejected, or unjudgeable.Each tuple is labeled by three annotators, and accuracy is defined as the portion of accepted tuples.Source: §4.1 · p. 6, §4.2 · pp. 6–7

4.2 Extracting Knowledge of Diverse New Relations

• Using 487 easy-to-collect Auto relations, RobertaNet extracted 122K knowledge items with 65% accuracy, comparable to COMET (230K) and TransOMCS (18.4M), despite differing settings.The Auto relation set is an order of magnitude larger than predefined relation sets; comparisons across paradigms are collected for reference because settings differ.Source: §4.2 · p. 7

4.3 Analyzing Automatic Prompt Creation

• Multi-Prompts improves accuracy by 4% over Top-1 Prompt, while Top-1 Prompt improves up to 9% over Human Prompt; AutoPrompt performs worse with few examples.The results are based on 1,000 harvested tuples per Human relation evaluated by human annotation.Source: §4.3 · p. 7
• Multi-prompt scoring performs best on ConceptNet and LAMA precision-recall curves, outperforming COMET and LPAQA because entity-pair scoring supports KG extraction.The comparison uses positive and synthesized negative tuples and evaluates precision and recall at ranking cutoffs.Source: §4.3 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

4.4 Analysis of Knowledge in Different LMs

• Larger models improve accuracy by around 7% for BERT and 3% for RoBERTa, while RoBERTa’s better pretraining yields 0.73 versus 0.70 for large models and 0.70 versus 0.63 for base models.Knowledge distillation also improves DistilBERT’s result-graph accuracy by around 4%, possibly by eliminating noisy teacher information.Source: §4.4 · pp. 8–9

5 Conclusion

• The framework efficiently and scalably extracts KGs from pretrained LMs, producing relation-specific graphs such as BERTNET and ROBERTANET.It supports arbitrary new relation types and entities without restriction to pre-existing knowledge or corpora.Source: §5 · p. 9
• The resulting KGs provide a symbolic interpretation of the source LMs.Source: §5 · p. 9
• The design and experiments are limited to generic-domain LMs rather than domain-specific models such as healthcare models.The paper leaves harvesting knowledge from other neural networks and application domains for future work.Source: §5 · p. 9
• Automatically generated knowledge tuples may be unethical, although filtering sensitive words is suggested as an effective mitigation for simple-phrase graphs.The authors report no foreseeable negative societal impacts caused by the method itself.Source: §5 · p. 9

A Detailed Results of Harvested Knowledge

• Tables 3 and 4 present human-annotated results for harvested knowledge under different settings, while Table 5 reports results separately for each relation.Source: A Detailed Results of Harvested… · p. 12

B Preprocessing of ConceptNet

• ConceptNet preprocessing filters linguistic and trivial relations, retains confidence above 1, and excludes relations with fewer than 1000 eligible tuples.The test set from Li et al. (2016) is avoided because its training allocation leaves a small and unbalanced test set.Source: B Preprocessing of ConceptNet · p. 12

C Efficient knowledge tuple search

• Candidate proposal uses minimum token log-likelihoods for pruning, and the algorithm performs exact search rather than beam search.This avoids biasing results toward more probable head entities.Source: C Efficient knowledge tuple search · p. 12
• A minimum heap stores the best candidate MTL scores, using the heap top as a threshold to prune partial tuples that cannot surpass existing candidates.For 100 searched tuples, the heap has maximum size 100, and pruning stops continuous search when a partial log-likelihood falls below the threshold.Source: C Efficient knowledge tuple search · p. 12
• Algorithm 1 takes a language model, entity number, candidate-tuple limit, and relation prompts, then returns a list of candidate entity tuples.Source: C Efficient knowledge tuple search · p. 12

D Detailed Experiment setting

• GPT-3 paraphrases relation prompts, entity repetition is capped at 10, and each relation produces at most 50,000 searched entity tuples.Source: D Detailed Experiment setting · p. 12
• The method uses relation-specific thresholds set at 10% of the k-th consistency, retaining tuples above 0.1 × consistencyk; settings are base-10 and base-100 for k=10 and 100.Source: D Detailed Experiment setting · p. 12
• Table 2 variants include RobertaNet configurations ranging from 2.2K to 230K tuples across Human, Auto, and ConceptNet settings.The listed variants combine 50%, base-10, and base-100 truncation choices.Source: D Detailed Experiment setting · pp. 12–13
• Table 5 reports accepted and rejected tuple proportions by model and prompting method, including Human, Auto, Top-1, and Multi settings.The abbreviations cover Roberta-large, DistilBert, Bert-large, Bert-base, and Roberta-base.Source: Table 5 · p. 13
• Figure 6 shows how PLM(h|p) and PLM(t|p,h) are calculated for single-token entities and how multi-token entity probabilities are decomposed.The example uses the relation prompt “A IS THE PLACE FOR B,” with library as a one-token head and study as a first head token.Source: Figure 6 · p. 13

• The annotator instruction is presented in Figure 7.Source: E Human evaluation · p. 13
• The questions posed to annotators are presented in Figure 8.Source: E Human evaluation · p. 13
• Inter-annotator agreement is 0.27 by Krippendorff’s Alpha, indicating fair agreement.Source: E Human evaluation · p. 13

F Compute resource

• F Compute resource: Harvesting one relation with RoBERTa-large takes about one hour on a single Nvidia GTX1080Ti GPU.Source: F Compute resource · p. 13

• The reported setup combines a single-GPU harvesting process with publicly available experimental assets.Source: F Compute resource · p. 13, G The license of the assets · p. 13

G The license of the assets

• G The license of the assets: Datasets, relation definitions, and seed entity pairs are officially public resources.Source: G The license of the assets · p. 13

• The system produces results only from source language models such as BERT and does not add to their known risks.Source: H Potential Risks · p. 13
• The authors discourage applying the methods to maliciously trained language models.Source: H Potential Risks · p. 13
• Figure 7 presents the instruction given to annotators, while Figure 8 presents the questions they answer.Source: Figure 7 · p. 14, Figure 8 · p. 14