Source-linked AI summary
KG-BERT: BERT for Knowledge Graph Completion
Liang Yao, Chengsheng Mao, Yuan Luo
TL;DR
Knowledge graphs are useful for AI but incomplete, creating a need to assess missing triples. KG-BERT converts entities, relations, and triples into textual sequences and fine-tunes BERT for completion tasks, achieving state-of-the-art results across multiple benchmarks while incurring expensive link-prediction evaluation.
Problem
Knowledge graphs support important AI tasks but are incomplete, motivating methods for assessing the plausibility of missing triples.
Method
KG-BERT represents entities, relations, and triples as textual sequences and fine-tunes BERT through sequence classification for triple or relation prediction.
Results
KG-BERT achieves state-of-the-art results in triple classification, relation prediction, and link prediction across multiple benchmark knowledge graphs.
Takeaways & Limitations
KG-BERT uses rich language information and highlights important words connected to a triple for knowledge graph completion.
Takeaways & Limitations
BERT makes link prediction evaluation very time consuming because nearly all corrupted triple sequences are processed by a 12-layer Transformer.
Abstract
from arXiv · showhide
Knowledge graphs are important resources for many artificial intelligence tasks but often suffer from incompleteness. In this work, we propose to use pre-trained language models for knowledge graph completion. We treat triples in knowledge graphs as textual sequences and propose a novel framework named Knowledge Graph Bidirectional Encoder Representations from Transformer (KG-BERT) to model these triples. Our method takes entity and relation descriptions of a triple as input and computes scoring function of the triple with the KG-BERT language model. Experimental results on multiple benchmark knowledge graphs show that our method can achieve state-of-the-art performance in triple classification, link prediction and relation prediction tasks.
Introduction
Knowledge graphs support important AI tasks but remain incomplete, motivating knowledge graph completion. KG-BERT addresses this problem by treating entities, relations, and triples as text and fine-tuning BERT for plausibility and relation prediction, achieving state-of-the-art results across several completion tasks.
- Knowledge graphs represent entities and relations as triples and support tasks including semantic search, recommendation, and question answering.
- Knowledge graph completion assesses the plausibility of triples that are absent from an incomplete graph.
- Existing embedding methods often rely on sparse observed graph structure, while text-enhanced models can ignore contextual information across triples.
- BERT learns contextualized word representations from large-scale text using a bidirectional Transformer encoder.
- KG-BERT treats entities, relations, and triples as textual sequences and fine-tunes BERT as a sequence classifier for triple plausibility or relation prediction.
- KG-BERT is presented as the first study to model triple plausibility with a pre-trained contextual language model.
- KG-BERT achieves state-of-the-art results in triple classification, relation prediction, and link prediction on several benchmark datasets.
Related Work
Prior knowledge graph completion methods primarily use structural information or incorporate text through specialized embeddings and attention mechanisms. KG-BERT differs by using pre-trained language models to learn context-aware text embeddings with richer language information.
- Knowledge graph embedding methods include translational distance and semantic matching models that score triples using vector-based functions.
- Structural embedding methods use observed triple facts, while external information such as entity types, logical rules, and textual descriptions can improve performance.
- TEKE, LSTM-attention models, and mutual-attention methods incorporate contextual or triple-specific textual information into knowledge graph embeddings.
- Compared with these methods, KG-BERT learns context-aware text embeddings with rich language information through pre-trained language models.
- Pre-trained language representation methods include feature-based and fine-tuning approaches, with BERT representing the latter category.
Method
KG-BERT fine-tunes a pre-trained bidirectional language model on textual representations of entities, relations, and triples for knowledge-graph completion. It supports triple plausibility prediction and direct relation prediction using sequence representations and task-specific classification objectives.
- Method: BERT uses a multi-layer bidirectional Transformer encoder initialized with pre-trained weights and fine-tuned using labeled downstream-task data.Its pre-training uses masked language modeling and next sentence prediction.
- Method: KG-BERT treats entity names or descriptions, relations, and triples as textual sequences and fine-tunes BERT for knowledge-graph completion.Triples are packed as a single BERT input sequence, while two-entity sequences support relation prediction.
- Triple plausibility prediction: KG-BERT(a) packs head, relation, and tail sentences with [CLS] and [SEP] tokens, segment embeddings, and position embeddings.The final [CLS] hidden state serves as the aggregate sequence representation for computing triple plausibility.
- Triple plausibility prediction: Triple plausibility is learned with cross-entropy over positive and negative triple labels, where negatives replace a positive triple’s head or tail with a random entity.Negatives already present in the positive set are excluded, and the pre-trained and new parameters are updated by gradient descent.
- Relation prediction: KG-BERT(b) predicts the relation between two entities from their sentences, using the [CLS] representation and a cross-entropy loss over relation labels.The method reports that direct two-entity relation prediction performed better in a preliminary experiment than relation corruption with KG-BERT(a).
Experiments
KG-BERT is evaluated on triple classification, link prediction, and relation prediction across benchmark knowledge graphs. It shows strong results, including state-of-the-art performance claims, while trading ranking quality against computational cost in link prediction.
- Experimental setup: Experiments cover triple classification, link prediction, and relation prediction on six benchmark knowledge graphs.The datasets include WN11, FB13, FB15K, WN18RR, FB15k-237, and UMLS.
- Experimental setup: KG-BERT uses BERT-Base with 12 layers, 12 self-attention heads, and hidden size H = 768, fine-tuned with Adam.Training uses task-specific epoch counts and standard BERT hyperparameters including batch size 32 and learning rate 5e-5.
- Triple classification: KG-BERT(a) clearly outperforms all baselines in triple classification, with improvements reported as significant at p < 0.01.The models were run 10 times, with standard deviations below 0.2; the paper reports the best results so far for this task.
- Triple classification: With only 5% of FB13 training triples, KG-BERT(a) reaches 88.1% test accuracy; with 10% of WN11 triples, it reaches 87.0%.These results exceed some baseline models trained on the full datasets and are attributed to using linguistic patterns from external text.
- Link prediction: KG-BERT(a) achieves the lowest mean ranks on WN18RR and FB15k-237, but its Hits@10 is lower than some state-of-the-art methods.The paper links this trade-off to semantic relatedness from text without explicitly modeling KG structure.
- Relation prediction: KG-BERT(b) achieves the highest Hits@1 reported for relation prediction on FB15K.This variant predicts the relation from the two entity sentences and is compared with both structural and text-enhanced baselines.
- Discussion: Link prediction evaluation is very time consuming because nearly all entities replace the head or tail and each corrupted sequence passes through a 12-layer Transformer.The paper suggests 1-N scoring models or lightweight language models as possible solutions.
Conclusion and Future Work
KG-BERT represents entities and relations as textual sequences for knowledge graph completion and reports strong results on multiple benchmark datasets. Future work proposes combining textual information with graph structure and using larger-text pre-trained models.
- KG-BERT represents entity and relation names or descriptions as textual sequences and formulates knowledge graph completion as sequence classification.
- KG-BERT uses free-text language information and highlights important words connected to a triple.
- KG-BERT outperforms state-of-the-art results on multiple benchmark knowledge graph datasets.
- Future directions include jointly modeling textual information with knowledge graph structures and using pre-trained models trained on more text data, such as XLNet.