Source-linked AI summary
SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
Liang Wang, Wei Zhao, Zhuoyu Wei, Jingming Liu
TL;DR
Knowledge graph completion needs to infer missing links, but text-based methods have lagged behind embedding-based methods despite their inductive potential. SimKGC improves text-based KGC through efficient contrastive learning with multiple negative types and InfoNCE, substantially outperforming state-of-the-art methods on several benchmarks. The paper identifies interpretability and false negatives from incomplete graphs as directions for future work.
Problem
Text-based knowledge graph completion methods have potential for inductive entity representation learning but still lag behind embedding-based methods.
Method
SimKGC uses in-batch, pre-batch, and self-negatives with a bi-encoder and InfoNCE loss to improve contrastive learning efficiency.
Results
MRR improves from 47.6 to 66.6 on WN18RR, from 29.0 to 35.8 on Wikidata5M transductive, and from 49.3 to 71.4 on Wikidata5M inductive.
Takeaways & Limitations
SimKGC substantially outperforms state-of-the-art methods and provides a strong baseline for further text-based knowledge graph completion research.
Takeaways & Limitations
SimKGC lacks the easy-to-understand interpretations of methods such as RotatE and TransE, and future work should address false negatives caused by incomplete graphs.
Abstract
from arXiv · showhide
Knowledge graph completion (KGC) aims to reason over known facts and infer the missing links. Text-based methods such as KGBERT (Yao et al., 2019) learn entity representations from natural language descriptions, and have the potential for inductive KGC. However, the performance of text-based methods still largely lag behind graph embedding-based methods like TransE (Bordes et al., 2013) and RotatE (Sun et al., 2019b). In this paper, we identify that the key issue is efficient contrastive learning. To improve the learning efficiency, we introduce three types of negatives: in-batch negatives, pre-batch negatives, and self-negatives which act as a simple form of hard negatives. Combined with InfoNCE loss, our proposed model SimKGC can substantially outperform embedding-based methods on several benchmark datasets. In terms of mean reciprocal rank (MRR), we advance the state-of-the-art by +19% on WN18RR, +6.8% on the Wikidata5M transductive setting, and +22% on the Wikidata5M inductive setting. Thorough analyses are conducted to gain insights into each component. Our code is available at https://github.com/intfloat/SimKGC .
1 Introduction
Knowledge graph completion addresses incomplete graphs, while text-based methods offer inductive entity representations but have lagged behind embedding-based approaches. SimKGC targets inefficient contrastive learning with expanded negative sampling and reports large benchmark gains.
- Knowledge graph completion supports the automatic construction and verification of incomplete knowledge graphs.
- Text-based methods’ performance degradation is hypothesized to stem from inefficient contrastive learning compared with embedding-based methods.Embedding-based systems can train with larger negative sample sizes, whereas RoBERTa-based KEPLER trains with a negative sample size of 1.
- SimKGC introduces in-batch, pre-batch, and self-negatives, using a bi-encoder and cached vectors to scale negative sampling to thousands.It also replaces margin-based ranking loss with InfoNCE to emphasize hard negatives.
- Text-based methods can represent unseen entities inductively, unlike embedding-based methods such as TransE that operate under transductive settings.They also use entity descriptions and can leverage pre-trained language models.
- Text-based models can underweight graph topology, so SimKGC proposes boosting scores for the head entity’s k-hop neighbors.The strategy uses the observation that entities connected by short paths are more likely to be related.
- MRR improves from 47.6 to 66.6 on WN18RR, from 29.0 to 35.8 on Wikidata5M transductive, and from 49.3 to 71.4 on Wikidata5M inductive.On FB15k-237, the results are described as competitive.
2 Related Work
Related work covers embedding-based, text-based, inductive, and pre-trained-language-model approaches to knowledge graph completion, alongside broader contrastive-learning research. Text-based methods incorporate entity descriptions but have remained behind methods such as RotatE in benchmark performance.
- Knowledge graph completion models multi-relational data to support automatic construction of large-scale knowledge graphs.
- Embedding-based methods represent entities and relations as vectors, while text-based methods encode entity descriptions with neural or pre-trained language models.Examples include TransE, RotatE, DKRL, KG-BERT, StAR, and BLP.
- Inductive relation prediction methods use subgraph or path information to reason about entities not seen during training.
- Text-based knowledge graph completion methods still underperform methods such as RotatE in benchmark performance.
- Pre-trained language models are first trained on large unlabeled corpora and then fine-tuned for downstream tasks, with research probing or injecting knowledge into them.
- Contrastive learning learns representations by contrasting task-specific positive and negative examples, with applications across vision, multimodal, and retrieval settings.
3 Methodology
SimKGC uses dual text encoders to score relation-aware head entities against tail entities, and improves training with shared, cached, and hard negatives under InfoNCE. It also supports graph-based re-ranking and efficient bi-encoder inference.
- Problem reformulation: SimKGC reformulates each triple with an inverse relation, reducing KGC to tail entity prediction.For (h,r,t), it adds (t,r^-1,h), so the model handles only the tail-prediction case.
- Model architecture: Two independently parameterized encoders initialized from the same pre-trained language model represent relation-aware heads and tails.The head encoder receives the head description and relation; the tail encoder receives only the tail description.
- Model architecture: Relation-aware head embeddings use mean pooling and L2 normalization, while tail embeddings are also L2-normalized.Different relations produce different head inputs and therefore different relation-aware head embeddings.
- Scoring and inference: Tail prediction ranks every entity by cosine similarity between the relation-aware head embedding and each candidate tail embedding.Because both embeddings are L2-normalized, cosine similarity equals their dot product.
- Negative sampling: Training combines in-batch, pre-batch, and self-negatives, with masking to remove known false negatives within the training data.In-batch negatives reuse embeddings, pre-batch negatives cache earlier embeddings, and self-negatives use the head entity as a hard negative for tail prediction.
- Re-ranking: Graph-based re-ranking increases scores for candidate tails within k-hop neighbors of the head entity in the training graph.This strategy incorporates spatial locality, an inductive bias that text-based KGC may not fully capture.
- Training objective: InfoNCE with additive margin raises the positive triple’s score and emphasizes hard negatives through the temperature parameter.Smaller temperature increases emphasis on hard negatives but can overfit label noise; the temperature is re-parameterized as learnable.
- Scoring and inference: SimKGC requires |E| + 2 × |T| BERT forward passes at inference, compared with |E| × 2 × |T| for cross-encoder KG-BERT.Entity embeddings are computed once, after which dot products produce rankings for test triples.
4 Experiments
SimKGC is evaluated on three benchmarks using filtered entity-ranking metrics, with strong results on WN18RR and Wikidata5M but weaker performance on FB15k-237. Analyses examine negative-sampling combinations, dataset characteristics, inductive performance, and inference efficiency.
- Experimental Setup: SimKGC is evaluated on WN18RR, FB15k-237, and Wikidata5M in both transductive and inductive settings.Textual descriptions come from KG-BERT for WN18RR and FB15k-237, while Wikidata5M provides descriptions for entities and relations.
- Experimental Setup: Evaluation ranks candidate entities for head and tail prediction using filtered MRR and Hits@k metrics averaged across both directions.The filtered setting ignores known true triples from the training, validation, and test sets.
- Negative-Sampling Analysis: Combining in-batch, pre-batch, and self-negatives generally gives the best results, although self-negatives improve H@1 while hurting H@10.The authors hypothesize that self-negatives reduce reliance on simple text matching, negatively affecting recall-oriented metrics such as H@10.
- Negative-Sampling Analysis: Self-negatives raise inductive Wikidata5M MRR from 60.3% to 71.3% by helping prevent prediction of the given head entity.Text-based models rely more heavily on text matching in inductive than transductive KGC.
5 Analysis
The analyses show that both abundant negatives and InfoNCE loss drive SimKGC’s performance, with the loss function appearing more influential. Additional analyses examine re-ranking, relation types, human evaluation, and embedding structure.
- Loss and negatives: 64.4 to 48.8 MRR when negatives decrease from 255 to 5, while replacing InfoNCE with margin loss reduces MRR to 39.5.Both the number of negatives and loss function matter, but the loss change has the larger observed impact.
- Loss and negatives: 48.8 to 67.1 MRR as more negatives are added, although larger batches require more GPU memory and may complicate optimization.The experiments use batch size 1024 and do not test larger batches.
- Re-ranking: Re-ranking head-entity k-hop neighbors produces a slight but stable increase across all metrics on Wikidata5M-Trans.The strategy incorporates topological information but does not apply to inductive KGC.
- Relation types: Predicting the “n” side of relation cardinality is generally harder because many plausible answers can confuse the model and the graph is incomplete.Human evaluation indicates some apparently incorrect predictions may nevertheless be correct.
- Human evaluation: 49% of predictions labeled wrong by H@1 were judged correct by human annotators, suggesting automatic metrics can underestimate performance.The evaluation randomly sampled 100 H@1-wrong predictions on Wikidata5M-Trans.
- Embedding visualization: Entity embeddings from eight categories are well separated in t-SNE visualization, with some overlap between Community and Village.The overlap is described as reasonable because the concepts are not mutually exclusive.
6 Conclusion
The conclusion presents SimKGC as a simple text-based KGC method built around efficient contrastive learning. Its experiments show substantial gains over state-of-the-art methods, while interpretability and false negatives remain future-work challenges.
- Method: SimKGC uses a bi-encoder architecture with in-batch, pre-batch, and self-negatives to improve contrastive learning efficiency.The method combines these negatives with text-based entity representation learning.
- Limitations and future work: SimKGC lacks the readily understandable relational interpretations available in methods such as RotatE and TransE.Improving interpretability is identified as a future-work direction.
- Limitations and future work: Future work should address false negatives arising from incomplete knowledge graphs.The conclusion also identifies interpretability improvement as a separate direction.
7 Broader Impacts
The paper positions SimKGC as a baseline for further progress in text-based KGC and identifies applications and research directions enabled by improved completion systems.
- Future directions: SimKGC can serve as a baseline for improving text-based knowledge graph completion systems.The paper highlights incorporating global graph structure and exploring alternatives to InfoNCE as future directions.
- Potential applications: Improved knowledge graph completion systems could support exploration in KBQA, information retrieval, and knowledge-grounded response generation.These applications are presented as opportunities for future exploration.
A Details on Hyperparameters
SimKGC uses dataset-specific learning rates and neighbor-based re-ranking, while augmenting relation and entity text to improve sparse descriptions and prevent label leakage.
- Table 10 lists shared SimKGC hyperparameters across datasets.The cited passages identify the table but do not provide all shared settings.
- Learning rates are 5 × 10−5 for WN18RR, 10−5 for FB15k-237, and 3 × 10−5 for Wikidata5M.
- Re-ranking uses 5-hop neighbors for WN18RR and 2-hop neighbors for the other datasets.
- Inverse relations receive the prefix word “inverse” in their relation descriptions.
- Entities with very short descriptions are augmented with names of training-set neighbors.The correct entity is dynamically excluded from input text during training to avoid label leakage.
B More Analysis Results
Additional analyses examine batch size, additive-margin InfoNCE, and prediction examples to clarify SimKGC’s behavior and training choices.
- Tables 11 and 12 evaluate how batch size affects SimKGCIB performance on Wikidata5M-Trans and FB15k-237.
- The additive-margin variant of InfoNCE performs consistently better than standard InfoNCE, but the improvement is marginal.The comparison is reported for the variant used in Equation 5 and illustrated in Table 13.
- Table 14 provides additional SimKGC prediction examples from the Wikidata5M-Trans test set.Full model predictions on test datasets are available in the public code repository.
- The implementation is based on the open-source transformers project.