Source-linked AI summary
Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework
Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Mikhail Galkin, Sahand Sharifzadeh, Asja Fischer, Volker Tresp, Jens Lehmann
TL;DR
Heterogeneous implementations and evaluation practices make knowledge graph embedding results difficult to reproduce and compare fairly. The paper re-implements 21 interaction models and benchmarks configurations across four datasets, finding that performance depends strongly on training choices and inverse-relation modeling, not architecture alone.
Problem
Heterogeneous implementations, hyper-parameter specifications, preprocessing, and evaluation metrics make prior knowledge graph embedding results difficult to reproduce and compare fairly.
Method
The authors re-implemented 21 interaction models in a unified framework and evaluated combinations of models, training approaches, loss functions, and explicit inverse relations across four datasets.
Results
Performance depends strongly on the combination of interaction model, training approach, loss function, and explicit inverse relations, with several configurations achieving competitive results when carefully configured.
Takeaways & Limitations
Interaction models should be assessed as parts of complete configurations, since even baseline architectures can outperform more sophisticated models under appropriate training choices.
Takeaways & Limitations
Essential implementation details, including inverse-triple preprocessing and score clamping, were sometimes absent from publications but affected reproducibility and evaluation.
Abstract
from arXiv · showhide
The heterogeneity in recently published knowledge graph embedding models' implementations, training, and evaluation has made fair and thorough comparisons difficult. In order to assess the reproducibility of previously published results, we re-implemented and evaluated 21 interaction models in the PyKEEN software package. Here, we outline which results could be reproduced with their reported hyper-parameters, which could only be reproduced with alternate hyper-parameters, and which could not be reproduced at all as well as provide insight as to why this might be the case. We then performed a large-scale benchmarking on four datasets with several thousands of experiments and 24,804 GPU hours of computation time. We present insights gained as to best practices, best configurations for each model, and where improvements could be made over previously published best configurations. Our results highlight that the combination of model architecture, training approach, loss function, and the explicit modeling of inverse relations is crucial for a model's performances, and not only determined by the model architecture. We provide evidence that several architectures can obtain results competitive to the state-of-the-art when configured carefully. We have made all code, experimental configurations, results, and analyses that lead to our interpretations available at https://github.com/pykeen/pykeen and https://github.com/pykeen/benchmarking
1 INTRODUCTION
The paper addresses two obstacles to fair knowledge graph embedding comparisons: difficulty reproducing published results and difficulty determining whether improvements arise from architectures or experimental choices.
- Reproducibility: Reproducing published knowledge graph embedding results is difficult because implementations, frameworks, source-code availability, and hyper-parameter specifications vary.Different results have even been reported for identical model–dataset combinations.
- Comparability: Fairly verifying novelty is difficult because performance gains may reflect training approaches, hyper-parameters, preprocessing, inverse relations, or evaluation-metric implementations rather than architecture.Fine-tuned baseline models can achieve competitive performance to more sophisticated models.
- Contributions: It also benchmarks 21 knowledge graph embedding models across four datasets while varying training approaches, loss functions, optimizers, hyper-parameters, and explicit inverse-relation modeling.The benchmark evaluates combinations of these choices rather than model architectures alone.
2 KNOWLEDGE GRAPHS
Knowledge graphs are directed, multi-relational collections of entity–relation–entity triples, but their incompleteness makes predicting missing links necessary for downstream use.
- Knowledge-graph structure: A knowledge graph represents triples (h, r, t), with entities as nodes and directed, typed relations as edges.Relation direction distinguishes the head and tail roles of entities.
- Knowledge-graph assumptions: Under the closed world assumption, absent triples are false, whereas under the open world assumption, their truth is unknown.Knowledge-graph construction and most relevant work rely on the open world assumption.
- Link prediction: Link prediction infers missing head or tail entities for queries such as (Sarah, studied at, ?) or (?, CEO of, Deutsche Bank).The task addresses incomplete and noisy knowledge graphs.
3 KNOWLEDGE GRAPH EMBEDDING MODELS
Knowledge graph embedding models learn latent entity and relation representations for scoring triples, while differing in interaction mechanisms and modeling assumptions. The paper evaluates shallow models and emphasizes that expressiveness, computational cost, and relation structure vary across architectures.
- Model framework: A knowledge graph embedding model comprises an interaction model, training approach, loss function, and use of explicit inverse relations.This abstraction enables studying each component separately and in combination.
- Scope: The paper excludes temporal knowledge graph embedding models and focuses on shallow embedding approaches using matrix lookups for entity and relation encoders.Temporal knowledge graphs are treated as a distinct research line with separate benchmarking datasets.
- Interaction models: Interaction models assign real-valued plausibility scores to triples, with larger scores generally indicating more plausible triples.The paper categorizes them as translational distance-based or semantic matching-based models.
- Translational distance models: TransE models relations as translations, making it computationally efficient but unable to inherently represent 1-N, N-1, and N-M relations.Its constraint h + r ≈ t can force multiple related entities toward similar embeddings.
- Translational distance models: TransH, TransR, and TransD extend translational models by projecting entities into relation-specific spaces or using entity-relation-specific projections.TransR uses relation-specific projection matrices, while TransD constructs projections from entity and relation vectors.
- Semantic matching models: RotatE represents relations as unit-modulus complex rotations and can model symmetry, antisymmetry, inversion, and composition.Entities and relations are embedded in complex space, with each relation element representing a rotation angle.
- Semantic matching models: ComplEx can model anti-symmetric relations because complex-valued Hadamard multiplication is non-commutative, unlike DistMult.DistMult is more computationally efficient because it restricts its matrices to diagonal form, but this reduces expressiveness.
sLCWA
The sLCWA samples negative triples rather than treating every unobserved triple as false. The section also distinguishes negative-sampling strategies and loss functions used to train KGEMs.
- Stochastic local closed world assumption: sLCWA randomly samples candidate triples from the sets of unobserved head, tail, or relation corruptions.Unlike exhaustive negative construction, it samples from these candidate sets.
- Negative sampling: Filtering all known positive triples from negative candidates is theoretically appropriate but often omitted because the candidate set is much larger.An unobserved corrupted triple can nevertheless represent a true fact.
- Negative sampling: Uniform negative sampling randomly chooses whether to corrupt the head or tail, then uniformly selects a replacement entity.
- Negative sampling: Bernoulli negative sampling sets corruption probabilities from relation-specific tph and hpt statistics, reducing the chance of generating true facts.One-to-many relations receive higher head-corruption probability, whereas many-to-one relations receive higher tail-corruption probability.
- Loss functions: Pointwise losses score each triple-label pair independently, while pairwise losses compare positive and corrupted triples through their score difference.Pairwise losses align with the open-world assumption by treating negatives as less positive rather than necessarily false.
- Loss functions: Setwise losses operate on more than two scores, including self-adversarial negative sampling, which weights higher-scoring negatives more strongly.This targets the limitation that many generated negatives are trivial.
4 EVALUATION METRICS FOR KGEMS
KGEM evaluation ranks the correct entity against corrupted alternatives for link-prediction queries. Metrics summarize these ranks, but incomplete knowledge graphs can leave unknown true triples among the negatives and distort results.
- Ranking evaluation: Link prediction evaluates whether a model ranks the correct head or tail entity highly for a query.For each test triple, corrupted-head and corrupted-tail candidates are scored and sorted.
- Ranking evaluation: Filtered evaluation removes known true triples from corrupted candidates so another known true entity does not lower the test triple’s rank.
- Evaluation caveats: Unknown false negatives can still distort metrics because true triples absent from the KG cannot be filtered out.The authors therefore recommend inspecting predicted scores rather than relying only on computed metrics.
- Metrics: AMR adjusts mean rank for the number of candidate triples and chance, with values from 0 to 1 where smaller values indicate better performance.
- Metrics: MRR averages reciprocal ranks and ranges from 0 to 1, with larger values indicating better performance.The paper notes that MRR remains useful for early stopping despite concerns about averaging ordinal reciprocal ranks.
- Metrics: AUC-ROC and AUC-PR are often unavailable because incomplete KGs usually do not provide the true-negative counts they require.
5 EXISTING BENCHMARK DATASETS
The benchmark suite spans heterogeneous knowledge graphs, including Freebase, WordNet, kinship, country relations, biomedical concepts, and YAGO-derived facts. FB15K and WN18 have inverse-relation leakage, motivating FB15K-237 and WN18RR.
- Datasets: FB15K is a Freebase subset with 14,951 entities, 1,345 relations, and more than half a million triples.Its facts cover domains including movies, actors, awards, sports, and sports teams.
- Leakage-controlled datasets: FB15K-237 removes inverse relations from FB15K and contains 14,541 entities and 237 relations.FB15K had approximately 81% test leakage through inverse or related training triples.
- Datasets: WN18 contains 40,943 synsets and 18 relations from a lexical knowledge base representing conceptual-semantic and lexical relationships.
- Leakage-controlled datasets: WN18RR removes inverse relations from WN18 because approximately 94% of its test set had leakage through inverse patterns.A simple rule-based system had achieved results competitive with state-of-the-art results on WN18.
- Datasets: Kinships contains 10,686 triples about relationships among members of the Alyawarra tribe, with 104 entities and 26 relationship types.
- Datasets: Nations, UMLS, and YAGO3-10 represent country relations, biomedical concepts, and general facts aligned with WordNet, respectively.
6 REPRODUCIBILITY STUDIES
The reproducibility study tested whether published KGEM results could be replicated using reported or recoverable experimental information. Reproduction often depended on unreported hyper-parameters, implementation details, and ranking conventions.
- Study design: Each experiment used the original training and evaluation settings, ran four random seeds, and reported MR, AMR, MRR, and Hits@K.The study accounted for differences among ranking-metric implementations.
- Reproduction outcomes: TransE on WN18 required 4000 training epochs instead of the reported 1000, while RotatE improved with an adapted learning rate.These discrepancies may reflect implementation details of the underlying training frameworks.
- Unreported hyper-parameters: Some ConvKB and HolE experiments could not be reproduced because essential setup information, including a TransE batch size and FB15K hyper-parameters, was unavailable.
- Publication versus implementation: ConvE added inverse triples and SimplE clamped predicted scores, although these implementation details were not described in the corresponding publications.Both details could materially affect reproduction or evaluation.
- Official implementations: DistMult and TransD reproduced on WN18 but not FB15K, while missing official implementations prevented checking framework-level differences; TransH also could not be reproduced.
- Ranking conventions: Different realistic, optimistic, and pessimistic ranking implementations can diverge substantially when many triples receive identical scores, as observed for SimplE.
7 BENCHMARKING
The benchmark evaluates 21 interaction models across datasets and configurations, showing that training approach, loss function, inverse-relation modeling, and optimizer choices substantially affect performance. Results also show that carefully configured simpler or smaller models can be competitive, while model performance varies by dataset.
- 21 interaction models were evaluated across four datasets, two training approaches, five loss functions, and configurations with or without explicit inverse relations.Random search optimized remaining hyper-parameters with early stopping for each configuration.
- 7.1 Results on the Kinships Dataset: Adam generally outperformed Adadelta on Kinships, so subsequent experiments used Adam to reduce computational costs.
- 7.1 Results on the Kinships Dataset: Training approach, loss function, and inverse-relation modeling materially changed performance and robustness across datasets.On Kinships, loss functions showed different variance, inverse relations reduced variance, and training approaches benefited models differently.
- 7.1 Results on the Kinships Dataset: Model size showed no strong correlation with performance on Kinships, with small models appearing alongside larger models on the performance skyline.
- 7.2 Results on the WN18RR Dataset: 60.09% hits@10 was achieved by RotatE on WN18RR, while several other models also obtained competitive results with suitable configurations.Examples include TransE at 56.98%, ConvE at 56.33%, and MuRE at 57.90%.
- 7.2 Results on the WN18RR Dataset: On WN18RR, LCWA produced higher maximum performance but greater variance than sLCWA for BCEL and SPL.SPL had lower overall variance because it is numerically more stable than BCEL.
8 RELATIONAL PATTERN ANALYSIS
The study analyzes how knowledge-graph relational patterns are represented and how model configurations affect performance across benchmark datasets. Symmetric relations are generally easier than anti-symmetric and composite relations, while RotatE and TransE perform best on the harder patterns.
- The analysis examines symmetry, anti-symmetry, inversion, and composition in FB15K-237, WN18RR, and YAGO3-10.
- No inverse relations were detected in FB15K-237 or WN18RR because preprocessing explicitly removed them.
- The evaluation computes hits@10 on test-triple subsets associated with each relational pattern.
- Symmetric relations are generally easier to model than anti-symmetric and composite relations, which are similarly challenging.
- RotatE and TransE obtain the best overall results on anti-symmetry and composition, whereas UM and NTN perform poorly.
9 DISCUSSION & FUTURE WORK
The discussion concludes that KGEM performance depends on the full configuration rather than the interaction model alone, while reproducibility remains constrained by incomplete reporting and ranking choices. The authors identify practical configuration trade-offs and several directions for further study.
- Reproduction is often difficult because publications may omit experimental details, lack official implementations, or depend on ranking choices.
- The interaction model, training approach, loss function, and explicit inverse relations jointly influence performance, so KGEM should be distinguished from its interaction model.
- No configuration performs best across all datasets, although several configurations can achieve comparable results depending on the dataset.
- Appropriate configurations can compress model size while balancing memory requirements and performance.
- TransE can outperform state-of-the-art interaction models with suitable training and loss choices, while RotatE, MuRE, TuckER, and ComplEx show strong or robust results in supported settings.
- The authors call for further analysis of inverse-relation representations and of whether prior findings can be attributed to interaction models alone.
ADDITIONAL RESULTS FROM REPRODUCIBILITY STUDY
The reproducibility study compares published and reproduced results under multiple ranking conventions and reports model-size information for selected configurations. It also documents cases where reproduction required altered settings or encountered numerical problems.
- The reproduction tables distinguish published, realistic, optimistic, and pessimistic ranking results, with additional categories for undefined or non-deterministic published results.
- MuRE produced non-finite loss values on WN18RR under the published setting, while a different setting outperformed the published results.
- The appendix reports model sizes in bytes for the best reported configurations studied in the reproducibility analysis.
ADDITIONAL RESULTS FROM BENCHMARKING STUDY
The benchmarking appendices provide Pareto-optimal models, best configurations by dataset, and configuration-level results across four datasets. The figures organize performance by model, loss, training approach, and inverse-relation usage, while model size and performance are generally weakly correlated.
- Pareto-optimal models are reported for FB15K-237, Kinships, WN18RR, and YAGO3-10 using model bytes and Hits@10.
- Best configurations are tabulated separately for each investigated model and dataset.
- Model size and Hits@10 show low correlation across trained models, with Pareto-optimal models highlighted.
- The configuration figures report results across interaction models, losses, training approaches, and explicit inverse-relation usage.
- Training-approach comparisons are shown for Kinships, WN18RR, and FB15K-237 using the best-performing configuration for each setting.