Source-linked AI summary
Knowledge Base Completion: Baselines Strike Back
Rudolf Kadlec, Ondrej Bajgar, Jan Kleindienst
TL;DR
Knowledge base completion research often attributes benchmark gains to new relation-learning architectures, but the evidence may also reflect tuning and training objectives. This paper reimplements and tunes DistMult, finding that it outperforms nearly all compared models and motivating more careful evaluation and reporting. The authors also identify filtered evaluation as unrealistic because knowledge bases can contain multiple correct answers.
Problem
It is unclear whether reported KBC improvements arise from architectural changes or from hyper-parameter tuning and different training objectives.
Method
The paper evaluates a reimplemented DistMult baseline with tuned hyperparameters, softmax normalization, and standard filtered evaluation on WN18 and FB15k.
Results
The tuned DistMult baseline outperforms 27 of 29 models evaluated on WN18 and FB15k using Hits@10.
Takeaways & Limitations
The findings support reconsidering how KBC performance is evaluated and reported and encourage large-scale empirical comparisons of algorithms.
Takeaways & Limitations
Filtered evaluation assumes a single correct candidate answer, which is unrealistic for knowledge bases; raw evaluation may therefore be worth emphasizing in future work.
Abstract
from arXiv · showhide
Many papers have been published on the knowledge base completion task in the past few years. Most of these introduce novel architectures for relation learning that are evaluated on standard datasets such as FB15k and WN18. This paper shows that the accuracy of almost all models published on the FB15k can be outperformed by an appropriately tuned baseline - our reimplementation of the DistMult model. Our findings cast doubt on the claim that the performance improvements of recent models are due to architectural changes as opposed to hyper-parameter tuning or different training objectives. This should prompt future research to re-consider how the performance of models is evaluated and reported.
1 Introduction
Knowledge base completion tests whether machines can reason over entity–relation–entity triplets by predicting missing information. The paper argues that a properly tuned DistMult baseline can outperform most published models on standard KBC benchmarks.
- Knowledge bases such as Wikidata and Freebase accumulate knowledge as entity–relation–entity triplets, motivating machine reasoning over them.
- Knowledge base completion predicts a missing head or tail entity in an incomplete relational triple.For example, a system may predict USA for ⟨Donald Trump, president of, ?⟩.
- KBC can both evaluate generic reasoning over knowledge bases and expand incomplete knowledge bases with deduced entries.
- The paper reports that a properly tuned DistMult model outperforms 27 of 29 models evaluated on WN18 and FB15k using Hits@10.
- The results suggest that hyper-parameter tuning may offer substantial improvement even for more complex relational-learning models.
2 The Model
The paper uses distributional representations and the DistMult scoring model, whose diagonal relation matrices impose a symmetry assumption. Its implementation changes training and normalization choices that earlier work found important.
- Distributional KBC models represent entities and sometimes relations as N-dimensional real vectors.
- DistMult assumes all relations are symmetric, an unrealistic restriction for directed relational data.
- Earlier studies improved DistMult empirically by changing training hyperparameters, replacing the L1-based max-margin loss with softmax negative log-likelihood, or tuning hyperparameters alone.
- DistMult represents each relation with a diagonal matrix whose diagonal contains the relation-vector elements.
- The implementation normalizes scores with a softmax over candidate answer entities for each query.
3 Experiments
Experiments evaluate DistMult on WN18 and FB15k using filtered ranking, negative-sampling training, and extensive hyper-parameter tuning. The tuned implementation performs strongly across reported benchmarks, while results vary by metric and dataset.
- Experimental setup: Experiments use WN18 and FB15k with the filtered evaluation protocol for ranking candidate triplets.Training converts each triplet into head and tail queries and ranks test candidates after excluding known true triplets.
- Experimental setup: The final search varies embedding dimension, batch size, and negative-sample count, with Adam optimization and early stopping on validation Hits@10.The best validation settings are N = 512, b = 2048, M = 2000 for FB15k and N = 256, b = 1024, M = 1000 for WN18.
- Results: The implementation achieves the best reported MRR on FB15k, although MRR is reported less often than other metrics.MRR is described as less sensitive to outliers than MR.
- Results: Performance depends on dataset and metric: DistMult is competitive in WN18 Hits@10 but performs poorly in WN18 MR and MRR, while it outperforms other models on FB15k Hits@1.On WN18, IRN and R-GCN+ achieve better Hits@10, whereas HolE and ComplEx outperform the implementation on Hits@1.
4 Conclusion
The paper argues that tuning and training objectives can substantially affect KBC results, while filtered evaluation and Hits@10 can obscure important limitations. It calls for broader empirical comparisons and attention to alternative metrics and evaluation scenarios.
- Hits@1 and MRR may be more informative than Hits@10 on WN18, where many models achieve similarly high Hits@10 scores.The authors note that their DistMult implementation is competitive in Hits@10 but underperforms in Hits@1.
- The filtered scenario is easy to evaluate but assumes exactly one correct answer among candidates, which is unrealistic for knowledge bases.The paper therefore suggests greater focus on the raw scenario, despite its requirement for metrics such as MAP.
- Table 2 separates knowledge-base-only models from systems using path information or additional textual data, with DistMult implementations listed separately for comparison with the first group.MR, Hits@10, and MRR are the reported entity-prediction metrics.
- The study contributes to debate over whether state-of-the-art gains reflect better algorithms or more extensive hyper-parameter search and advocates a large-scale comparative KBC study.The proposed comparison is modeled on a prior comparative study of word-embedding models.