Source-linked AI summary
A Re-evaluation of Knowledge Graph Completion Methods
Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha Talukdar, Yiming Yang
TL;DR
Recent neural KGC methods report unusually high and inconsistent benchmark performance, raising concerns about evaluation validity. The paper reexamines their score functions and protocols, proposes RANDOM evaluation, and finds that it fairly distinguishes affected methods while remaining robust across seeds. It recommends RANDOM for KGC comparisons.
Problem
Recent neural KGC methods report high but inconsistent benchmark gains, while their complex score functions and evaluation procedures require reexamination.
Method
The paper analyzes score-function behavior, compares evaluation tie-breaking protocols, and proposes RANDOM evaluation for fair model comparisons.
Results
The study finds that inappropriate evaluation inflates some methods’ performance, while RANDOM distinguishes affected methods and is robust to different random seeds.
Takeaways & Limitations
The authors strongly encourage using RANDOM evaluation for all KGC performance comparisons.
Takeaways & Limitations
The paper leaves TransGate’s re-evaluation for future work because no open-source implementation was found.
Abstract
from arXiv · showhide
Knowledge Graph Completion (KGC) aims at automatically predicting missing links for large-scale knowledge graphs. A vast number of state-of-the-art KGC techniques have got published at top conferences in several research fields, including data mining, machine learning, and natural language processing. However, we notice that several recent papers report very high performance, which largely outperforms previous state-of-the-art methods. In this paper, we find that this can be attributed to the inappropriate evaluation protocol used by them and propose a simple evaluation protocol to address this problem. The proposed protocol is robust to handle bias in the model, which can substantially affect the final results. We conduct extensive experiments and report the performance of several existing methods using our protocol. The reproducible code has been made publicly available
1 Carnegie Mellon University, 2 Indian Institute of Science
The listed authors are affiliated with Carnegie Mellon University and the Indian Institute of Science.
- Authors are affiliated with Carnegie Mellon University and the Indian Institute of Science.
1 Introduction
KGC predicts missing facts in incomplete knowledge bases, but recent neural methods report unusually high and inconsistent gains. The paper attributes these results to inappropriate evaluation and proposes a fairer protocol.
- Knowledge Graph Completion predicts missing links in incomplete multi-relational knowledge bases.
- Recent neural KGC methods use complex CNN, RNN, GNN, and capsule-network score functions.
- Several neural methods report high gains on some benchmark datasets but inconsistent improvements across datasets.
- The paper attributes these results to inappropriate evaluation, which can give a constant-output model a perfect score.
- The authors propose a simple protocol for fair comparison and publicly release the source code.
2 Background
KGC evaluates candidate facts by ranking model scores, while prior work and the paper’s examples motivate scrutiny of evaluation and reported improvements.
- KGC infers missing facts from known facts using entity and relation embeddings with a score function.
- During evaluation, models rank all candidate tails and measure the valid triplet’s rank; filtered evaluation removes other known correct triplets.
- Prior studies questioned whether reported gains reflected architectural changes rather than tuning or different training objectives.
- Figure 1 illustrates a ConvKB score distribution in which negative samples may receive the same score as a valid triplet.
3 Observations
The observations identify inconsistent benchmark gains and repeated tied scores in some neural KGC methods. High proportions of zero ReLU activations provide a proposed explanation for these ties.
- 3.1 Inconsistent Improvements over Benchmark Datasets: ConvKB improves MRR by 21.8% over ConvE on FB15k-237 but degrades by 42.3% on WN18RR.
- 3.2 Observations on Score Functions: Evaluation ranks candidate tails by scoring all triplets sharing the query head and relation.
- 3.2 Observations on Score Functions: On FB15k-237, ConvKB and CapsE show many negative triplets tied with valid-triplet scores, whereas ConvE rarely does.
- 3.2 Observations on Score Functions: ConvKB averages 125 tied entities and CapsE 197 across FB15k-237, compared with approximately 0.002 for ConvE.
- 3.2 Observations on Score Functions: ConvKB and CapsE have unusually high fractions of zeroed ReLU neurons—87.3% and 92.2%—versus 41.1% for ConvE.
- 3.2 Observations on Score Functions: The authors link extensive neuron zeroing to similar triplet representations and identical scores during the forward pass.
4 Evaluation Protocols for KGC
The paper compares TOP, BOTTOM, and RANDOM tie-breaking protocols for KGC evaluation. RANDOM is proposed as the fairest protocol because TOP rewards tied scores while BOTTOM penalizes them.
- Tie-breaking protocols: KGC evaluation must decide how to rank multiple triplets receiving the same score.The protocols differ only in where the correct triplet is placed among tied candidates.
- Tie-breaking protocols: TOP places the correct triplet first, BOTTOM places it last, and RANDOM places it randomly in the candidate list.
- Experimental comparison: Table 2 reports TOP and BOTTOM performance changes relative to RANDOM on FB15k-237.
- Protocol properties: TOP gives an inappropriate advantage to models biased toward equal scores, whereas BOTTOM can unfairly assign the lowest possible rank.
- Protocol properties: RANDOM is proposed for model comparisons because it is both rigorous and fair when several candidates receive the same score.
5 Experiments
The experiments re-evaluate existing KGC methods across evaluation protocols and datasets using author-provided implementations and standard filtered metrics. Non-Affected methods remain stable, while Affected methods vary substantially and perform best with TOP and worst with BOTTOM.
- Experimental setup: The study conducts extensive experiments to compare existing KGC methods using the proposed evaluation protocols.
- Method categories: Methods are categorized as Non-Affected when performance is consistent across protocols and Affected when protocol choice changes performance.
- Experimental setup: The evaluation uses author-provided code and hyperparameters, with MRR, MR, and H@10 measured under the filtered setting.
- Experimental setup: The experiments compare TOP and BOTTOM against RANDOM and estimate RANDOM variability across five runs with different random seeds.
- Scope: The re-evaluation of TransGate is left for future work because no open-source implementation was found.
- Evaluation results: Non-Affected methods remain consistent across protocols, whereas Affected methods perform best with TOP and worst with BOTTOM.
- Evaluation results: The RANDOM protocol is robust to random seeds, with randomness having little effect on evaluation results for real-world large-scale knowledge graphs.
6 Conclusion
The paper concludes that inappropriate evaluation protocols and problematic score functions can inflate reported KGC performance. It proposes RANDOM evaluation to distinguish affected methods and recommends it for KGC evaluation.
- Conclusion: Many neural-network-based KGC models have score-function issues that, combined with inappropriate evaluation, inflate reported performance.
- Conclusion: The proposed RANDOM protocol distinguishes affected methods from other methods in evaluation.
- Conclusion: The authors strongly encourage using RANDOM evaluation for all KGC evaluation purposes.
A Results on WN18RR dataset
The paper evaluates its protocols on WN18RR, a lexical-relations subset of WN18 with inverse relations removed. The results support conclusions similar to those obtained on FB15k-237, while Figure 4 identifies a strong equal-score pattern for ConvKB.
- Dataset and results: WN18RR is a WN18 subset containing lexical relations between words, with inverse relations removed.
- Dataset and results: The WN18RR results support conclusions similar to those drawn from the FB15k-237 experiments.
- Experimental comparison: Table 3 compares performance under TOP, BOTTOM, and RANDOM protocols on WN18RR.
- Score ties: Figure 4 shows the frequency of negative triplets receiving the same score as the valid triplet during WN18RR evaluation.
- Score ties: Unlike FB15k-237, only ConvKB has a large number of negative triplets tied with valid triplets on WN18RR.