PaperlayerView exact revision on arXiv

Source-linked AI summary

Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study

Farahnaz Akrami, Mohammed Samiul Saeef, Qingheng Zhang, Wei Hu, Chengkai Li

arXiv:2003.08001v1cs.AIcs.CL

TL;DR

The paper asks whether link-prediction methods remain effective in realistic settings, given the reverse triples in widely used benchmarks. It systematically examines redundancy and related dataset defects, finding that model performance is much weaker on realistic evaluations and that absolute accuracy is poor.

  • Problem

    The paper asks whether link-prediction methods remain effective in realistic settings, given reverse triples in widely used benchmarks such as FB15k and WN18.

    Source: §4.2.1 · p. 7

  • Method

    The authors investigate data redundancy and Cartesian-product relations in benchmark datasets and evaluate their impact on link-prediction models.

    Source: §1 · p. 4, §6 · p. 14

  • Results

    Embedding-model performance degenerates significantly on realistic datasets, and the authors describe absolute link-prediction accuracy as poor.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    The authors conclude that link prediction lacks a truly effective automated solution and call for re-investigation of approaches to knowledge-graph completion.

    Source: §1 · p. 3

    Existing performance measures rely on the closed-world assumption and can penalize correct triples absent from the ground-truth dataset.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

In the active research area of employing embedding models for knowledge graph completion, particularly for the task of link prediction, most prior studies used two benchmark datasets FB15k and WN18 in evaluating such models. Most triples in these and other datasets in such studies belong to reverse and duplicate relations which exhibit high data redundancy due to semantic duplication, correlation or data incompleteness. This is a case of excessive data leakage---a model is trained using features that otherwise would not be available when the model needs to be applied for real prediction. There are also Cartesian product relations for which every triple formed by the Cartesian product of applicable subjects and objects is a true fact. Link prediction on the aforementioned relations is easy and can be achieved with even better accuracy using straightforward rules instead of sophisticated embedding models. A more fundamental defect of these models is that the link prediction scenario, given such data, is non-existent in the real-world. This paper is the first systematic study with the main objective of assessing the true effectiveness of embedding models when the unrealistic triples are removed. Our experiment results show these models are much less accurate than what we used to perceive. Their poor accuracy renders link prediction a task without truly effective automated solution. Hence, we call for re-investigation of possible effective approaches.

1 INTRODUCTION

Knowledge graphs remain incomplete despite serving AI applications, motivating completion methods based on embeddings and observable features. This introduction argues that benchmark results overstate link-prediction effectiveness because reverse, duplicate, and Cartesian-product relations make many predictions unrealistic.

1 INTRODUCTION

  • After reverse triples are removed, all existing embedding models degrade significantly; TuckER reaches FMRR 0.355 on FB15k-237 versus 0.79 on FB15k.Figure 1 compares embedding-model performance on FB15k versus FB15k-237 and WN18 versus WN18RR using FMRR.Source: Figure 1 · p. 2, §1 · p. 2
  • Reverse pairs cover 70% of FB15k training triples and 92.5% of WN18 training triples; a simple rule attains FHits@1 71.6% and 96.4%, respectively.The simple rule's results are on par with the best-performing embedding models on those datasets.Source: §1 · p. 2
  • Duplicate and reverse-duplicate relations add further redundancy; FB15k has 142 Cartesian-product relations among 1,345, where every applicable subject-object pairing is true.The paper reports that 60% of its Cartesian-product relations are due to Freebase mediator nodes and edge concatenation in FB15k.Source: §1 · p. 3
  • On nine FB15k Cartesian-product relations, a statistics-based rule averages FHits@10 98.3%, above TransE's 96.3%.The rule identifies relations with many observed subject-object pairs and predicts the remaining applicable pairs.Source: §1 · p. 3
  • AMIE also degrades on realistic datasets: FMRR falls from 0.797 to 0.308 on FB15k/FB15k-237 and from 0.94 to 0.357 on WN18/WN18RR.AMIE is an observed-feature model using rules discovered by a rule-mining system.Source: §1 · p. 3
  • The authors call FB15k and WN18 misleading and report that most YAGO3-10 triples are duplicates; they conclude no truly effective automated link-prediction solution exists.They call for re-investigation of effective approaches to knowledge-graph completion.Source: §1 · p. 3

2 BACKGROUND: KNOWLEDGE GRAPH COMPLETION METHODS

Knowledge-graph completion methods score triples and learn entity and relation representations; alternatives use observable features or reasoning paths to predict missing links.

2 BACKGROUND: KNOWLEDGE GRAPH COMPLETION METHODS

  • The section represents triples as (h, r, t), with entities in E and relations in R, and defines the multilinear dot product component-wise.Vectors and matrices use bold lowercase and uppercase notation, respectively.Source: §2 · p. 4

2.1 Latent Feature Models

  • Embedding methods learn entity and relation representations by scoring triple plausibility and optimizing scores to separate correct from corrupted triples.The section describes margin-based and logistic loss functions for this optimization.Source: §2.1 · p. 4
  • TransE represents relations as translations, while TransH, TransR, TransD, and RotatE introduce relation-specific representations, projection spaces, or rotations.TransE has difficulty modeling 1-to-n, n-to-1, and m-to-n relations; TransD decomposes the projection matrix and assigns one to each entity-relation pair.Source: §2.1 · p. 4
  • Tensor-factorization models score triples from learned factors; DistMult models only symmetric relations, while ComplEx handles symmetric and anti-symmetric relations.The section also describes RESCAL's relation matrices, TuckER's Tucker decomposition, and ConvE's convolutional and fully connected layers.Source: §2.1 · p. 4

2.2 Other Approaches

  • AMIE mines rules whose instantiated body atoms support predictions of the rule's head.A rule consists of antecedent atoms and a consequent atom.Source: §2.2 · p. 5
  • Multi-hop methods learn to walk from a query entity through labeled relations, conditioning each step on the query relation and path history.The aim is to find complex patterns by following reasoning paths in the graph.Source: §2.2 · p. 5

3 EXISTING EVALUATION FRAMEWORK

The framework evaluates link prediction on FB15k, WN18, and YAGO3-10 by ranking corrupted triples and measuring ranking accuracy with raw and filtered metrics.

3.1 Evaluation Datasets

  • FB15k contains 14,951 entities, 1,345 relations, and 592,213 triples, selected using minimum appearance thresholds for entities and relations.Its triples were randomly split into training, validation, and test sets.Source: §3.1 · p. 5
  • WN18 is extracted from WordNet, while YAGO3-10 is a YAGO3 subset containing entities involved in at least 10 relations.The section describes WN18 as a lexical database and YAGO3 as derived from Wikipedia and WordNet.Source: §3.1 · p. 5

3.2 Evaluation Methods and Measures

  • For each test triple, link prediction ranks the original against triples formed by replacing its head or tail with other dataset entities.The ideal ranking places the original test triple at the top.Source: §3.2 · p. 5
  • Accuracy uses Hits@1, Hits@10, MR, and MRR, with filtered versions excluding corrupted triples already present in the dataset.Higher Hits and reciprocal-rank values and lower mean-rank values indicate better accuracy.Source: §3.2 · p. 5

4 INADEQUACY OF BENCHMARKS AND EVALUATION MEASURES

Section 4 examines how reverse, duplicate, and Cartesian product relations distort link-prediction evaluations, and traces some FB15k artifacts to its likely Freebase source snapshot. These relations make predictions easier or less meaningful than realistic completion tasks.

4 INADEQUACY OF BENCHMARKS AND EVALUATION MEASURES

  • Reverse, duplicate, and Cartesian product relations substantially overestimate embedding-model accuracy in FB15k, WN18, and YAGO3-10.The section examines these relation types across the three datasets.Source: §4 · pp. 5–6

4.1 Identifying the Most Probable Freebase Snapshot Used for Producing FB15k

  • The May 5, 2013 Freebase snapshot contains 99.54% of FB15k’s triples, making it the most likely source snapshot identified by the authors.FB15k also retains most concatenated edges: 396,148 of its 592,213 triples are concatenated, and 394,947 appear in that snapshot.Source: §4.1 · p. 6

4.2.1 Data Leakage Due to Reverse Triples.

  • On FB15k, a simple reverse-relation rule reaches 71.6% FHits@1, versus 73.8% for the best embedding model; on WN18, it reaches 96.4% and exceeds embedding models.Reverse triples exist in training for 70.3% of FB15k test triples and 4,658 of 5,000 WN18 test triples.Source: §4.2.1 · p. 7

4.2.2 Other Redundant Triples.

  • On FB15k, ConvE’s FMRR is 0.734 on redundant relations versus 0.454 on relations without redundancy; on YAGO3-10, RotatE’s is 0.612 on two duplicate relations versus 0.304 on others.The YAGO3-10 relations isAffiliatedTo and playsFor account for 35% and 30% of training triples, respectively.Source: §4.2.2 · p. 8

4.3 Cartesian Product Relations

  • Cartesian product relations make prediction trivial or unmeaningful, and 3,506 of the 4,683 identified relations are concatenated edges arising from simplified multiary relationships.The section notes that FB15k removes mediator nodes while retaining concatenated edges, reducing multiary facts to binary relations.Source: §4.3 · pp. 8–9
  • On nine detected FB15k Cartesian product relations, the rule-based property method outperforms TransE; for r2, FMRR is 0.80 versus 0.42 using FB15k as ground truth, and 1 using Freebase.The higher result with Freebase ground truth reflects its broader coverage and reduced closed-world evaluation defect.Source: Table 3 · p. 9, §4.3 · p. 10
  • The closed-world ranking measures penalize valid but unrecorded triples, a flaw that is especially apparent when evaluating methods that exploit Cartesian product relations.A predictor may rank triples absent from the dataset above the test triple even when those triples are correct.Source: §4.3 · p. 9

5 EXPERIMENTS

Across the more realistic benchmark datasets, performance drops substantially for embedding models and AMIE, while performance differences vary by relation type. The experiments compare cleaned and original datasets, examine relation-specific results, and evaluate embedding and rule-based methods.

5.1 FB15k-237, WN18RR, YAGO3-10-DR

  • The experiments use FB15k-237, WN18RR, and YAGO3-10-DR; these datasets remove selected reverse, duplicate, or symmetric triples, with WN18RR retaining symmetric relations.FB15k-237 also removes test and validation triples whose entity pairs are linked in training; YAGO3-10-DR removes playsFor and selected redundant symmetric-relation triples.Source: §5.1 · p. 11

5.2 Experiment Setup

  • The study compares embedding models with AMIE rules generated from training data, using the source-code developers’ hyperparameter settings for the embedding models.AMIE ranks predictions by maximum rule confidence, breaking ties by the number of applicable rules.Source: §5.2 · p. 11

5.3 Results

  • Removing reverse relations sharply reduces ConvE’s FMRR: 0.698 to 0.305 on FB15k to FB15k-237 and 0.945 to 0.429 on WN18 to WN18RR.The paper also reports substantial performance drops on YAGO3-10-DR after removing duplicate and reverse triples.Source: §5.3 · pp. 13–14
  • On FB15k-237, successor models’ FMRR advantage over TransE narrows, while RotatE and TuckER stand out on FHits@1 for FB15k-237 and WN18RR.The authors note that retained reverse triples from symmetric relations may contribute to successor models’ wider edge on WN18RR.Source: §5.3 · pp. 13–14
  • AMIE’s FHits@10 falls from 88.1% to 47.7% on FB15k to FB15k-237 and from 94.0% to 35.6% on WN18 to WN18RR.On FB15k and WN18, embedding models and AMIE have comparable FHits@1; on FB15k-237 and WN18RR, RotatE and TuckER stand out.Source: §5.3 · pp. 13–14
  • On FB15k-237, RotatE excels on 1-to-1 and 1-to-n relations, while TuckER excels on n-to-1 and n-to-m relations; WN18RR’s relation breakdown is less robust.TransE remains strong on FB15k-237 1-to-1 relations; on both datasets, all methods do better predicting the “1” side of 1-to-n and n-to-1 relations.Source: §5.3 · p. 14
  • On YAGO3-10, AMIE outperforms embedding models on FHits@1 and FMRR; on relation categories, it leads on 1-to-1 and n-to-m while RotatE and TuckER lead on 1-to-n and n-to-1.The two near-duplicate relations are n-to-m, and the self-reciprocal relation isMarriedTo is 1-to-1.Source: §5.3 · p. 14
  • On FB15k-237, RotatE and TuckER lead on different relation categories; on YAGO3-10-DR, all models drop significantly after duplicate and reverse triples are removed.The paper recommends YAGO3-10-DR instead of YAGO3-10 for evaluating embedding models.Source: §5.3 · p. 14

6 CONCLUSIONS

The study finds that data redundancy and Cartesian product relations produce unrealistic evaluation conditions, under which simple statistical rules can challenge complex models. After removing straightforward prediction cases, model performance drops significantly, and the authors conclude that link prediction lacks a truly effective automated solution.

6 CONCLUSIONS

  • The study finds that removing straightforward prediction tasks significantly reduces model performance, while Cartesian product relations also produce unrealistic evaluation results.The authors conclude that link prediction has no truly effective automated solution and call for re-investigating approaches to knowledge graph completion.Source: §6 · p. 14
  • Simple rules based on data statistics can challenge complex machine learning models under the data characteristics studied.The paper also reports that existing evaluation metrics can penalize methods for generating correct predictions absent from the ground-truth dataset.Source: §6 · p. 14
Loading 2003.08001v1…