Source-linked AI summary
CoDEx: A Comprehensive Knowledge Graph Completion Benchmark
Tara Safavi, Danai Koutra
TL;DR
KGC needs stronger benchmarks because widely used Freebase-derived datasets have quality and difficulty limitations. CoDEx introduces Wikidata- and Wikipedia-derived datasets with varied structures, multilingual information, and verified hard negatives, then evaluates embedding models and compares CoDEx with FB15K-237. The results show broader content and a more challenging link prediction benchmark, while the evaluation focuses on embeddings and hard-negative annotation remains an ongoing direction for CODEX-L.
Problem
KGC evaluation relies heavily on established Freebase-derived benchmarks despite documented quality, leakage, and difficulty concerns.
Method
The paper constructs CoDEx from Wikidata and Wikipedia, analyzes its logical relation patterns, benchmarks five embedding models, and compares it with FB15K-237.
Results
CoDEx covers more diverse and interpretable content and is a more challenging link prediction benchmark than FB15K-237.
Takeaways & Limitations
CoDEx provides a benchmark for multiple KGC tasks, with data, code, and pretrained models released for community use.
Takeaways & Limitations
The benchmarking section focuses on embedding methods, and high-quality crowdsourced negative annotations for CODEX-L remain under investigation.
Abstract
from arXiv · showhide
We present CoDEx, a set of knowledge graph completion datasets extracted from Wikidata and Wikipedia that improve upon existing knowledge graph completion benchmarks in scope and level of difficulty. In terms of scope, CoDEx comprises three knowledge graphs varying in size and structure, multilingual descriptions of entities and relations, and tens of thousands of hard negative triples that are plausible but verified to be false. To characterize CoDEx, we contribute thorough empirical analyses and benchmarking experiments. First, we analyze each CoDEx dataset in terms of logical relation patterns. Next, we report baseline link prediction and triple classification results on CoDEx for five extensively tuned embedding models. Finally, we differentiate CoDEx from the popular FB15K-237 knowledge graph completion dataset by showing that CoDEx covers more diverse and interpretable content, and is a more difficult link prediction benchmark. Data, code, and pretrained models are available at https://bit.ly/2EPbrJs.
1 Introduction
CoDEx addresses limitations in widely used KGC benchmarks by introducing Wikidata- and Wikipedia-derived datasets with broader scope and greater difficulty. The paper combines dataset construction, logical analysis, benchmarking, and comparison with FB15K-237.
- Motivation: Knowledge graph completion research needs high-quality benchmarks because existing evaluation has remained largely static and often relies on outdated Freebase subsets.Freebase had known quality issues, and FB15K was shown to contain train/test leakage.
- Contribution: CoDEx is a set of knowledge graph completion datasets extracted from Wikidata and Wikipedia.
- Contribution: CoDEx improves upon existing Freebase-based benchmarks in scope and level of difficulty.
- Data: The benchmark contains three knowledge graphs varying in size and structure, entity types, multilingual labels and descriptions, and manually verified hard negative triples.
- Evaluation: The paper reports baseline link prediction and triple classification results for five widely used embedding models, then compares CoDEx with FB15K-237.
2 Existing datasets
Existing KGC benchmarks span encyclopedic, web-mined, and domain-specific datasets, but their coverage and evaluation quality vary substantially. The section surveys these datasets and highlights limitations including leakage, narrow domains, duplicate relations, and questionable triples.
- Survey scope: Existing KGC benchmark surveys cover datasets and tasks across artificial intelligence, machine learning, and natural language processing.
- Encyclopedic datasets: FB15K and FB15K-237 are popular Freebase-derived benchmarks, with FB15K-237 created to remedy inverse-triple leakage in FB15K.
- Encyclopedic datasets: NELL-995 covers many domains but contains low-quality or overly generic triples, while its reported mean average precision was below 50% around its 1000th iteration.
- Encyclopedic datasets: YAGO3-10 is high precision but was reported to be too easy for link prediction because it contains many duplicate relations.
- Domain-specific datasets: KGC evaluations also use smaller domain-specific datasets from WordNet, UMLS, kinship, Countries, and synthetic family-tree resources.
3 Data collection
CoDEx is constructed from Wikidata using broad domain seeds, k-core filtering, leakage-reducing splits, multilingual auxiliary information, and manually verified hard negatives. Three size-scaled datasets support different evaluation settings.
- Collection: The pipeline defines knowledge graphs as entities, relations, and factual (head, relation, tail) triples, then collects an initial Wikidata graph through snowball sampling.Seeds cover 13 domains and retrieve 380,038 entities, 75 relations, and 1,156,222 triples.
- Dataset construction: Three k-core snapshots form CODEX-S with 36k triples, CODEX-M with 206k, and CODEX-L with 612k.CODEX-S targets testing and debugging, CODEX-M is all-purpose, and CODEX-L supports general and few-shot evaluation.
- Leakage control: The datasets remove inverse relations and use 90/5/5 train/validation/test splits whose validation and test entities and relations occur in training.Relations with more than 50% entity-pair overlap with another relation are removed to reduce leakage.
- Auxiliary information: CODEX includes entity types, multilingual labels and descriptions, and Wikipedia introduction extracts in Arabic, German, English, Spanish, Russian, and Chinese.These resources support joint learning over graph structure and auxiliary information.
- Hard negatives: Crowdsourced negative annotations for CODEX-L were still under investigation, limiting the stated annotation coverage across datasets.CODEX-S and CODEX-M received manually labeled validation and test negatives.
4 Analysis of relation patterns
The analysis characterizes CODEX through symmetry and compositionality, while omitting inversion because inverse relations were removed to prevent train/test leakage. Composition yields increasingly many discovered rules in larger datasets, especially CODEX-L.
- Overview: CODEX analyzes symmetry and compositionality as learnable relation patterns, omitting inversion because inverse relations were removed to avoid train/test leakage.Compositionality requires multi-hop reasoning over paths of relations.
- Symmetry: Five relations meet the paper’s 50% overlap criterion for symmetry: diplomatic relation, shares border with, sibling, spouse, and unmarried partner.The criterion compares overlapping head-tail and tail-head entity pairs for each relation.
- Symmetry: Symmetric patterns are more prevalent in CODEX-S, whereas the larger datasets are mostly antisymmetric.For antisymmetric relations, a relation generally does not hold in the reversed direction.
- Composition: AMIE3 identifies 26, 44, and 93 compositional rules in CODEX-S, CODEX-M, and CODEX-L, with average confidences of 0.630, 0.556, and 0.459.Only paths of lengths two and three are analyzed because longer paths are relatively costly to compute.
- Composition: CODEX-L has especially prevalent composition, including a rule linking an entity’s founder and the founder’s citizenship country to the entity’s country of origin.The example rule has confidence 0.709, and the released rules support further rule-based KGC development.
5 Benchmarking
The benchmark evaluates five embedding models on link prediction and triple classification under extensive model selection, including plausible manually verified hard negatives. Results show that relation structure and training choices strongly affect performance, while hard negatives make triple classification substantially more difficult.
- Benchmark setup: Five embedding models are benchmarked on CODEX for link prediction and triple classification using extensive hyperparameter tuning.The models span linear, translational, and nonlinear architectures.
- Evaluation tasks: Link prediction ranks candidate head or tail entities for each test query using filtered MRR and hits@k.Known positive triples are excluded from ranking so they do not artificially lower scores.
- Link prediction results: ComplEx performs best on CODEX-S, whose relation structure is highly symmetric, while TuckER performs best on compositional CODEX-L.On the most frequent symmetric relation in CODEX-S, ComplEx reaches 0.859 MRR; on CODEX-L’s most frequent compositional relation, TuckER reaches 0.465 MRR.
- Model selection: Validation MRR varies by over 30 percentage points across training strategies and input configurations, showing that hyperparameters strongly affect link prediction.Each model reaches its peak performance with cross-entropy loss, while no negative-sampling strategy is consistently dominant.
- Triple classification results: Triple classification on uniformly random negatives reaches nearly 100% accuracy for all models, making that evaluation nearly solved.Performance remains strong with frequency-based negatives but changes substantially on CODEX hard negatives.
- Triple classification results: Hard negatives reduce classification performance by about 8–11 percentage points relative to frequency-based sampling and 13–19 points relative to uniform random sampling.On hard negatives, RESCAL is slightly strongest on CODEX-S and ConvE on CODEX-M, unlike the link prediction rankings.
6 Comparative case study
The case study compares CODEX-M with FB15K-237, finding that CODEX-M covers more diverse and interpretable content and is more difficult for link prediction.
- CODEX-M is compared with FB15K-237 because the datasets are closest in size and FB15K-237 is a widely used benchmark.
- Diversity: Over 25% of FB15K-237 triples belong to the /award relation domain, whereas CODEX-M’s frequent relations indicate more diverse content.
- Interpretability: Wikidata relations use concise natural-language labels, while many Freebase relations have hierarchical labels and n-nary structure that complicates reasoning.
- Difficulty: The frequency baseline ranks candidate entities by relation-specific training frequency after filtering already observed triples.
- Difficulty: The embedding’s median MRR improvement is 8.27 percentage points on FB15K-237 versus 20.04 on CODEX-M.
- Difficulty: Nearly 40% of FB15K-237 test triples have less than five percentage points of improvement in both MRR and Hits@10, with zero or negative improvement 15% of the time.
- Conclusion: CODEX is more appropriately difficult for link prediction and supports testing symmetry and compositionality beyond frequency-based reasoning.
7 Conclusion and outlook
The paper concludes that CODEX provides a reusable benchmark for knowledge graph completion and outlines directions for model analysis, triple classification, and multimodal information use.
- CODEX comprises knowledge graph completion datasets extracted from Wikidata and Wikipedia and is intended for multiple KGC tasks.
- The released data, code, and pretrained models support community use of the benchmark.
- Future directions: CODEX can support analysis of hyperparameters, training strategies, and model architectures in KGC.
- Future directions: The authors encourage triple classification alongside link prediction because it directly tests discriminative power.
- Future directions: Including text in link prediction and triple classification could improve performance and support few-shot link prediction.
B Seeds for data collection
CODEX seed entities are collected from Wikidata using different relations for people and things.
- People-related seed entities are retrieved by querying Wikidata with the occupation relation.
- Thing-related seed entities are retrieved using the instance of and subclass of relations.
C Negative annotation guidelines
The negative annotation guidelines require judging triples as true or false using entity and relation evidence, with specificity and factual support guiding difficult cases.
- Annotators label each triple true or false using Wikipedia and Wikidata links, while unsupported claims should be labeled false.
- Examples: False triples may be grammatically incorrect because their entity or relation types do not make sense.
- Examples: Grammatically valid triples can still be false when their factual content is incorrect.
- Examples: Matching entity types do not establish truth, as illustrated by false claims involving citizenship, foundational texts, and influence.
- Tips: Occupation and genre annotations should use the most specific interpretation of what a person is mainly known for.
- Embedding models: The appendix describes RESCAL, TransE, ComplEx, ConvE, and TuckER as the five models used in link prediction and triple classification.
E Content comparison
CoDEx is more diverse in content than FB15K-237, spanning more domains and entity types, while both benchmarks remain biased toward developed Western countries and cultures.
- Both benchmarks are biased toward developed Western countries and cultures, but CoDEx covers more diverse domains.CoDEx includes academia, entertainment, journalism, politics, science, and writing, whereas FB15K-237 focuses mostly on entertainment and sports.
- Five of FB15K-237’s top-30 entities are specific to the United States, indicating stronger U.S. bias than in CoDEx.
- CoDEx covers people, places, organizations, movies, and abstract concepts among its top entity types.
- FB15K-237 has many overlapping entity types that are mostly related to entertainment.
F Hyperparameter search
The experiments use dataset-specific hyperparameter searches, early stopping, and validation-based model selection, with different search budgets for CODEX-S, CODEX-M, and CODEX-L.
- Search strategies: Ax combines quasi-random trials and Bayesian optimization for hyperparameter search across CoDEx datasets.
- Search strategies: CODEX-S uses 30 quasi-random trials followed by 10 Bayesian-optimization trials per negative-sampling and loss combination.Models train for up to 400 epochs with patience-5 early stopping; trials terminate after 50 epochs below 0.05 MRR.
- Search strategies: CODEX-M uses 20 quasi-random trials per negative-sampling and loss combination with the same epoch limit and early-stopping criteria as CODEX-S.
- Search strategies: CODEX-L uses 10 quasi-random trials of 20 epochs to reduce resource usage, then retrains the best model for up to 400 epochs.The reduced search budget reflects that MRR typically plateaus after 20–30 epochs.
- Model selection: Models are selected by validation MRR for link prediction and by validation accuracy for triple classification.Triple classification selects among pretrained models across negative-sampling and loss-function combinations.
- Reproducibility: The released pretrained LibKGE models include accompanying configuration files in the centralized CODEX repository.