Source-linked AI summary
MMKG: Multi-Modal Knowledge Graphs
Ye Liu, Hui Li, Alberto Garcia-Duran, Mathias Niepert, Daniel Onoro-Rubio, David S. Rosenblum
TL;DR
Knowledge graphs are incomplete, while existing resources do not combine relational structure with numerical features, images, and cross-graph alignments. MMKG constructs three such graphs and evaluates multimodal learning for sameAs link prediction. The experiments provide evidence that multiple feature types benefit the task.
Problem
Knowledge graphs are often incomplete, and existing visual-relational resources do not provide MMKG’s combination of relational reasoning, images, numerical features, and cross-graph alignments.
Method
MMKG constructs FREEBASE15K, DBPEDIA15K, and YAGO15K with aligned entities, relational triples, numerical literals, and image information.
Results
Experiments validate MMKG’s utility for sameAs link prediction and show that performance benefits from learning multiple feature types.
Takeaways & Limitations
MMKG provides a benchmark resource for developing and evaluating multimodal approaches to link prediction and entity matching.
Takeaways & Limitations
Freebase has deprecated RDF URIs, and image access is handled through learned VGG16 embeddings because image URLs are not permanent.
Abstract
from arXiv · showhide
We present MMKG, a collection of three knowledge graphs that contain both numerical features and (links to) images for all entities as well as entity alignments between pairs of KGs. Therefore, multi-relational link prediction and entity matching communities can benefit from this resource. We believe this data set has the potential to facilitate the development of novel multi-modal learning approaches for knowledge graphs.We validate the utility ofMMKG in the sameAs link prediction task with an extensive set of experiments. These experiments show that the task at hand benefits from learning of multiple feature types.
1 Introduction
MMKG introduces three heterogeneous knowledge graphs enriched with numerical features, image links, and cross-graph entity alignments. It targets link prediction and entity matching, with experiments examining how modalities affect sameAs completion.
- Resource and motivation: Knowledge graphs encode entities and directed relationships as multi-relational triples, but are often incomplete.This incompleteness motivates knowledge graph completion research, including link prediction and cross-graph alignment.
- Resource and motivation: MMKG contains three knowledge graphs with numerical features, images, and entity alignments between graph pairs.These resources are designed for link prediction and entity matching research.
- Contributions: The contributions include releasing graph structure, sameAs predicates, numerical literals, and image links in separate files.The paper also analyzes modality combinations across different alignment percentages and KG pairs.
- Evaluation: The paper evaluates sameAs queries linking entities from different knowledge graphs, treating the task as intermediate between link prediction and entity matching.Experiments vary known alignment percentages and compare individual modalities with feature combinations.
2 Relevance
MMKG targets knowledge graph completion through link prediction and entity matching, extending evaluation beyond relational structure by combining numerical and visual information. Its heterogeneous graphs and alignment coverage support robustness and active-learning studies across varying data conditions.
- MMKG targets link prediction and entity matching as two knowledge graph completion tasks.
- Multi-relational link prediction methods typically learn scoring functions that assign high scores to true triples and low scores to presumed false triples.
- 2.1 Relevance for Multi-Relational Link Prediction Research: MMKG extends relational graph evaluation by associating every entity with numerical and visual data.
- 2.1 Relevance for Multi-Relational Link Prediction Research: The three MMKG graphs are heterogeneous in relation types and sparsity, enabling measurement of link prediction robustness across graph characteristics.
- 2.2 Relevance for Entity Matching Research: MMKG supports entity matching that combines relational structure with literals, images, and known alignments between entities from different graphs.
- 2.2 Relevance for Entity Matching Research: Its alignment coverage permits evaluation against different numbers of given alignments and can support active learning strategies for selecting queried alignments.
3 MMKG: Dataset Generation
MMKG is generated from FB15K by aligning entities with DBPEDIA and YAGO, then adding relational, numerical, and visual information. The resulting datasets preserve heterogeneous graph statistics while providing reusable image embeddings and alignment files.
- Dataset generation: MMKG construction starts from FB15K N-Triples and aligns its entities with DBPEDIA and YAGO to create DB15K and YAGO15K.DB15K and YAGO15K are versions of the source knowledge graphs aligned through sameAs links.
- Dataset generation: DB15K includes aligned entities plus highly connected additional entities, with triples retained when both endpoints belong to the dataset.This produces the relational graph structure while preserving entities that cannot be aligned across knowledge graphs.
- Dataset generation: Numerical literals are collected as RDF triples, including relations such as latitude, and the most common numerical relationships are summarized for DB15K and YAGO15K.The repository stores numerical information as RDF literals alongside entity and relation URIs.
- Dataset generation: Images are gathered through disambiguated search queries derived from Wikipedia URIs, reducing noise from polysemous entity labels.For example, distinct Wikipedia URIs are used to distinguish entities sharing the label “Paris”.
- Dataset generation: YAGO15K repeats the DB15K construction sequence but uses YAGO sameAs predicates with DBPEDIA and existing FB15K alignments to create cross-graph alignments.Entity and relation frequencies in YAGO15K and DB15K are reported on logarithmic scales, illustrating heterogeneity between the graphs.
- Availability and sustainability: Image URLs are complemented by 4096-dimensional VGG16 embeddings, distributed in HDF5 format, while repository files provide relational, numerical, visual, and alignment data.The embeddings are obtained by removing VGG16’s softmax layer; users can alternatively crawl images from the search engines.
4 Technical Quality of MMKG
MMKG extends a Product of Experts framework with visual information and combines latent, relational, numerical, and image-based evidence for sameAs prediction. The framework evaluates modality combinations and alternative fusion strategies for cross-knowledge-graph entity alignment.
- 4.2 Model: Products of Experts: MMKG's Product of Experts extends previous work by incorporating visual information into multi-modal sameAs prediction.The model defines experts for latent, relational, numerical, and visual feature types.
- 4.2 Model: Products of Experts: The PoE assigns high probability to true triples and low probability to triples assumed to be false.In this setting, each data vector is a knowledge-graph triple d = (h, r, t).
- Visual Experts: The visual expert scores sameAs pairs using cosine similarity between 4096-dimensional image feature vectors.The visual expert is learned only for the sameAs relation type.
- Learning: Negative sampling generates N corrupted triples by replacing the tail with entities sampled uniformly from the full entity set.The sampled triples support approximation of the training objective, and the procedure is referred to as negative sampling.
- Combining Modalities: Alternative fusion methods include logistic regression over concatenated modality features and an ensemble that adds independently trained expert scores at test time.The ensemble differs from PoE because its experts are trained separately rather than jointly and end-to-end.
- Visual Experts: Figure 7 identifies sameAs queries for which visual experts led to good performance, with image pairs shown for FB15K and DB15K.The figure compares the left and right images within each pair across the two knowledge graphs.
5 Experiments
Experiments evaluate MMKG on sameAs prediction across two KG pairs, varying the percentage of known alignments and comparing modalities, PoE variants, and baselines. Results support multi-modal benefits, while the strongest PoE variant depends on alignment coverage and KG pair.
- Experimental Setup: Experiments cover FB15K–DB15K and FB15K–YAGO15K sameAs prediction using PoE instances and baseline methods.The evaluation uses link-prediction metrics for completion queries across the two KG pairs.
- Experimental Setup: Known alignments are varied across low, medium, and high settings corresponding to 20%, 50%, and 80% of sameAs predicates.These alignments are combined with the two knowledge graphs as observed information.
- Evaluation Metrics: Evaluation ranks the correct entity after replacing one endpoint with candidates from the other KG, reporting Mean Rank, hits@n, and Mean Reciprocal Rank.Mean Rank is lower-is-better, whereas hits@n and Mean Reciprocal Rank are higher-is-better.
- Results: PoE-lrni tends to perform best at low alignment percentages, while PoE-rni tends to perform best at high percentages.Performance varies across alignment percentages and knowledge-graph pairs, limiting broad generalizations.
- Results: The embedding expert response often dominates, and adding it can harm performance when it is not the best-performing expert.Tables 4 and 5 nevertheless provide empirical evidence that including additional modalities benefits performance.
- Results: The best-performing PoE instance significantly outperforms CONCAT and ENSEMBLE at P = 80%.This result is presented as validation of PoE for incorporating modalities into link prediction in a principled manner.
6 Conclusion
MMKG provides three heterogeneous multi-modal knowledge graphs for benchmarking link prediction and entity matching. Extensive experiments validate the dataset's utility for sameAs link prediction.
- Conclusion: MMKG contains three heterogeneous knowledge graphs with multi-modal data for benchmarking link prediction and entity matching.The graphs differ in properties including the number of relation types and degree of sparsity.
- Conclusion: Extensive experiments validate MMKG's utility for the sameAs link prediction task.