Source-linked AI summary
A Semantic Matching Energy Function for Learning with Multi-relational Data
Xavier Glorot, Antoine Bordes, Jason Weston, Yoshua Bengio
TL;DR
Statistical relational learning faces a central data challenge. The paper introduces semantic matching, a neural architecture embedding entities and relation types into a shared continuous space, and reports that its bilinear formulation outperforms its linear formulation across three tasks, with mixed benchmark results overall.
Problem
Data is now a central challenge in statistical relational learning.
Method
Semantic matching embeds entities and relation types as vectors in an embedding space and combines them in a structured fashion.
Results
The bilinear formulation of SME outperforms the linear formulation on all three tasks, while benchmark comparisons are mixed across UMLS, Nations, and Kinships.
Takeaways & Limitations
SME models relation types in the same space as entities, allowing relation types to be treated as standard entities and supporting application where any entity can be a relation type.
Takeaways & Limitations
Experimental results on the benchmark datasets are mixed.
Abstract
from arXiv · showhide
Large-scale relational learning becomes crucial for handling the huge amounts of structured data generated daily in many application domains ranging from computational biology or information retrieval, to natural language processing. In this paper, we present a new neural network architecture designed to embed multi-relational graphs into a flexible continuous vector space in which the original data is kept and enhanced. The network is trained to encode the semantics of these graphs in order to assign high probabilities to plausible components. We empirically show that it reaches competitive performance in link prediction on standard datasets from the literature.
1 Introduction
Statistical relational learning faces the challenge of handling growing amounts of data, including link prediction beyond observed valid triplets.
- Large-scale data is now a central challenge in statistical relational learning.
- Link prediction generalizes beyond the set of given valid triplets.
2 Semantic Matching Energy Function
The paper represents multi-relational databases as directed graphs and embeds entities and relation types in a continuous space. Its semantic matching energy function creates relation-dependent embeddings and compares transformed graph arguments during training.
- Graph representation: Multi-relational databases are modeled as directed graphs whose links represent several possible relation types between entities.
- Embedding space: Each entity and relation type is associated with a d-dimensional vector in a shared embedding space.The model allows general mappings from entities to embeddings.
- Semantic matching: The energy function maps triplet symbols to embeddings, constructs relation-dependent representations for both entities, and matches them with a structured criterion.The left and right transformations use parametrized functions conditioned on the relation embedding.
- Training: Training uses stochastic gradient descent with a ranking objective that makes the energy lower for training examples than for other possible configurations.
- Neural architecture: The neural architecture processes each entity in parallel, like a siamese network, before comparing the resulting representations.Its intuition is that the relation type should help extract relevant components from each argument’s embedding.
- Model variants: SME has linear and bilinear variants, with the latter using 3-mode tensors as core weights.The linear form uses linear layers, while the bilinear form uses tensor weights and biases.
3 Empirical Evaluation
SME is evaluated on link prediction across UMLS, Nations, and Kinships using area under the precision-recall curve, with results varying by dataset and interaction complexity. The experiments also compare SME variants and establish its modeling scope relative to prior methods.
- Evaluation setup: SME is evaluated for link prediction on UMLS, Nations, and Kinships using area under the precision-recall curve (AUC).The comparison includes RESCAL, MRC, IRM, CP, and LFM.
- Model variants: SME(bilinear) outperforms the linear SME formulation on all three tasks.
- Interaction structure: Joint interactions among lhs, rel, and rhs are crucial for representing Nations and Kinships, where relations cannot be reduced to sums of bigrams.The result is especially pronounced for the complex kinship systems of the Alyawarra.
- Interaction structure: UMLS interactions can be represented using entity-entity and entity-relation-type bigrams, unlike the stronger ternary interaction requirements observed for Nations and Kinships.
- Comparative results: SME(bilinear) performs similarly to LFM on UMLS, slightly outperforms it on Nations, and is outperformed by CP, RESCAL, and LFM on Kinships.The Kinships result is attributed to the dataset's complex ternary interactions, where reconstruction-based tensor factorization or combined bigram-trigram interactions may be beneficial.
- Model scope: SME models relation types as vectors in the same space as entities, enabling direct application when any entity can also create relationships between other entities.The paper describes this as a conceptual distinction from previous methods.