Source-linked AI summary
Embedding Entities and Relations for Learning and Inference in Knowledge Bases
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, Li Deng
TL;DR
Knowledge-base embedding models lacked a unified comparison, while graph-based Horn-rule mining faced scalability challenges. This paper unifies embedding models and mines rules from learned relation representations, achieving 73.2% top-10 accuracy on Freebase versus 54.7% for TransE and outperforming AMIE in rule extraction.
Problem
Existing embedding models had unclear comparative effects from representation choices, while traditional Horn-rule mining faced scalability issues on large knowledge-base graphs.
Method
The paper unifies multi-relational embedding models and uses learned relation embeddings, including bilinear representations, to mine compositional Horn rules.
Results
The bilinear model achieves 73.2% top-10 accuracy versus TransE’s 54.7% on Freebase, while embedding-based rule extraction outperforms AMIE.
Takeaways & Limitations
Bilinear embeddings capture relational and compositional semantics sufficiently well for both knowledge-base link prediction and Horn-rule extraction.
Takeaways & Limitations
The rule-extraction assumptions may not hold in the implementations, and effectively enforcing them remains future work.
Abstract
from arXiv · showhide
We consider learning representations of entities and relations in KBs using the neural-embedding approach. We show that most existing models, including NTN (Socher et al., 2013) and TransE (Bordes et al., 2013b), can be generalized under a unified learning framework, where entities are low-dimensional vectors learned from a neural network and relations are bilinear and/or linear mapping functions. Under this framework, we compare a variety of embedding models on the link prediction task. We show that a simple bilinear formulation achieves new state-of-the-art results for the task (achieving a top-10 accuracy of 73.2% vs. 54.7% by TransE on Freebase). Furthermore, we introduce a novel approach that utilizes the learned relation embeddings to mine logical rules such as "BornInCity(a,b) and CityInCountry(b,c) => Nationality(a,c)". We find that embeddings learned from the bilinear objective are particularly good at capturing relational semantics and that the composition of relations is characterized by matrix multiplication. More interestingly, we demonstrate that our embedding-based rule extraction approach successfully outperforms a state-of-the-art confidence-based rule mining approach in mining Horn rules that involve compositional reasoning.
1 INTRODUCTION
The introduction motivates scalable relational learning for large knowledge bases and identifies interpretability limits in embedding-based link prediction. It presents a unified framework for multi-relational embeddings and evaluates entity and relation representations on link prediction.
- Motivation: Large knowledge bases store billions of RDF-triple facts about entities across many relation types, creating a need for scalable relational learning methods.Freebase is cited as containing millions of entities and billions of facts.
- Related approaches: Tensor factorization and neural-embedding models learn low-dimensional entity and relation representations while aiming to scale and support relational reasoning.The introduction identifies these as two popular approaches to encoding relational information.
- Problem: Neural-embedding models such as TransE and NTN achieve strong prediction performance, but their relational properties are difficult to interpret from link prediction alone.Link prediction evaluates correctness of unseen triples only indirectly as evidence that embeddings are meaningful.
- Contributions: The paper introduces a general framework that unifies most prior multi-relational embedding models, including NTN and TransE.This is identified as the first of three main contributions.
- Contributions: The paper empirically compares entity and relation representation choices within this framework on the canonical link prediction task.The introduction states that this evaluation supports comparison under the unified framework.
2 RELATED WORK
Related work contrasts scalable representation-learning approaches for multi-relational knowledge with traditional statistical methods, and characterizes neural embedding models by how they parameterize relation operators. It also situates embedding-based rule extraction within continuous-space logical inference and emphasizes relational compositionality and comparative evaluation on Freebase.
- Multi-relational learning: Traditional statistical approaches to multi-relational learning, including Markov-logic networks, usually suffer from scalability issues.
- Multi-relational learning: Representation-learning methods embed multi-relational knowledge into low-dimensional representations of entities and relations, including tensor or matrix factorization.
- Neural embedding models: Existing neural embedding models represent entities as low-dimensional vectors and relations as operators combining two entity representations, differing in relation-operator parameterization.
- Neural embedding models: The Neural Tensor Network represents each relation with a bilinear tensor operator followed by a linear matrix operator.
- Embedding-based rule extraction: The work connects embedding-based rule extraction to earlier logical inference with learned continuous-space representations and shows that learned relation embeddings capture compositional semantics.
- Embedding-based rule extraction: The approach is systematically evaluated on Freebase and compared favorably with a state-of-the-art rule mining approach for rule extraction.
3 MULTI-RELATIONAL REPRESENTATION LEARNING
This section presents a neural-network framework for learning entity and relation representations in multi-relational knowledge bases. It unifies linear, bilinear, and combined scoring models and trains them by ranking observed triplets above corrupted alternatives.
- Entity representations: The framework projects high-dimensional entity inputs into low-dimensional vectors, using either one-hot or feature-based representations and a linear or nonlinear projection.The projection matrix may be randomly initialized or initialized with pretrained vectors.
- Relation representations: Relation representations determine scoring functions that can be expressed through linear transformations, bilinear transformations, or their combination.These formulations unify most existing scoring functions for relation triplets.
- Model comparison: NTN is the most expressive model because it contains linear and bilinear relation operators as special cases, whereas TransE is the simplest because it uses one-dimensional linear relation vectors.The comparison concerns model expressiveness and parameterization within the unified framework.
- Bilinear formulation: The basic bilinear model is a simplified NTN without nonlinear and linear operators, using a matrix relation operator that can be restricted to a diagonal matrix to reduce parameters.This formulation is also connected to prior matrix-factorization models with different regularization choices.
- Training objective: Models are learned with a margin-based ranking objective that encourages positive triplets to score higher than negative triplets created by corrupting either relation argument.The objective is defined over observed positive triplets and constructed negative triplets.
4 INFERENCE TASK I: LINK PREDICTION
The link-prediction study ranks candidate entities for corrupted test triplets across several embedding models and datasets. Simpler bilinear models generally outperform more complex alternatives, while suitable entity-vector initialization further improves performance.
- Evaluation setup: Link prediction is evaluated by ranking the correct entity against all corrupted dictionary entities for each test triplet.Each entity is treated as the target in turn, with scores ranked in descending order.
- Embedding models: The comparison includes NTN, Bilinear+Linear, TransE, Bilinear, and Bilinear-diag in decreasing order of model complexity.TransE is a special case of Bilinear+Linear, while Bilinear-diag restricts the relation matrix to be diagonal.
- Multiplicative vs. Additive Interactions: Bilinear-diag and TransE have equal parameter counts but differ in interaction: multiplicative weighted element-wise products versus additive element-wise subtraction with bias.The paper refers to these variants as DISTMULT and DISTADD, respectively.
- Initialization of Entity Vectors: DISTMULT-tanh-EV-init provides the best performance on all metrics, whereas DISTMULT-tanh-WV-init reduces performance on FB15k-401.The authors attribute the drop to word vectors being unsuitable for non-compositional entity phrases; over 73% of entities are person names, locations, organizations, or films.
5 INFERENCE TASK II: RULE EXTRACTION
The section introduces embedding-based Horn-rule extraction to avoid the scalability limits of graph-based mining, modeling relation composition through vector addition or matrix multiplication. Experiments show that bilinear embeddings support effective rule selection and generally outperform additive composition and AMIE.
- Motivation: Embedding-based rule mining addresses the scalability problems of traditional approaches that search large KB graphs for statistically significant closed paths.Traditional methods prune candidate rules directly on the KB graph, whereas the proposed approach uses learned embeddings to explore the rule space.
- Rule Representation: Rules are constrained to distinct relation types, closed paths, and lengths 2 or 3 because longer rules face an exponential search space.Argument-domain information can further restrict candidate rules by limiting which entities may occupy relation positions.
- Composition and Extraction: For length-2 rules, relation composition is modeled through vector addition or matrix multiplication, then ranked by embedding similarity to the candidate head relation.Vector models use Euclidean distance, while matrix models use the Frobenius norm; the extraction algorithm also applies global and heuristic thresholds.
- Length-2 Results: EMBEDRULE using bilinear embeddings consistently outperforms AMIE on length-2 rule predictions, indicating that relation embeddings enable rule selection without entity-level inspection.Bilinear variants can recover rules that AMIE misses when relying on observed rule instantiations in the KB.
- Composition Comparison: Multiplicative composition from DISTMULT and BILINEAR generally performs better than additive composition from DISTADD because it places composed relations closer to their heads.DISTADD can assign large embedding-space distances to valid body-head compositions, causing rule-retrieval failures.
- Length-3 Results: For length-3 rules, BILINEAR consistently outperforms DISTMULT and DISTADD on the top 1K predictions, while DISTMULT-TANH-EV-INIT tends to lead as prediction volume increases.The initial length-3 rules extracted by EMBEDRULE provide good precision in general.
6 CONCLUSION
The paper presents a general framework for learning entity and relation representations in knowledge bases and evaluates embedding models on knowledge-inference tasks. It finds that a simple bilinear model can outperform state-of-the-art embedding models for Freebase link prediction and uses learned embeddings to extract logical rules.
- 6 CONCLUSION: The framework learns representations of entities and relations in knowledge bases and supports empirical comparison across knowledge-inference tasks.The paper evaluates different embedding models within this framework.
- 6 CONCLUSION: A simple bilinear formulation outperforms state-of-the-art embedding models for link prediction on Freebase.The conclusion identifies this as a central empirical result of the model comparison.
- 6 CONCLUSION: The learned embeddings are also used to extract logical rules from knowledge bases.This extends the framework beyond link prediction to examining embeddings through rule extraction.
APPENDIX · A EXAMPLES OF THE EXTRACTED HORN RULES
This appendix presents Horn rules extracted by EMBEDRULE using embeddings learned from DISTMULT-tanh-EV-init. The examples include both length-2 and length-3 compositional rules.
- A EXAMPLES OF THE EXTRACTED HORN RULES: The appendix introduces examples of length-2 Horn rules extracted by EMBEDRULE with DISTMULT-tanh-EV-init embeddings.These examples instantiate two-step relational compositions.
- A EXAMPLES OF THE EXTRACTED HORN RULES: AwardInCeremany(a, b) and CeremanyEventType(b, c) imply AwardInEventType(a, c).
- A EXAMPLES OF THE EXTRACTED HORN RULES: AtheletePlayInTeam(a, b) and TeamPlaySport(b, c) imply AtheletePlaySport(a, c).
- A EXAMPLES OF THE EXTRACTED HORN RULES: TV ProgramInTV Network(a, b) and TV NetworkServiceLanguage(b, c) imply TV ProgramLanguage(a, c).
- A EXAMPLES OF THE EXTRACTED HORN RULES: LocationInState(a, b) and StateInCountry(b, c) imply LocationInCountry(a, c).
- A EXAMPLES OF THE EXTRACTED HORN RULES: BornInLocation(a, b) and LocationInCountry(b, c) imply Nationality(a, c).
- A EXAMPLES OF THE EXTRACTED HORN RULES: The appendix also gives length-3 Horn rules extracted by EMBEDRULE with DISTMULT-tanh-EV-init embeddings.These rules compose three premises into a conclusion.
- A EXAMPLES OF THE EXTRACTED HORN RULES: SportPlayByTeam(a, b), TeamInClub(b, c), and ClubHasPlayer(c, d) imply SportPlayByAthelete(a, d).
B VISUALIZATION OF THE RELATION EMBEDDINGS
t-SNE visualization of 189 FB15k-401 relation embeddings shows that DISTMULT captures interpretable clustering structure, whereas DISTADD yields harder-to-interpret structure.
- B VISUALIZATION OF THE RELATION EMBEDDINGS: t-SNE visualization of 189 FB15k-401 relations shows DISTMULT reflecting clustering structure, including /film/release region near /film/country, while DISTADD is harder to interpret.The embeddings were learned using DISTMULT and DISTADD.