Source-linked AI summary
Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction
Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, Jie Wang
TL;DR
Knowledge graph embeddings are effective for link prediction, but many existing models do not model ubiquitous semantic hierarchies. HAKE maps entities into polar coordinates, using radial and angular information for hierarchy levels and same-level distinctions, and significantly outperforms prior state-of-the-art methods on benchmark link-prediction datasets.
Problem
Many knowledge graph embedding models focus on relation patterns but fail to model semantic hierarchies that commonly occur in knowledge graphs.
Method
HAKE maps entities into a polar coordinate system, using modulus for hierarchy levels and phase for distinguishing entities at the same level.
Results
HAKE significantly outperforms existing state-of-the-art methods on benchmark datasets for link prediction.
Takeaways & Limitations
HAKE can model entities at both different and same levels of semantic hierarchies.
Takeaways & Limitations
HAKE restricts relation-embedding entries to be positive while allowing entity-embedding entries to be negative.
Abstract
from arXiv · showhide
Knowledge graph embedding, which aims to represent entities and relations as low dimensional vectors (or matrices, tensors, etc.), has been shown to be a powerful technique for predicting missing links in knowledge graphs. Existing knowledge graph embedding models mainly focus on modeling relation patterns such as symmetry/antisymmetry, inversion, and composition. However, many existing approaches fail to model semantic hierarchies, which are common in real-world applications. To address this challenge, we propose a novel knowledge graph embedding model -- namely, Hierarchy-Aware Knowledge Graph Embedding (HAKE) -- which maps entities into the polar coordinate system. HAKE is inspired by the fact that concentric circles in the polar coordinate system can naturally reflect the hierarchy. Specifically, the radial coordinate aims to model entities at different levels of the hierarchy, and entities with smaller radii are expected to be at higher levels; the angular coordinate aims to distinguish entities at the same level of the hierarchy, and these entities are expected to have roughly the same radii but different angles. Experiments demonstrate that HAKE can effectively model the semantic hierarchies in knowledge graphs, and significantly outperforms existing state-of-the-art methods on benchmark datasets for the link prediction task.
1 Introduction
Knowledge graph embeddings support link prediction, but semantic hierarchies remain difficult to model automatically without additional hierarchy information. HAKE addresses this gap by representing hierarchy through radial and angular coordinates.
- Motivation: Knowledge graphs encode factual triples, while link prediction seeks to recover missing links in incomplete graphs.The task must determine both whether entities are related and which relation connects them.
- Motivation: Knowledge graph embeddings represent entities and relations as low-dimensional vectors, matrices, or tensors for efficient computation and semantic modeling.They are used to preserve semantic meaning and inherent entity and relation structures.
- Motivation: Semantic hierarchy is ubiquitous in knowledge graphs, but existing approaches often require additional data or processing to obtain hierarchy information.Examples include hypernym relations in WordNet and containment relations in Freebase.
- HAKE: HAKE maps entities into polar coordinates, using modulus for hierarchy levels and phase for distinguishing entities at the same level.Entities at higher levels are associated with smaller radii, while same-level entities can share radii but differ in angles.
2 Related Work
Prior knowledge graph embedding models use translational, bilinear, or neural-network scoring, while hierarchy-aware methods often need extra hierarchy information. HAKE instead learns hierarchy from triples and explicitly models modulus information alongside relation patterns.
- Model Category: Knowledge graph embedding models are commonly categorized as translational distance, bilinear, or neural-network based models.These categories differ in how they define scores for entity-relation triples.
- Model Category: Translational models represent relations as transformations between entities, while RotatE models relations as rotations in complex space.TransE uses h + r ≈ t, whereas RotatE uses complex Hadamard multiplication with unit-modulus relations.
- Model Category: Bilinear models use product-based score functions, with examples ranging from full-rank relation matrices in RESCAL to constrained matrices in DistMult and ANALOGY.The constraints reduce the overfitting risk associated with full-rank matrices.
- HAKE and RotatE: HAKE belongs to translational distance models but differs from RotatE by explicitly modeling modulus information to distinguish hierarchy levels.RotatE encourages linked entities to share modulus, whereas HAKE explicitly represents modulus differences.
- Hierarchy Modeling: Unlike prior hierarchy methods, HAKE targets link prediction, automatically learns hierarchy without clustering, and requires only knowledge graph triples.Its design also aims to model relation patterns such as symmetry, antisymmetry, inversion, and composition.
3 The Proposed HAKE
HAKE models semantic hierarchies by separating entities across hierarchy levels from entities at the same level. It combines modulus-based depth information with phase-based distinctions in a polar-coordinate representation.
- Entity categories: HAKE distinguishes entities at different hierarchy levels from entities at the same level.Examples include “mammal” versus “dog” and “rose” versus “peony”.
- Modulus part: The modulus part models hierarchy levels by using entity moduli as depth-like values, with smaller moduli expected at higher levels.Relation moduli act as scaling transformations between entity moduli.
- Phase part: The phase part distinguishes entities at the same hierarchy level by assigning different phases to entities with similar moduli.Phase transformations are represented by relation phase embeddings.
- Polar representation: HAKE combines modulus and phase components into polar-coordinate embeddings, where each paired modulus and phase forms a two-dimensional point.An entity embedding is the concatenation of its modulus and phase vectors.
- Combined distance: Combining both components lets HAKE model different-level and same-level entities, whereas modulus-only representations can make same-level entities indistinguishable.The model uses a weighted combination of modulus and phase distances in its score function.
4 Experiments and Analysis
HAKE is evaluated on three benchmark knowledge graphs and analyzed for its ability to represent hierarchical levels and same-level distinctions. Across benchmark results, visualizations, and ablations, the evidence supports combining modulus and phase information, with mixture bias providing an additional improvement.
- Experimental Settings: HAKE, ModE, and prior embedding models are evaluated on WN18RR, FB15k-237, and YAGO3-10, with WN18RR and FB15k-237 used as benchmark datasets because their larger parent datasets suffer from test-set leakage.The compared methods include TransE, DistMult, ComplEx, ConvE, and RotatE.
- Main Results: HAKE significantly outperforms existing state-of-the-art methods on all three datasets.ModE also outperforms TransE on all datasets and beats RotatE on FB15k-237 and YAGO3-10.
- Main Results: 0.021 higher MRR, 2.4% higher H@1, and 2.4% higher H@3 against RotatE are reported for HAKE on WN18RR.This comparison concerns the dataset’s relations that include hierarchical links alongside symmetric relations.
- Main Results: 0.050 higher MRR, 6.0% higher H@1, and 4.6% higher H@3 than RotatE are reported for HAKE on YAGO3-10.The dataset contains high relation-specific indegree, and the paper interprets its many-to-one links as reflecting semantic hierarchy.
- Analysis on Relation Embeddings: Modulus distributions separate relations connecting different hierarchy levels, while phase distributions distinguish entities at the same level.Relations with scaling values below one make head moduli smaller than tail moduli; values above one produce the converse, while same-level relations have moduli around one and phase values including π.
- Analysis on Entity Embeddings: Clear concentric circles appear in HAKE visualizations, whereas RotatE embeddings are mixed across the three WN18RR entity-pair cases.The three cases compare head entities at lower, similar, and higher semantic levels than their tails.
- Ablation Studies: 4.7% higher H@1 on YAGO3-10 is obtained when mixture bias is added, while combining modulus and phase outperforms either part alone on all datasets.The modulus-only component cannot distinguish same-level entities, whereas the phase-only version degenerates to pRotatE.
- Comparison with Hierarchy Models: .884 H@10 is achieved by HAKE on FB15k, compared with .734 for the best TKRL version, without requiring additional information.The best TKRL result is obtained by the WHE+STC version.
5 Conclusion
HAKE maps entities into the polar coordinate system to model semantic hierarchies, and experiments show it outperforms existing state-of-the-art methods on link prediction benchmarks.
- HAKE maps entities into the polar coordinate system to model semantic hierarchies.
- Experiments show that HAKE significantly outperforms several existing state-of-the-art methods on benchmark link prediction datasets.
- Further investigation shows that HAKE models entities at both different and the same levels of semantic hierarchies.
Appendix
The appendix analyzes relation patterns, negative entity embeddings, and entity-embedding moduli, and provides additional visualizations of semantic hierarchies.
- The appendix analyzes relation patterns, negative entity embeddings, and moduli of entity embeddings.
- It also provides additional visualization results on semantic hierarchies.
A. Analysis on Relation Patterns
The appendix presents propositions and proofs that HAKE can infer symmetry, antisymmetry, inversion, and composition relation patterns, alongside visual analyses of its embeddings.
- A. Analysis on Relation Patterns: HAKE can infer the symmetry and antisymmetry relation patterns.
- A. Analysis on Relation Patterns: HAKE can infer the inversion relation pattern.
- A. Analysis on Relation Patterns: HAKE can infer the composition relation pattern.
- A. Analysis on Relation Patterns: Figure 5 plots the frequency against the number of embedding coordinates with different signs for linked and unlinked entity pairs.Linked pairs are labeled when a relation connects the entities; unlinked pairs contain no triple in the train, validation, or test data.
- A. Analysis on Relation Patterns: HAKE's entity-embedding moduli are more dispersed than RotatE's, giving it greater potential to model semantic hierarchies.
B. Analysis on Negative Entity Embeddings
The negative-embedding analysis compares sign patterns in linked and unlinked entity pairs and supports using negative signs to distinguish positive from negative triples.
- Unlinked entity pairs may still contain valid triples because the knowledge graph is incomplete.
- For linked entity pairs, most embedding entries have the same sign.
- For sampled unlinked pairs, around half of the embedding entries have different signs, consistent with random initialization.
- The results support the hypothesis that negative signs in entity embeddings help distinguish positive and negative triples.
C. Analysis on Moduli of Entity Embeddings
The modulus patterns differ between RotatE and HAKE: RotatE produces uniform embedding moduli, whereas HAKE produces more dispersed moduli that may better model semantic hierarchies.
- RotatE encourages entity embeddings to have the same modulus because relations are modeled as rotations in complex space.
- HAKE produces more dispersed entity-embedding moduli than RotatE.
- The dispersed moduli give HAKE more potential to model semantic hierarchies.
D. More Results on Semantic Hierarchies
Additional WN18RR visualizations compare HAKE with RotatE across entities in different and identical hierarchies, with HAKE modeling both more effectively.
- The visualizations plot head and tail entities on 2D planes for additional WN18RR triples.
- Compared with RotatE, HAKE better models entities in different hierarchies and within the same hierarchy.
- Figure 7 visualizes several entity embeddings from the WN18RR dataset.