Source-linked AI summary
Low-Dimensional Hyperbolic Knowledge Graph Embeddings
Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, Christopher Ré
TL;DR
Knowledge graphs require embeddings that preserve both hierarchical structure and diverse logical relation patterns, while existing hyperbolic methods do not capture the latter. The paper introduces ATTH, which combines trainable hyperbolic curvature with reflections, rotations, and attention, achieving strong low- and high-dimensional benchmark results. Its reported scope includes standard KG link prediction benchmarks and future extensions to other tasks and geometric operations.
Problem
Existing hyperbolic KG embedding methods preserve hierarchical structure but do not account for the rich logical patterns found in knowledge-graph relations.
Method
ATTH learns relation-specific trainable curvatures and combines hyperbolic isometries, including rotations and reflections, with attention-based transformations.
Results
ATTH improves over previous Euclidean and hyperbolic methods by up to 6.1% MRR in low dimensions and reaches 49.6% MRR on WN18RR and 57.7% MRR on YAGO3-10 in high dimensions.
Takeaways & Limitations
Different hyperbolic transformations capture different logical relation patterns, while attention-based transformations generalize across multiple relations.
Takeaways & Limitations
The authors identify exploring other tasks and extending attention-based transformations to other geometric operations as future directions.
Abstract
from arXiv · showhide
Knowledge graph (KG) embeddings learn low-dimensional representations of entities and relations to predict missing facts. KGs often exhibit hierarchical and logical patterns which must be preserved in the embedding space. For hierarchical data, hyperbolic embedding methods have shown promise for high-fidelity and parsimonious representations. However, existing hyperbolic embedding methods do not account for the rich logical patterns in KGs. In this work, we introduce a class of hyperbolic KG embedding models that simultaneously capture hierarchical and logical patterns. Our approach combines hyperbolic reflections and rotations with attention to model complex relational patterns. Experimental results on standard KG benchmarks show that our method improves over previous Euclidean- and hyperbolic-based efforts by up to 6.1% in mean reciprocal rank (MRR) in low dimensions. Furthermore, we observe that different geometric transformations capture different types of relations while attention-based transformations generalize to multiple relations. In high dimensions, our approach yields new state-of-the-art MRRs of 49.6% on WN18RR and 57.7% on YAGO3-10.
1 Introduction
Knowledge graphs combine hierarchical and logical relation patterns, creating a need for compact embeddings that preserve both. ATTH addresses this with hyperbolic geometry, relation-specific transformations, attention, and strong benchmark performance.
- KGs represent factual knowledge as entity–relationship–entity triples and support applications including word sense disambiguation, question answering, and information extraction.
- Relations differ logically: “married to” is symmetric, whereas hypernym relations are not, and representing each behavior can require high-dimensional embeddings with substantial memory costs.
- Hyperbolic geometry can represent hierarchical data with low-dimensional embeddings, but prior hyperbolic methods target simpler graphs and cannot express diverse KG relationships.
- ATTH combines relation-specific curvatures, hyperbolic isometries, and attention-based combinations to preserve hierarchies and encode logical patterns such as symmetry and anti-symmetry.
- 6.1% MRR improvement over prior Euclidean and hyperbolic methods is achieved by ATTH in low dimensions across standard KG benchmarks.
- 16% to 24% relative improvement versus Euclidean baselines is observed for hierarchical relationships such as hypernym and member meronym.
- 49.6% MRR on WN18RR and 57.7% MRR on YAGO3-10 are reported as new state-of-the-art results in 500 dimensions.
2 Related Work
KG embedding research has advanced through richer spaces, operations, and neural architectures, but existing approaches retain limitations in logical expressiveness, dimensionality, or hyperbolic optimization.
- Prior KG embedding methods improve either the representation space or the geometric operation, whereas this approach advances both directions.
- Euclidean translation and tensor-factorization methods are simple and parameter-efficient but fail to encode important logical properties such as symmetry.
- RotatE captures symmetry, anti-symmetry, composition, and inversion through complex-space rotations, but complex and quaternion embeddings require very high-dimensional spaces.
- Neural-network approaches use relation-specific aggregation, convolutions, or graph attention to learn KG embeddings.
- MuRP targets hierarchical data in hyperbolic space and achieves promising results with fewer dimensions than Euclidean analogues.
- MuRP remains limited as a translation model with fixed curvature and cumbersome Riemannian optimization, while the proposed method uses hyperbolic isometries, tangent-space optimization, and trainable relation-specific curvature.
3 Problem Formulation and Background
The paper formulates KG embedding as scoring entity–relation–entity triples and introduces the hyperbolic-geometric concepts needed to represent them. Its background spans the learning objective, the Poincaré ball, tangent spaces, maps, and distance operations.
- 3 Problem Formulation and Background: The section introduces the KG embedding problem setting and provides background on hyperbolic geometry.
- 3.1 Knowledge graph embeddings: The KG embedding problem maps entities and relationships to vectors in a chosen space while preserving graph structure.
- 3.1 Knowledge graph embeddings: Training optimizes a scoring function on training triples so that test triples receive higher scores than triples absent from the graph.
- 3.2 Hyperbolic geometry: Hyperbolic geometry is non-Euclidean geometry with constant negative curvature, and the paper uses a d-dimensional Poincaré ball with curvature −c, c > 0.
- 3.2 Hyperbolic geometry: The exponential map sends tangent-space vectors to the hyperbolic ball, while the logarithmic map sends points on the ball back to the tangent space.
- 3.2 Hyperbolic geometry: At a point x, the tangent space T_x is a d-dimensional vector space containing possible path directions leaving x.
- 3.2 Hyperbolic geometry: At the origin, the exponential and logarithmic maps have closed-form expressions that support computation between tangent space and the hyperbolic manifold.
- 3.2 Hyperbolic geometry: Because ordinary vector addition is not well-defined in the ball, Möbius addition provides its hyperbolic analogue, alongside an explicit hyperbolic-distance formula.
4 Methodology
ATTH combines hyperbolic geometry with relation-specific isometries, attention, and curvature to represent both hierarchical structure and logical relation patterns in low dimensions.
- Hyperbolic embeddings: ATTH learns relation-specific curvatures so different relations can inhabit geometries ranging from tree-like to Euclidean-like.Fixing curvature rather than learning it can significantly degrade performance.
- Hyperbolic isometries: Rotations and reflections encode logical relation properties such as composition, inversion, symmetry, and anti-symmetry.Rotations can model several logical patterns, while reflections naturally represent symmetric relations.
- Hyperbolic isometries: Reflection transformations can infer reciprocal facts for symmetric relations, such as deriving Bob sibling of Alice from the reverse triple.Reflections are self-inverse because applying one twice gives the identity.
- Isometry parameterization: Givens transformations parameterize relation-specific rotations and reflections as block-diagonal matrices that preserve hyperbolic distances.These isometries can be computed in linear time in the embedding dimension.
- Attention: Attention selects or combines rotations and reflections, allowing ATTH to represent symmetric, anti-symmetric, and mixed-behavior relations.The model maps hyperbolic representations to tangent space, computes attention scores, and forms a weighted average before translation.
- The ATTH model: The ATTH architecture applies relation-specific rotations and reflections to head embeddings, combines them with hyperbolic attention, and then applies translation.Query and target tail embeddings are compared using hyperbolic distance; total parameters scale as O(|V|d), with extra cost proportional to the number of relations.
5 Experiments
Experiments evaluate ATTH across low- and high-dimensional KG link prediction settings, relation types, geometric transformations, and curvature choices. Results show that hyperbolic geometry benefits hierarchical relations in low dimensions, attention selects effective transformations for varied logical patterns, and trainable curvature supports strong high-dimensional performance.
- Experimental setup: ATTH is evaluated on WN18RR, FB15k-237, and YAGO3-10 using standard KG link prediction benchmarks.The experiments include inverse-relation augmentation and report dataset statistics in Table 1.
- Low-dimensional results: 6.1% points: ATTH improves over previous Euclidean and hyperbolic methods in MRR on YAGO3-10 at d = 32.On WN18RR, the corresponding improvement is 0.7% points; ATTH and MuRP perform similarly on FB15k-237.
- Relation-specific results: Hyperbolic embeddings perform better on hierarchical relations such as hypernym and has part, while Euclidean and hyperbolic models perform similarly on non-hierarchical relations such as verb group.Learned curvature in low dimensions correlates with estimated graph curvature, assigning more curved spaces to tree-like relations.
- Curvature: 2.3% in MRR: ATTH improves over ATTE on YAGO3-10, while fixed-curvature MuRP underperforms its Euclidean analogue MuE.These results support the importance of learning the appropriate curvature rather than fixing it.
- Geometric transformations: Symmetric relations favor reflections, anti-symmetric relations favor rotations, and attention-based transformations recover the best operation across datasets.For relations that are neither symmetric nor anti-symmetric, ATTH can outperform rotations and reflections by combining operators with attention.
- High-dimensional results: Fixed curvature degrades high-dimensional performance, whereas trainable curvature lets ROTH perform as well or better than previous methods in both dimensional regimes.Fixed curvature clusters embeddings near the ball boundary, while trainable curvature distributes points more effectively throughout the ball.
6 Conclusion
ATTH combines hyperbolic geometry with attention-based transformations to improve low-dimensional knowledge-graph representations. Trainable curvatures let it learn relationship-specific geometry, and it achieves new state-of-the-art results on WN18RR and YAGO3-10.
- ATTH leverages hyperbolic space and attention-based geometric transformations to learn improved knowledge-graph representations in low dimensions.
- Trainable hyperbolic curvatures allow ATTH to learn the appropriate geometry for each relationship and generalize across embedding dimensions.
- ATTH achieves new state-of-the-art results on WN18RR and YAGO3-10, whose real-world graphs exhibit hierarchical structures.
- Future work includes applying hyperbolic geometry to other tasks, such as hypernym detection, and extending attention-based transformations to other geometric operations.
A Appendix
The appendix supplies additional mathematical, evaluation, optimization, and comparison details for the proposed approach.
- The appendix begins with the hyperbolic analogue of addition and additional background on hyperbolic geometry.
- It then explains metrics used to determine how hierarchical a dataset is.
- The appendix concludes with experimental details, tangent-space optimization, and an additional comparison against the Dihedral model.
A.1 M¨obius addition
Möbius addition provides the hyperbolic analogue of vector addition used in the appendix. Unlike Euclidean addition, it is neither commutative nor associative and is connected to parallel transport.
- Möbius addition has a closed-form expression for combining points in hyperbolic space.
- Unlike Euclidean addition, Möbius addition is neither commutative nor associative.
- Parallel transport maps a vector from T_c_x to T_c_y while preserving its angle with the geodesic connecting x and y.
- Möbius addition provides an analogous operation through the lens of parallel transport.
A.2 Hierarchy estimates
The appendix uses curvature and the Krackhardt hierarchy score to characterize relation-level and global hierarchy. Curvature is estimated from sampled relation graphs, while the score reflects local loop structure.
- The curvature estimate captures global hierarchical behavior, whereas the Krackhardt hierarchy score captures local behavior through the number of small loops.
- Relation curvature is estimated on the undirected graph spanned by a relation using sampled triangles from its connected components.
- Triangle curvature is positive in circles, negative in trees, and zero in lines, while the midpoint of the shortest path between b and c defines the estimate.
- The full-graph curvature is a weighted average of relation curvatures, with weights based on the connected components.
- The Krackhardt score uses the directed relation graph and its adjacency matrix; fully symmetric relations score 0, while antisymmetric relations score 1.
A.3 Experimental details
The experiments tune learning rate, optimizer, negative sample size, and batch size for Euclidean and hyperbolic models, using 500 training epochs with early stopping.
- Four hyperparameters—learning rate, optimizer, negative sample size, and batch size—are searched for every Euclidean and hyperbolic model.The optimizers considered are Adam and Adagrad.
- Models are trained for 500 epochs, with early stopping after 100 epochs when validation MRR stops increasing.
- The best hyperparameters for each dataset are reported in Table 7.
A.4 Tangent space optimization
ATTH avoids Riemannian stochastic gradient descent by optimizing parameters in the tangent space and mapping them to the Poincaré ball. Its attention-based transformations outperform Dihedral across datasets.
- Tangent space optimization: ATTH uses tangent space optimization instead of Riemannian stochastic gradient descent for hyperbolic embeddings.Parameters are defined at the origin, optimized with standard Euclidean techniques, and recovered through the exponential map.
- Tangent space optimization: Tangent space optimization is exact in hyperbolic space because a global bijection exists between the tangent space and the manifold.Accordingly, the procedure does not reduce representational power.
- Tangent space optimization: Entity and relation embeddings are mapped to the Poincaré ball while trainable model parameters remain Euclidean parameters.
- Comparison to Dihedral: ATTE significantly outperforms Dihedral on all datasets when both methods combine rotations and reflections.ATTE learns attention-based transformations, whereas Dihedral uses one parameter to select the transformation.