Source-linked AI summary
Poincaré Embeddings for Learning Hierarchical Representations
Maximilian Nickel, Douwe Kiela
TL;DR
Existing embedding methods can lack the capacity to represent complex symbolic data with latent hierarchies. This paper introduces Poincaré embeddings in hyperbolic space and an optimization procedure for learning them, finding strong performance across hierarchical data tasks.
Problem
Complex graph patterns can require computationally infeasible embedding dimensionality, limiting large-scale representations without information loss.
Method
Poincaré embeddings represent symbolic objects in hyperbolic space, using the Poincaré ball and Riemannian optimization to learn hierarchical representations.
Results
Poincaré embeddings outperform Euclidean embeddings on hierarchical data in representation capacity and generalization, including strong taxonomy and network results and state-of-the-art lexical-entailment performance.
Takeaways & Limitations
The results indicate that hyperbolic geometry can provide a useful structural bias for parsimonious representations of complex symbolic data with hierarchy.
Takeaways & Limitations
The study evaluates general properties of hyperbolic geometry, while application-specific models and full Riemannian optimization remain future work.
Abstract
from arXiv · showhide
Representation learning has become an invaluable approach for learning from symbolic data such as text and graphs. However, while complex symbolic datasets often exhibit a latent hierarchical structure, state-of-the-art methods typically learn embeddings in Euclidean vector spaces, which do not account for this property. For this purpose, we introduce a new approach for learning hierarchical representations of symbolic data by embedding them into hyperbolic space -- or more precisely into an n-dimensional Poincaré ball. Due to the underlying hyperbolic geometry, this allows us to learn parsimonious representations of symbolic data by simultaneously capturing hierarchy and similarity. We introduce an efficient algorithm to learn the embeddings based on Riemannian optimization and show experimentally that Poincaré embeddings outperform Euclidean embeddings significantly on data with latent hierarchies, both in terms of representation capacity and in terms of generalization ability.
1 Introduction
Embedding methods organize symbolic objects by similarity, but their representation capacity can be insufficient for complex, hierarchical datasets. The paper addresses this by using hyperbolic embeddings for latent hierarchies and evaluates them across several tasks.
- Embedding methods represent words, graph nodes, and multi-relational objects so embedding-space similarity reflects semantic or functional similarity.
- Complex graph patterns can require computationally infeasible embedding dimensionality, limiting large-scale embeddings without information loss.
- Hierarchical structure is common in complex symbolic data, including natural language, scale-free networks, and many tree-like real-world networks.
- The paper proposes Poincaré embeddings in hyperbolic space, using Riemannian optimization to compute scalable representations of hierarchical symbolic data.
- The approach is evaluated on taxonomy embedding, network link prediction, and lexical entailment.
2 Embeddings and Hyperbolic Geometry
Hyperbolic geometry provides a negatively curved space whose expanding metric and area naturally accommodate tree-like hierarchies. The paper situates this geometric property within network modeling and symbolic-data embeddings.
- Hyperbolic geometry studies spaces of constant negative curvature and has been used to model hierarchical data.
- A regular tree’s exponentially growing number of nodes across levels can be represented in two-dimensional hyperbolic space by placing levels on spheres with radii proportional to depth.
- Hyperbolic disc area and circle length grow exponentially with radius, providing the geometric capacity for tree-like structures.
- Hyperbolic spaces have been applied to greedy routing, Internet-topology embeddings, and models of heterogeneous, highly clustered networks.
- Machine-learning work has primarily used Euclidean embeddings, while related approaches include holographic and complex embeddings for knowledge-graph completion.
3 Poincaré Embeddings
Poincaré embeddings represent symbolic objects in a hyperbolic ball so that semantic similarity and latent hierarchy are captured together. The method uses Riemannian optimization while preserving valid points inside the ball.
- 3 Poincaré Embeddings: Hyperbolic space provides a structural bias for parsimonious embeddings, targeting improved generalization and reduced runtime and memory complexity.The motivation is to increase representation capacity for complex patterns without relying on computationally infeasible dimensionality.
- 3 Poincaré Embeddings: Poincaré embeddings represent symbolic objects in hyperbolic space, using the Poincaré ball to reflect similarity and latent hierarchy.The approach does not require direct access to hierarchical relationships and instead infers them fully unsupervised.
- 3 Poincaré Embeddings: The Poincaré ball is the open d-dimensional unit ball equipped with a Riemannian metric, with its boundary representing infinitely distant points.Geodesics are circles orthogonal to the boundary, including diameters.
- 3 Poincaré Embeddings: Its distance grows rapidly near the boundary, allowing roots to lie near the origin and leaves to occupy positions close to the boundary.This locality property supports continuous embeddings of hierarchies.
- 3.1 Optimization: Embeddings are learned by minimizing a problem-specific loss with stochastic Riemannian optimization, rescaling Euclidean gradients by the inverse metric tensor.Updates use a retraction and projection to maintain the unit-ball constraint; the implementation uses ε = 10^-5 for numerical stability.
4 Evaluation
The evaluation tests Poincaré embeddings on taxonomies, networks, and graded lexical entailment, comparing them primarily with Euclidean and translational baselines. Across these settings, the experiments assess representation capacity, generalization, and hierarchical relationship modeling.
- Evaluation setup: The evaluation compares Poincaré distance with Euclidean and translational scoring functions across taxonomy embedding, network link prediction, and lexical entailment tasks.Translational embeddings are excluded for undirected network edges.
- Embedding Taxonomies: The WORDNET experiments measure reconstruction capacity and link-prediction generalization using mean rank and MAP, with held-out links excluding root and leaf nodes.The transitive closure contains 82,115 nouns and 743,241 hypernymy relations, and its hierarchy must be inferred from the directed acyclic graph.
- Embedding Taxonomies: Poincaré embeddings show greatly improved taxonomy performance while using an embedding smaller by an order of magnitude, and their link-prediction results are robust to embedding dimension.The authors attribute this robustness to the structural bias of Poincaré embeddings, which could reduce overfitting on data with clear latent hierarchy.
- Embedding Taxonomies: A two-dimensional visualization of the WORDNET mammals subtree displays ground-truth is-a relations as blue edges; a d = 5 Poincaré embedding achieves mean rank 1.26 and MAP 0.927.The visualization is trained only on the mammals subtree for clarity.
- Network Embeddings: On four social collaboration networks, Poincaré embeddings perform especially well in the low-dimensional regime and outperform Euclidean embeddings.Evaluation uses randomly split train, validation, and test sets, with MAP reported for test-set link prediction and reconstruction.
- Lexical Entailment: On HYPERLEX, Poincaré embeddings clearly outperform the evaluated state-of-the-art methods for graded lexical-entailment ranking and achieve state-of-the-art WBLESS accuracy of 0.86.The embeddings were trained on WORDNET with d = 5 rather than specifically trained for lexical entailment.
5 Discussion and Future Work
Poincaré embeddings simultaneously capture similarity and hierarchy, delivering parsimonious representations and strong performance on hierarchical data. The authors identify broader applications and improved optimization as future directions.
- Parsimonious Poincaré representations support high-quality embeddings of large-scale taxonomies.
- Excellent link prediction results indicate that hyperbolic geometry provides a structural bias for embedding complex symbolic data.
- Future work includes multi-relational data, application-specific models, and full Riemannian optimization for potentially higher quality and faster convergence.