PaperlayerView exact revision on arXiv

Source-linked AI summary

Poincaré GloVe: Hyperbolic Word Embeddings

Alexandru Tifrea, Gary Bécigneul, Octavian-Eugen Ganea

arXiv:1810.06546v2cs.CL

TL;DR

The paper addresses the difficulty of representing asymmetric and hierarchical word relations while preserving useful embedding operations. It adapts GloVe to products of hyperbolic spaces, connects these embeddings to Gaussian geometry, and introduces a hypernymy score. The resulting unsupervised embeddings are reported to perform competitively across similarity, analogy, and hypernymy, including a new unsupervised WBLESS accuracy state of the art.

  • Problem

    Existing point embeddings struggle with asymmetric and hierarchical word relations, while Gaussian embeddings encode generality through variance but lose point-embedding arithmetic properties.

  • Method

    The paper adapts GloVe to products of hyperbolic spaces, connects hyperbolic embeddings to Gaussian geometry, and uses that connection to define a principled hypernymy score.

  • Results

    The unsupervised model is reported as the first to simultaneously achieve state-of-the-art or near-state-of-the-art results on similarity, analogy, and hypernymy detection, with a new unsupervised WBLESS accuracy state of the art.

  • Takeaways & Limitations

    Hyperbolic Poincaré GloVe provides one unsupervised embedding framework that competitively addresses similarity, analogy, and hypernymy detection.

  • Takeaways & Limitations

    The paper identifies limitations in variance-based Gaussian hypernymy scoring and norm-based hyperbolic entailment scoring, including nonmonotonicity and dependence on non-intrinsic coordinates.

Abstract

from arXiv · show

Words are not created equal. In fact, they form an aristocratic graph with a latent hierarchical structure that the next generation of unsupervised learned word embeddings should reveal. In this paper, justified by the notion of delta-hyperbolicity or tree-likeliness of a space, we propose to embed words in a Cartesian product of hyperbolic spaces which we theoretically connect to the Gaussian word embeddings and their Fisher geometry. This connection allows us to introduce a novel principled hypernymy score for word embeddings. Moreover, we adapt the well-known Glove algorithm to learn unsupervised word embeddings in this type of Riemannian manifolds. We further explain how to solve the analogy task using the Riemannian parallel transport that generalizes vector arithmetics to this new type of geometry. Empirically, based on extensive experiments, we prove that our embeddings, trained unsupervised, are the first to simultaneously outperform strong and popular baselines on the tasks of similarity, analogy and hypernymy detection. In particular, for word hypernymy, we obtain new state-of-the-art on fully unsupervised WBLESS classification accuracy.

1 INTRODUCTION & MOTIVATION

Existing unsupervised point embeddings capture word meaning but struggle with asymmetric relations and latent hierarchy. The paper motivates hyperbolic geometry as a way to represent these structures while retaining broad embedding utility.

  • Unsupervised GloVe, Word2Vec, and FastText learn word vectors from word co-occurrence statistics for continuous NLP representations.
  • Point embeddings largely fail to reveal asymmetric word relations and latent word hierarchies, limiting automatic text understanding.
  • Gaussian embeddings encode generality or specificity through variance, but lose point-embedding arithmetic properties needed for analogy reasoning.
  • The proposed model is presented as the first word-embedding model to competitively address similarity, analogy, and hypernymy simultaneously.
  • Words are motivated as an aristocratic community whose few generic terms connect many more specific, hierarchically organized terms.

2 RELATED WORK

Related work uses supervision, density-based representations, or hyperbolic embeddings to model hierarchical word relations. The paper positions its approach as addressing limitations in the scalability, competitiveness, and evidential grounding of earlier unsupervised hyperbolic models.

  • Other supervised approaches embed trees or directed acyclic graphs by imposing partial orders, using hyperbolic geometry, or combining both.
  • Supervised methods combine external resources such as WordNet with raw text to learn embeddings that encode hypernymy or hierarchy.
  • Fully unsupervised alternatives replace point embeddings with probability densities to represent hierarchical information from text.
  • Earlier unsupervised hyperbolic models were reported as noncompetitive in high dimensions, insufficiently demonstrative for asymmetric relations, or trained on unrealistically small corpora.

3 HYPERBOLIC SPACES AND THEIR CARTESIAN PRODUCT

The paper uses the Poincaré ball and products of hyperbolic balls as word-embedding spaces, with distances defined by hyperbolic geometry. It also uses an isometry to relate the disk to the half-plane model.

  • Words are embedded in the Poincaré ball D^n = {x ∈ R^n | ∥x∥2 < 1}, one of five isometric hyperbolic models.
  • Figure 1 depicts an isometric deformation ϕ from the two-dimensional disk D^2 into the half-plane H^2.
  • A product of p hyperbolic balls (D^n)^p uses induced product geometry, combining the component-space distances.
  • The Poincaré half-plane H^2 is another hyperbolic model used by the paper, with its own hyperbolic distance function.
  • The isometry maps the disk’s vertical segment to the positive vertical axis of the half-plane, fixes (0,1), and sends the radius toward infinity.

4 ADAPTING GLOVE

GloVe learns Euclidean word representations from co-occurrence statistics by fitting biased inner products. The paper replaces this Euclidean formulation with differentiable metric distances and selects the distance transformation as a model hyperparameter.

  • GloVe learns word representations from corpus co-occurrence statistics to geometrically capture word meaning and relations.
  • The standard GloVe formulation uses target and context vectors whose biased inner products fit log co-occurrence counts.
  • The loss uses a weighting function that down-weights frequent-word signals, typically with α = 3/4 and x_m = 100.
  • Because hyperbolic spaces lack a direct Euclidean inner-product analogue, the paper rewrites GloVe using a differentiable distance and absorbs squared norms into biases.
  • The distance transformation h is a model hyperparameter; although h(x) = x^2/2 matches GloVe directly, h = cosh^2 sometimes performs better.

5 CONNECTING GAUSSIAN EMBEDDINGS & HYPERBOLIC EMBEDDINGS

The paper establishes a direct correspondence between diagonal Gaussian embeddings and products of hyperbolic spaces through the Fisher geometry of Gaussians. This connection also motivates intrinsic Riemannian optimization for learning the embeddings.

  • Gaussian–hyperbolic correspondence: The Fisher geometry of one-dimensional Gaussians is hyperbolic, linking Gaussian parameters to points in R × R∗+.A one-dimensional Gaussian N(µ, σ^2) is represented by its mean and standard deviation coordinates.
  • Gaussian–hyperbolic correspondence: For n-dimensional Gaussians with diagonal covariance matrices, the Fisher-distance construction extends componentwise across the hyperbolic product.The diagonal covariance is written as Σ = diag(σ)^2, enabling the product-space representation.
  • Gaussian–hyperbolic correspondence: Diagonal Gaussian embeddings correspond directly to the product space (H2)^n.The covariance is represented diagonally, with each Gaussian dimension associated with a two-dimensional hyperbolic component.
  • Fisher distance, KL & Gaussian embeddings: KL divergence and Fisher distance agree locally to second order through the Fisher information metric.This provides a local geometric connection between the KL-based WORD2GAUSS objective and Fisher-distance-based hyperbolic embeddings.
  • Riemannian optimization: Riemannian adaptive optimization such as RADAGRAD is used for hyperbolic representations and empirically performs better than Riemannian SGD.The approach parallels GLOVE’s use of ADAGRAD while adapting optimization to the manifold geometry.

6 ANALOGIES FOR HYPERBOLIC/GAUSSIAN EMBEDDINGS

The paper extends analogy computations from Euclidean vector arithmetic to hyperbolic and Gaussian embeddings using gyro-translations and geodesic operations. Curvature makes the two directional constructions differ, so the method selects a point between them.

  • Analogy computations: The connection between Gaussian and hyperbolic embeddings enables mathematically grounded analogy computations for Gaussian embeddings.It also supports hypernymy detection for hyperbolic embeddings using Gaussian distributions.
  • Analogy computations: Euclidean analogy solving uses vector arithmetic such as d = c + (b − a), but this construction is not intrinsic to Gaussian distributions.The paper uses the Gaussian–hyperbolic correspondence to define an intrinsic alternative.
  • Analogy computations: In the Poincaré ball, gyro-translation yields two curvature-aware analogy candidates, d1 and d2.The candidates correspond to the two Euclidean constructions, with gyration terms accounting for the geometry.
  • Analogy computations: Because curvature makes d1 and d2 differ, evaluation selects a point on the geodesic connecting them.The selected point is parameterized by t ∈ [0, 1], and t = 1/2 gives the gyro-midpoint.
  • Analogy computations: The analogy construction recovers its Euclidean counterpart when the Poincaré ball radius tends to infinity.This continuous deformation provides a geometric sanity check for the hyperbolic formulation.

7 TOWARDS A PRINCIPLED SCORE FOR ENTAILMENT/HYPERNYMY

The paper develops an isometry-aware hypernymy score by mapping hyperbolic word embeddings to Gaussian embeddings, using variance-related geometry to represent generality. It also specifies how to identify and align the relevant hyperbolic direction before scoring entailment.

  • Gaussian connection and hypernymy score: Hyperbolic embeddings are mapped to Gaussian embeddings before applying a principled is-a score for lexical entailment.The score targets relations such as is-a(dog, animal).
  • Gaussian connection and hypernymy score: Distance-based training is invariant to hyperbolic isometries, but norm-based entailment scores are not, so an isometry must be fixed first.The paper argues that the embedding parametrization must be aligned before defining non-intrinsic hypernymy scores.
  • Gaussian connection and hypernymy score: The Gaussian–hyperbolic connection assigns Gaussian variance to the positive vertical coordinate of the Fisher-geometric upper half-plane, which is isometric to the Poincaré disk.This provides the geometric basis for encoding generality through scale.
  • Isometry identification: The alignment centers embeddings using generic and specific word means, then rotates the centered configuration so the generic direction maps to (0, 1).All words are Möbius-translated by the global midpoint before rotation.
  • Isometry identification: The alignment can use either frequency-based unsupervised sets or WordNet-based weak supervision to identify generic and specific words.The unsupervised method uses frequent versus less frequent words; the weakly supervised method uses upper and lower WordNet levels.
  • Gaussian entailment score: For Gaussian embeddings, the proposed volume score removes dependence on the mean and is invariant to common covariance rescaling.The scale invariance follows from using a difference of logarithms.

8 EMBEDDING SYMBOLIC DATA IN A CONTINUOUS SPACE WITH MATCHING

The paper motivates hyperbolic embeddings by measuring the tree-likeness of co-occurrence data and constructs distances from similarity statistics. The resulting co-occurrence metric spaces exhibit very low hyperbolicity, supporting hyperbolic or product-hyperbolic representations.

  • Motivation: Selecting a word-embedding metric space is framed as choosing an inductive bias for representing symbolic co-occurrence data.The relevant symbolic input is the matrix of co-occurrence counts Xij.
  • Hyperbolicity of symbolic data: Average δ-hyperbolicity quantifies tree-likeness; low δavg indicates an approximate hyperbolic structure that is well suited to isometric embedding.The scale-invariant ratio 2 ∗δavg/davg provides a related measure of hyperbolicness.
  • Hyperbolicity of symbolic data: Because co-occurrence data provide similarities rather than symmetric distances, the paper transforms counts into distances through h(d(i, j)) := −log(Xij) + bi + bj.The choice bi := log(Xi) is used to obtain the distance construction.
  • Hyperbolicity of symbolic data: The induced co-occurrence metric spaces have very low hyperbolicity, suggesting that words should be embedded in products of hyperbolic spaces.The paper evaluates multiple choices of h in Table 1.

9 EXPERIMENTS: SIMILARITY, ANALOGY, ENTAILMENT

Across similarity, analogy, and hypernymy evaluation, the hyperbolic models generally outperform vanilla GloVe, with fully unsupervised 50x2D embeddings achieving state-of-the-art unsupervised hypernymy results.

  • Experimental setup: 1.4 billion tokens and 189,533 unique tokens define the Wikipedia corpus used to train all models.
  • Similarity: 100D and 50x2D hyperbolic models outperform vanilla GloVe in nearly every word-similarity setting.
  • Analogy: The vast majority of hyperbolic models outperform vanilla GloVe on analogy benchmarks, with 100D hyperbolic embeddings performing best overall.
  • Hypernymy: The fully unsupervised 50x2D model with h(x) = x2 sets a new state-of-the-art on unsupervised WBLESS accuracy and matches the prior best unsupervised HyperLex Spearman correlation.
  • Overall comparison: The 50x2D model with h(x) = x2 and the initialization trick achieves state-of-the-art hypernymy detection while remaining close to the best similarity and analogy models.
  • Hypernymy: A 50x2D Poincaré GloVe model with the initialization trick and 190k vocabulary reaches 0.652 accuracy under the unsupervised 5k+5k setting.

10 CONCLUSION

The paper adapts GloVe to products of hyperbolic spaces, connects them to Gaussian statistical manifolds, and reports strong unsupervised performance across similarity, analogy, and hypernymy detection.

  • The proposed hyperbolic GloVe model leverages Gaussian-geometry connections to interpret entailment relations between embeddings.
  • The model is reported as the first to simultaneously obtain state-of-the-art or close results on word similarity, analogy, and hypernymy detection.

A MORE EXPERIMENTS

The appendix evaluates the method across dimensions, product structures, vocabularies, and training functions, while documenting evaluation choices and vocabulary-related caveats.

  • More experiments: The experiments vary embedding dimensions, product structures, vocabulary sizes, and h functions to test the method under multiple settings.
  • Similarity: Similarity scores are Spearman correlations, with minus Poincaré distance used to rank neighbors.
  • Similarity: Restricted 50k-vocabulary similarity results lose benchmark word pairs and should therefore be interpreted alongside the unrestricted results.
  • Analogy: Poincaré analogy solutions use Riemannian operations, including parallel transport, rather than Euclidean 3COSADD.
  • Analogy: Restricting the vocabulary to 190k or 50k words removes some test instances from benchmark analogy datasets.
  • Analogy: The best analogy interpolation parameter was t = 0.3 in both Google analogy partitions, selected through cross-validation.

A.3 HYPERNYMY

The hypernymy analysis examines how hyperbolic embeddings map to Gaussian representations and how geometry, distance transforms, and hyperbolicity affect interpretation and performance.

  • Mapping embeddings: The figures illustrate centering, rotation, and isometric mapping from product Poincaré disks to diagonal Gaussians across ten 2D spaces.
  • Distance transforms: Embeddings trained with h = cosh2 are closer together than those trained with h = (·)2 because the inverse distance transform is smaller.
  • Hyperbolicity: Low graph hyperbolicity indicates approximately tree-like structure, whereas high hyperbolicity suggests long cycles or distortion in low-dimensional hyperbolic embeddings.
  • Hyperbolicity: The product D2 × D2 is ∞-hyperbolic under worst-case hyperbolicity, motivating average hyperbolicity as a more informative and robust measure.
  • Additional experiments: Higher powers of cosh improved most similarity benchmarks beyond WordSim but also contracted word distances, leaving their source of benefit unresolved.
Loading 1810.06546v2…