Source-linked AI summary
Embedding Text in Hyperbolic Spaces
Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, George E. Dahl
TL;DR
Natural language has hierarchical structure, but prior hyperbolic embedding methods did not directly support practical parametric encoders for text. This paper introduces a constrained reparameterization for word and sentence embeddings, finding intuitive hierarchy signals and mixed downstream-task results.
Problem
The paper asks how hierarchical structure in natural language can be incorporated into unsupervised embeddings when text objects are too numerous for lookup-table representations.
Method
The paper reparameterizes hyperbolic embeddings by separately modeling direction and norm, using a sigmoid-constrained norm to train parametric text encoders with standard optimizers.
Results
Hyperbolic embeddings encode word-context frequency and phrase constituency, improving some downstream tasks but not all; phrase norms correlate 0.67 with constituency height.
Takeaways & Limitations
Hyperbolic organization appears useful for some natural-language relations and tasks, but specialized embeddings may be needed for different lexical and sentence-level objectives.
Takeaways & Limitations
The learned continuous hierarchy is difficult to interrogate because the parametric encoder covers a practically infinite set without clear edges between items.
Abstract
from arXiv · showhide
Natural language text exhibits hierarchical structure in a variety of respects. Ideally, we could incorporate our prior knowledge of this hierarchical structure into unsupervised learning algorithms that work on text data. Recent work by Nickel & Kiela (2017) proposed using hyperbolic instead of Euclidean embedding spaces to represent hierarchical data and demonstrated encouraging results when embedding graphs. In this work, we extend their method with a re-parameterization technique that allows us to learn hyperbolic embeddings of arbitrarily parameterized objects. We apply this framework to learn word and sentence embeddings in hyperbolic space in an unsupervised manner from text corpora. The resulting embeddings seem to encode certain intuitive notions of hierarchy, such as word-context frequency and phrase constituency. However, the implicit continuous hierarchy in the learned hyperbolic space makes interrogating the model's learned hierarchies more difficult than for models that learn explicit edges between items. The learned hyperbolic embeddings show improvements over Euclidean embeddings in some -- but not all -- downstream tasks, suggesting that hierarchical organization is more useful for some tasks than others.
1 Introduction
The paper investigates hyperbolic embeddings for natural language, whose implicit hierarchical structure is linked to semantic specificity and context breadth. It introduces a parameterization for trainable text encoders and evaluates the resulting word and sentence embeddings.
- Natural language contains explicit hierarchies such as WordNet and implicit hierarchies in sentences and social networks.
- Hyperbolic spaces can represent hierarchical relationships and previously improved WordNet and scientific-network embedding results over Euclidean spaces.
- The paper hypothesizes that general concepts occur in broader context ranges than specific concepts, enabling unsupervised hierarchical text embeddings.
- Parametric encoders are needed because lookup tables are impractical for the many sentences and phrases in natural language.
- The proposed direction-and-norm parameterization keeps embeddings inside the Poincaré ball, supports standard optimizers such as Adam, and matches Nickel and Kiela’s WordNet reconstruction error.
- Word embeddings improved lexical entailment but worsened word similarity, while sentence embeddings showed a 0.67 correlation between phrase norms and Penn Treebank constituency height.
2 Background – Poincar´e Embeddings
Poincaré embeddings use hyperbolic geometry to represent hierarchical structure efficiently. The method optimizes distances between similar and dissimilar objects, while a reparameterization removes projection requirements and permits common deep-learning optimizers.
- Hyperbolic space is efficient for trees because circumference and disc area grow exponentially with radius, matching exponential growth in hierarchical depth.
- Poincaré embeddings model objects as points inside the unit ball of hyperbolic space.
- Effective tree representations place roots near the origin and leaves near the boundary, where distances to other points increase sharply.
- The training loss minimizes hyperbolic distance for similar objects and maximizes it for dissimilar objects.
- Nickel and Kiela optimize with Riemannian-SGD and project embeddings back into the ball after steps that leave its boundary.
- The proposed reparameterization removes projection and allows popular deep-learning optimizers such as Adam.
3 Parametric Poincar´e Embeddings
The paper develops a re-parameterization that maps arbitrary encoder outputs into the Poincaré ball, enabling parametric hyperbolic embeddings for words and multi-word text. It combines this representation with unsupervised objectives based on co-occurrence or surrounding context.
- Parametric re-parameterization: The re-parameterization maps arbitrary encoder outputs from R^d′ into the Poincaré ball B^d and can be added to existing encoders.It computes a direction and norm from the original representation before producing the hyperbolic embedding.
- Parametric re-parameterization: A sigmoid constrains the embedding norm to (0, 1), ensuring the resulting embedding remains inside B^d.This removes the need for a projection step after gradient updates.
- Training objectives: Training minimizes hyperbolic distance for similar objects and maximizes it for dissimilar objects, using Adam once embeddings are guaranteed to lie in B^d.The objective is applied to model parameters in the encoder and re-parameterization functions.
- Word embeddings: Word embeddings are learned from a co-occurrence graph whose repeated edges preserve pair frequencies after downsampling.The graph contains word pairs occurring within a fixed context window.
- Sentence embeddings: Sentence embeddings are trained by predicting surrounding text, using a modified Skip-Thoughts objective that directly incorporates hyperbolic distance.The modification removes the decoder GRU and predicts a bag of surrounding words conditioned on the source sentence.
- Sentence embeddings: The hyperbolic sentence likelihood combines distance-based and Euclidean terms, with learned coefficients controlling their relative importance.After training, λ2 > λ1, indicating that local context receives greater weight in prediction.
4 Experiments & Results
Experiments show that the re-parameterized hyperbolic method matches the original on WordNet reconstruction while converging faster, and that hyperbolic word and sentence embeddings encode hierarchical signals with task-dependent downstream results.
- WordNet: The re-parameterized Poincaré method has comparable WordNet reconstruction error to the original and significantly outperforms Euclidean embeddings.The comparison uses WordNet’s transitive closure containing 82,114 nouns and 743,241 hypernym-hyponym edges.
- WordNet: The re-parameterized method reaches its best error around epoch 20, whereas the original reaches its best error after hundreds of epochs.The re-parameterized model uses batch size 1024 versus 50 for the original; the authors hypothesize that Adam contributes to the speed-up.
- Word Embeddings: Hyperbolic word embeddings place words occurring in fewer contexts at higher norms, corresponding to greater lexical specificity.Examples include “bulldogs” having a higher norm than “dog” and “greatest” having a higher norm than “great.”
- Word Embeddings: Poincaré embeddings outperform SGNS on lexical entailment but underperform SGNS on word similarity.The best Poincaré lexical-entailment score is 0.259, compared with 0.2795 for the Frequency Ratio baseline.
- Sentence Embeddings: Sentence and phrase embedding norms correlate with constituent-tree height, with Spearman correlation 0.671, and generally increase upward through parses.The embeddings were evaluated on gold-standard constituent parses from Section 23 of the Penn Treebank.
- Discussion: Overall, hyperbolic embeddings produce mixed downstream results: improvements on some tasks and degradation on others.The discussion concludes that different architectures may be needed for different lexical and sentence-level relations.
5 Related Work
The paper differs from prior work by learning generic word and sentence representations unsupervised, using re-parameterization to make hyperbolic distance compatible with standard deep-learning optimization.
- Prior work applied hyperbolic distance to question-answer retrieval, whereas this paper learns generic word and sentence representations from text corpora.
- Re-parameterization lets the hyperbolic distance function operate like another nonlinear deep-network layer, removing the need for Riemannian-SGD.
- Earlier hierarchical word-embedding methods include supervised approaches such as Order Embeddings, LEAR, and HyperVec, alongside unsupervised methods such as DIVE.
- Sentence-embedding research has also improved Skip-Thoughts through faster objectives, layer normalization, and discourse-based supervision.
6 Conclusion
The paper presents a re-parameterization method for Poincaré embeddings over arbitrary encoders and distance-based losses, finding intuitive hierarchy in language embeddings but mixed downstream-task gains.
- The re-parameterization method supports Poincaré embeddings on arbitrary encoder modules with arbitrary distance-based loss functions.
- On explicitly hierarchical data, the re-parameterized method achieves performance comparable to Nickel and Kiela’s original method.
- Word- and sentence-level language embeddings show intuitive notions of hierarchy, but improve performance on some rather than all downstream tasks.
- The authors identify alternative unsupervised formulations and downstream tasks as directions for future work.
A.1 WordNet Experiments
The appendix reports optimization settings for the re-parameterized Poincaré embeddings used in the WordNet experiments.
- The re-parameterized Poincaré embeddings used Adam with batch size 1024, learning rate 0.005, no burn-in period, and ten sampled negatives per positive.
- Embeddings were initialized in U[−0.001, 0.001], with norms initialized around σ(−5), and gradients clipped to norm 5.
A.2 Word Embedding Experiments
The word-embedding experiments construct a subsampled co-occurrence graph from TEXT8 and train embeddings with negative sampling and Adam.
- The TEXT8 corpus contains around 17M tokens after lowercase preprocessing, number spelling, and replacement of non-a-z characters by whitespace.
- After removing stopwords, the authors connect words appearing within five tokens and subsample frequent edges with c = 0.25.
- The word-embedding model uses Adam, batch size 512, learning rate 0.005, and 50 negative samples per training step.
- Word-embedding norms are initialized around σ(−5), and hyperparameters are tuned to maximize the stated objective.
A.3 Sentence Embedding Experiments
The experiments retained the 20,000 most frequent types, optimized with Adam, and used a local two-word context window during decoding.
- The preprocessing retained the top 20,000 most frequent types and replaced all remaining types with UNK.
- Training used Adam with batch size 64 and an exponentially decayed learning rate selected from 0.005, 0.0008, and 0.0001.
- Decoding used a local context window of K = 2 words around the target word.
- Embedding norms were initialized around σ(−2).