Source-linked AI summary
Spherical Text Embedding
Yu Meng, Jiaxin Huang, Guangyuan Wang, Chao Zhang, Honglei Zhuang, Lance Kaplan, Jiawei Han
TL;DR
Euclidean-trained text embeddings are often used with directional similarity, creating a training-use mismatch. The paper jointly learns word and paragraph embeddings with a spherical generative model, reporting state-of-the-art performance across text embedding applications.
Problem
Euclidean-trained embeddings are often used with directional similarity, creating a mismatch between how they are trained and used.
Source: §1 · p. 1Method
A two-step spherical generative model jointly learns word and paragraph embeddings, trained with Riemannian optimization.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Paragraph embeddings are directly obtained during training, while word learning exploits both word-word and word-paragraph co-occurrence statistics.
Source: §1 · p. 2The model assumes independent center-word and context-word representations, with independent generation processes for the two roles.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Unsupervised text embedding has shown great power in a wide range of NLP tasks. While text embeddings are typically learned in the Euclidean space, directional similarity is often more effective in tasks such as word similarity and document clustering, which creates a gap between the training stage and usage stage of text embedding. To close this gap, we propose a spherical generative model based on which unsupervised word and paragraph embeddings are jointly learned. To learn text embeddings in the spherical space, we develop an efficient optimization algorithm with convergence guarantee based on Riemannian optimization. Our model enjoys high efficiency and achieves state-of-the-art performances on various text embedding tasks including word similarity and document clustering.
1 Introduction
Text embeddings are often trained in Euclidean space but used with directional similarity, creating a mismatch. The paper directly learns spherical embeddings with a two-step generative model and an efficient Riemannian optimization procedure.
1 Introduction
- Euclidean training can mismatch practice because cosine similarity or normalized vectors are commonly used for word similarity and document clustering.After projection to spherical space, the original-space loss optimum may no longer be optimal.Source: §1 · p. 1
- The model directly learns spherical text embeddings by imposing unit-norm constraints, rather than normalizing Euclidean embeddings afterward.Its two-step generative process first generates a word from paragraph semantics, then surrounding words from the center word’s semantics.Source: §1 · pp. 1–2
- Joint learning uses word-word and word-paragraph co-occurrence statistics to learn word embeddings and paragraph embeddings together.Paragraph embeddings are obtained during training, and the model exploits both types of co-occurrence to learn word embeddings.Source: §1 · p. 2
- The model reports state-of-the-art performance on diverse text embedding applications, including word similarity and document clustering.The paper also reports an efficient optimization algorithm with a convergence guarantee.Source: §1 · p. 2
2 Related Work
Most unsupervised text embedding models use Euclidean space, while prior non-Euclidean work has often targeted hierarchical representations. Spherical models motivate this paper’s focus on directional similarity in general text embedding tasks.
2.1 Text Embedding
- Most unsupervised text embedding models are trained in Euclidean space, while Poincaré, Lorentz, and hyperbolic-cone models have targeted hierarchical representations.The paper positions its spherical model for general text embedding applications, including word similarity and document clustering.Source: §2.1 · p. 2
2.2 Spherical Space Models
- Spherical representations are motivated by directional-similarity tasks, with prior work reporting benefits for document clustering and neural-network optimization.The paper aims to align training and use by learning embeddings in spherical space.Source: §2.2 · p. 2
3 Spherical Text Embedding
The model generates a center word from paragraph semantics and surrounding words from the center word, with spherical directions representing textual semantics. Its objective contrasts positive context tuples with negative samples using a margin.
3.1 The Generative Model
- A center word is generated from paragraph semantics, then surrounding words from the center word’s semantics, with direction on the sphere representing textual semantics.The model treats greater directional similarity as implying higher co-occurrence probability.Source: §3.1 · p. 3
- With infinite vocabulary, the conditional word and center-word distributions take von Mises–Fisher forms centered on their respective conditioning embeddings.The stated concentration parameter is 1.Source: §3.1 · p. 3
3.2 Objective
- The objective raises the probability of a positive tuple (u, v, d) over a negative tuple using a max-margin loss.The positive tuple has v in u’s local context window in paragraph d; u′ is a randomly sampled negative word, and the loss uses margin m > 0.Source: §3.2 · pp. 3–4
- The model assumes each word’s center-word and context-word representations are independent, so their generation processes are independent.Source: §3.2 · p. 3
4 Optimization
The method constrains text embeddings to the unit sphere and uses Riemannian optimization to update them on that curved surface. Its training design combines an efficient retraction with angularly adjusted gradient steps and has a convergence guarantee.
4 Optimization
- The optimization section frames text-embedding training as constrained optimization on the unit hypersphere.Source: §4 · p. 4, §4.1 · p. 4
4.1 The Constrained Optimization Problem
- Because embeddings are constrained to the unit sphere, Euclidean SGD is unsuitable; the parameters must be updated using optimization on the curved surface.Source: §4.1 · p. 4
4.2 Preliminaries
- On the sphere, the intrinsic distance between points is the angle arccos(x⊤y), and tangent vectors provide the local space for updates.The sphere inherits its Riemannian metric from Euclidean space; the tangent hyperplane and exponential mapping describe how tangent vectors relate to points on the sphere.Source: §4.2 · p. 4
4.3 Riemannian Optimization
- Riemannian SGD updates parameters on the sphere using a tangent-space gradient, obtained by projecting the Euclidean gradient onto the tangent hyperplane.The gradient projection uses I − xx⊤.Source: §4.3 · p. 4
4.4 Training Details
- A retraction replaces the costly exponential mapping with an ambient-space addition followed by projection onto the sphere.The retraction approximates the exponential mapping with distance error O(∥z∥^2).Source: §4.4 · p. 5
- The update scales the Riemannian gradient by cosine distance so that larger angular discrepancies produce larger steps; negative samples instead use negative cosine similarity.The authors motivate the negative-sample multiplier by noting that these samples are usually semantically irrelevant and ideally orthogonal to the center word.Source: §4.4 · p. 5
- Theorem 2 guarantees almost-sure convergence to a critical point, with the Riemannian gradient converging to zero under the stated learning-rate condition.The theorem applies the update rule to L(x) when the learning rates satisfy the stated stochastic-approximation condition.Source: §4.4 · p. 6
5 Evaluation
The evaluation tests JoSE on word similarity, document clustering, document classification, and training efficiency. Across the reported comparisons, JoSE performs best on classification, remains optimal across clustering algorithms, and is described as superior on word similarity and more efficient than prior embedding systems.
5 Evaluation
- The evaluation trains models for 10 iterations with a context window of 10 and 100-dimensional embeddings; JoSE uses margin 0.15, two negative samples, and initial learning rate 0.04.Training efficiency is assessed on the latest Wikipedia dump because embeddings are building blocks for downstream tasks that must scale to large datasets.Source: §5 · p. 6
5.1 Word Similarity
- Spherical-space training is reported as essential for superior word-similarity performance across WordSim353, MEN, and SimLex999.The comparison includes Euclidean-trained Word2Vec, GloVe, fastText, and BERT, plus Poincaré GloVe.Source: §5.1 · p. 6, Table 1 · p. 6
5.2 Document Clustering
- JoSE remains optimal with either K-Means or spherical K-Means, while switching to spherical K-Means gives only a marginal boost.The result supports the importance of paragraph-feature quality for clustering on 20 Newsgroups.Source: §5.2 · p. 7, Table 2 · p. 7
5.3 Document Classification
- JoSE achieves the best k-NN classification performance on both 20 Newsgroups and movie reviews, covering topical and sentiment semantics.The evaluation uses Macro-F1 and Micro-F1 with k = 3; similar comparisons were observed for k from 1 to 10.Source: §5.3 · p. 7, Table 3 · p. 8
5.4 Training Efficiency
- JoSE has the highest training efficiency among the compared embedding models, attributed to a simple objective and inexpensive Riemannian updates.The comparison reports per-iteration training time on the latest Wikipedia dump; BERT is not specifically designed for learning text embeddings.Source: §5.4 · p. 8, Table 4 · p. 8
6 Conclusions and Future Work
The paper addresses the mismatch between Euclidean training and directional-similarity use by learning spherical word and paragraph embeddings jointly. It reports state-of-the-art results on common embedding applications and identifies extensions to other tasks and signals as future work.
6 Conclusions and Future Work
- The spherical generative model jointly learns word and paragraph embeddings, and Riemannian optimization trains them on the unit hypersphere.The paper reports state-of-the-art results on common text-embedding applications, including word similarity and document clustering, and better efficiency than previous embedding systems.Source: §6 · p. 8
- Future work could extend spherical embeddings to lexical entailment, incorporate subword information, or add norm constraints or Riemannian optimization when fine-tuning word embeddings.For lexical entailment, the paper suggests learning each word's vMF concentration parameter or designing other generative models.Source: §6 · p. 8
A Proof of Theorem 1
The proof establishes that, as vocabulary size tends to infinity, the model's conditional distributions converge to von Mises-Fisher distributions. It evaluates the normalizing integral through polar coordinates and a modified Bessel function.
A Proof of Theorem 1
- The integral is evaluated using the power-of-sine integral lemma and the modified Bessel function, yielding the stated vMF density.The derivation substitutes the integral expression into the density to obtain the limiting form.Source: A Proof of Theorem 1 · pp. 11–12
- As vocabulary size tends to infinity, both conditional distributions converge to vMF distributions with the prior embedding as mean direction and concentration 1.The theorem gives the limits for p(v | u) and p(u | d).Source: A Proof of Theorem 1 · p. 12
- The proof converts the normalization integral to polar coordinates using an orthogonal transformation whose first row is the direction vector.The transformation expresses the angle through the dot product of the unit vectors.Source: A Proof of Theorem 1 · p. 12
B Proof of Theorem 2
Under the stated stochastic-approximation learning-rate condition, the update converges almost surely to a critical point with vanishing Riemannian gradient. The proof first establishes convergence for the exponential-map update, then proves the result for the update rule in Theorem 2.
B Proof of Theorem 2
- The proof first shows that the update using the exponential map converges almost surely to a critical point with vanishing Riemannian gradient.The preliminary result is established before proving Theorem 2 for the update using a retraction.Source: B Proof of Theorem 2 · p. 13
- The proof establishes almost-sure loss convergence by showing the loss sequence is a non-negative supermartingale.The proof also bounds the Riemannian Hessian and gradient while deriving the convergence argument.Source: B Proof of Theorem 2 · p. 14
- The Riemannian gradient converges almost surely to 0 by combining its convergence with the finite accumulated squared-gradient bound.The proof obtains gradient convergence and then uses the summed bound to conclude the limit is zero.Source: B Proof of Theorem 2 · p. 14
- With ∑t ηt = ∞, the update rule for L(x) converges almost surely to a critical point, and grad L(x) converges almost surely to 0.This is Theorem 2's guarantee under the usual stochastic-approximation learning-rate condition.Source: B Proof of Theorem 2 · p. 14