Source-linked AI summary
Word Representations via Gaussian Embedding
Luke Vilnis, Andrew McCallum
TL;DR
Point-based word embeddings inadequately represent uncertainty and asymmetric relationships such as entailment. This paper learns word representations as Gaussian distributions and evaluates them across similarity, entailment, hierarchy, and qualitative tasks, finding effectiveness on asymmetric linguistic tasks and standard benchmarks. The approach remains limited by covariance parameterization and difficult optimization settings.
Problem
Point vectors do not naturally express uncertainty or asymmetric relationships, limiting their representation of inclusion and entailment.
Method
The paper learns word types as Gaussian distributions with learned means and variances, using energy functions and KL divergence for asymmetric comparisons.
Results
The embeddings perform effectively on an asymmetric linguistic task, standard word-similarity benchmarks, synthetic hierarchies, and qualitative evaluations.
Takeaways & Limitations
Gaussian density embeddings represent uncertainty and provide richer geometry for modeling inclusion, entailment, and asymmetric relationships.
Takeaways & Limitations
The current approach is constrained by diagonal or spherical covariances and difficult nonconvex optimization, especially for supervised hierarchies.
Abstract
from arXiv · showhide
Current work in lexical distributed representations maps each word to a point vector in low-dimensional space. Mapping instead to a density provides many interesting advantages, including better capturing uncertainty about a representation and its relationships, expressing asymmetries more naturally than dot product or cosine similarity, and enabling more expressive parameterization of decision boundaries. This paper advocates for density-based distributed embeddings and presents a method for learning representations in the space of Gaussian distributions. We compare performance on various word embedding benchmarks, investigate the ability of these embeddings to model entailment and other asymmetric relationships, and explore novel properties of the representation.
1 INTRODUCTION
The paper replaces point-based word embeddings with Gaussian densities to represent uncertainty and asymmetric relationships. It evaluates these embeddings qualitatively and quantitatively on similarity and entailment tasks.
- Motivation: Point vectors do not naturally express uncertainty or asymmetric relationships such as inclusion and entailment.Their common similarity measures also impose symmetric distance geometry through the triangle inequality.
- Approach: Gaussian embeddings learn both means and variances, representing words as densities rather than points.The Gaussian formulation provides object-specific distances and supports asymmetric KL-divergence comparisons.
- Approach: Embedding words as Gaussian potential functions creates soft regions that model uncertainty, inclusion, entailment, and richer latent-space geometry.The approach builds on earlier distribution-valued representations such as radial basis functions.
- Evaluation: Figure 1 visualizes learned diagonal variances and mean positions for words near Bach, including other composers such as Mozart.The projection uses generalized eigenvectors based on mixture means and Bach’s variance.
- Evaluation: The paper evaluates the method on qualitative and quantitative datasets, seven standard similarity tasks, unsupervised lexical entailment, and supervised asymmetric training.The evaluation includes comparisons with Skip-Gram and demonstrations of concept containment and specificity.
2 RELATED WORK
The paper situates Gaussian word embeddings within distributed semantics, probabilistic matrix factorization, tensor factorization, metric learning, and distributional inclusion work. Its distinction is to learn directly in distribution space while extending inclusion ideas to dense embeddings.
- Foundations: The approach builds on distributed and distributional semantic word-vector research and the broader representation-learning literature.This includes neural, count-based, and related embedding methods.
- Probabilistic representations: Probabilistic matrix factorization also represents rows and columns as Gaussians, but infers latent distributions from observed data using Bayesian methods.The paper instead works directly in the space of probability distributions and discriminatively trains representations.
- Related embedding methods: Tensor factorization and metric-learning methods jointly learn representations or distances but do not effectively learn a distance function for each item.Gaussian mixture models have also been applied to embeddings for document-level Fisher kernels.
- Distributional inclusion: The paper extends distributional inclusion ideas from count-based vector regions to dense, low-dimensional distributed embeddings.This preserves the use of asymmetric distances for modeling semantic inclusion.
3 BACKGROUND
The paper frames word embedding as energy-based prediction from contexts, then replaces point-vector scoring with ranking-based training for Gaussian distributions. Its contribution is a pair of energy functions for these representations.
- Embedding setup: The goal is to map word and context types to latent-space representations whose properties and relationships capture linguistic properties.The paper distinguishes word types from observed word tokens in context.
- Embedding setup: Unsupervised word-vector learning maps words to vectors so that types appearing in similar contexts receive similar representations.Training uses observed context sets associated with each word type.
- Energy-based learning: Energy-based learning trains scores for positive and negative input-output pairs using a supervision-defined loss.In prediction-based word embedding, inputs are word tokens and outputs are contexts, which may be observed or negatively sampled.
- Training objective: Unlike Skip-Gram’s dot-product scoring and binary-classification loss, the paper uses ranking loss because Gaussian energies have limited and parameter-dependent dynamic ranges.The ranking objective avoids relying on absolute energy values.
- Training objective: The contribution is a pair of energy functions for training Gaussian distributions to represent word types.These functions replace the point-vector energy used in standard prediction-based embeddings.
4 WARMUP: EMPIRICAL COVARIANCES
The warmup constructs Gaussian variances from context-vector variation around a fixed word mean. Although empirical covariances provide a simple estimator, their distributions lack desired entailment structure, motivating discriminative Gaussian training.
- Empirical construction: Empirical variances can be constructed from the context vectors associated with each word in a pretrained embedding set.The construction uses context vector sets and a context-window size.
- Empirical construction: The empirical covariance estimates context dispersion while holding the word vector fixed as the distribution mean.This makes the resulting variance a context-based estimator rather than a jointly learned parameter.
- Regularization: A positive ridge term must be added to the covariance diagonal to regularize inversion and avoid numerical problems.The regularizer is denoted δ > 0.
- Limitation and motivation: Empirical-estimator distributions do not exhibit desired Gaussian-embedding properties such as unsupervised entailment represented by ellipsoid inclusion.The paper therefore turns to discriminative embedding in Gaussian space to improve performance.
5 ENERGY-BASED LEARNING OF GAUSSIANS
The paper trains Gaussian embeddings with symmetric expected-likelihood and asymmetric KL-based energies, using closed-form computations and constraints suited to covariance geometry. These energies support similarity, entailment-oriented inclusion, and stochastic optimization with regularization.
- Training setup: Gaussian embeddings are trained to predict contexts using energy functions over positively and negatively sampled word-context pairs.The training setup selects an energy function, a max-margin loss, positive and negative pairs, and regularization for the nonconvex objective.
- Symmetric similarity: The expected-likelihood energy uses the inner product of two Gaussian densities, whose Gaussian identity yields a tractable logarithmic form.Taking logarithms improves interpretability for ranking ratios and numerical stability when probabilities become exponentially small.
- Symmetric similarity: The expected-likelihood log-energy combines mean separation under joint inverse covariance with a log-determinant term that regularizes ellipse volume.The Mahalanobis-distance term measures mean proximity, while the log-determinant term prevents reducing distance solely through covariance changes.
- Asymmetric similarity: KL divergence supplies an asymmetric energy for entailment: low KL from x to y indicates that y entails x, corresponding to soft inclusion of y within x.The model defines negative energy because KL is a distance, and interprets negative KL through expected log-likelihood ratios and Gaussian ellipsoid level sets.
- Optimization: Both Gaussian energy functions have closed-form gradients, while diagonal or spherical covariance structures make matrix inverses trivial to compute.Low-rank plus diagonal structure can further reduce storage and computation through the matrix inversion lemma.
- Optimization: Training constrains means by ℓ2 norm and covariance eigenvalues to [m, M]d, because small covariances can otherwise dominate expected-likelihood scores through the log-determinant term.For diagonal covariances, the eigenvalue constraint is enforced elementwise by clipping each diagonal entry.
6 EVALUATION
The evaluation tests Gaussian embeddings on uncertainty, entailment, synthetic hierarchies, and word similarity, comparing learned and empirical variances plus Skip-Gram baselines. Results show learned variances support asymmetric entailment, hierarchical structure, and competitive similarity performance, while covariance choice affects similarity outcomes.
- 6.1 SPECIFICITY AND UNCERTAINTY OF EMBEDDINGS: Variance broadly tracks specificity and ambiguity: specific words have smaller variance, whereas broader or polysemous words have larger variance.Examples include joviality and electroclash with smaller variance, and mix, mind, graph, and coldness with larger variance.
- 6.2 ENTAILMENT: 79.34 Average Precision and .78 best F1 are obtained by Learned (S) E KL, compared with 70.05 and .68 for Empirical (D) E KL.The reported table also gives 79.01 and .80 for Learned (D) E KL, showing that the best metric differs by covariance setting.
- 6.2 ENTAILMENT: Learned variances outperform empirical variances for KL-based entailment evaluation, while empirical variances perform poorly with KL divergence.The comparison includes diagonal and spherical covariance settings on the Baroni et al. entailment dataset.
- 6.2 ENTAILMENT: Gaussian embeddings capture all hierarchical relationships in synthetic trees, while sibling leaves are pushed into overlapping regions.The trees are embedded directly in two dimensions using KL divergence between parent and child Gaussians.
- 6.4 WORD SIMILARITY BENCHMARKS: 50-dimensional Gaussian embeddings match or beat reported 80-dimensional Skip-Gram results and the authors’ Skip-Gram implementation across seven word-similarity benchmarks.The comparison is qualified because the experiments use an older Wikipedia version and are not directly comparable to all reported numbers.
- 6.4 WORD SIMILARITY BENCHMARKS: Spherical covariances have a slight overall edge over diagonal covariances for word similarity, reversing the pattern observed in entailment experiments.Diagonal-variance embeddings benefit more than spherical ones when distribution variance is included in the similarity measure.
7 CONCLUSION AND FUTURE WORK
The paper embeds word types as Gaussian distributions, representing uncertainty and enabling richer geometry than point vectors. Future work targets more expressive covariance structures, alternative distributions, improved optimization, and applications beyond word embeddings.
- Gaussian embeddings represent words as densities over latent space, directly modeling uncertainty and enabling richer geometry.
- More expressive covariance structures could reduce axis alignment and increase model capacity and expressivity.
- Alternative distributions and multimodal representations are proposed because Gaussian density concentration can behave counterintuitively in high dimensions.
- Combining kernel methods and manifold learning with deep learning and linguistic representation learning is identified as an exciting frontier.
- Future applications include relational learning, similarity-in-context, sentence and paragraph comparison, and common-sense reasoning.