Source-linked AI summary
Probabilistic FastText for Multi-Sense Word Embeddings
Ben Athiwaratkun, Andrew Gordon Wilson, Anima Anandkumar
TL;DR
Word embeddings need to represent rare or unseen words while capturing uncertainty and multiple meanings. Probabilistic FastText combines Gaussian or Gaussian-mixture representations with n-gram-based means, and it outperforms FastText and dictionary-level probabilistic models across word-similarity evaluations, including sense-discrimination benchmarks.
Problem
Dictionary-level embeddings have limited ability to represent rare or out-of-vocabulary words, while existing representations do not jointly provide subword structure, uncertainty, and multiple senses.
Method
Probabilistic FastText represents words with Gaussian or Gaussian-mixture densities whose component means are sums of n-gram vectors.
Results
Across word-similarity benchmarks, PFT achieved a 3.7% average weighted improvement over FASTTEXT, a 3.1% improvement over dictionary-level density models, and a 1.0% improvement over a recent density model on SCWS.
Takeaways & Limitations
The models provide subword-aware probabilistic representations that capture multiple meanings and achieve high semantic quality for rare words.
Takeaways & Limitations
Future work must examine the trade-off among full covariance matrices, computational complexity, and performance.
Abstract
from arXiv · showhide
We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information. In particular, we represent each word with a Gaussian mixture density, where the mean of a mixture component is given by the sum of n-grams. This representation allows the model to share statistical strength across sub-word structures (e.g. Latin roots), producing accurate representations of rare, misspelt, or even unseen words. Moreover, each component of the mixture can capture a different word sense. Probabilistic FastText outperforms both FastText, which has no probabilistic model, and dictionary-level probabilistic embeddings, which do not incorporate subword structures, on several word-similarity benchmarks, including English RareWord and foreign language datasets. We also achieve state-of-art performance on benchmarks that measure ability to discern different meanings. Thus, the proposed model is the first to achieve multi-sense representations while having enriched semantics on rare words.
1 Introduction
Probabilistic FastText combines subword structure with probabilistic, multi-sense word representations. It targets rare and unseen words while improving semantic quality across word-similarity and sense-disambiguation benchmarks.
- Dictionary-based embeddings struggle with rare words, whereas character n-grams share strength across related words, misspellings, and languages.FastText can construct representations from shared roots, such as “circum” and “navigation” for “circumnavigation.”
- Probabilistic FastText represents each word with a Gaussian or Gaussian mixture whose component means decompose into n-gram vectors.PFT-G denotes the Gaussian model, PFT-GM the Gaussian mixture model, and mixture components can represent different senses.
- PFT supports uncertainty, hierarchical representations, multiple senses, and subword-aware representations for rare and out-of-vocabulary words.Its training procedure is energy-based, max-margin, efficient, and designed for the proposed probabilistic formulation.
- 3.7% average weighted improvement over FASTTEXT and 3.1% over dictionary-level density models were reported across word-similarity benchmarks.The comparisons included FASTTEXT, W2G, and W2GM.
- PFT also outperformed FASTTEXT on many foreign word-similarity datasets and achieved a 1.0% improvement over a recent density model on SCWS.SCWS evaluates the ability to separate different word meanings.
- The authors identify the work as the first multi-sense embedding approach with high semantic quality for rare words.Code and embeddings are publicly available.
2 Related Work
Related work spans vector embeddings, subword-aware embeddings, and multi-prototype representations. These approaches motivate combining flexible word structure with multiple representations of meaning.
- WORD2VEC and GLOVE established efficient vector-based word embeddings using negative sampling and co-occurrence matrix factorization.
- Subword modeling makes embeddings more flexible for unseen words and supports applications such as language modeling, named entity recognition, and machine translation.
- Multi-prototype embedding methods represent words with multiple vectors learned from clustered contexts, multiple latent representations, or non-parametric sense assignments.
3 Probabilistic FastText
Probabilistic FastText combines subword-based mean vectors with Gaussian or Gaussian-mixture word representations, allowing uncertainty and multiple senses while sharing information across related word forms. It defines density similarity through component interactions and trains representations with margin-based context ranking.
- 3.1 Probabilistic Subword Representation: Unlike deterministic FastText, PFT uses multi-prototype probabilistic embeddings and trains by maximizing similarity between word probability densities.Dictionary-based mixture components in PFT-GM reduce the constraint imposed by subword structure and promote independence for meaning discovery.
- 3 Probabilistic FastText: Probabilistic FastText represents words with Gaussian or Gaussian-mixture densities, with mixture components able to encode different word senses.PFT-G uses subword estimates for Gaussian means, while PFT-GM uses one subword-based component and dictionary-based components for other meanings.
- 3.2 Similarity Measure between Words: The expected likelihood kernel defines word energy as a density-space analogue of the vector dot product, with partial energies measuring pairwise component similarity.Figure 2 illustrates these interactions between Gaussian components of two words.
- 3.3 Training: Training raises the energy of true context pairs above negative pairs by a margin using a hinge loss optimized with Adagrad.Contexts are sampled from nearby words, while frequent words receive lower sampling probabilities through subsampling and a 3/4-power negative-sampling distribution.
- 3.3 Training: With K > 1, separate density modes can represent distinct meanings, such as “star” near celebrity-related words in one mode and constellation-related words in another.The model’s component interactions similarly identify the most similar sense pair for “rock” and “pop.”
- 3.4 Energy Simplification: The simplified energy function assumes spherical covariances, an efficiency-oriented choice that is equivalent to the general form up to an additive constant when covariances are shared.The scale of the inverse covariance term is controlled by the hyperparameter α.
4 Experiments
The experiments show that Probabilistic FastText combines subword structure with probabilistic, multi-sense representations and performs strongly across word-similarity, foreign-language, and rare-word evaluations.
- Model and evaluation: Probabilistic FastText combines density embeddings with subword structure to represent uncertainty and disentangle multiple word meanings.PFT-G uses Gaussian representations, while PFT-GM uses Gaussian mixtures whose components can capture different senses.
- Model and evaluation: The models are evaluated on English and foreign-language corpora using word-similarity benchmarks, nearest-neighbor analyses, and rare-word evaluations.The English corpus contains 3.376 billion words, while foreign-language experiments use French, German, and Italian corpora.
- Qualitative analyses: Nearest-neighbor analyses show separated senses for polysemous words in English and foreign languages, including financial versus river senses of “bank” and two meanings of Italian “piano”.Subword representations also support rare-word semantics through shared n-grams, as illustrated by related forms such as “abnormal” and “abnormality”.
- Word-similarity results: PFT-GM achieves the highest average score among competing models, outperforming FASTTEXT and dictionary-level density embeddings.The comparison uses weighted averages of Spearman correlations across word-similarity datasets.
- Word-similarity results: PFT-GM outperforms W2GM by 3.1% in 300 dimensions and 8.7% in 50 dimensions, while outperforming FASTTEXT by 3.7%.PFT-G also outperforms W2GM and FASTTEXT, with a 1.2% improvement over FASTTEXT.
- Sense-discrimination results: On SCWS, PFT-GM outperforms dictionary-based W2GM in both 50 and 300 dimensions and achieves state-of-the-art results comparable to Neelakantan et al. (2014).SCWS evaluates similarity for words with potentially many meanings using maximum-similarity scoring.
5 Numbers of Components
The paper argues that two mixture components often balance flexibility and parsimony, while still representing richer distributions and multiple meanings. Subword contributions also support shared representations for related rare words.
- Choosing the number of components: K = 2 provides a good balance between mixture flexibility and Occam’s razor because most words can be flexibly modeled with two Gaussians.Although K > 2 allows greater flexibility, prior dictionary-level mixtures with K = 3 did not improve overall word-similarity results.
- Subword contributions: The final representations of “abnormal” and “abnormality” share the same top-five n-grams, illustrating how subword structure transfers information across related words.This structure can improve representations for rare words composed of common subwords.
- Choosing the number of components: Two components can form a heavy-tailed unimodal distribution that captures one dominant meaning and relationships to many other words.This richer distribution can benefit even words with a single meaning compared with K = 1.
6 Conclusion and Future Work
The paper concludes that Probabilistic FastText combines probabilistic, multimodal, and subword representations for rare and out-of-vocabulary words. It identifies richer covariance modeling and multilingual co-training as directions for future work.
- Conclusion: Probabilistic FastText combines flexible subword structures with uncertainty-aware multimodal representations that can uncover multiple meanings.The paper presents these models as suitable for rare and out-of-vocabulary words, with interpretable and disentangled representations.
- Conclusion: The multimodal models are presented as the first multi-prototype embeddings that can handle rare words.The conclusion also reports better semantic quality than competing models on word-similarity benchmarks.
- Future work: Future work will examine the trade-off among full covariance matrices, computational complexity, and performance.The paper identifies variance-sensitive tasks such as hierarchical modeling as a potential area of impact.
- Future work: Future work will explore co-training Probabilistic FastText on many languages to help align semantics for polysemous words.The authors suggest that the multi-prototype structure may facilitate semantic alignment.