Source-linked AI summary
Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space
Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, Andrew McCallum
TL;DR
Single-vector embeddings overlook polysemy, motivating a scalable extension of Skip-gram that learns multiple word senses. The method jointly assigns senses and learns embeddings, while its non-parametric variant estimates sense counts; it achieves state-of-the-art word-similarity results and trains on nearly a billion tokens in less than 6 hours.
Problem
Most word-embedding methods use one vector per word type, ignoring polysemy and limiting representation of distinct contextual meanings.
Method
MSSG jointly clusters token contexts and learns multiple sense embeddings, while NP-MSSG creates a varying number of senses per word type online.
Results
The methods achieve new state-of-the-art results in word similarity in context and train on nearly a billion tokens in less than 6 hours.
Takeaways & Limitations
The approach provides multiple embeddings per word type with joint sense learning, non-parametric sense discovery, and scalable training.
Abstract
from arXiv · showhide
There is rising interest in vector-space word embeddings and their use in NLP, especially given recent methods for their fast estimation at very large scale. Nearly all this work, however, assumes a single vector per word type ignoring polysemy and thus jeopardizing their usefulness for downstream tasks. We present an extension to the Skip-gram model that efficiently learns multiple embeddings per word type. It differs from recent related work by jointly performing word sense discrimination and embedding learning, by non-parametrically estimating the number of senses per word type, and by its efficiency and scalability. We present new state-of-the-art results in the word similarity in context task and demonstrate its scalability by training with one machine on a corpus of nearly 1 billion tokens in less than 6 hours.
1 Introduction
Single-vector word embeddings ignore polysemy, while prior multi-sense methods rely on costly preprocessing and fixed sense counts. MSSG jointly learns sense assignments and embeddings, and NP-MSSG can vary the number of senses while scaling efficiently.
- Motivation: Single-vector representations ignore polysemy and homonymy, causing words such as plant to average distinct contextual meanings.This can pull unrelated words associated with different senses toward one another.
- Prior limitations: Prior multi-sense methods pre-cluster token contexts, relabel the corpus, and then learn embeddings in a separate second step.Pre-clustering prevents joint learning of sense-discriminated vectors and clustering.
- Prior limitations: Prior approaches also use a fixed number of senses per word type and can be computationally expensive at billion-token scale.Huang et al. took one week to learn multiple embeddings for 6,000 vocabulary words on nearly a billion tokens.
- Proposed approach: MSSG jointly learns sense-discriminated vectors and token-context assignments, using emerging sense representations to improve clustering.The method performs sense discrimination and embedding learning together during online training.
- Proposed approach: NP-MSSG non-parametrically discovers a varying number of senses per word type, while online joint training improves speed and scalability.The paper reports training on nearly a billion tokens in less than 6 hours, a 27x improvement over Huang et al.
- Results: The methods surpass previous state of the art on SCWS and WordSim353 and dramatically outperform Huang et al. on Google Analogy.The paper also compares qualitative nearest-neighbor results across single-sense, parametric, and non-parametric models.
2 Related Work
Related work established dense word embeddings and multiple representations for polysemous words, but existing multi-sense approaches typically separate context clustering from embedding learning.
- Neural embedding models: Neural language models replace n-gram conditional tables with networks that represent each word token using a small vector.This line of work estimates neural-network parameters and embeddings from data.
- Neural embedding models: Skip-gram-style log-linear models remove hidden layers, use larger context windows and aggressive subsampling, and target useful word representations for downstream tasks.Their emphasis is representation quality rather than low perplexity.
- Alternative methods: High-quality word vectors can also be estimated without neural or log-linear models, including with Canonical Correlation Analysis.Dhillon and Ungar use CCA to estimate word vector representations.
- Applications: Word embeddings and Brown-cluster features have been applied across tasks including named entity recognition, parsing, chunking, sentiment analysis, and paraphrase detection.Embeddings have also been used to represent paragraphs and documents.
- Multi-sense embeddings: Reisinger and Mooney and Huang et al. learn multiple embeddings after clustering contexts as a preprocessing step.Huang et al. add global document context and recursive neural networks for dense, low-dimensional embeddings.
3 Background: Skip-gram model
Skip-gram learns word vectors by predicting surrounding words from local contexts, using observed word-context pairs and randomly sampled noisy contexts in its training objective.
- Model: Skip-gram learns word embeddings that are useful for predicting surrounding words in a sentence.Each word w has a vector v(w) in R^d, where d is the embedding dimensionality.
- Training objective: The training data pairs each target word with observed context words and randomly sampled noisy context words.D+ contains observed word-context pairs, while D− contains pairs with sampled noisy contexts.
- Training objective: The embeddings are learned by maximizing an objective over the observed positive and sampled negative word-context pairs.The objective is defined over the training sequence w1, w2, ..., wT.
- Context construction: For each training word, the context contains up to Rt words on both the left and right, with Rt sampled uniformly from 1 through the maximum window size N.The context is ct = {w_t−R_t, ..., w_t−1, w_t+1, ..., w_t+R_t}.
- Negative sampling: Noisy context words are sampled independently for each observed context word from a distribution based on the unigram frequencies and a normalization constant.The number of noisy samples per target context is S.
4 Multi-Sense Skip-gram (MSSG) model
MSSG extends Skip-gram by maintaining multiple sense vectors and assigning each token to the closest context cluster before updating the selected sense and shared context representations.
- Joint learning: MSSG performs sense discrimination and embedding learning jointly, allowing current sense representations to participate in context clustering.This joint procedure is the central difference from previous approaches.
- Model representation: MSSG associates each word with one global vector, K sense vectors, and K context-cluster centers, all with embedding dimension d.K is a model hyperparameter.
- Sense assignment: For each token, MSSG represents its context by averaging the global vectors of the surrounding words.Global context vectors are used instead of context-word sense vectors to avoid predicting every context word’s sense.
- Sense assignment: The token’s sense is assigned to the context cluster whose center is most similar to the averaged context representation.The hard assignment resembles k-means, and experiments use cosine similarity.
- Online updates: After assigning a sense, training updates that sense vector, the global vectors of observed and noisy context words, and the selected context-cluster center.The observed and noisy pairs follow the same construction as Skip-gram.
5 Non-Parametric MSSG model (NP-MSSG)
NP-MSSG extends MSSG with online, non-parametric sense induction, learning a varying number of senses for each word from contextual similarity. It creates new senses when a context is insufficiently similar to existing cluster centers while retaining MSSG’s objective and probabilistic model.
- Non-parametric sense learning: NP-MSSG learns a varying number of senses per word type instead of using MSSG’s fixed number.The number of senses is learned during training rather than specified uniformly in advance.
- Online clustering: A new sense and context cluster are created online when a word’s context has similarity below λ to every existing cluster center.The threshold λ is a model hyperparameter governing when additional senses are introduced.
- Context representation: For each word occurrence, the context is represented by the average of its surrounding words’ global vectors.The surrounding context consists of words within the selected window around the target token.
- Sense assignment: The method selects the closest existing sense by comparing context similarity with each cluster center, or creates a new sense when none meets the threshold.The selected cluster center is computed as the average of the vector representations of contexts assigned to that cluster.
- Model relationship: NP-MSSG changes only word-sense discrimination; its objective function and probabilistic model remain those of MSSG.Thus, the non-parametric variant differs from MSSG in how senses are selected or created, not in the underlying word-context model.
6 Experiments
The experiments evaluate MSSG and NP-MSSG on nearest-neighbor coherence, WordSim-353, SCWS, and word analogy tasks. The models achieve strong similarity results, with SCWS state-of-the-art performance and substantial gains over Huang et al. on analogies.
- Experimental setup: The evaluation uses a 990-million-token Wikipedia corpus, removes words occurring fewer than 20 times, and uses a maximum context window of five words on each side.MSSG uses three senses, while NP-MSSG learns the number of senses during training.
- Efficiency: Training time improves substantially over Huang et al. while remaining within an order of magnitude of Skip-gram on similarly sized corpora and single-machine implementations.The experiments compare single-machine training times across models.
- Nearest neighbors: Table 2 reports semantically coherent nearest neighbors for discovered senses, while NP-MSSG creates a reasonable number of senses.Neighbors are computed using cosine similarity between each sense embedding and other words’ context embeddings.
- Word similarity: SCWS evaluates in-context similarity using Spearman correlation between model scores and human judgments; avgSimC weights sense-pair similarity by contextual fit.GlobalSim ignores senses, whereas localSim independently selects one context-supported sense for each word.
- Word similarity: 69.3 is MSSG’s best SCWS avgSimC score, exceeding Huang et al.’s 65.7% previous state of the art; performance also improves with larger embedding dimensions.NP-MSSG performs best for globalSim, avgSim, and localSim according to Table 4.
7 Conclusion
The paper extends Skip-gram to efficiently learn multiple embeddings per word type, jointly learning sense discrimination and embeddings while estimating the number of senses non-parametrically. It reports state-of-the-art word similarity in context results and trains on nearly 1 billion tokens in less than 6 hours.
- MSSG jointly learns word-sense discrimination and multiple embeddings for each word type.
- The non-parametric method automatically estimates a varying number of senses per word type.
- The approach achieves new state-of-the-art results on word similarity in context.
- Nearly 1 billion tokens are processed in less than 6 hours on one machine.