Source-linked AI summary
Breaking Sticks and Ambiguities with Adaptive Skip-gram
Sergey Bartunov, Dmitry Kondrashkin, Anton Osokin, Dmitry Vetrov
TL;DR
Skip-gram and related word-representation methods generally use one representation per word, leaving ambiguity and the required number of meanings unresolved. AdaGram adds a Bayesian nonparametric, context-dependent prototype model with online variational learning, and it consistently outperforms the compared approaches on the reported word-sense-induction datasets. Its semantic resolution is controlled by α, while its practical scope includes assumptions such as a constant context size for non-boundary words.
Problem
Existing word-representation methods commonly use one representation per word, while multi-prototype alternatives may require a fixed number of meanings or heuristic allocation.
Method
AdaGram is a Bayesian nonparametric extension of Skip-gram that uses context-dependent prototypes and online variational inference to learn word meanings.
Results
AdaGram consistently outperforms the compared approaches on all reported word-sense-induction datasets and significantly improves performance on the WWSI test set.
Takeaways & Limitations
The resolution parameter α controls prototype granularity, with α = 0.1 −0.2 generally suitable for practical use and T = 30 sufficient in the reported setting.
Takeaways & Limitations
The Skip-gram formulation assumes a constant context size for all non-boundary words.
Abstract
from arXiv · showhide
Recently proposed Skip-gram model is a powerful method for learning high-dimensional word representations that capture rich semantic relationships between words. However, Skip-gram as well as most prior work on learning word representations does not take into account word ambiguity and maintain only single representation per word. Although a number of Skip-gram modifications were proposed to overcome this limitation and learn multi-prototype word representations, they either require a known number of word meanings or learn them using greedy heuristic approaches. In this paper we propose the Adaptive Skip-gram model which is a nonparametric Bayesian extension of Skip-gram capable to automatically learn the required number of representations for all words at desired semantic resolution. We derive efficient online variational learning algorithm for the model and empirically demonstrate its efficiency on word-sense induction task.
1 INTRODUCTION
Word representations capture useful semantic and linguistic information, but standard methods usually assign one representation per word and therefore ignore ambiguity. AdaGram extends Skip-gram to learn multiple representations and automatically determine how many are needed.
- Motivation: Continuous-valued word representations support NLP applications by serving as features and capturing semantic relationships between concepts.CBOW and Skip-gram additionally provide high-dimensional representations, linguistic regularities, computational efficiency, and online processing.
- Motivation: Standard word-representation methods usually ignore ambiguity, although a word can have multiple meanings.The passage illustrates this issue with “apple,” which may refer to a fruit.
- Contribution: AdaGram extends Skip-gram with multiple prototypes that can automatically learn the required number of representations for each word.The model is designed to retain fast online learning and high-quality representations while addressing word ambiguity.
- Contribution: Multi-prototype representations could improve representation-based applications such as dependency parsing, named-entity recognition, and sentiment analysis.The passage links these applications to the ability to handle word ambiguity.
2 SKIP-GRAM MODEL
Skip-gram learns word representations by predicting context words from input words, with training performed online through stochastic updates. Hierarchical soft-max replaces the expensive vocabulary-wide soft-max computation with a binary-tree formulation.
- Skip-gram objective: Skip-gram groups prediction tasks that estimate context words from corresponding input words.Training examples pair an input word with the words within a specified context window.
- Skip-gram objective: The input text is treated as a sequence of training objects, each containing a word and its context words.The context consists of nearby positions within the window, excluding the input position.
- Training: Skip-gram optimizes its context likelihood stochastically by sampling words and contexts, estimating gradients, and updating parameters.Sentence and document boundaries may be ignored so the input is processed as a word stream.
- Representations: The original model uses one input representation per word, treating those representations as features that capture semantic similarity.The single-representation design motivates later extensions for ambiguity.
- Assumption: For notational simplicity, the context size is assumed constant for non-boundary words.This is an explicit modeling assumption rather than a property of all text positions.
- Hierarchical soft-max: Hierarchical soft-max replaces vocabulary-wide soft-max with predictions along paths in a binary tree.Output representations are associated with tree nodes, and Huffman coding is used for computational efficiency.
- Hierarchical soft-max: The model uses a binary tree whose leaves represent dictionary words and whose paths encode left- or right-child decisions.The resulting probabilities form a valid distribution over words.
3 ADAPTIVE SKIP-GRAM
AdaGram extends Skip-gram with Bayesian nonparametric prototypes whose number adapts to each word and whose assignments depend on context. Variational inference and stochastic updates make this infinite-capacity formulation tractable for online training and meaning inference.
- Model motivation: AdaGram addresses the inadequacy of one prototype per word by automatically allocating a word-specific number of prototypes.The model uses a Bayesian nonparametric approach instead of requiring the correct number of meanings in advance.
- Model structure: The model predicts context words using the meaning-dependent prototype of the input word, while context representations remain meaning-independent.Making context words meaning-aware is described as possible but substantially more complicated.
- Nonparametric prior: AdaGram permits infinitely many potential prototypes, while finite data limit the number of prototypes with non-zero prior probability to the word’s occurrences.The concentration parameter α controls the prior allocation of prototypes.
- Nonparametric prior: Larger α values produce more granular meanings, and the expected number of prototypes grows proportionally to α log(n_w).The model can discover more distinctive meanings as larger text corpora provide more data.
- Variational inference: The AdaGram objective is trained through a variational lower bound because latent assignments and infinite-dimensional parameters make direct marginal-likelihood optimization intractable.The variational approximation is truncated to T representations per word.
- Variational inference: Variational updates use expected sense-assignment counts as sufficient statistics for the posterior parameters.The natural parameters are tied to expected assignments n_wk.
- Online learning: Stochastic variational inference preserves online efficiency by updating global posterior parameters from individual training data points.The model separates global parameters from local sense assignments.
- Online learning: A conservative initialization starts each word with one allocated meaning, while additional meanings can be introduced during training.Representations are initialized with random values, and updates modify the posterior counts.
4 RELATED WORK
Prior multi-prototype word-representation methods either rely on context clustering, external sense inventories, or heuristic procedures for discovering meanings. AdaGram is positioned against these approaches as an automatically adaptive alternative.
- Earlier neural methods learn multiple word prototypes by clustering contexts as a preprocessing or intermediate step.This clustering must be applied across a large number of words.
- WordNet-based methods use external lexical senses to learn multi-prototype representations, limiting applicability where high-quality linguistic resources are unavailable.The described approach first pre-trains single-prototype Skip-gram representations, then uses WordNet meanings for ambiguous words.
- NP MSSG automatically discovers multiple meanings but uses an ad-hoc greedy learning procedure, unlike AdaGram’s Bayesian nonparametric approach.AdaGram is presented as an alternative to methods that require a predefined number of prototypes or rely on greedy allocation.
5 EXPERIMENTS
The experiments evaluate AdaGram’s learned prototypes, semantic resolution, predictive likelihood, word-sense induction, and web-search diversification. Across these tests, the model produces interpretable meaning representations and improves over competing approaches in the reported evaluations.
- Experimental setup: AdaGram was trained on nearly 2 million Wikipedia articles with 990 million tokens, using context width C = 10, truncation T = 30, and 300-dimensional representations.Words occurring fewer than 20 times were excluded.
- Nearest neighbours of learned prototypes: The model learns interpretable prototypes whose predictive probabilities reflect how frequently each meaning occurs in the training corpus.Nearest neighbours were computed using cosine similarity between meaning prototypes.
- Semantic resolution: Larger α values generally produce more induced meanings, while α = 0.1 was reported as most interpretable for most words.For “light,” increasing α progresses from physical and military meanings toward narrower meanings such as light tanks and light troops.
- Word prediction: AdaGram models Wikipedia text better than original Skip-gram by average test log-likelihood across a wide range of α values.The comparison used the last 200 megabytes of a December 2014 English Wikipedia snapshot, with contexts of size C = 10.
- Word prediction: AdaGram with α = 0.15 outperforms 600-dimensional Skip-gram despite the parameter-matched comparison, and α = 0.1 also outperforms that baseline.The α = 0.15 model has approximately two learned prototypes per word on average.
- Word-sense induction and web search: AdaGram consistently outperforms competing approaches on the reported word-sense induction datasets and achieves significant improvement on the WWSI test set.The evaluation compares AdaGram with MSSG and NP-MSSG using Adjusted Rand Index; AdaGram also yields higher curves than concurrent models on the SemEval-2013 diversification task.
6 CONCLUSION
AdaGram extends Skip-gram with Bayesian nonparametric prototype allocation, allowing context-dependent representations that capture word ambiguity. Its learned prototypes correspond to word meanings, with semantic resolution controlled by α and scalable online variational training.
- AdaGram is a Bayesian nonparametric extension of Skip-gram that uses different prototypes for a word depending on context.
- Learned representations correspond to different word meanings, and α controls how many prototypes are extracted from the same corpus.
- Online variational learning makes AdaGram scalable, requiring only several times the training time of Skip-gram.
- The model was evaluated on several word-sense induction datasets, including a new large dataset built automatically from Wikipedia disambiguation pages.
7 APPENDIX
The appendix reports evaluation metrics, dataset-construction details, and implementation procedures for AdaGram. It also documents the WWSI dataset and the limitations of V-Measure and F-Score.
- Algorithm 1 Training AdaGram model: AdaGram training alternates local variational updates over prototype assignments with global parameter updates using stochastic learning rates.
- Metrics: V-Measure and F-Score are reported for comparability, although V-Measure favors many clusters and F-Score favors few clusters.
- WWSI dataset construction: WWSI senses are constructed from Wikipedia disambiguation pages after filtering categories and words lacking suitable semantic content.
- Prototype inspection: Table 4 lists nearest neighbors and predictive probabilities for meaning prototypes learned with α = 0.1.
8 EXPERIMENTS ON CONTEXTUAL WORD SIMILARITY
The contextual similarity evaluation uses SCWS and compares context-sensitive prototype-selection strategies across multi-prototype and single-prototype models. The appendix also identifies the reported WSI metrics and contextual-similarity result table.
- SCWS contains 2003 word pairs, each with 10 human semantic-similarity judgments evaluated using Spearman’s rank correlation.
- MaxSimC compares the cosine similarity of the most probable prototypes selected for each word’s context.
- The comparison includes NP-MSSG, MSSG, MPSG, and original Skip-gram alongside AdaGram.
- For original Skip-gram, AvgSimC and MaxSimC coincide because it learns only one representation per word.
- Table 8 reports Spearman’s rank correlation results for contextual similarity on SCWS, with numbers multiplied by 100.