Source-linked AI summary
A Mixture Model for Learning Multi-Sense Word Embeddings
Dai Quoc Nguyen, Dat Quoc Nguyen, Ashutosh Modi, Stefan Thater, Manfred Pinkal
TL;DR
Word embeddings often overlook lexical ambiguity, motivating representations that account for multiple word senses. The paper proposes MSWE, which learns context-dependent mixtures of sense embeddings with induced weights. MSWE outperforms Word2Vec and achieves highly competitive standard-task results, although comparisons are limited by smaller training corpora and differing preprocessing.
Problem
Word embedding models usually represent an ambiguous word with one vector, while prior multi-sense methods assign equal weights to its senses.
Method
MSWE jointly learns word and sense embeddings, using topic-model probabilities to induce context-dependent mixture weights over topical senses.
Results
MSWE outperforms Word2Vec and produces highly competitive results on standard evaluation tasks, including word similarity benchmarks.
Takeaways & Limitations
Inducing mixture weights lets the model represent word meaning through multiple senses rather than a single fixed representation.
Takeaways & Limitations
Comparisons are incomplete because many prior models use much larger training corpora, and preprocessing differences may affect results.
Abstract
from arXiv · showhide
Word embeddings are now a standard technique for inducing meaning representations for words. For getting good representations, it is important to take into account different senses of a word. In this paper, we propose a mixture model for learning multi-sense word embeddings. Our model generalizes the previous works in that it allows to induce different weights of different senses of a word. The experimental results show that our model outperforms previous models on standard evaluation tasks.
1 Introduction
Word embeddings capture syntactic and semantic properties but typically represent ambiguous words with a single vector. MSWE addresses this by learning context-sensitive mixtures of sense representations with induced sense weights.
- Previous multi-sense methods commonly assign the same weight to every sense, whereas this model reflects each sense’s association degree with a context.
- The model uses topic-model topic-to-word and document-to-topic probabilities to infer sense weights and compose target-word representations for context prediction.
- MSWE considers all possible meanings of a word rather than relying only on the most suitable topic in context.
- MSWE introduces a mixture model that jointly learns word and sense embeddings while inducing different weights for a word’s senses.
- MSWE outperforms Word2Vec Skip-gram and other embedding models on word analogy and word similarity tasks.
2 The mixture model
The mixture model treats topics as word senses and represents each word through weighted topical representations. It defines variants that select the most suitable topic or marginalize over all senses, then trains them to predict context words.
- Topics serve as senses, and each word is represented by a mixture of its topical representations.
- Topic modeling supplies topic-to-word and document-to-topic probabilities, which are combined to infer each target word’s topic weights.
- MSWE-1 uses the most suitable topic for a word in a document, whereas MSWE-2 marginalizes over all possible senses.
- The model learns representations by minimizing negative log-likelihood for target-context word prediction, using a context window around each target word.
- Negative sampling approximates the expensive softmax log-probability during training, and the models are optimized with stochastic gradient descent.
3 Experiments
MSWE is evaluated on word similarity and analogy benchmarks against Word2Vec Skip-gram and prior embedding models. It improves standard similarity results and analogy accuracy, while contextual similarity remains mixed against specialized baselines.
- 3 Experiments: Comparisons are constrained because many prior studies use larger training corpora or different preprocessing, and the authors note that larger corpora could improve results.
- 3.1 Experimental Setup: The experiments evaluate word similarity through Spearman’s rank correlation and word analogy through exact-match accuracy on benchmark datasets.Similarity evaluations use GlobalSim, AvgSim, and AvgSimC; the analogy benchmark contains 8,869 semantic and 10,675 syntactic questions grouped into 14 categories.
- 3.2 Word Similarity: 2.92 Spearman points, or about 8.5% relative improvement, is achieved by MSWE over Word2Vec Skip-gram on the RW similarity dataset.
- 3.2 Word Similarity: MSWE obtains the highest published results on RW, SCWS, WS353, and MEN, and the second-highest result on SIMLEX using GlobalSim.
- 3.2.2 Results for contextual word similarity: On contextual SCWS similarity, MSWE outperforms Cheng et al. (2015) but is outperformed by Neelakantan et al. (2014) and Chen et al. (2014) with AvgSim and AvgSimC.MSWE uses a fixed number of topics as word senses, whereas the cited competing methods induce or initialize senses differently.
- 3.3 Word Analogy: 69.7% analogy accuracy is reached by MSWE, compared with 68.6% for Word2Vec Skip-gram.Table 4 reports that all MSWE results are significantly higher than Word2Vec Skip-gram, with two-tail p < 0.001 using McNemar’s test.
4 Conclusions
The paper concludes that MSWE represents words in context as mixtures of sense embeddings with induced mixture weights. It reports better scores than Word2Vec and highly competitive results on standard evaluation tasks.
- MSWE represents a word in context as a mixture of its sense representations, with induced mixture weights.
- The model scores better than Word2Vec on the reported evaluation tasks.
- MSWE produces highly competitive results on standard evaluation tasks.
- Future work will improve contextual-information modeling and explore alternative ways to compute or learn mixture weights.