Factors Influencing the Surprising Instability of Word Embeddings

1 Introduction

• Word embeddings are dense, low-dimensional representations used in diverse NLP tasks, but their stability remains incompletely understood.Examples include text similarity, part-of-speech tagging, sentiment analysis, and machine translation.Source: §1 · p. 1
• Medium-frequency words show huge stability variance that frequency alone cannot explain, despite lower-frequency words being less stable (Figure 1).The figure measures word2vec stability across ten randomized embedding spaces trained on the PTB.Source: §1 · p. 1, Figure 1 · p. 1
• The study examines how data, algorithm, and word properties affect embedding stability and explores its effects on downstream tasks.The authors describe this as the first comprehensive study of factors behind instability.Source: §1 · p. 1

2 Related Work

• Earlier studies examined parameter effects across PPMI, word2vec, and GloVe, word2vec geometry, and how ambiguity affects embedding quality.Levy et al. (2015), Mimno and Thompson (2017), and Hellrich and Hahn (2016) are cited in these discussions.Source: §2 · pp. 1–2
• Antoniak and Mimno (2018) study document properties and embedding stability, while this paper broadens the factors considered and evaluates downstream task effects.Their work focuses on using embeddings to analyze language rather than perform tasks.Source: §2 · p. 2
• Tan et al. (2015) compare spaces through linear transformation and show that word usage differs between Twitter and Wikipedia.Source: §2 · p. 2
• Prior work validates second-order embedding properties by showing comparable task performance to typical first-order embeddings; this paper uses them to quantify stability.Newman-Griffis and Fosler-Lussier (2017) are cited for that validation.Source: §2 · p. 2

3 Defining Stability

• Stability is the percent overlap between a word’s ten nearest neighbors in two spaces, with 100% indicating agreement and 0% complete disagreement.Cosine similarity identifies neighbors, and the measure generalizes to averages over pairs of embedding spaces.Source: §3 · p. 2
• The stability metric was also explored concurrently by Antoniak and Mimno (2018).Source: §3 · p. 2
• GloVe stability stays consistent across neighbor counts within frequency buckets; ten neighbors performs approximately as well as 100 while requiring less computation.Figure 2 varies the frequency bucket and number of nearest neighbors used to calculate stability.Source: §3 · p. 3, Figure 2 · p. 3
• The authors therefore use ten nearest neighbors throughout the rest of the paper.Source: §3 · p. 3

4.1 Methodology

• The model uses ridge regression to associate word, training-data, and algorithm properties with stability measured across pairs of embedding spaces.It models 2,521 words across seven domains and three algorithms, yielding 27,794,025 training instances; the authors emphasize interpreting feature associations rather than task performance.Source: §4 · p. 3, §4.1 · pp. 3–4

4.2 Word Properties

• Word features include primary and secondary POS, POS- and WordNet-based polysemy measures, and syllable count.POS comes from the Brown corpus; Table 2 illustrates features for international across word2vec on NYT Arts and PPMI on Europarl.Source: §4.2 · p. 4, Table 2 · p. 4

4.3 Data Properties

• The seven domains range from narrow NYT sections to broader All NYT and large, single-topic Europarl, enabling comparisons across data settings.Data features include word frequencies and their difference, corpus vocabulary size and overlap, domain identity, and whether corpora share a domain; first-occurrence positions capture word2vec training order.Source: §4.3 · pp. 4–5

4.4 Algorithm Properties

• Algorithm features represent word2vec, GloVe, and PPMI, along with their parameter settings and embedding dimensions.Word2vec uses window size 5 and minimum count 5; dimensions are 50, 100, 200, 400, or 800. First-appearance percentages from multicore word2vec are approximate, though single-core results were comparable.Source: §4.4 · pp. 5–6

• The regression model achieves R²=0.301 on the training data, which the authors describe as a reasonable fit.Source: §5 · p. 6
• Removing curriculum-learning features lowers training R² from 0.301 to 0.291, while first-appearance positions are the regression model’s two highest-magnitude features.Figure 3 contrasts the relationship between starting position and stability for word2vec and order-invariant GloVe on the PTB.Source: §5 · p. 6, Figure 3 · p. 6
• POS is one of the biggest stability factors: numerals, verbs, and determiners are most stable, while punctuation, adpositions, and particles are least stable.Table 5 orders the POS categories by decreasing stability.Source: §5 · pp. 6–7, Table 5 · p. 7
• Within-domain stability is substantially higher than cross-domain stability; All NYT generalizes across NYT domains better than Europarl, but not as well as in-domain data.This holds even though Europarl is much larger than All NYT.Source: §5 · p. 7, Figure 4 · p. 7
• GloVe is the most stable algorithm, PPMI is similar, and word2vec lags; word2vec stability also varies with frequency and neighbor count.For lower-frequency words, word2vec stability rises approximately exponentially as the number of neighbors increases; for high-frequency words, the fewest and most neighbors show the greatest stability.Source: §5 · p. 7, Figure 5 · p. 7, Figure 6 · p. 8
• With all other features included, frequency contributes little: raw-frequency and frequency-removed models both score R²=0.301, while normalized frequency scores 0.303.Frequency-only models score R²=0.008 with raw frequency and 0.0059 with normalized frequency, so frequency is not negligible on its own.Source: §5 · pp. 7–8

6 Impact of Stability on Downstream Tasks

• The study uses word similarity and POS tagging because both offer intuitive word-level evaluations of stability’s downstream impact.Source: §6 · p. 8

6.1 Word Similarity

• Word-similarity absolute error decreases slightly as the two words’ combined stability increases, despite the datasets containing substantially more unstable than stable words.Figure 7 shows absolute error by word stability; pairs with words absent from the PTB are omitted.Source: §6.1 · p. 8, Figure 7 · p. 8

6.2 Part-of-Speech Tagging

• With fixed word vectors, POS tagging scores 85.0% on the development set, versus 95.5% for the model that trains its vectors; lower-stability vectors shift more during training.Figure 8 breaks down POS error and vector shift by word frequency and stability.Source: §6.2 · p. 9, Figure 8 · p. 9

• Domain and part of speech are key factors behind word-embedding instability, even among relatively high-frequency words.Source: §7 · p. 9
• Instability affects tasks using embedding-based features, but allowing embeddings to shift during training can reduce that effect.Source: §7 · p. 9
• The authors recommend GloVe or a good word2vec training-data curriculum for stable embeddings, and in-domain embeddings whenever possible.Source: §7 · p. 9
• The code used in the experiments is publicly available.Source: §7 · p. 9