Source-linked AI summary
Towards Understanding Linear Word Analogies
Kawin Ethayarajh, David Duvenaud, Graeme Hirst
TL;DR
Word analogies can often be solved with vector arithmetic, but it is unclear why arithmetic works for nonlinear embedding models, and earlier explanations make strong assumptions. This paper formalizes conditions for linear analogies using co-occurrence shifted PMI and contextual geometry, then establishes implications for vector addition and Euclidean distance. Experiments support the assumptions and show analogy transformations correspond to different csPMI values.
Problem
It is unclear why vector arithmetic solves word analogies in nonlinear models such as SGNS, while earlier theories make strong assumptions about embedding spaces or word distributions.
Source: §1 · p. 1Method
The paper formalizes analogies as transformations and analyzes GloVe and SGNS through their implicit factorization of a word-context co-occurrence matrix.
Source: §1 · p. 1, §3.1 · p. 3Results
Under the stated conditions, a linear analogy holds iff the relevant word pairs have equal csPMI and their corresponding rows are coplanar; experiments support the assumptions and associate analogy transformations with different csPMI values.
Source: §1 · pp. 1–2Takeaways & Limitations
The theory proves the Pennington et al. (2014) conjecture, explains how SGNS addition down-weights more frequent words, and links Euclidean distance to word dissimilarity.
Source: §1 · p. 2, §6 · p. 9Infrequent word pairs have csPMI values more sensitive to small frequency changes and more reconstruction error, making analogies less likely to hold in practice.
Source: §5 · p. 9
Abstract
from arXiv · showhide
A surprising property of word vectors is that word analogies can often be solved with vector arithmetic. However, it is unclear why arithmetic operators correspond to non-linear embedding models such as skip-gram with negative sampling (SGNS). We provide a formal explanation of this phenomenon without making the strong assumptions that past theories have made about the vector space and word distribution. Our theory has several implications. Past work has conjectured that linear substructures exist in vector spaces because relations can be represented as ratios; we prove that this holds for SGNS. We provide novel justification for the addition of SGNS word vectors by showing that it automatically down-weights the more frequent word, as weighting schemes do ad hoc. Lastly, we offer an information theoretic interpretation of Euclidean distance in vector spaces, justifying its use in capturing word dissimilarity.
1 Introduction
The paper explains when word analogies can be solved by vector arithmetic in GloVe and SGNS, without the strong assumptions of earlier theories. Its account connects linear analogies to csPMI and contextual geometry, and derives implications for word-vector addition and Euclidean distance.
1 Introduction
- The paper studies GloVe and SGNS because they implicitly factorize a word-context matrix, making word-context inner products interpretable.It calls PMI(x,y) + log p(x,y) the co-occurrence shifted pointwise mutual information, or csPMI.Source: §1 · p. 1
- The authors frame analogies as transformations between words and call them linear analogies when the transformation is a displacement-vector addition.They report that experiments support their assumptions and show that analogy transformations correspond to different csPMI values.Source: §1 · pp. 1–2
- Under exact matrix reconstruction, a linear analogy holds iff all relevant word pairs have equal csPMI and the corresponding row vectors are coplanar.The paper extends the discussion to practice by explaining why analogies can hold with some reconstruction noise.Source: §1 · pp. 1–2
- The authors formally prove the Pennington et al. (2014) conjecture that analogies correspond to approximately matching ratios of conditional word probabilities.The conjecture states that p(w|a)/p(w|b) approximately matches p(w|x)/p(w|y) for every vocabulary word w.Source: §1 · p. 2
- In SGNS with no reconstruction error, adding two word vectors automatically down-weights the more frequent word, offering a justification for ad hoc weighting schemes.If the sum vector were in the vocabulary, its csPMI similarity to one component would equal the other component’s log probability plus a model-specific constant.Source: §1 · p. 2
- For SGNS and GloVe spaces with no reconstruction error, squared Euclidean distance decreases linearly with csPMI, rigorously linking vector distance to word dissimilarity.Words with greater csPMI similarity have smaller vector distance.Source: §1 · p. 2
2 Related Work
Related work connects word embeddings to co-occurrence statistics and describes two prior formal accounts of arithmetic analogies. The paper contrasts their assumptions with its focus on SGNS and GloVe factorization.
2 Related Work
- Pointwise mutual information captures how much more frequently two words co-occur than chance would predict.Source: §2 · p. 2
- Word embeddings arise either from neural networks predicting words and contexts or from low-rank approximations of word-context co-occurrence matrices.The paper situates both approaches in the distributional view that word meaning depends on the company a word keeps (Firth, 1957).Source: §2 · p. 2
- Levy and Goldberg (2014) showed that SGNS implicitly factorizes a word-context PMI matrix shifted by −logk, despite not explicitly calculating co-occurrence statistics.The SGNS objective maximizes observed word-context pair probabilities and minimizes probabilities for k sampled negative examples.Source: §2 · p. 2
- The latent variable model (Arora et al., 2016) assumes vectors arise from a discourse-vector random walk on the unit sphere and relies on isotropy, which experiments found often fails for SGNS.The account also assumes word vectors are known a priori and generated by randomly scaling uniformly sampled unit-sphere vectors.Source: §2 · p. 2
- The paraphrase model derives vector addition under a uniform word distribution, an unrealistic assumption given Zipf-distributed frequencies.Allen and Hospedales (2019) derive linear analogies from paraphrase relations with explicit error terms, but provide no systematic empirical evidence that the mapping holds in practice.Source: §2 · p. 3
3 The Structure of Word Analogies
The section formalizes linear analogies as shared vector displacements and characterizes when they hold in SGNS or GloVe spaces. It connects this structure to csPMI and explains why analogies can remain effective with reconstruction noise.
3.1 Formalizing Analogies
- Under exact reconstruction and the stated assumptions, a linear analogy holds iff its pairs share a csPMI value and satisfy contextual coplanarity and cross-pair csPMI equality.The theorem applies to SGNS or GloVe word and context spaces.Source: §3.1 · p. 4
3.2 Analogies as Parallelograms
- A shared displacement makes each two-pair analogy a parallelogram, but in higher dimensions equal side lengths alone do not guarantee one, so coplanarity is also required.With no reconstruction error, coplanarity can be expressed as a constraint on the factorized matrix rather than checked directly in the embeddings.Source: §3.2 · p. 4
3.3 Analogies in the Context Space
- With full rank and exact reconstruction, a linear analogy in word space has a corresponding context-space analogy; under the word-context assumption, their displacement vectors differ by a global scale.The paper motivates the geometric assumption by evidence that word and context spaces perform equally well on analogy tasks.Source: §3.3 · p. 5
3.4 Proof of the csPMI Theorem
- The proof translates the analogy constraints into constant csPMI conditions, with the theorem’s exact result relying on self-cooccurrence; experiments report a moderately strong correlation between csPMI and γ′.The reported Pearson correlation is r > 0.50 in normalized and unnormalized SGNS spaces.Source: §3.4 · pp. 5–6
3.5 Robustness to Noise
- In practice, analogy accuracy is higher for frequent pairs: capital-country pairs (median frequency 3436.5) reach 95.4%, versus 9.2% for currency-country pairs (median frequency 19).The paper attributes this pattern to lower reconstruction-error variance for frequent pairs.Source: §3.5 · p. 6
4 Vector Addition as a Word Analogy
The section derives consequences of the csPMI theorem for word addition, Euclidean distance, and linear analogies. It formalizes how addition favors the rarer word, relates squared distance to negative csPMI, and proves the ratio conjecture for exact SGNS factorization.
4.1 Formalizing Addition
- Corollary 1 formalizes the sum of two words under the csPMI Theorem’s assumptions, with its stated offset independent of x and y.The corollary considers the case where the sum is itself a vocabulary word.Source: §4.1 · p. 6
4.2 Automatically Weighting Words
- Addition gives the sum greater csPMI with the rarer word, so it automatically down-weights the more frequent word.This provides a theoretical justification for composing words by addition, as frequency-based weighting schemes also down-weight frequent words.Source: §4.2 · p. 7
4.3 Interpreting Euclidean Distance
- Squared Euclidean distance is a linear function of negative csPMI, and −csPMI correlates with distance at Pearson’s r > 0.50 in normalized and unnormalized SGNS.The paper interprets the relationship as closer embeddings corresponding to words with greater csPMI similarity.Source: §4.3 · p. 7
4.4 Are Relations Ratios?
- SGNS reconstruction noise is roughly normally distributed, with variance decreasing as word frequency increases.The paper notes that analogies often involve frequent pairs, where noise is often negligible.Source: Figure 2 · p. 8
- The ratio conjecture holds for exact SGNS factorization: equal linear displacements correspond to equal conditional-probability ratios for every context word.The proof assumes exact factorization and context vectors that span the embedding space.Source: §4.4 · pp. 7–8
5 Experiments
Experiments connect analogy accuracy to csPMI patterns and support the predicted relationship between word similarity and Euclidean distance. They also identify polysemy, corpus-specific pair statistics, and rare-pair noise as practical constraints on analogies.
Measuring Noise
- Noise in SGNS estimates was approximately zero-centered Gaussian, with lower variance at higher frequencies, supporting the assumptions used to explain analogy robustness.The authors link frequent pairs' often-negligible noise to analogies typically holding over frequent pairs.Source: §5 · p. 8
Estimating csPMI
- −0.70: csPMI variance correlated with analogy accuracy (Pearson correlation, statistically significant at the 1% level), while analogy types gradated with mean csPMI but not mean PMI.Similar analogy types had similar mean csPMI; the progression ran from geography to verb tense to adjectives.Source: §5 · p. 8, Table 1 · p. 9
Euclidean Distance
- 0.502: negative csPMI and squared Euclidean distance were positively correlated for SGNS word pairs; the normalized space showed a similar correlation of 0.514.Higher csPMI similarity corresponded to smaller vector distances.Source: Figure 3 · p. 8, §5 · p. 9
Unsolvability
- Polysemy can obstruct intuitive analogies because the theorem applies across all senses of a word, even when only one sense is relevant.The paper gives (open,closed)::(high,low) as an example whose words are all highly polysemous.Source: §5 · p. 9
- Intuitive analogies are not guaranteed to meet the required csPMI relationships in a training corpus, and infrequent pairs are especially sensitive to small frequency changes.Rare pairs also have more reconstruction error; currency pairs co-occurred with their country at a median frequency of 19 in Wikipedia.Source: §5 · p. 9
6 Conclusion
Under formalized conditions, analogies hold when csPMI values and contextual geometry satisfy the theorem's requirements. The paper also derives implications for ratio-based relations, vector addition, and Euclidean distance.
6 Conclusion
- Under the stated conditions, an analogy holds iff csPMI is constant across its word pairs and the corresponding words are contextually coplanar with matching cross-pair csPMI values.The conclusion states the theorem's conditions without enumerating their assumptions.Source: §6 · p. 9
- The paper formally proves the Pennington et al. (2014) conjecture as an explanation of linear analogy structure.Source: §6 · p. 9
- SGNS vector addition automatically down-weights the more frequent word, providing a theoretical justification for composing words by addition.Source: §6 · p. 9
- The paper gives the first rigorous explanation of Euclidean distance as a proxy for word dissimilarity.Source: §6 · p. 9