Source-linked AI summary
Analogies Explained: Towards Understanding Word Embeddings
Carl Allen, Timothy Hospedales
TL;DR
Word embeddings display linear analogy structure despite being trained from pairwise co-occurrence data, and existing explanations rely on assumptions that fail in practice. The paper defines paraphrasing probabilistically, recasts it as word transformation, and proves that these relationships yield linear structure in PMI-derived embeddings, including W2V and Glove, subject to explicit error terms.
Problem
The paper addresses the lack of a rigorous explanation for why analogous word pairs exhibit linear relationships in embeddings trained from pairwise co-occurrence data.
Method
The paper defines paraphrasing through induced context-word distributions, interprets paraphrases as additive word transformations, and analyses analogies through shared transformations.
Results
The paper proves that paraphrase relationships produce linear relationships in PMI-factorising embeddings and extends the result to embeddings that approximate PMI factorisation, including W2V and Glove.
Takeaways & Limitations
Word transformations provide a mathematical interpretation of analogies and connect semantic relationships to properties of word embeddings.
Takeaways & Limitations
The results are subject to statistical dependence and can produce false positives when error terms cancel or project into a lower-rank embedding’s null space.
Abstract
from arXiv · showhide
Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing that we re-interpret as word transformation, a mathematical description of "$w_x$ is to $w_y$". From these concepts we prove existence of linear relationships between W2V-type embeddings that underlie the analogical phenomenon, identifying explicit error terms.
1. Introduction
Neural word embeddings such as W2V and Glove often exhibit linear relationships that solve analogies, despite training based only on pairwise co-occurrence data. The paper explains these relationships through probabilistic paraphrasing, word transformations, and PMI factorisation.
- W2V and Glove are prominent neural embedding methods for representing words in low-dimensional spaces.
- Word embeddings often satisfy wa∗−wa + wb ≈ wb∗, enabling analogy questions to be solved by vector addition and subtraction.
- This analogical structure is surprising because the embeddings are trained from pairwise word co-occurrence data rather than higher-order relationships.
- The paper defines paraphrasing probabilistically, interprets it as additive word transformation, and uses shared transformations to formalise analogies.
- The contributions include rigorous linear-relationship proofs with explicit error terms and extensions from PMI vectors to embeddings approximating PMI factorisation.
2. Previous Work
Prior explanations of embedding analogies rely on assumptions that can fail in practice. The paper positions its contribution as the first rigorous mathematical account of why analogous word pairs exhibit linear embedding relationships.
- Earlier explanations proposed latent-variable or co-occurrence-based accounts of linear analogies, but imposed strong assumptions about word vectors or distributions.
- Arora et al. assume equal W2V embedding matrices, while the paper states that this equality is false in practice.
- Gittens et al. assume uniform word frequencies, conditional-distribution learning, and zero higher-order joint probabilities, which the paper says fail in practice.
- The paper states that no previous work rigorously explains why analogous word pairs produce linear relationships between corresponding embeddings.
3. Background
The paper relates embedding objectives to co-occurrence statistics and argues that PMI matrix factorisation underlies linear analogical structure, with Glove offering greater flexibility than W2V.
- W2V constructs target-context word pairs from a text corpus using words within a symmetric context window.
- Glove shares W2V’s architecture, has comparably performing embeddings, and also exhibits linear analogical structure.
- Glove’s biases give its loss function greater flexibility than W2V and potentially a wider range of solutions.
- The paper argues that PMI matrix factorisation causes linear analogical structure in embeddings, as approximately achieved by W2V.
4. Preliminaries
The paper treats PMI factorisation, reconstruction error, and projection assumptions as the basis for relating co-occurrence statistics to word embeddings. It also removes the arbitrary W2V shift and states practical conditions for the analysis.
- Impact of the Shift: The W2V shift −log k is arbitrary because k is a chosen hyper-parameter unrelated to word properties, and it can materially affect embeddings.The paper therefore focuses on embeddings that factorise the unshifted PMI matrix.
- Reconstruction Error: Exact PMI factorisation is unavailable in practice because W⊤C is rank-constrained relative to the factored matrix, creating reconstruction error.The analysis assumes these linear relationships remain sufficiently preserved after projection into the lower-dimensional embedding space.
- Assumptions: The analysis assumes C has full row rank and that relevant probabilities are non-zero, ensuring PMI values are well-defined.It also notes that rare-word zero co-occurrences can make empirical PMI undefined, although larger corpora or context windows reduce these zeros.
- Reconstruction Error: The projection from PMI columns to embeddings is assumed approximately additive, so sums of PMI columns remain approximately sums of their corresponding embeddings.A least-squares projection satisfies this additivity exactly under the stated full-row-rank condition.
5. Paraphrases
The paper defines paraphrasing probabilistically through similarity between induced context-word distributions, then connects this relation to sums of word embeddings. The resulting theorem identifies paraphrase and dependence errors that govern when embedding addition is accurate, while also describing false positives and W2V-specific loss of accuracy.
- Defining a Paraphrase: A word w∗ paraphrases a set W when it is semantically interchangeable with W and induces a similar distribution over context words.The paper measures this relationship through the conditional distributions p(c_j|W) and p(c_j|w∗), with KL-based paraphrase error.
- Paraphrase = Embedding Sum + Error: The paraphrase theorem decomposes the difference between an embedding of w∗ and the sum of embeddings for W into paraphrase and dependence errors.The dependence component reflects conditional and mutual dependencies among words in W, rather than the relationship between W and w∗.
- Dependence Error: The dependence error vanishes exactly when the words in W are mutually independent and conditionally independent given every context word.The scalar term τ_W measures mutual independence, while σ_W captures conditional dependencies.
- Paraphrase = Embedding Sum + Error: Embedding addition identifies a paraphrase when w∗ paraphrases W and the words in W are materially independent, so the net dependence error is small.For example, the framework explains why the sum of embeddings for man and royal can approximate the embedding for king.
- Do Linear Relationships Identify Paraphrases?: The converse fails: a small embedding difference does not imply paraphrasing because errors may cancel or be projected into the null space of C†.The projection-based false-positive mechanism applies to lower-rank embeddings, whereas cancellation can also arise in PMI vectors but is described as highly unlikely in high dimensions.
- Paraphrasing in W2V Embeddings: For W2V embeddings, paraphrase relationships retain an additional error term linear in |W|, making them less accurate for |W|>1 than PMI-factorizing embeddings.The theorem also applies to explicit PMI embeddings, where the paper notes the absence of reconstruction error and projection-based false positives.
6. Analogies
The paper explains analogy linearity by defining paraphrase probabilistically, recasting it as word transformation, and connecting those transformations to additive relationships between embeddings. The resulting relationships hold under explicit paraphrase and dependence-error conditions, including for W2V-type embeddings.
- Paraphrasing word sets: Theorem 2 extends single-word paraphrasing to word sets, with the single-word case recovered when |W*| = 1.Under this definition, paraphrasing is reflexive and symmetric because the paraphrase-error magnitude is symmetric.
- Paraphrasing word sets: Generalised paraphrasing relates an embedding wx* to wx + wW+ − wW− through equivalent word-set distributions and explicit paraphrase and dependence errors.The approximation holds when the word sets paraphrase and within-set dependence is negligible or cancels.
- From paraphrases to transformations: Paraphrase sets can be interpreted as transformations that add context through W+ and remove or broaden context through W−, producing additive embedding differences.This transformation view makes W+ and W− parameters describing how one word is changed into another.
- Interpreting analogies: An analogy means that common transformation parameters simultaneously transform each source word into its corresponding target word.The paper formalises “wa is to wa* as wb is to wb*” by requiring parameters W+ and W− that transform both pairs.
- Interpreting analogies: These transformations establish the observed linear embedding relationships when paired word sets paraphrase and exhibit similar dependence, subject to dependence error.The result links the generalised paraphrase theorem directly to the analogy equations.
- Example: For the king–man–woman example, wQ ≈ wK − wM + wW depends on how closely {man, queen} paraphrases {king, woman} and on statistical dependencies within those pairs.The paper also notes that W2V embeddings exhibit the same linear analogy relationships, with higher-order statistical relationships determining their accuracy.
7. Conclusion
The paper gives a probabilistically principled account of paraphrasing and word transformation, proving that these relationships generate linear structure in PMI-based embeddings and approximations such as W2V and Glove. It identifies this as a rigorous explanation for analogical linearity while noting extensions to other factorisation-based representations remain future work.
- Paraphrasing is defined through equivalence between words and word sets based on the distributions they induce over surrounding words.
- Paraphrase relationships give rise to linear relationships between PMI-factorising embeddings and approximations including W2V and Glove, subject to statistical dependencies.
- Reinterpreting paraphrases as word transformations mathematically defines analogies and connects semantic properties with embedding properties.
- The paper presents its result as the first rigorous explanation for linear relationships between embeddings of analogies.
- Future work will extend this analysis to graph embeddings and recommender systems that rely on underlying matrix factorisation.
Appendices
The appendix compares two KL-divergence objectives for selecting a word that represents a word set through its induced context distribution. The objectives differ in whether they prioritize covering all probability mass associated with the set or avoiding additional meanings.
- Minimizing DKL[p(cj|W) || p(cj|wi)] selects the word whose context distribution is closest to the word-set distribution while assigning mass wherever the set does.
- The first objective selects a word that most closely reflects all aspects of the word set and may occur in contexts absent from individual set members.
- Minimizing DKL[p(cj|wi) || p(cj|W)] selects a word whose distribution is broadly assigned only to contexts receiving mass under the word set.
- The second objective favors a word reflecting as many aspects of the word set as possible without adding meanings absent from it.
A.1. Weakening the paraphrase assumption
The weak paraphrasing analysis studies the relationship between a word-set embedding sum and the word minimizing one KL divergence. Unlike the stronger assumption, this weaker relationship identifies a hyperplane rather than a unique word and cannot explain additive paraphrase embeddings.
- The analysis compares a word-set embedding sum with the embedding of the word minimizing the KL divergence.
- The weaker assumption tests whether the stronger paraphrase requirement is excessive and examines embedding relationships as paraphrase error increases.
- Under weak paraphrasing, the minimizing word lies on a hyperplane and is therefore not uniquely determined.
- Because it does not uniquely specify the word, weak paraphrasing cannot explain embedding addition for paraphrases.
- The theorem may help locate words that loosely paraphrase a set, corresponding to increased paraphrase error.
B. Proof of Lemma 1
The proof introduces error terms while deriving relationships between word-set paraphrases and PMI-based embeddings. It then relates those terms through paraphrasing and independence, including the case where two paraphrasing word sets need not correspond to single words.
- The proof tracks introduced error terms whose statistical meaning is established through relationships involving paraphrasing and independence.
- The lemma applies to word sets W and W* whose sizes are bounded by l.
- Set-to-set paraphrasing differs from direct paraphrasing because neither word set must correspond to a single word.
- The paraphrase error ρW,W* compares the context distributions induced by the two word sets, following the principles of direct paraphrasing.
D. Alternate Proof of Corollary 2.1
The alternate proof states Corollary 2.1 for arbitrary words and sufficiently small positive and negative word sets, then completes the argument by multiplying by C†.
- Corollary 2.1 applies to any words wx and wx∗ in E with word sets W+ and W− contained in E.
- The corollary assumes both word sets have cardinality less than l − 1.
- The proof simplifies notation by defining W = {wx} ∪ W+ and W∗ = {wx∗} ∪ W−.
- The alternate proof is completed by multiplying by C†.