Source-linked AI summary
Linking GloVe with word2vec
Tianze Shi, Zhiyuan Liu
TL;DR
The paper examines whether GloVe and SGNS have similar training objectives, comparing their objective forms and GloVe bias terms. It concludes that SGNS resembles a specialized GloVe objective, though their cost functions and weighting strategies differ.
Problem
Method
Results
SGNS's training objective is similar to that of a specialized GloVe form, while the models differ mainly in cost functions and weighting strategies.
Source: §4 · p. 4Takeaways & Limitations
GloVe bias terms tend toward SGNS's corresponding terms, which the authors suggest may approximate the globally optimized value.
Source: §4 · p. 4The effect of weighting-function choices on the two models remains a focus for future investigation.
Source: §4 · p. 4
Abstract
from arXiv · showhide
The Global Vectors for word representation (GloVe), introduced by Jeffrey Pennington et al. is reported to be an efficient and effective method for learning vector representations of words. State-of-the-art performance is also provided by skip-gram with negative-sampling (SGNS) implemented in the word2vec tool. In this note, we explain the similarities between the training objectives of the two models, and show that the objective of SGNS is similar to the objective of a specialized form of GloVe, though their cost functions are defined differently.
1 Introduction and Notation
Word embeddings represent words and contexts as vectors, enabling direct calculation of similarities and other features. The section introduces the vocabulary vectors, matrices, and corpus counts used to describe them.
1 Introduction and Notation
- Vector representations let similarities between words and other valuable features be calculated directly with vector arithmetic.Embedding algorithms seek word and context vectors that meet a predefined criterion, such as predicting a word's surrounding context.Source: §1 · p. 1
- Word and context vocabularies have d-dimensional vectors, collected respectively as rows of matrices W and C.The row for each word or context represents its embedding.Source: §1 · p. 1
- The count #(w, c) records corpus observations of a word-context pair; #(w) and #(c) sum pair counts for each word and context.Summing either set of counts gives the total number of word-context pairs.Source: §1 · p. 1
2 Training Objectives of the Two Models
GloVe explicitly factorizes word-context co-occurrences, while SGNS implicitly factorizes a shifted-PMI matrix. Their objectives have similar forms, but differ in cost functions and weighting strategies.
2.1 GloVe
- GloVe explicitly factorizes word-context co-occurrences, fitting word-context dot products to log co-occurrence values adjusted by word and context biases.Its weighting function down-weights rare co-occurrences.Source: §2.1 · p. 2
2.2 Skip-gram with Negative Sampling (SGNS)
- SGNS implicitly factorizes a word-context matrix whose cells are shifted PMI, with the shift determined by the number of negative samples.The objective's stationary solution is log #(wi, cj) − log #(wi) − log #(cj) + log Σw#(w) − log k.Source: §2.2 · p. 2
2.3 Similarities between the Two Objectives
- The objectives have similar forms: SGNS's word and context log-count terms can be absorbed into GloVe's biases, and its index-independent offset can be split between them.GloVe's biases are learned during matrix factorization and may not converge to the corresponding SGNS values, making GloVe more general with a wider optimization domain.Source: §2.3 · p. 3
2.4 Differences between the Two Objectives
- The models use different cost functions, which may affect performance at low vector dimensionality, and different weighting strategies for rare and unobserved pairs.GloVe down-weights rare observed pairs and ignores unobserved pairs; whether defining an objective for unobserved pairs and using negative sampling improves performance remains open.Source: §2.4 · p. 3
3 Observations of the Bias Terms in the GloVe Model
The section examines whether GloVe’s learned bias terms track word and context frequencies, and reports that word-bias correlation with word frequency is high after 50 iterations. It also notes that GloVe’s word-vector product moves toward shifted PMI.
3 Observations of the Bias Terms in the GloVe Model
- The experiment compares trained bias terms with log word and context counts, using a 1.5-billion-token Wikipedia dump and 300-dimensional vectors trained for 50 iterations.The vocabulary includes words occurring at least 100 times; xmax is 10 or 100 and α is 3/4.Source: §3 · p. 3
- Figure 2 shows bWi against log #(wi) after the first and all 50 iterations; the two bands may reflect truncation of less frequent words.Source: §3 · p. 4
- After 50 iterations, bWi correlates well with log #(wi), and smaller xmax produces higher correlation; Wi·CT moves toward shifted PMI.The text describes this as similar to the SGNS model’s optimization.Source: §3 · p. 4
4 Discussion
GloVe and SGNS have similar, but not identical, training objectives, with differences in cost functions and weighting strategies. The discussion reports that GloVe bias terms tend toward SGNS’s corresponding terms and identifies weighting-function choices as a subject for future study.
4 Discussion
- GloVe explicitly factorizes co-occurrences while SGNS implicitly factorizes shifted PMI, yet their objectives are similar but not identical.Their main differences are their cost functions and weighting strategies.Source: §4 · p. 4
- GloVe’s bias terms tend to converge toward SGNS’s corresponding terms, which the authors suggest may approximate globally optimized values.Source: §4 · p. 4
- Future investigation may examine weighting-function choices and their effects on the two models.Source: §4 · p. 4
- Figure 2 plots bWi against log #(wi) after the first and all 50 iterations and gives Pearson correlation coefficients R.Source: Figure 2 · p. 5